January 2018 - syslog-ng - lists.balabit.hu

One source multiple destinations ?
by Scot 10 Mar '18

10 Mar '18

I'm having a problem where I am trying to take input source(s) and write them out to multiple destinations. Before I go barking up the wrong tree I just wanted to make sure I wasn't missing something. We should be able to take a source and send it to file, elastic-search and SPLUNK and sumologic all at the same time right ? Troubleshooting an odd behavior where only one network destination will work but then I switch the order the other starts working. I know it's vague but has anyone seen this behavior? Thanks Scot

4 10

syslog-ng 3.13 key-value parser crashes and aborts
by Don C 19 Feb '18

19 Feb '18

Hi, I was upgrading syslog-ng from 3.12 to 3.13 using the prebuilt unofficial RPMs. I was testing my config on 3.13 and ran into the following issue where the key-value parsing aborts.This appears to be a regression in 3.13, the abort never happened in 3.12. If I remove the kv_parser from my config, there is no issue. Here's the end of the debug and verbose output I get. Notice the garbage values in the key-value names parsed from the message. [2018-01-25T17:33:08.924895] Filter rule evaluation result; msg='0x7fc0a807b370', result='match', rule='f_compliant_hosts', location='/etc/syslog-ng/syslog-ng.conf:66:27' [2018-01-25T17:33:08.924919] Setting value; msg='0x7fc0a80612f0', name='@\x15\x01pÀ', value='2017-11-21 19:11:24.817041' [2018-01-25T17:33:08.924930] Setting value; msg='0x7fc0a80612f0', name='', value='debug' [2018-01-25T17:33:08.924936] Setting value; msg='0x7fc0a80612f0', name='f_compliant_hosts', value='[robotnats kafka transport prepared]' [2018-01-25T17:33:08.924944] Setting value; msg='0x7fc0a80612f0', name='PK\x06¨À', value='ROBOT_serviceTracking_neo4j-topo-svc-2717019760-1sdn1' [2018-01-25T17:33:08.924952] Setting value; msg='0x7fc0a80612f0', name='', value='nats://nats:4222' [2018-01-25T17:33:08.924957] Setting value; msg='0x7fc0a80612f0', name='', value='[robot-kafka:9092]' [2018-01-25T17:33:08.924963] Setting value; msg='0x7fc0a80612f0', name='kv_kafka', value='robotnats_kafka' [2018-01-25T17:33:08.924969] Setting value; msg='0x7fc0a80612f0', name=' »\x08¬À', value='ROBOT_serviceTracking_neo4j-topo-svc-2717019760-1sdn1' [2018-01-25T17:33:08.924975] Message parsing complete; result='1', rule='p_kv', location='/etc/syslog-ng/syslog-ng.conf:75:5' [2018-01-25T17:33:08.924998] Incoming log entry; line='time="2017-11-21 19:11:24.817095" level=info msg="[skeleton core configuration parse stage complete]" config=/etc/robot/servicetracking.conf debug=true logfile= maxthreads=16 tag=ROBOT_serviceTracking_neo4j-topo-svc-2717019760-1sdn1 version=' [2018-01-25T17:33:08.925030] Setting value; msg='0x7fc0a807bd60', name='MESSAGE', value='time="2017-11-21 19:11:24.817095" level=info msg="[skeleton core configuration parse stage complete]" config=/etc/robot/servicetracking.conf debug=true logfile= maxthreads=16 tag=ROBOT_serviceTracking_neo4j-topo-svc-2717019760-1sdn1 version=' [2018-01-25T17:33:08.925040] Setting value; msg='0x7fc0a807bd60', name='HOST_FROM', value='syslog-ng-logging-1040168119-qsmrf' [2018-01-25T17:33:08.925046] Setting value; msg='0x7fc0a807bd60', name='HOST', value='syslog-ng-logging-1040168119-qsmrf/syslog-ng-logging-1040168119-qsmrf' [2018-01-25T17:33:08.925053] Setting value; msg='0x7fc0a807bd60', name='FILE_NAME', value='/mnt/logfs/neo4j-topo-svc/1/servicetracking_stdout.log' [2018-01-25T17:33:08.925057] Setting value; msg='0x7fc0a807bd60', name='SOURCE', value='s_file_stdout' [2018-01-25T17:33:08.925062] Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:81:5' [2018-01-25T17:33:08.824606] Incoming log entry; line='time="2017-11-21 19:10:56.399341" level=debug msg="[section not required]" section=etcd tag=ROBOT_serviceTracking_robot-topo-svc-2107321943-48rnw' [2018-01-25T17:33:09.022629] Filter rule evaluation begins; msg='0x7fc0a807bd60', rule='f_compliant_hosts', location='/etc/syslog-ng/syslog-ng.conf:66:27' [2018-01-25T17:33:08.824700] Setting value; msg='0x7fc0a409b810', name='HOST_FROM', value='syslog-ng-logging-1040168119-qsmrf' # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # The crash happened outside the Java Virtual Machine in native code. # See problematic frame for where to report the bug. # ... [2018-01-25T17:33:09.022911] Requesting flow control; location='/etc/syslog-ng/syslog-ng.conf:81:5' [2018-01-25T17:33:09.022916] Filter rule evaluation begins; msg='0x7fc08c0f5550', rule='f_compliant_hosts', location='/etc/syslog-ng/syslog-ng.conf:66:27' [2018-01-25T17:33:09.022922] Filter node evaluation result; msg='0x7fc08c0f5550', result='not-match', type='==' [2018-01-25T17:33:09.022927] Filter rule evaluation result; msg='0x7fc08c0f5550', result='not-match', rule='f_compliant_hosts', location='/etc/syslog-ng/syslog-ng.conf:66:27' [2018-01-25T17:33:09.022932] Filter rule evaluation begins; msg='0x7fc08c0f5550', rule='f_noncompliant_hosts', location='/etc/syslog-ng/syslog-ng.conf:70:30' [2018-01-25T17:33:09.022937] Filter node evaluation result; msg='0x7fc08c0f5550', result='not-match', type='==' [2018-01-25T17:33:09.022942] Filter node evaluation result; msg='0x7fc08c0f5550', result='match', type='filter(f_compliant_hosts)' [2018-01-25T17:33:09.022947] Filter rule evaluation result; msg='0x7fc08c0f5550', result='match', rule='f_noncompliant_hosts', location='/etc/syslog-ng/syslog-ng.conf:70:30' Aborted Is this a known issue with the key-value parser in 3.13? Regards, Don

3 2

Re: [syslog-ng] Help with ES destination error
by Marco Mignone 01 Feb '18

01 Feb '18

Hi Fabien, I have tried the following: - emptying all index/docs in ES - create the test/test index with the CURL in my email - pointing syslog ES destination to the test/test index This resulted in the same error again. I have tried to change the template to just output all nv-pairs and use a complete new index - same error. Grabbing some packet capture now to see if I can spot anything wrong. Marco > On 28 Jan 2018, at 14:19, Fabien Wernli <wernli(a)in2p3.fr> wrote: > > Hi, > > The reason I asked you to configure syslogng to index to "test" was to make > sure you are in the same conditions as your curl command. > You might for instance have a mapping template matching fw-* but not test. > > Please either configure syslogng to index to test, or use the same fw- index > on the curl cmdline. > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq >

2 5

Syslog-ng as basic realtime reliable logfile replicator. Possible?
by Declan White 31 Jan '18

31 Jan '18

Hullo. I'm trying to fit syslog-ng around a basic problem and looking for tips. I have log files growing on one machine that I want to follow and reliably replicate to a central machine, so it's effectively a basic 'tail -f' job. It seems simple, but as I try and close out the possible error conditions it's getting hairier and hairier. e.g. - by default, there is nothing you can make with syslog-ng alone that will not lose data during a network or endpoint outage. - transporting metadata can tell you which file the data is from, but not where in the file it's from, so you can't really tell if you have duplicate data, or missed data. (The inode number might be handy too) - behaviour around input file truncation is fuzzy. That a truncation has occured might be useful metadata to send (if you're looking for people fiddling logs). - It doesn't seem to be able to encode binary/NULs in the logs, so it cannot relay data from 'untrusted' application logs? - Not sure what it does with very long lines. Loses data? I'm not necessarily looking to get syslog-ng to recreate the file exactly, just to send enough information to allow something else to work out the full order of events. Googling around to see how others solve this problem, I see people doing infinite rsync loops, or installing large Java beasties, or paying someone else to make it all go away. I tried using rsyslog, but it melted down into a screaming puddle of nondeterministic threading. Is what I'm attempting really as hard as it seems? - D

5 5

Re: [syslog-ng] Help with ES destination error
by Fabien Wernli 28 Jan '18

28 Jan '18

Hi, The reason I asked you to configure syslogng to index to "test" was to make sure you are in the same conditions as your curl command. You might for instance have a mapping template matching fw-* but not test. Please either configure syslogng to index to test, or use the same fw- index on the curl cmdline.

1 0

Help with ES destination error
by Marco Mignone 26 Jan '18

26 Jan '18

Hi all, Hope you can help me solve this scenario. I am receiving messages from a firewall, extracting KV pairs with a custom parser_db, sending them to an elasticsearch destination applying a JSON template. All works fine until I apply the JSON template to the Elasticsearch destination - when I do this in the config and a message is received I get the following from Syslog: syslog-ng | [2018-01-25T17:59:00.873601] Outgoing message; message='{"timestamp":"2018-01-25 17:59:00","src":{"port":"62118","ip":"192.168.xx.xx","if":"X1","hostname":"hostname.domain.co.uk"},"sn":"xxxx","sid":"5165","priority":"1","nipspri":"3","msg":"IPS Detection Alert","ipscat":"WEB-TLS SSLv2.0 Client Hello 2","fw":{"ip":"x.xxx.xxx.xx","action":"NA"},"dst":{"port":"443","ip":"40.xxx.xxx.xx","if":"X5"},"_classifier":{"rule_id":"71593655-6fa7-4fca-9617-480e79703215","class":"IPSDetection"},"PROGRAM":"id=Firewall","LEGACY_MSGHDR":"id=Firewall ","HOST_FROM":"xx.xx.xx.xx","HOST":"xx.xx.xx.xx"}\x0a’ syslog-ng | 17:59:00.897 [?3?] ERROR - {"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse"}],"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"not_x_content_exception","reason":"Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"}} My setup is all on Ubuntu/Docker and is using Syslog-ng 3.13.2 and ES 6.1. Configuration is quite simple: source s_net { udp( ip(0.0.0.0),port(514) #,flags(no-parse) ); }; #this parse all the field from the previous message parser sonicwall { db-parser(file("/etc/syslog-ng/patterndb.d/sonicwall-pattern.xml")); }; #some more parsing due to firewall sending dynamic lenght messages for sources and destinations parser split-sonicwall-srcdata { csv-parser( columns("src.ip", "src.port", "src.if","src.hostname") delimiters(chars(":"), strings(" ")) template("${src.data}") ); }; parser split-sonicwall-dstdata { csv-parser( columns("dst.ip", "dst.port", "dst.if","dst.hostname") delimiters(chars(":"), strings(" ")) template("${dst.data}") ); }; #finally the template applied template t_sonicwall { template("$(format-json --scope all-nv-pairs --exclude MESSAGE,SOURCE,src.data,dst.data,mfield,cfield,nfield)\n"); }; destination d_elasticsearch { elasticsearch2( client-lib-dir("/jarfiles/*.jar:/usr/lib/syslog-ng/3.13/java-modules/elastic-jest-client/*.jar:/usr/lib/syslog-ng/3.13/java-modules/") index("fw-${YEAR}.${MONTH}.$(lowercase '${.classifier.class}')") type("syslog") client_mode("http") cluster("docker-cluster") cluster_url("http://elasticsearch:9200") template(t_sonicwall) flush-limit("1") ); }; log { source(s_net); parser(sonicwall); parser(split-sonicwall-srcdata); parser(split-sonicwall-dstdata); destination(d_elasticsearch); destination(d_file); }; Although if I try to manually put the content of the message with a curl POST request it works (not some info is masked with xx): curl -X POST -H "Content-Type: application/json" -H "Cache-Control: no-cache" -d '{ "timestamp": "2018-01-25 17:59:00", "src.port": "62118", "src.ip": "xx.xx.xx.xx", "src.if": "X1", "src.hostname": "hostname.domain.co.uk", "sn": “xxxx", "sid": "5165", "priority": "1", "nipspri": "3", "msg": "IPS Detection Alert", "ipscat": "WEB-TLS SSLv2.0 Client Hello 2", "fw.ip": "xx.xx.xx.xx", "fw.action": "NA", "dst.port": "443", "dst.ip": "xx.xx.xx.xx", "dst.if": "X5", "_classifier.rule_id": "71593655-6fa7-4fca-9617-480e79703215", "_classifier.class": "IPSDetection", "PROGRAM": “id=Firewall", "LEGACY_MSGHDR": "id=Firewall ", "HOST_FROM": "xx.xx.xx.xx", "HOST": "xx.xx.xx.xx" }' "http://es:9200/test/test/“ {"_index":"test","_type":"test","_id":"ZzaCLmEB5A1B2qoXkU_p","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":1,"_primary_term":1} Any suggestions would be much appreciated. Thanks, Marco Mignone

2 2

Excessive memory usage
by Patrick Hemmer 26 Jan '18

26 Jan '18

We're seeing an issue on our systems where syslog-ng is consuming an excessive amount of memory, approximately 380mb (RSS) right now on one of our hosts. I'm not sure what's causing this as our config is pretty minimal, and there are no destinations unavailable which would cause it to buffer. We do have `log-msg-size(65536)` and `log-fifo-size(10000)` which might have caused it to buffer a large amount some time in the past, but would it not release these buffers? Version 3.13.2 on CentOS/7 (RPM from https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng313/) Config: @version:3.13 @include "scl.conf" # This configuration is meant to utilize systemd journald options { flush_lines(0); time_reopen(10); log-msg-size(65536); log_fifo_size(10000); chain_hostnames(off); use_dns(no); use_fqdn(yes); keep_hostname(yes); create_dirs(no); stats_freq(3600); }; source s_system { system(); }; source s_internal { internal(); }; destination d_journald { unix-dgram("/dev/log" flush-lines(1) template("${MSG}\n")); }; log { source(s_internal); destination(d_journald); }; destination d_loghost { syslog("syslog.ecom.chewy.com" frac-digits(3) log-fifo-size(10000) ); }; log { source(s_system); destination(d_loghost); }; Stats: # syslog-ng-ctl stats SourceName;SourceId;SourceInstance;State;Type;Number dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;dropped;163025 dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;processed;489030242 dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;queued;1 dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;written;488867216 destination;d_journald;;a;processed;1997 src.internal;s_internal#0;;a;processed;1997 src.internal;s_internal#0;;a;stamp;1515709115 center;;received;a;processed;1997 destination;d_loghost;;a;processed;489030242 source;s_internal;;a;processed;1997 center;;queued;a;processed;489032239 global;scratch_buffers_count;;a;queued;25769803782 dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;dropped;0 dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;processed;845 dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;queued;0 dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;written;845 global;payload_reallocs;;a;processed;2932894697 src.journald;;journal;a;processed;489030242 src.journald;;journal;a;stamp;1515710081 global;sdata_updates;;a;processed;0 global;scratch_buffers_bytes;;a;queued;0 global;msg_clones;;a;processed;135 global;internal_queue_length;;a;processed;0 Thanks -Patrick

2 3

SUMO Logic Integration
by Scot 25 Jan '18

25 Jan '18

Hi all, Thought I had asked already but I don't see it, so sorry if it's a repost. I now have the need to switch all my syslog-ng -> SPLUNK feeds over to SUMO. Including json/beats going through syslog-ng :( Sumo supports a syslog tcp/udp collection, I setup an "Installed Collector" local to the environment. Seems to work fine until you you need to match categories in sumo. The sumo config seems to be pretty limited per instance so looking at spinning up multiple instances of sumo just to support Category mapping. Not happy with the product. Anyone working with syslog-ng and SUMO who would like to collaborate? Scot

1 0

pulling in "sar" data
by Jim Hendrick 24 Jan '18

24 Jan '18

Hi list, Has anyone added sar data to syslog-ng? I know of a couple "brute-force" ways like running a cron to run the sar command and dump it to a file. But before I do that I wanted to check and see if there was a more elegant solution. The overall problem would be to run this across 10,000+ servers with minimal need to change configurations. (for example - if there was a need to change the parameters to sar, or change the frequency of the pull). I guess I was hoping for some add-on that either directly pulled the performance data from the kernel or could include a config that would specify what data elements would be included. FYI - we do use syslog-ng PE but this seems fairly generic so I thought I'd ask the list. Given the scale of the problem, pulling "all" data even relatively frequently (say every 1-5 minutes) would result in a huge volume increase in our logging solution (where we pay by the ingested GB...) Thoughts? Advice? Thanks in advance! Jim

2 1

new build of syslog-ng 3.13.2 has new dependencies
by Evan Rempel 23 Jan '18

23 Jan '18

The copr repo at https://copr-be.cloud.fedoraproject.org/results/czanik/syslog-ng313/epel-7-… has a new build of syslog-ng. The actual version of syslog-ng has not changed, just the build. The package has gone from version 3.13.2-1 to 3.13.2-2 and new dependencies have been added. Specifically the 4 below --> Processing Dependency: libmongoc-1.0.so.0(LIBMONGOC_1.0)(64bit) for package: syslog-ng-3.13.2-2.el7.centos.x86_64 --> Processing Dependency: libbson-1.0.so.0(LIBBSON_1.0)(64bit) for package: syslog-ng-3.13.2-2.el7.centos.x86_64 --> Processing Dependency: libmongoc-1.0.so.0()(64bit) for package: syslog-ng-3.13.2-2.el7.centos.x86_64 --> Processing Dependency: libbson-1.0.so.0()(64bit) for package: syslog-ng-3.13.2-2.el7.centos.x86_64 Since the version of syslog-ng has not changed, how come the dependencies have changed? Is this a mistake and these dependencies are not actually needed? I ask because I am running the package 3.13.2-1 and not having any difficulties (I am not using the mondo module). If the dependencies are not required, can they be removed and a new build created? -- Evan Rempel

2 3