- syslog-ng - lists.balabit.hu

Re: [syslog-ng] Help with ES destination error
by Marco Mignone 01 Feb '18

01 Feb '18

Hi Fabien, I have tried the following: - emptying all index/docs in ES - create the test/test index with the CURL in my email - pointing syslog ES destination to the test/test index This resulted in the same error again. I have tried to change the template to just output all nv-pairs and use a complete new index - same error. Grabbing some packet capture now to see if I can spot anything wrong. Marco > On 28 Jan 2018, at 14:19, Fabien Wernli <wernli(a)in2p3.fr> wrote: > > Hi, > > The reason I asked you to configure syslogng to index to "test" was to make > sure you are in the same conditions as your curl command. > You might for instance have a mapping template matching fw-* but not test. > > Please either configure syslogng to index to test, or use the same fw- index > on the curl cmdline. > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq >

2 5

Syslog-ng as basic realtime reliable logfile replicator. Possible?
by Declan White 31 Jan '18

31 Jan '18

Hullo. I'm trying to fit syslog-ng around a basic problem and looking for tips. I have log files growing on one machine that I want to follow and reliably replicate to a central machine, so it's effectively a basic 'tail -f' job. It seems simple, but as I try and close out the possible error conditions it's getting hairier and hairier. e.g. - by default, there is nothing you can make with syslog-ng alone that will not lose data during a network or endpoint outage. - transporting metadata can tell you which file the data is from, but not where in the file it's from, so you can't really tell if you have duplicate data, or missed data. (The inode number might be handy too) - behaviour around input file truncation is fuzzy. That a truncation has occured might be useful metadata to send (if you're looking for people fiddling logs). - It doesn't seem to be able to encode binary/NULs in the logs, so it cannot relay data from 'untrusted' application logs? - Not sure what it does with very long lines. Loses data? I'm not necessarily looking to get syslog-ng to recreate the file exactly, just to send enough information to allow something else to work out the full order of events. Googling around to see how others solve this problem, I see people doing infinite rsync loops, or installing large Java beasties, or paying someone else to make it all go away. I tried using rsyslog, but it melted down into a screaming puddle of nondeterministic threading. Is what I'm attempting really as hard as it seems? - D

5 5

Re: [syslog-ng] Help with ES destination error
by Fabien Wernli 28 Jan '18

28 Jan '18

Hi, The reason I asked you to configure syslogng to index to "test" was to make sure you are in the same conditions as your curl command. You might for instance have a mapping template matching fw-* but not test. Please either configure syslogng to index to test, or use the same fw- index on the curl cmdline.

1 0

Help with ES destination error
by Marco Mignone 26 Jan '18

26 Jan '18

Hi all, Hope you can help me solve this scenario. I am receiving messages from a firewall, extracting KV pairs with a custom parser_db, sending them to an elasticsearch destination applying a JSON template. All works fine until I apply the JSON template to the Elasticsearch destination - when I do this in the config and a message is received I get the following from Syslog: syslog-ng | [2018-01-25T17:59:00.873601] Outgoing message; message='{"timestamp":"2018-01-25 17:59:00","src":{"port":"62118","ip":"192.168.xx.xx","if":"X1","hostname":"hostname.domain.co.uk"},"sn":"xxxx","sid":"5165","priority":"1","nipspri":"3","msg":"IPS Detection Alert","ipscat":"WEB-TLS SSLv2.0 Client Hello 2","fw":{"ip":"x.xxx.xxx.xx","action":"NA"},"dst":{"port":"443","ip":"40.xxx.xxx.xx","if":"X5"},"_classifier":{"rule_id":"71593655-6fa7-4fca-9617-480e79703215","class":"IPSDetection"},"PROGRAM":"id=Firewall","LEGACY_MSGHDR":"id=Firewall ","HOST_FROM":"xx.xx.xx.xx","HOST":"xx.xx.xx.xx"}\x0a’ syslog-ng | 17:59:00.897 [?3?] ERROR - {"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse"}],"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"not_x_content_exception","reason":"Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"}} My setup is all on Ubuntu/Docker and is using Syslog-ng 3.13.2 and ES 6.1. Configuration is quite simple: source s_net { udp( ip(0.0.0.0),port(514) #,flags(no-parse) ); }; #this parse all the field from the previous message parser sonicwall { db-parser(file("/etc/syslog-ng/patterndb.d/sonicwall-pattern.xml")); }; #some more parsing due to firewall sending dynamic lenght messages for sources and destinations parser split-sonicwall-srcdata { csv-parser( columns("src.ip", "src.port", "src.if","src.hostname") delimiters(chars(":"), strings(" ")) template("${src.data}") ); }; parser split-sonicwall-dstdata { csv-parser( columns("dst.ip", "dst.port", "dst.if","dst.hostname") delimiters(chars(":"), strings(" ")) template("${dst.data}") ); }; #finally the template applied template t_sonicwall { template("$(format-json --scope all-nv-pairs --exclude MESSAGE,SOURCE,src.data,dst.data,mfield,cfield,nfield)\n"); }; destination d_elasticsearch { elasticsearch2( client-lib-dir("/jarfiles/*.jar:/usr/lib/syslog-ng/3.13/java-modules/elastic-jest-client/*.jar:/usr/lib/syslog-ng/3.13/java-modules/") index("fw-${YEAR}.${MONTH}.$(lowercase '${.classifier.class}')") type("syslog") client_mode("http") cluster("docker-cluster") cluster_url("http://elasticsearch:9200") template(t_sonicwall) flush-limit("1") ); }; log { source(s_net); parser(sonicwall); parser(split-sonicwall-srcdata); parser(split-sonicwall-dstdata); destination(d_elasticsearch); destination(d_file); }; Although if I try to manually put the content of the message with a curl POST request it works (not some info is masked with xx): curl -X POST -H "Content-Type: application/json" -H "Cache-Control: no-cache" -d '{ "timestamp": "2018-01-25 17:59:00", "src.port": "62118", "src.ip": "xx.xx.xx.xx", "src.if": "X1", "src.hostname": "hostname.domain.co.uk", "sn": “xxxx", "sid": "5165", "priority": "1", "nipspri": "3", "msg": "IPS Detection Alert", "ipscat": "WEB-TLS SSLv2.0 Client Hello 2", "fw.ip": "xx.xx.xx.xx", "fw.action": "NA", "dst.port": "443", "dst.ip": "xx.xx.xx.xx", "dst.if": "X5", "_classifier.rule_id": "71593655-6fa7-4fca-9617-480e79703215", "_classifier.class": "IPSDetection", "PROGRAM": “id=Firewall", "LEGACY_MSGHDR": "id=Firewall ", "HOST_FROM": "xx.xx.xx.xx", "HOST": "xx.xx.xx.xx" }' "http://es:9200/test/test/“ {"_index":"test","_type":"test","_id":"ZzaCLmEB5A1B2qoXkU_p","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":1,"_primary_term":1} Any suggestions would be much appreciated. Thanks, Marco Mignone

2 2

Excessive memory usage
by Patrick Hemmer 26 Jan '18

26 Jan '18

We're seeing an issue on our systems where syslog-ng is consuming an excessive amount of memory, approximately 380mb (RSS) right now on one of our hosts. I'm not sure what's causing this as our config is pretty minimal, and there are no destinations unavailable which would cause it to buffer. We do have `log-msg-size(65536)` and `log-fifo-size(10000)` which might have caused it to buffer a large amount some time in the past, but would it not release these buffers? Version 3.13.2 on CentOS/7 (RPM from https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng313/) Config: @version:3.13 @include "scl.conf" # This configuration is meant to utilize systemd journald options { flush_lines(0); time_reopen(10); log-msg-size(65536); log_fifo_size(10000); chain_hostnames(off); use_dns(no); use_fqdn(yes); keep_hostname(yes); create_dirs(no); stats_freq(3600); }; source s_system { system(); }; source s_internal { internal(); }; destination d_journald { unix-dgram("/dev/log" flush-lines(1) template("${MSG}\n")); }; log { source(s_internal); destination(d_journald); }; destination d_loghost { syslog("syslog.ecom.chewy.com" frac-digits(3) log-fifo-size(10000) ); }; log { source(s_system); destination(d_loghost); }; Stats: # syslog-ng-ctl stats SourceName;SourceId;SourceInstance;State;Type;Number dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;dropped;163025 dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;processed;489030242 dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;queued;1 dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;written;488867216 destination;d_journald;;a;processed;1997 src.internal;s_internal#0;;a;processed;1997 src.internal;s_internal#0;;a;stamp;1515709115 center;;received;a;processed;1997 destination;d_loghost;;a;processed;489030242 source;s_internal;;a;processed;1997 center;;queued;a;processed;489032239 global;scratch_buffers_count;;a;queued;25769803782 dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;dropped;0 dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;processed;845 dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;queued;0 dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;written;845 global;payload_reallocs;;a;processed;2932894697 src.journald;;journal;a;processed;489030242 src.journald;;journal;a;stamp;1515710081 global;sdata_updates;;a;processed;0 global;scratch_buffers_bytes;;a;queued;0 global;msg_clones;;a;processed;135 global;internal_queue_length;;a;processed;0 Thanks -Patrick

2 3

SUMO Logic Integration
by Scot 25 Jan '18

25 Jan '18

Hi all, Thought I had asked already but I don't see it, so sorry if it's a repost. I now have the need to switch all my syslog-ng -> SPLUNK feeds over to SUMO. Including json/beats going through syslog-ng :( Sumo supports a syslog tcp/udp collection, I setup an "Installed Collector" local to the environment. Seems to work fine until you you need to match categories in sumo. The sumo config seems to be pretty limited per instance so looking at spinning up multiple instances of sumo just to support Category mapping. Not happy with the product. Anyone working with syslog-ng and SUMO who would like to collaborate? Scot

1 0

pulling in "sar" data
by Jim Hendrick 24 Jan '18

24 Jan '18

Hi list, Has anyone added sar data to syslog-ng? I know of a couple "brute-force" ways like running a cron to run the sar command and dump it to a file. But before I do that I wanted to check and see if there was a more elegant solution. The overall problem would be to run this across 10,000+ servers with minimal need to change configurations. (for example - if there was a need to change the parameters to sar, or change the frequency of the pull). I guess I was hoping for some add-on that either directly pulled the performance data from the kernel or could include a config that would specify what data elements would be included. FYI - we do use syslog-ng PE but this seems fairly generic so I thought I'd ask the list. Given the scale of the problem, pulling "all" data even relatively frequently (say every 1-5 minutes) would result in a huge volume increase in our logging solution (where we pay by the ingested GB...) Thoughts? Advice? Thanks in advance! Jim

2 1

new build of syslog-ng 3.13.2 has new dependencies
by Evan Rempel 23 Jan '18

23 Jan '18

The copr repo at https://copr-be.cloud.fedoraproject.org/results/czanik/syslog-ng313/epel-7-… has a new build of syslog-ng. The actual version of syslog-ng has not changed, just the build. The package has gone from version 3.13.2-1 to 3.13.2-2 and new dependencies have been added. Specifically the 4 below --> Processing Dependency: libmongoc-1.0.so.0(LIBMONGOC_1.0)(64bit) for package: syslog-ng-3.13.2-2.el7.centos.x86_64 --> Processing Dependency: libbson-1.0.so.0(LIBBSON_1.0)(64bit) for package: syslog-ng-3.13.2-2.el7.centos.x86_64 --> Processing Dependency: libmongoc-1.0.so.0()(64bit) for package: syslog-ng-3.13.2-2.el7.centos.x86_64 --> Processing Dependency: libbson-1.0.so.0()(64bit) for package: syslog-ng-3.13.2-2.el7.centos.x86_64 Since the version of syslog-ng has not changed, how come the dependencies have changed? Is this a mistake and these dependencies are not actually needed? I ask because I am running the package 3.13.2-1 and not having any difficulties (I am not using the mondo module). If the dependencies are not required, can they be removed and a new build created? -- Evan Rempel

2 3

format-json incorrectly parsing some events
by Alicia Smith 19 Jan '18

19 Jan '18

I'm using syslog-ng rpm version 3.12.1-2 on CentOS 7 When we receive events remotely from another CentOS 7 host it uses the RFC5424 format and parses the messages correctly. However we have some hosts that are older and still using rsyslog which is using the RFC3164 format - those events do not parse correctly. My question is what is the best way to get syslog-ng to parse them? This is how they come out: {"TAGS":".source.test","SOURCEIP":"127.0.0.1","SOURCE":"test","SEQNUM":"26","PROGRAM":"info","PRIORITY":"notice","MESSAGE":" mig-agent 10430 - - - [info] refreshing agent environment","LEGACY_MSGHDR":"info ","HOST_FROM":" syslog-dev1.private.mdc1.mozilla.com","HOST":"sanvmadm1.ops.mdc1.mozilla.com","FILE_NAME":"/var/log/test.log","FACILITY":"user","DATE":"Jan 17 23:57:52","CATEGORY":"syslog"] Notice the Program says "info" and the mig-agent and pid are in the message key's value. This is a correctly parsed event that has those fields parsed properly: {"TAGS":".source.moz_net","SOURCEIP":"127.0.0.1","SOURCE":"moz_net","SEQNUM":"35","PROGRAM":"mig-agent","PRIORITY":"info","PID":"2698","MESSAGE":"- - - [info] Public IP retrieval failed through proxy http://proxy.dmz.scl3.mozilla.com:3128 - Get https://api.mig.mozilla.org/api/v1//ip: proxyconnect tcp: dial tcp 10.22.74.78:3128: i/o timeout","LEGACY_MSGHDR":"mig-agent[2698]: ","HOST_FROM":"localhost6.localdomain","HOST":" syslog-dev1.private.mdc1.mozilla.com","FACILITY":"daemon","DATE":"Jan 18 00:02:25","CATEGORY":"syslog"} destination d_amqp { amqp( vhost("/") host("localhost") port(5672) exchange("eventtask") exchange-type("direct") routing-key("eventtask") body("$(format-json --scope selected_macros --scope nv_pairs)") persistent(no) username("rabbituser") password("*****") ); }; -- Alicia Smith @phrozyn Information Security Engineer asmith(a)mozilla.com

2 1

Balabit acquired by One Identity
by Scheidler, Balázs 18 Jan '18

18 Jan '18

Dear syslog-ng users, You might have already heard that Balabit, the company behind syslog-ng has been acquired by One Identity (http://www.oneidentity.com) and you may be wondering what would happen to syslog-ng in the future. I'd like to share a bit of personal details: I've founded the syslog-ng project back in 1998, during my university years, and a bit later on, I was also one of the founders of Balabit. I've worked on syslog-ng as a personal hobby until about 2007, at which point Balabit has officially started sponsoring the project, building products on top and also providing professional services. Although I was CEO and later CTO of Balabit, I always remained an active member of the syslog-ng community. Whenever I've found some leisure time (e.g. on a plane :), I was working on syslog-ng, and I've kept doing that parallel to Balabit establishing a professional dev team that worked on syslog-ng and related products. All of this work went straight to the open source project: https://github.com/balabit/syslog-ng/graphs/contributors With that story, I’d like to assure you that this acquisition will not negatively affect syslog-ng. One Identity considers the log management products of Balabit an important part of the acquisition and will continue to develop both the commercial and the open source branches further. Of course, I also have a role in the combined entity, and will continue to keep working on syslog-ng personally. I seem to be expected to be sitting on planes much more often :) One Identity as a business will also keep working on syslog-ng, and are committed to keeping it open source, adding new features, and so on, just like we did in the past. As a first step of this process, the syslog-ng product line will receive its own homepage in the coming weeks. Stay tuned, and keep using syslog-ng -- Bazsi

1 0

RHEL 6 / CentOS 6 packages
by Czanik, Péter 18 Jan '18

18 Jan '18

Hi, There were many requests for RHEL 6 / CentOS 6 support. The last version compiling on these ancient platforms was syslog-ng 3.9. Bazsi provided a fix: https://github.com/balabit/syslog-ng/pull/1837 which I applied to the syslog-ng 3.13 branch. Packages are available at: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng313epel6/ Compared to the 3.9 package other than making it to compile I enabled GeoIP2 (libmaxminddb) support and fixed any problems in packaging I ran into. I consider these packages ready to use, but if you run into any problems let me know! Please RT https://twitter.com/PCzanik/status/953931953669050368 to reach more people$ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

2 3

match filter scale issue
by Kavita Salve 14 Jan '18

14 Jan '18

Hello, I have an issue related to the syslog-ng “match” filter on which I need some inputs here. I have 800 “match” filters used in syslog configuration to filter syslogs with priority 3, 4, 5 and 6 based on the content of the syslog. So any incoming syslog’s will get forwarded only if the match is found in those 800 filters. I am using the match filter with the macro value(MSGHDR) and value(MSG) depending on where I need to search the given regex for the syslog. E.g. · match (“%LINEPROTO-5-UPDOWN” value(MSGHDR)) – tries to find match in header · match (“STANDBY: kernel” value(MSG)) - tries to find match in message body (this is same as using message filter) The filters work perfectly fine, but we see that there is a severe performance degradation even when we use the match filter with macros - value(MSGHDR) and value(MSG). From documentation, I see that we are not supposed to use plain match filter (E.g. match (“%LINEPROTO-5-UPDOWN”) that matches both header and message) due to performance issues, but we should be able to use the match filter with macros. We did a scale test to find the eps that is getting processed by the syslog-ng server. The scale was done on Centos server with RAM - 8GB, 4 CPU and 80 GB hard disk. Following are the scale test results we are seeing for various combinations of message and match filters: 1. When absolutely no filters are used (i.e. no match /message filters used, its only plain syslog forwarding everything from source to destination), we got eps for processing syslogs around *6667 eps.* 2. When I used ‘match’ filter (plain match filter without macros), it resulted in eps of processing syslogs being around *1150-1200.* 3. When I used message filter, it resulted in eps of processing syslogs as *4730 eps* 4. When I used ‘match’ filter with value’ (macro value(MSG) and value(MSGHDR) used), it resulted in eps of *~1080eps* *My analysis and questions:* 1. I am seeing that even if I use one match filter in syslog-ng configuration, the eps is coming down drastically. This happens even when I use the match filter with macros. *How do I solve this issue? Is this expected? Is there anything wrong in the way I am using the filters?* 2. I cannot use the alternative “message” filter as I need to match syslogs based on the MSGHDR also in many cases and “message” filter cannot do this*. Is there any other way/filter to use here?* *3. *I want to find out why match filter with value macro is also causing performance issues. Syslog-ng gives warning if I use match without value that this will lead to low performance, but even with using value macro I am seeing the performance hit*. I want help on how to solve this issue. Can we get comparable eps with match filter where compared to message filter?* *PS:* I am filtering syslogs generated by Cisco IOS devices that are in the format: *%FACILITY-SEVERITY-MNEMONIC: Message-text* E.g.: *Mar 6 22:48:34.452 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback0, changed state to up For these kind of syslogs, we are seeing that syslog-ng treats *“%FACILITY-SEVERITY-MNEMONIC:*” as the *Message Header* and “*Message-text*” as the *Message Body*. Thanks Kavita

2 2

Anyone sourcing from beats
by Evan Rempel 11 Jan '18

11 Jan '18

Looking for a clean way to get beats products to send data to syslog-ng. Does anyone have a working example? -- Evan

4 4

SYSLOG-NG issue with ES 6.X
by hari ram 10 Jan '18

10 Jan '18

Hi I have installed SYSLOG-NG 3.13.2 on ubunutu, try to send logs to ES 6.0 i failed to do so, here is my inputs. root@ES6:/etc/syslog-ng# syslog-ng -V syslog-ng 3 (3.13.2) Config version: 3.13 Installer-Version: 3.13.2 Revision: 3.13.2-1 Compile-Date: Dec 5 2017 13:24:07 Module-Directory: /usr/lib/syslog-ng/3.13 Module-Path: /usr/lib/syslog-ng/3.13 Available-Modules: afuser,mod-python,afstomp,http,afsql,disk-buffer,mod-java,cef,pseudofile,sdjournal,kvformat,xml,csvparser,snmptrapd-parser,appmodel,confgen,pacctformat,linux-kmsg-format,dbparser,system-source,map-value-pairs,add-contextual-data,date,syslogformat,afamqp,geoip2-plugin,tfgetent,graphite,afmongodb,cryptofuncs,geoip-plugin,afsmtp,afsocket,redis,affile,stardate,basicfuncs,riemann,json-plugin,tags-parser,afprog Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Systemd: on === root@ES6:/etc/syslog-ng# more syslog-ng.conf @version:3.13 @module mod-java @include "scl.conf" options { flush_lines(0); keep_hostname(yes); normalize_hostnames(yes); threaded(yes); }; source s_local { system(); internal(); }; source s_network { syslog(transport(tcp)); }; destination d_all { file ("/var/log/all.log"); }; destination d_elastic { elasticsearch2( client-lib-dir("/usr/share/elasticsearch/lib/:/usr/lib/syslog-ng/3.13/java-modules/") client_mode("http") cluster_url("http://192.168.1.75:9200") index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("syslog") cluster("test") flush-limit("1000") template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)") time-zone("UTC") ); }; log { source(s_network); destination(d_elastic); }; log { source(s_local); destination(d_all); }; === root@ES6:/etc/syslog-ng# ls /usr/share/elasticsearch/lib/ elasticsearch-6.0.1.jar jackson-dataformat-smile-2.8.6.jar jopt-simple-5.0.2.jar lucene-analyzers-common-7.0.1.jar lucene-join-7.0.1.jar lucene-sandbox-7.0.1.jar plugin-cli-6.0.1.jar HdrHistogram-2.1.9.jar jackson-dataformat-yaml-2.8.6.jar jts-1.13.jar lucene-backward-codecs-7.0.1.jar lucene-memory-7.0.1.jar lucene-spatial3d-7.0.1.jar securesm-1.2.jar hppc-0.7.1.jar java-version-checker-6.0.1.jar log4j-1.2-api-2.9.1.jar lucene-core-7.0.1.jar lucene-misc-7.0.1.jar lucene-spatial-7.0.1.jar snakeyaml-1.15.jar jackson-core-2.8.6.jar jna-4.4.0-1.jar log4j-api-2.9.1.jar lucene-grouping-7.0.1.jar lucene-queries-7.0.1.jar lucene-spatial-extras-7.0.1.jar spatial4j-0.6.jar jackson-dataformat-cbor-2.8.6.jar joda-time-2.9.5.jar log4j-core-2.9.1.jar lucene-highlighter-7.0.1.jar lucene-queryparser-7.0.1.jar lucene-suggest-7.0.1.jar t-digest-3.0.jar ==== root@ES6:/etc/syslog-ng# ls /usr/lib/syslog-ng/3.13/java-modules/ elastic.jar elastic-jest-client elastic-v2.jar hdfs.jar http.jar kafka.jar log4j-1.2.16.jar syslog-ng-common.jar syslog-ng-core.jar == root@ES6:/etc/syslog-ng# syslog-ng -Fevd [2017-12-14T23:04:21.552408] Compiling #unnamed sequence [log] at [source generator system:14:12] [2017-12-14T23:04:21.552510] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:10:35] [2017-12-14T23:04:21.552632] Compiling d_all reference [destination] at [/etc/syslog-ng/syslog-ng.conf:27:24] [2017-12-14T23:04:21.552715] Compiling d_all sequence [destination] at [/etc/syslog-ng/syslog-ng.conf:12:1] [2017-12-14T23:04:21.552781] Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf:12:20] [2017-12-14T23:04:21.552884] Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf:12:21] [2017-12-14T23:04:21.553211] Module loaded and initialized successfully; module='syslogformat' [2017-12-14T23:04:21.553425] Processing the time zone file (32bit part); filename='/usr/share/zoneinfo/UTC' [2017-12-14T23:04:21.671696] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/syslog-ng-core.jar; [2017-12-14T23:04:21.672418] Add path to classpath: /usr/share/elasticsearch/lib/; [2017-12-14T23:04:21.673641] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/syslog-ng-core.jar; [2017-12-14T23:04:21.673912] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/syslog-ng-common.jar; [2017-12-14T23:04:21.674218] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/hdfs.jar; [2017-12-14T23:04:21.674704] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/http.jar; [2017-12-14T23:04:21.675858] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/kafka.jar; [2017-12-14T23:04:21.676116] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/log4j-1.2.16.jar; [2017-12-14T23:04:21.676322] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/elastic-v2.jar; [2017-12-14T23:04:21.676484] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/elastic.jar; [2017-12-14T23:04:21.741649] Add path to classpath: /usr/lib/syslog-ng/3.13/java-modules/syslog-ng-core.jar; [2017-12-14T23:04:21.746168] Error initializing message pipeline; plugin name='java', location='#buffer:2:3' Any suggestions ? R!

3 6

3.13 maintenance branch
by Scheidler, Balázs 09 Jan '18

09 Jan '18

Hi, I have just pushed a branch containing a number of bugfixes for the 3.13 branch, that I felt would be useful to be published in various syslog-ng binary packages over the existing 3.13.2 release that we produced a few weeks ago. https://github.com/balabit/syslog-ng/tree/3.13/master The most important change is this: https://github.com/balabit/syslog-ng/commit/7e96705ec0b3570857ed6cd66913999… Which is bug that causes the elasticsearch2 destination SCL wrapper to fails to start up if any __VARARGS__ is used. (see https://github.com/balabit/syslog-ng/issues/1804) The other patches are just my selection of fixes that were deemed low risk to me. We expect the next official syslog-ng release to be 3.14 at the end of February, and I've felt this set of fixes would be useful even in the meantime. At this point, I take this as an experiment: if publishing fixes this way "works" (e.g. fixes get published in the OS packages that we don't maintain ourselves), then we can get some form of maintenance to older open source releases as time permits, with collaboration from our fellow OS maintainers. Also, power users, who are able to compile source code for themselves get help too. Let me know if this is useful. Cheers, -- Bazsi

2 2

Does anyone use kafka as a source
by Evan Rempel 08 Jan '18

08 Jan '18

I would like to source log messages from kafka with syslog-ng so that I can leverage some message parsing components. I may place the post parsed messages back into a kafka pipe but I am having problems determining a good way to get messages from kafka to syslog-ng. Any recommendations? -- Evan

3 2

syslog-ng inside LXC guest receives kernel messages from host
by webman＠manfbraun.de 03 Jan '18

03 Jan '18

Hello! It's the first time, that I use syslog-ng (although the plan ist old - due to the ability to use rabbitmq ...). The host (which is a VM too - do not know exactly which type) has the normal rsyslog installed (was "shipped" with it and not directly of my interest - so I kept it). What I am getting from the host are kernel messages generated from iptables logging - I know the log prefix. The guest has just now no iptables rules at all, but a running ulog2, which (no iptables rules at the moment) just runs, but has nothing to log and messages continue to arrive, after I've stopped it. I had a reboot in between, just to be sure, iptables has not something in its memory. There is a bridge to the host and the outside. While the iptables rules were active, I blocked port 514, but this does not change anything. As told, the messages now continue, even iptables has no active rules. A tcpdump inside the lxc guest does not show packages on port 514. BTW, the messages are logged with the hostname of the guest. syslog-ng uses the standards for it input (system, internal). Probably someone could shed some light on it. It is nothing more worrying, then messages from unknown source! Thanks anyway and best regards, Manfred

2 1

Syslog-ng versions available with RHEL 6
by Carter, Jed 02 Jan '18

02 Jan '18

Good morning, I am currently running syslog-ng OSE version 3.7 and RHEL 6. I'm trying to update my syslog-ng OSE through EPEL. It looks like the only way to get syslog-ng v3.12 is from RHEL 7. What is the latest version of syslog-ng OSE that is supported by RHEL 6? Thank you, Jed -- Jed Carter GCIH, GSEC, CEH Information Security Engineer Partners Health Care The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.

5 5

Problem using Python Parser
by Ronald Fenner 30 Dec '17

30 Dec '17

When I try to load my config with a python parser in it I'm getting this error message: Starting /opt/syslog-ng/sbin/syslog-ng: [2017-12-29T23:00:05.813945] Error loading Python module; module='_syslogng', exception='exceptions.ImportError: No module named _syslogng' [2017-12-29T23:00:05.814066] Error looking Python parser class; parser='p_php_fpm', class='PhpFpmParser', exception='None' [2017-12-29T23:00:05.814116] Error initializing message pipeline; plugin name='python', location='/opt/syslog-ng/etc/syslog-ng.conf:52:20' I build syslog-ng from source with the python options. Here is the -V output syslog-ng 3 (3.11.1) Installer-Version: 3.11.1 Revision: Compile-Date: Dec 29 2017 21:24:13 Module-Directory: /opt/syslog-ng/lib/syslog-ng Module-Path: /opt/syslog-ng/lib/syslog-ng Available-Modules: snmptrapd-parser,affile,cef,afstomp,basicfuncs,pseudofile,tfgetent,afsocket,mod-python,json-plugin,afuser,kvformat,stardate,graphite,dbparser,csvparser,date,afmongodb,system-source,disk-buffer,confgen,linux-kmsg-format,afamqp,map-value-pairs,http,afprog,add-contextual-data,sdjournal,cryptofuncs,syslogformat Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off Enable-Systemd: off Not sure how to fix this as from what I can tell this module is supposed to be compiled in and automatically imported. Ronald Fenner Programmer Game Circus LLC. rfenner(a)gamecircus.com

3 4

Host name issue while logging
by hari ram 16 Dec '17

16 Dec '17

Hi It was running ok till yesterday all the logs are stored with Hostname and date format after upgrade the hostname not able to log in the log format, what is the fix ? Looks like after upgrade to new 3.13 my logs no longer able to use hostname it is storing like this :-2017-12-11.log before INET-2017-12-11.log ( INET Missing why ) any changes ? in the new code 3.13 ? R!

2 2

Insider 2017-12: 3.13; Splunk HEC; Application Adapters; Graylog;
by Czanik, Péter 14 Dec '17

14 Dec '17

Dear syslog-ng users, This is the 64th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng 3.13 released -------------------------- The latest version of syslog-ng, 3.13 is now available. It now parses collected messages automatically using application adapters and can easily forward name-value pairs using the enterprise-wide message model. Support for Graylog and the GELF message format was also added. There are many more smaller features and bug fixes. For a complete list check the release announcements: https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.13.1 and https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.13.2 Sending logs to Splunk through HTTP ----------------------------------- For quite some time, Splunk has recommended to collect syslog messages using syslog-ng, save them to files, and send them to Splunk using forwarders. Unless you have a very high message rate, the HTTP destination of syslog-ng can greatly simplify this logging architecture. Instead of writing messages to files and reading them by a forwarder, syslog-ng can forward messages to Spunk HTTP Event Collector (HEC) directly, using HTTP or HTTPS connections. And if you parse messages using syslog-ng, you can send the resulting name-value pairs to Splunk in JSON format and be able to search them instantly. https://www.balabit.com/blog/sending-logs-splunk-http/ Application Adapters & Enterprise-wide Message Model ---------------------------------------------------- Do you want to simplify parsing your log messages? Try the new “application adapter” and “enterprise-wide message model” frameworks in syslog-ng: you can automatically parse log messages and forward the results to another syslog-ng instance. Optionally, you can also include the original, raw message that you can forward unmodified to a SIEM system for further analysis. Learn how to use these new features from https://www.balabit.com/blog/application-adapters-enterprise-wide-message-m… Graylog as destination in syslog-ng ----------------------------------- Version 3.13 of syslog-ng introduced a graylog2() destination and a GELF (Graylog Extended Log Format) template to make sending syslog messages to Graylog easier. You can also use them to forward simple name-value pairs where the name starts with a dot or underscore. If names of your name-value pairs include dots other than the first character, you should use JSON formatting directly instead of the GELF template and send logs to a raw tcp port in Graylog, which can then extract fields from nested JSON. https://www.balabit.com/blog/graylog-destination-syslog-ng/ Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

"Error initializing message pipeline; plugin name='java', location='#buffer:2:3'" after update to syslog-ng 3.12
by Roegels Ruben 14 Dec '17

14 Dec '17

Hi there, I'm using the OSE of syslog-ng for some weeks. Today I updated syslog-ng to 3.13 and afterwards I started seeing -- Unit syslog-ng.service has begun starting up. Dez 13 11:53:59 syslog syslog-ng[23803]: [2017-12-13T11:53:59.572246] Error initializing message pipeline; plugin name='java', location='#buffer:2:3' Dez 13 11:53:59 syslog systemd[1]: syslog-ng.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Dez 13 11:53:59 syslog systemd[1]: Failed to start System Logger Daemon. -- Subject: Unit syslog-ng.service has failed -- Defined-By: systemd-13T11:53:59.572246] Error initializing message pipeline; plugin name='java', location='#buffer:2:3' In "journalctl -xe" If I disable the elasticsearch destination syslog-ng starts up fine. I don't know what to look at as the error message is not very meaningful to me. Did I miss some configuration format changes regarding the "elasticsearch2" destination? The header of my "/etc/syslog-ng/syslog-ng.conf": === Snip === @version: 3.13 @module mod-java @include "scl.conf" === Snip === My elasticsearch config for syslog-ng: https://pastebin.com/g6UB3gBi "ls -l {/usr/share/elasticsearch/lib/,/usr/lib/syslog-ng/3.13/java-modules/}" https://pastebin.com/nqYAVhHL "cat /etc/systemd/system/multi-user.target.wants/syslog-ng.service" https://pastebin.com/UzfQXvZL I really appreciate any help. Thank you! Best regards, Ruben ____________________________________________________________________________ netsys IT GmbH Ruben Roegels Nelly-Sachs-Str. 4 79111 Freiburg Tel. +49 761 888 691 0 Fax +49 761 888 691 10 <http://www.netsys-it.de> Ruben.Roegels(a)netsys-it.de ____________________________________________________________________________ Amtsgericht Freiburg HRB 5752 Geschaeftsfuehrer: Juergen Dufner ____________________________________________________________________________

3 2

SYSLOG-NG config issue and suggestion
by hari ram 08 Dec '17

08 Dec '17

Hi Now my syslog server running. Just try to use in my network and export the logs Logstash or elastic Search Take example I have SWITCH1, SWITCH2 ....100 How can i send logs to elasticsearch, as a ID of switch name and so i can search in Elastic search and make Kibana dashboards R!

2 1

Elastic 5 Support For More Than HTTP?
by JONATHAN PRICE 08 Dec '17

08 Dec '17

All, A project I'm working on is wanting to leverage syslog-ng as our log shipper, but we're ideally wanting it to communicate with Elasticsearch 5 over HTTPS or searchguard. My understanding is current support is only for HTTP for Elastic 5. Are there plans to add elastic 5 support for other client modes than HTTP anytime soon? I'm fine just doing HTTP for now and switching to secured communications later. If not, I'm okay spending a bit of time trying to get that working myself; I haven't delved too deep into the source, but the changes involving updates to the elastic module do not seem to be too too intensive and only include Java changes. Is this a correct interpretation? It doesn't look like it would be too difficult to do, but since it hasn't been done I was wondering if there was something else involved I'm missing. -- Thanks, Jonathan Price Associate Scientist, Cyber Security Raytheon BBN Technologies

2 1

Installation issue
by hari ram 07 Dec '17

07 Dec '17

Hi I have ubuntu Server install 16.X with all dependency packages #124-Ubuntu SMP Fri Nov 10 18:29:59 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux # more /etc/debian_version stretch/sid when i try to install syslog-ng with debian package as mentioned below steps. ~# more /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/xUb… ./ # apt-get update Hit:1 http://gb.archive.ubuntu.com/ubuntu xenial InRelease Hit:2 http://gb.archive.ubuntu.com/ubuntu xenial-updates InRelease Get:3 http://gb.archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB] Ign:4 http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/xUb… ./ InRelease Hit:5 http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/xUb… ./ Release Get:6 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB] Fetched 204 kB in 0s (350 kB/s) Reading package lists... Done # apt-get install syslog-ng-core Reading package lists... Done Building dependency tree Reading state information... Done Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: syslog-ng-core : Depends: libjson-c3 (>= 0.10) but it is not installable E: Unable to correct problems, you have held broken packages. But i have already install libjason package # apt-get install libjson-c-dev Reading package lists... Done Building dependency tree Reading state information... Done libjson-c-dev is already the newest version (0.11-4ubuntu2). 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. what is the issue, how can i install latest version of syslog-ng default syslog-ng only come with 3.5.6 version, but iam looking 3.11 and above version to install any suggestions ? R!

2 2