September 2016 - syslog-ng - lists.balabit.hu

Re: [syslog-ng] disk based buffering
by Scheidler, Balázs 28 Sep '16

28 Sep '16

Syslog-ng attempts to address application level failures with reliable disk buffer but kernel level crashes/power failures are not covered, at least you can suffer message loss, but the queue in general should stay intact. There's a tool for reading disk queue files, iirc the name is dqtool, should be included in your package. On Sep 27, 2016 8:35 PM, <thejaguar(a)tutanota.de> wrote: Hi, I have been using disk based buffering with reliable turned on yes as suggested here :- https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng- ose-guide-admin/html/configuring-diskbuffer-reliable.html This has been working great for me on an embedded linux device which does not have internet connection except when the application running on it turns on the modem/pppd when it has to send some data, basically to save battery power. Now syslog-ng is brilliant and sends all the stored/queued logs immediately upon detecting network connection as long as system stays alive. Now the challenge is if the device has a system reset or kernel crash in between network connection availability, will syslog-ng send unsent logs upon next system reboot when it gets the network connection ? Or it resets the queue and tracking upon system reset/boot ? I noticed any logs generated in between power resets and which are not sent are not transmitted on next net connection. Is it expected behaviour ? If not then what wrong I am doing ? also how can I read whats in /var/lib/syslog-ng/syslog-ng-00000.rqf or syslog-ng.persist ? ======================= destination d_net { network ( "`myloghost`" port(`mylogport`) transport("tls") tls( ca-dir("/etc/syslog-ng/ca") peer-verify(required-trusted) ssl-options(no-sslv3,no-tlsv1) ) disk-buffer( reliable(yes) mem-buf-size(1M) disk-buf-size(5M) qout-size(64) ) template("<$PRI> $FACILITY $ISODATE $HOST $PROGRAM $MSG\n") ); }; syslog-ng 3.8.1 Installer-Version: 3.8.1 Revision: Module-Directory: /usr/lib/syslog-ng Module-Path: /usr/lib/syslog-ng Available-Modules: cef,affile,basicfuncs,system-source,cryptofuncs,graphite, pseudofile,afuser,kvformat,add-contextual-data,date, csvparser,linux-kmsg-format,confgen,syslogformat,afprog, disk-buffer,dbparser,afsot Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: off Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off ======================= Thanks ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

2 2

Getting 100k EPS into syslog-ng
by Clayton Dukes 28 Sep '16

28 Sep '16

I have searched high and low for actual configs on how to do this, but can't nail down anything solid. Does anyone have a good document or site I can reference which outlines all the details of getting syslog-ng to receive at 100k eps? Including things like what is needed to add/change in sysctl.conf, etc. I've found pages, like a blog entry by Etsy claiming it, but they left out the details on how to actually do it unfortunately. [cid:image001.png@01D218DE.0E88D320] Clayton Dukes / Founder and CTO (818) 869-4552/ cdukes(a)logzilla.net<mailto:cdukes@logzilla.net> LogZilla Corporation http://www.logzilla.net<http://www.logzilla.net/>

2 1

disk based buffering
by thejaguar＠tutanota.de 27 Sep '16

27 Sep '16

Hi,I have been using disk based buffering with reliable turned on yes as suggested here :-https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-diskbuffer-reliable.html This has been working great for me on an embedded linux device which does not have internet connection except when the application running on it turns on the modem/pppd when it has to send some data, basically to save battery power. Now syslog-ng is brilliant and sends all the stored/queued logs immediately upon detecting network connection as long as system stays alive. Now the challenge is if the device has a system reset or kernel crash in between network connection availability, will syslog-ng send unsent logs upon next system reboot when it gets the network connection ? Or it resets the queue and tracking upon system reset/boot ?I noticed any logs generated in between power resets and which are not sent are not transmitted on next net connection. Is it expected behaviour ? If not then what wrong I am doing ? also how can I read whats in /var/lib/syslog-ng/syslog-ng-00000.rqf or syslog-ng.persist ? ======================= destination d_net { network ( "`myloghost`" port(`mylogport`) transport("tls") tls( ca-dir("/etc/syslog-ng/ca") peer-verify(required-trusted) ssl-options(no-sslv3,no-tlsv1) ) disk-buffer( reliable(yes) mem-buf-size(1M) disk-buf-size(5M) qout-size(64) ) template("<$PRI> $FACILITY $ISODATE $HOST $PROGRAM $MSG\n") );}; syslog-ng 3.8.1Installer-Version: 3.8.1Revision:Module-Directory: /usr/lib/syslog-ngModule-Path: /usr/lib/syslog-ngAvailable-Modules: cef,affile,basicfuncs,system-source,cryptofuncs,graphite,pseudofile,afuser,kvformat,add-contextual-data,date,csvparser,linux-kmsg-format,confgen,syslogformat,afprog,disk-buffer,dbparser,afsotEnable-Debug: offEnable-GProf: offEnable-Memtrace: offEnable-IPv6: offEnable-Spoof-Source: offEnable-TCP-Wrapper: offEnable-Linux-Caps: off ======================= Thanks

1 0

syslog message without timestamp
by Csuti Istvan 23 Sep '16

23 Sep '16

Hi All! I would like send UDP message to the syslog server without timestamp. I readed RFC3164 and RFC5424 and syslog-ng-ose-guide-admin.pdf. The new "RFC 5424 (also called the IETF-syslog protocol): see Section 2.8.2, IETF-syslog messages (p. 13)" support NIL values "-" in HEADER. I sended this "<110>1 - hostname procname procid - - Message" The syslog-ng server logged this: "Sep 21 19:29:54 h164-19 1: - hostname procname procid - - Message" Question: What is wrong in my message string? The syslog-ng server why do not decode "1" after ">" as version? Can i send old syslog message (RFC3164) without timestamp? Best Regards Istvan Csuti

3 5

Listen backlog issues?
by Claus Albøge 19 Sep '16

19 Sep '16

Hi, I have ~6000 hosts sending syslog-TLS to a syslog-ng server. I’m seeing a lot of reconnects from the clients, and on the syslog-ng servers I get the following messages: TCP: request_sock_TCP: Possible SYN flooding on port 6514. Sending cookies. Check SNMP counters. netstat -s show the following: $ netstat -s | grep -i listen 10480620 times the listen queue of a socket overflowed 11144791 SYNs to LISTEN sockets dropped A few resources suggest this could be due to the applications socket listen backlog. Starting syslog-ng with strace reveals the following: 30067 listen(5, 255) = 0 30067 listen(11, 255) = 0 30067 listen(12, 255) = 0 30067 listen(17, 255) = 0 It seems like the backlog is hardcoded to 255 in the source. A bit more info: Saving the output from: “netstat -n | grep -c SYN_RECV” to a file a few times per second, shows the following: 2016-09-19 09:54:14.767469015: 0 2016-09-19 09:54:14.906638364: 0 2016-09-19 09:54:15.048591252: 0 2016-09-19 09:54:15.184325070: 0 2016-09-19 09:54:15.324150368: 0 2016-09-19 09:54:15.459749187: 0 2016-09-19 09:54:15.587983284: 0 2016-09-19 09:54:15.722558975: 42 2016-09-19 09:54:15.873601766: 256 2016-09-19 09:54:16.020145083: 247 2016-09-19 09:54:16.177231109: 178 2016-09-19 09:54:16.340875439: 178 2016-09-19 09:54:16.488506916: 178 2016-09-19 09:54:16.637815500: 176 2016-09-19 09:54:16.790781389: 149 2016-09-19 09:54:16.950254517: 141 2016-09-19 09:54:17.125316109: 131 2016-09-19 09:54:17.286839687: 131 2016-09-19 09:54:17.445583267: 127 2016-09-19 09:54:17.598144758: 127 2016-09-19 09:54:17.747967473: 124 2016-09-19 09:54:17.905359412: 2 2016-09-19 09:54:18.062665358: 2 2016-09-19 09:54:18.239973822: 0 2016-09-19 09:54:18.398021514: 0 2016-09-19 09:54:18.548074304: 0 This matches the entry in /var/log/messages Sep 19 09:54:15 log03 kernel: [1116811.081666] TCP: request_sock_TCP: Possible SYN flooding on port 6514. Sending cookies. Check SNMP counters. Are there any tunings in the syslog-ng configuration I need to implement, or do I have to compile syslog-ng myself with a larger listen backlog? Perhaps this should be configurable? My source definition for syslog-TLS is like this: source s_syslog_tls { syslog( ip(0.0.0.0) port(6514) log_fetch_limit(100) log_iw_size(11000000) max-connections(10000) transport("tls") tls( key-file("/etc/pki/tls/certs/logserver.key") cert-file("/etc/pki/tls/certs/logserver.crt") peer-verify(optional-untrusted) ) ); }; $ syslog-ng -V syslog-ng 3.8.1 Installer-Version: 3.8.1 Revision: Module-Directory: //usr/lib64/syslog-ng Module-Path: //usr/lib64/syslog-ng Available-Modules: disk-buffer,sdjournal,afstomp,json-plugin,cryptofuncs,graphite,cef,kvformat,add-contextual-data,dbparser,pseudofile,curl,csvparser,syslogformat,confgen,afsocket,afuser,date,linux-kmsg-format,system-source,basicfuncs,afamqp,affile,afprog Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: off syslog-ng-3.8.1-1.el7.centos.x86_64 from https://copr-be.cloud.fedoraproject.org/results/czanik/syslog-ng38/epel-7-x… Please let me know if more info is needed. /Claus A

3 2

syslog-ng in ubuntu 16.10
by Gil Bahat 15 Sep '16

15 Sep '16

Ubuntu 16.10 is now in feature freeze and there is (apparently) one bug on s390x platform that prevents v3.7.3 from making it into the release in lieu of the older 3.5.6. that's a real shame, I hope someone from this list can help out. here is the failing build log on ubuntu 16.10: https://launchpadlibrarian.net/281969183/buildlog_ubuntu- yakkety-s390x.syslog-ng-incubator_0.5.0-1_BUILDING.txt.gz this does build correctly on debian sid though: https://buildd.debian.org/status/fetch.php?pkg=syslog- ng-incubator&arch=s390x&ver=0.5.0-1&stamp=1468967484 Regards, Gil Bahat

2 1

syslog-ng 3.8.1 rpm packages
by Czanik, Péter 15 Sep '16

15 Sep '16

Hi, This week my packaging focus was to make sure, that 3.8.1 will be available as part of the next openSUSE release. My initial unofficial 3.8.1 RPM packages with Java modules, etc. are available but still need some polishing: - move curl (HTTP) module into a separate sub package - update rust modules and on Fedora / EPEL: - build Java modules without Internet access - re-enable missing features, like riemann (which were turned off to make local test builds easier) Fedora/EPEL7: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng38/ EPEL6: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng38epel6/ openSUSE/SLES: https://build.opensuse.org/project/show/home:czanik:syslog-ng38 I hope to have more time working on this by the end of next week. Any feedback, suggestions, patches are welcome Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

4 6

Insider 2016-09: 3.8.1 released; GSoC; Turris Omnia; meetups;
by Czanik, Péter 15 Sep '16

15 Sep '16

Dear syslog-ng users, This is the 51st issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng 3.8.1 released ------------------------ I am happy to announce that after almost a year of development, syslog-ng 3.8.1 has been released. This is the first stable release from the syslog-ng 3.8 branch. Version 3.8.1 adds some long-awaited features to syslog-ng and improves the performance of name-value-pair related operations. Announcement: https://www.balabit.com/blog/syslog-ng-hits-major-milestone-with-version-3-… Development, packaging related changes: https://czanik.blogs.balabit.com/2016/08/syslog-ng-3-8-what-changed/ Google Summer of Code 2016 -------------------------- The syslog-ng project has a long tradition of participating in the Google Summer of Code program. Each year some interesting new features originate from GSoC students. Here are some of the blogs, where students explain what they developed during the Summer: Websocket destination: https://litterbear.github.io/2016-08-26-GSOC2016/ Release automation: http://black-perl.in/gsoc16-syslog-ng-autorel-summary/ Kafka source: https://vithulanmv.wordpress.com/2016/08/21/syslog-ng-kafka-source-in-depth… As command line tool: https://kvch.github.io/syslog-ng-as-a-command-line-tool.html If you are at the GSoC mentor summit (Sunnyvale, California, October 28-30, https://sites.google.com/site/2016gsocmentorsummit/home) you can even meet one of our mentors, Viktor Juhász, at the event. As a syslog-ng developer, he can talk to you about syslog-ng, even get you started developing syslog-ng. Coming up: All Things Open -------------------------- My next syslog-ng presentation will be in Raleigh, NC, at the All Things Open conference. I will talk about scaling a logging infrastructure using syslog-ng: https://allthingsopen.org/talk/scaling-your-logging-infrastructure/ syslog-ng in the Turris Omnia Linux router ------------------------------------------ The Turris Omnia is a Linux-based router running on an ARM CPU. As it turned out during a conference discussion, this popular little device is running syslog-ng: https://czanik.blogs.balabit.com/2016/06/turris-omnia-a-linux-router-powere… If you know of anything that could be added to https://syslog-ng.org/powered-by-syslog-ng/, let me know! syslog-ng at meetups -------------------- There were two meetups this summer where a member of Balabit’s crew talked about syslog-ng. Both were quite a positive experience. If you know of a meetup where participants might be interested in syslog-ng, please let me know! I can be reached by e-mail or on Twitter as @PCzanik San Francisco: https://bazsi.blogs.balabit.com/2016/07/syslog-ng-in-san-francisco/ London: https://www.balabit.com/blog/linuxing-in-london/ NEW RELEASES syslog-ng 3.8.1: https://www.balabit.com/blog/syslog-ng-hits-major-milestone-with-version-3-… SSB 4F4: https://istvanszabo.blogs.balabit.com/2016/07/improve-the-efficiency-of-sec… Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

pattern_db ES use case.
by Scot Needy 12 Sep '16

12 Sep '16

Hello List, I’m trying to understand the use case of pattern_db when the destination will be ES. My initial understanding was that I could use patterndb as an engine to tag my log message data with attributes, but it doesn’t seem to work that way. I have a json output like this in Kibana. In a loghost deployment, It looks like I would need to manually align a patterndb filter with each host_message type even before patterned comes into play. Q) What is the right solution for enriching message data into ES ? Example JSON from Kibana MESSAGE is not parsed. ======================= { "_index": "syslog-ng_2016.09.12", "_type": "syslog-ng", "_id": "AVcdnzJla9VjMdxDYo8Z", "_score": null, "_source": { "PROGRAM": “###-asa11", "PRIORITY": "warning", "MESSAGE": "%ASA-4-106023: Deny tcp src outside:###.###.31.2/33553 dst public:###.###.7.191/443 by access-group \"outside_access_in\" [0x2c1c6a65, 0x0]", "ISODATE": "2016-09-12T13:57:03-04:00", "HOST": “###.###.###.###", "FACILITY": "local5", "@timestamp": "2016-09-12T13:57:03-04:00" }, "fields": { "ISODATE": [ 1473703023000 ], "@timestamp": [ 1473703023000 ] }, "sort": [ 1473703023000 ] }

2 5

Archive module ?
by Scot Needy 12 Sep '16

12 Sep '16

Hi All, Is there a better way to archive my syslog data once it’s been received and injected into elk,spunk or whatever ? I see how I can log my data to flat files using date macros then compress and archive as normal data but is there a better way to write out the data in the same format it was received to facilitate re-injesting ? For example: My syslog data on file does not include the original syslog header data . Syslog is just one example.

2 6