- syslog-ng - lists.balabit.hu

Converting multiline text input to concatanated single-line syslog format
by Newport, Brendan (Contractor - Security Operations - Development & Support) 30 Sep '16

30 Sep '16

Classification: Public Hi! Thanks for the responses. The multi-line-garbage() line was only in there to aid testing. I will look at the LF/CR elements in the input. We have a support contract with BalaBit with a local Var defined and so I'll raise a support ticket. Version 5.0.6LTS is the version rolled-out across our enterprise, but an upgrade to 6LTS will likely occur before October 2017. So testing was performed with the version in use. Lloyds Banking Group plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC95000. Telephone: 0131 225 4555. Lloyds Bank plc. Registered Office: 25 Gresham Street, London EC2V 7HN. Registered in England and Wales no. 2065. Telephone 0207626 1500. Bank of Scotland plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC327000. Telephone: 03457 801 801. Cheltenham & Gloucester plc. Registered Office: Barnett Way, Gloucester GL4 3RL. Registered in England and Wales 2299428. Telephone: 0345 603 1637 Lloyds Bank plc, Bank of Scotland plc are authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and Prudential Regulation Authority. Cheltenham & Gloucester plc is authorised and regulated by the Financial Conduct Authority. Halifax is a division of Bank of Scotland plc. Cheltenham & Gloucester Savings is a division of Lloyds Bank plc. HBOS plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC218813. This e-mail (including any attachments) is private and confidential and may contain privileged material. If you have received this e-mail in error, please notify the sender and delete it (including any attachments) immediately. You must not copy, distribute, disclose or use any of the information in it or any attachments. Telephone calls may be monitored or recorded.

1 0

attempting to get parsed logs into elasticsearch
by Russell Fulton 30 Sep '16

30 Sep '16

Hi folks I am trying to get some parsed logs into elasticssearch but have ended up with a cryptic error message: Starting syslog-ng /usr/lib/jvm/java-1.7.0-oracle-1.7.0.91.x86_64/jre/lib/amd64/server [2016-09-30T12:43:43.649899] Error initializing message pipeline; which almost certainly relates to the ES set up but I have no idea what is actually wrong. The ES logs do not show anything. Config file: @version: 3.8 @module mod-java @include "scl.conf" options { use_dns (no); use_fqdn (no); keep_hostname (yes); }; source s_loghost { tcp(flags(no-multi-line) port(1514) keep-alive(yes)); }; destination d_syslog { file("/var/log/syslog.log"); }; destination d_elastic { elasticsearch( index("auth_${YEAR}.${MONTH}.${DAY}") type("auth") cluster("security") flush-limit("1000") ); }; parser p_patterns { db-parser( file("/etc/syslog-ng/merged.xml")); }; log { source(s_loghost); parser (p_patterns); destination(d_elastic ); }; The same configuration with a json file destination works fine. Any hints on what to look at appreciated. The ES instance running on the host is set to data: no and I expect it to ship the data to one of the other nodes which has storage. Russell (who admits to being an ES novice)

2 3

moving debian directory from master to release branch on GitHub
by Noémi Ványi 29 Sep '16

29 Sep '16

Hello! Debian directory is being moved from master to another branch named release. Thus, the source code is separated from the build. If someone wants to create a Debian package, he/she must checkout that branch first. If someone has any concerns or notes, please share either here or in the PR on GitHub. PR: https://github.com/balabit/syslog-ng/pull/1215 kvch

2 1

experimental syslog-ng incubator rpm-s for 3.8
by Czanik, Péter 29 Sep '16

29 Sep '16

Hi, I just updated my unofficial syslog-ng 3.8 fedora/rhel repo with an experimental incubator package. I used the latest git HEAD snapshot together with this patch: https://github.com/balabit/syslog-ng-incubator/pull/158 Notable changes since my packages for 3.7: - bugfixes & code changes needed to compile on 3.8 - grok support removed - librdkafka updated to 0.9.1 Due to lack of Kafka, ZeroMQ, etc. I did not test the packages, just compiled them. If you need grok, let me know and I'll try to fix the package... As usual, packages are available at https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng38/ Your feedback, be it negative or positive are very welcome! Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

Re: [syslog-ng] disk based buffering
by thejaguar＠tutanota.de 29 Sep '16

29 Sep '16

So with dqtool ( cool) I noticed that actual problem is that syslog-ng is not writing to disk queue immediately when a message is generated. The message goes to usual place like var/log/message etc but not in the queue where it should because it has not been transmitted to remote host. Now I am thinking there has to be a memory limit before it starts flushing/dumping unsent message to disk queue ? I tried mem-buf-size(1) and log-fifo-size(1) but none of them works. Is it a static value or is it configurable ? Thanks 27. Sep 2016 12:41 by thejaguar(a)tutanota.de: > > Thanks.> So if the queue stays intact, syslog-ng will try to send unsent messages as and when it starts ? even after 2-3 days ? it does not reset the queue or tracking ever ? > Thanks again > > 27. Sep 2016 12:11 by > balazs.scheidler(a)balabit.com> : > > >> >> Syslog-ng attempts to address application level failures with reliable disk buffer but kernel level crashes/power failures are not covered, at least you can suffer message loss, but the queue in general should stay intact. >> >> There's a tool for reading disk queue files, iirc the name is dqtool, should be included in your package. >> >> On Sep 27, 2016 8:35 PM, <>> thejaguar(a)tutanota.de>> > wrote: >> >>> >>> Hi,>>> I have been using disk based buffering with reliable turned on yes as suggested here :->>> https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-… >>> >>> This has been working great for me on an embedded linux device which does not have internet connection except when the application running on it turns on the modem/pppd when it has to send some data, basically to save battery power. Now syslog-ng is brilliant and sends all the stored/queued logs immediately upon detecting network connection as long as system stays alive. Now the challenge is if the device has a system reset or kernel crash in between network connection availability, will syslog-ng send unsent logs upon next system reboot when it gets the network connection ? Or it resets the queue and tracking upon system reset/boot ?>>> I noticed any logs generated in between power resets and which are not sent are not transmitted on next net connection. Is it expected behaviour ? If not then what wrong I am doing ? also how can I read whats in /var/lib/syslog-ng/syslog-ng-00000.rqf or syslog-ng.persist ? >>> ======================= >>> >>> destination d_net {>>> network (>>> "`myloghost`" port(`mylogport`) transport("tls")>>> tls( ca-dir("/etc/syslog-ng/ca") peer-verify(required-trusted) ssl-options(no-sslv3,no-tlsv1) )>>> disk-buffer( reliable(yes) mem-buf-size(1M) disk-buf-size(5M) qout-size(64) )>>> template("<$PRI> $FACILITY $ISODATE $HOST $PROGRAM $MSG\n")>>> );>>> }; >>> syslog-ng 3.8.1>>> Installer-Version: 3.8.1>>> Revision:>>> Module-Directory: /usr/lib/syslog-ng>>> Module-Path: /usr/lib/syslog-ng>>> Available-Modules: cef,affile,basicfuncs,system-source,cryptofuncs,graphite,pseudofile,afuser,kvformat,add-contextual-data,date,csvparser,linux-kmsg-format,confgen,syslogformat,afprog,disk-buffer,dbparser,afsot>>> Enable-Debug: off>>> Enable-GProf: off>>> Enable-Memtrace: off>>> Enable-IPv6: off>>> Enable-Spoof-Source: off>>> Enable-TCP-Wrapper: off>>> Enable-Linux-Caps: off >>> ======================= >>> >>> Thanks >>> >>> >>> ______________________________________________________________________________ >>> Member info: >>> https://lists.balabit.hu/mailman/listinfo/syslog-ng >>> Documentation: >>> http://www.balabit.com/support/documentation/?product=syslog-ng >>> FAQ: >>> http://www.balabit.com/wiki/syslog-ng-faq >>> >>> >>> >> >>

5 6

Re: [syslog-ng] disk based buffering
by Scheidler, Balázs 28 Sep '16

28 Sep '16

Syslog-ng attempts to address application level failures with reliable disk buffer but kernel level crashes/power failures are not covered, at least you can suffer message loss, but the queue in general should stay intact. There's a tool for reading disk queue files, iirc the name is dqtool, should be included in your package. On Sep 27, 2016 8:35 PM, <thejaguar(a)tutanota.de> wrote: Hi, I have been using disk based buffering with reliable turned on yes as suggested here :- https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng- ose-guide-admin/html/configuring-diskbuffer-reliable.html This has been working great for me on an embedded linux device which does not have internet connection except when the application running on it turns on the modem/pppd when it has to send some data, basically to save battery power. Now syslog-ng is brilliant and sends all the stored/queued logs immediately upon detecting network connection as long as system stays alive. Now the challenge is if the device has a system reset or kernel crash in between network connection availability, will syslog-ng send unsent logs upon next system reboot when it gets the network connection ? Or it resets the queue and tracking upon system reset/boot ? I noticed any logs generated in between power resets and which are not sent are not transmitted on next net connection. Is it expected behaviour ? If not then what wrong I am doing ? also how can I read whats in /var/lib/syslog-ng/syslog-ng-00000.rqf or syslog-ng.persist ? ======================= destination d_net { network ( "`myloghost`" port(`mylogport`) transport("tls") tls( ca-dir("/etc/syslog-ng/ca") peer-verify(required-trusted) ssl-options(no-sslv3,no-tlsv1) ) disk-buffer( reliable(yes) mem-buf-size(1M) disk-buf-size(5M) qout-size(64) ) template("<$PRI> $FACILITY $ISODATE $HOST $PROGRAM $MSG\n") ); }; syslog-ng 3.8.1 Installer-Version: 3.8.1 Revision: Module-Directory: /usr/lib/syslog-ng Module-Path: /usr/lib/syslog-ng Available-Modules: cef,affile,basicfuncs,system-source,cryptofuncs,graphite, pseudofile,afuser,kvformat,add-contextual-data,date, csvparser,linux-kmsg-format,confgen,syslogformat,afprog, disk-buffer,dbparser,afsot Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: off Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off ======================= Thanks ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

2 2

Getting 100k EPS into syslog-ng
by Clayton Dukes 28 Sep '16

28 Sep '16

I have searched high and low for actual configs on how to do this, but can't nail down anything solid. Does anyone have a good document or site I can reference which outlines all the details of getting syslog-ng to receive at 100k eps? Including things like what is needed to add/change in sysctl.conf, etc. I've found pages, like a blog entry by Etsy claiming it, but they left out the details on how to actually do it unfortunately. [cid:image001.png@01D218DE.0E88D320] Clayton Dukes / Founder and CTO (818) 869-4552/ cdukes(a)logzilla.net<mailto:cdukes@logzilla.net> LogZilla Corporation http://www.logzilla.net<http://www.logzilla.net/>

2 1

disk based buffering
by thejaguar＠tutanota.de 27 Sep '16

27 Sep '16

Hi,I have been using disk based buffering with reliable turned on yes as suggested here :-https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-diskbuffer-reliable.html This has been working great for me on an embedded linux device which does not have internet connection except when the application running on it turns on the modem/pppd when it has to send some data, basically to save battery power. Now syslog-ng is brilliant and sends all the stored/queued logs immediately upon detecting network connection as long as system stays alive. Now the challenge is if the device has a system reset or kernel crash in between network connection availability, will syslog-ng send unsent logs upon next system reboot when it gets the network connection ? Or it resets the queue and tracking upon system reset/boot ?I noticed any logs generated in between power resets and which are not sent are not transmitted on next net connection. Is it expected behaviour ? If not then what wrong I am doing ? also how can I read whats in /var/lib/syslog-ng/syslog-ng-00000.rqf or syslog-ng.persist ? ======================= destination d_net { network ( "`myloghost`" port(`mylogport`) transport("tls") tls( ca-dir("/etc/syslog-ng/ca") peer-verify(required-trusted) ssl-options(no-sslv3,no-tlsv1) ) disk-buffer( reliable(yes) mem-buf-size(1M) disk-buf-size(5M) qout-size(64) ) template("<$PRI> $FACILITY $ISODATE $HOST $PROGRAM $MSG\n") );}; syslog-ng 3.8.1Installer-Version: 3.8.1Revision:Module-Directory: /usr/lib/syslog-ngModule-Path: /usr/lib/syslog-ngAvailable-Modules: cef,affile,basicfuncs,system-source,cryptofuncs,graphite,pseudofile,afuser,kvformat,add-contextual-data,date,csvparser,linux-kmsg-format,confgen,syslogformat,afprog,disk-buffer,dbparser,afsotEnable-Debug: offEnable-GProf: offEnable-Memtrace: offEnable-IPv6: offEnable-Spoof-Source: offEnable-TCP-Wrapper: offEnable-Linux-Caps: off ======================= Thanks

1 0

syslog message without timestamp
by Csuti Istvan 23 Sep '16

23 Sep '16

Hi All! I would like send UDP message to the syslog server without timestamp. I readed RFC3164 and RFC5424 and syslog-ng-ose-guide-admin.pdf. The new "RFC 5424 (also called the IETF-syslog protocol): see Section 2.8.2, IETF-syslog messages (p. 13)" support NIL values "-" in HEADER. I sended this "<110>1 - hostname procname procid - - Message" The syslog-ng server logged this: "Sep 21 19:29:54 h164-19 1: - hostname procname procid - - Message" Question: What is wrong in my message string? The syslog-ng server why do not decode "1" after ">" as version? Can i send old syslog message (RFC3164) without timestamp? Best Regards Istvan Csuti

3 5

Listen backlog issues?
by Claus Albøge 19 Sep '16

19 Sep '16

Hi, I have ~6000 hosts sending syslog-TLS to a syslog-ng server. I’m seeing a lot of reconnects from the clients, and on the syslog-ng servers I get the following messages: TCP: request_sock_TCP: Possible SYN flooding on port 6514. Sending cookies. Check SNMP counters. netstat -s show the following: $ netstat -s | grep -i listen 10480620 times the listen queue of a socket overflowed 11144791 SYNs to LISTEN sockets dropped A few resources suggest this could be due to the applications socket listen backlog. Starting syslog-ng with strace reveals the following: 30067 listen(5, 255) = 0 30067 listen(11, 255) = 0 30067 listen(12, 255) = 0 30067 listen(17, 255) = 0 It seems like the backlog is hardcoded to 255 in the source. A bit more info: Saving the output from: “netstat -n | grep -c SYN_RECV” to a file a few times per second, shows the following: 2016-09-19 09:54:14.767469015: 0 2016-09-19 09:54:14.906638364: 0 2016-09-19 09:54:15.048591252: 0 2016-09-19 09:54:15.184325070: 0 2016-09-19 09:54:15.324150368: 0 2016-09-19 09:54:15.459749187: 0 2016-09-19 09:54:15.587983284: 0 2016-09-19 09:54:15.722558975: 42 2016-09-19 09:54:15.873601766: 256 2016-09-19 09:54:16.020145083: 247 2016-09-19 09:54:16.177231109: 178 2016-09-19 09:54:16.340875439: 178 2016-09-19 09:54:16.488506916: 178 2016-09-19 09:54:16.637815500: 176 2016-09-19 09:54:16.790781389: 149 2016-09-19 09:54:16.950254517: 141 2016-09-19 09:54:17.125316109: 131 2016-09-19 09:54:17.286839687: 131 2016-09-19 09:54:17.445583267: 127 2016-09-19 09:54:17.598144758: 127 2016-09-19 09:54:17.747967473: 124 2016-09-19 09:54:17.905359412: 2 2016-09-19 09:54:18.062665358: 2 2016-09-19 09:54:18.239973822: 0 2016-09-19 09:54:18.398021514: 0 2016-09-19 09:54:18.548074304: 0 This matches the entry in /var/log/messages Sep 19 09:54:15 log03 kernel: [1116811.081666] TCP: request_sock_TCP: Possible SYN flooding on port 6514. Sending cookies. Check SNMP counters. Are there any tunings in the syslog-ng configuration I need to implement, or do I have to compile syslog-ng myself with a larger listen backlog? Perhaps this should be configurable? My source definition for syslog-TLS is like this: source s_syslog_tls { syslog( ip(0.0.0.0) port(6514) log_fetch_limit(100) log_iw_size(11000000) max-connections(10000) transport("tls") tls( key-file("/etc/pki/tls/certs/logserver.key") cert-file("/etc/pki/tls/certs/logserver.crt") peer-verify(optional-untrusted) ) ); }; $ syslog-ng -V syslog-ng 3.8.1 Installer-Version: 3.8.1 Revision: Module-Directory: //usr/lib64/syslog-ng Module-Path: //usr/lib64/syslog-ng Available-Modules: disk-buffer,sdjournal,afstomp,json-plugin,cryptofuncs,graphite,cef,kvformat,add-contextual-data,dbparser,pseudofile,curl,csvparser,syslogformat,confgen,afsocket,afuser,date,linux-kmsg-format,system-source,basicfuncs,afamqp,affile,afprog Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: off syslog-ng-3.8.1-1.el7.centos.x86_64 from https://copr-be.cloud.fedoraproject.org/results/czanik/syslog-ng38/epel-7-x… Please let me know if more info is needed. /Claus A

3 2