June 2018 - syslog-ng - lists.balabit.hu

Unofficial syslog-ng 3.16 Debian/Ubuntu packages
by Laszlo Budai 18 Jun '18

18 Jun '18

Hi, syslog-ng 3.16.1[1] packages are available in OBS repo[2]. List of supported OSs: * Debian 8.0 * Debian 9.0 * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 16.10 * Ubuntu 17.04 * Ubuntu 17.10 * Ubuntu 18.04 In case of Debian 9.0, armv7l architecture also supported, so you can install it to your Raspberry Pi. (Note: as of writing this mail, armv7l is still building.) Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.16.1 [2] https://build.opensuse.org/project/show/home:laszlo_budai:syslog-ng regards, Laszlo Budai

1 0

java 10+ support for elasticsearch2
by Nik Ambrosch 15 Jun '18

15 Jun '18

Oracle is pushing to deprecate Java 8/9 - I was wondering if anyone has gotten syslog-ng 3.15 working with Oracle JDK 10.0.1? I'm seeing this on startup which seems to prevent the loading of the elasticsearch2 destination: Jun 15 01:51:39 centos3 syslog-ng[55366]: WARNING: An illegal reflective access operation has occurred Jun 15 01:51:39 centos3 syslog-ng[55366]: WARNING: Illegal reflective access by org.syslog_ng.SyslogNgClassLoader (file:/usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar) to method java.net.URLClassLoader.addURL(java.net.URL) Jun 15 01:51:39 centos3 syslog-ng[55366]: WARNING: Please consider reporting this to the maintainers of org.syslog_ng.SyslogNgClassLoader Jun 15 01:51:39 centos3 syslog-ng[55366]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations Jun 15 01:51:39 centos3 syslog-ng[55366]: WARNING: All illegal access operations will be denied in a future release Jun 15 01:51:39 centos3 syslog-ng[55366]: [2018-06-15T01:51:39.333858] Error while expanding path list: object is not an instance of declaring class; Jun 15 01:51:39 centos3 syslog-ng[55366]: java.lang.IllegalArgumentException: object is not an instance of declaring class Jun 15 01:51:39 centos3 syslog-ng[55366]: at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) Jun 15 01:51:39 centos3 syslog-ng[55366]: at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) Jun 15 01:51:39 centos3 syslog-ng[55366]: at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) Jun 15 01:51:39 centos3 syslog-ng[55366]: at java.base/java.lang.reflect.Method.invoke(Method.java:564) Jun 15 01:51:39 centos3 syslog-ng[55366]: at org.syslog_ng.SyslogNgClassLoader.expandClassPath(SyslogNgClassLoader.java:144) Jun 15 01:51:39 centos3 syslog-ng[55366]: at org.syslog_ng.SyslogNgClassLoader.loadClass(SyslogNgClassLoader.java:65) Jun 15 01:51:39 centos3 syslog-ng[55366]: [2018-06-15T01:51:39.335187] Can't find class; class_name='org.syslog_ng.elasticsearch_v2.ElasticSearchDestination' Jun 15 01:51:39 centos3 syslog-ng[55366]: [2018-06-15T01:51:39.337627] Error initializing message pipeline; plugin_name='java', location='#buffer:2:3'

1 0

Insider 2018-06: 3.15; ARM; Big Data; journald; Elasticsearch; conferences;
by Czanik, Péter 14 Jun '18

14 Jun '18

Dear syslog-ng users, This is the 67th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng 3.15 is released -------------------------- Version 3.15 of syslog-ng is now available. Support was added for if / elif / else blocks, which can simplify syslog-ng configuration considerably. Improved debug and error messages make finding configuration problems easier. Support for basic client-side failover was added. For a complete list of changes, check https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.15.1. For binary packages, check https://syslog-ng.com/3rd-party-binaries. Debian ARM builds ----------------- Starting with version 3.15, the unofficial syslog-ng Debian build is also available for ARM v7. It has all of the features of the x86 edition. The build is made for Debian 9.0 and tested on a Raspberry Pi, but it most likely works on recent editions of Raspbian as well. It is available at https://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/De…. Big Data: save all or save costs? --------------------------------- When starting a new project, Big Data vendors usually recommend a “save all” and “save raw” approach, as you never know what data might come in handy later on and in what format. Companies starting those projects also often have the same approach, as they still have their infrastructure under heavy development. Both go on the assumption that storage is practically free compared to the value of data. But is it really so? Learn more at https://syslog-ng.com/blog/big-data-save-save-costs/. syslog-ng vs. systemd’s journald -------------------------------- People often ask what to use: systemd’s journald or syslog-ng? The quick answer is that most likely both, but it depends on how you use your computer(s). If you have a single standalone machine, journald is probably enough. There is even a nice desktop application to view the logs in the journal. But once you have multiple machines to manage, using syslog-ng has many advantages. Read more at https://news.opensuse.org/2018/04/30/syslog-ng-vs-systemds-journald/. Storing logs in Elasticsearch using syslog-ng --------------------------------------------- Get started with sending logs directly to Elasticsearch using syslog-ng. Find out how to parse data with syslog-ng, store it in Elasticsearch, and analyze it with Kibana. Learn about the basics of using syslog-ng and Elasticsearch 6 on Red Hat Enterprise Linux / CentOS. Discover how to send netdata metrics through syslog-ng to Elasticsearch. Read our technical white paper at https://pages.balabit.com/storing-logs-in-elasticsearch-using-syslog-ng.html Containers and automation: five conferences in two words -------------------------------------------------------- In the past couple of weeks I visited five different conferences in four different countries either as a speaker or as booth staff. Altogether I had a chance to present syslog-ng to thousands of people and discuss syslog-ng and logging in general with hundreds. Except for a Big Data event in Budapest, I could summarize the conferences in two words: containers and automation. Of course, all events covered other diverse topics as well, but the main themes were these two. Let me give you a quick overview of the events: https://syslog-ng.com/blog/containers-automation-five-conferences-two-words/. Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://syslog-ng.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

Issue with Juniper SRX structured stream logs getting parsed by syslog-ng
by Brad Riemann 13 Jun '18

13 Jun '18

Hello, I've been banging my head against the wall for a while with this issue, I have a Juniper SRX that sends traffic logs to syslog-ng (centos 6, 6, version 3.13.2-2), the key being they are structured so the look like the following. [2018-06-12T19:03:54.384014] Incoming log entry; line='<14>1 2018-06-12T19:03:54.125Z corefw-site-0 RT_FLOW - RT_FLOW_SESSION_CLOSE [junos(a)2636.1.1.1.2.40 reason="idle Timeout" source-address="10.1.1.1" source-port="56998" destination-address="0.0.0.0" destination-port="443" service-name="junos-https" nat-source-address="10.100.1.1" nat-source-port="39155" nat-destination-address="0.0.0.0" nat-destination-port="443" src-nat-rule-type="source rule" src-nat-rule-name="test-servers" dst-nat-rule-type="N/A" dst-nat-rule-name="N/A" protocol-id="6" policy-name="allow-outbound" source-zone-name="test-zone" destination-zone-name="untrust" session-id-32="499236" packets-from-client="3" bytes-from-client="180" packets-from-server="0" bytes-from-server="0" elapsed-time="19" application="UNKNOWN" nested-application="UNKNOWN" username="N/A" roles="N/A" packet-incoming-interface="reth0.1" encrypted="UNKNOWN"]' I can confirm that syslog-ng is seeing the whole message as part of the incoming log entry, however it's just not showing up. I've dug through RFC 5424 and these logs DO conform but my confusion still remains. When I check the logs that were written the line looks like the following, which seems to just trunk everything after the RT_FLOW of the message header. 2018-06-12T19:03:54.125+00:00 corefw-site-0 RT_FLOW: I've run syslog-ng in debug mode, which was how I was able to confirm the outgoing message matches the prior line as well as the incoming message matching the expected data from my srx. [2018-06-12T19:03:54.385505] Outgoing message; message='2018-06-12T19:03:54.125+00:00 corefw-site-0 RT_FLOW: \x0a' Here is a snippet of my syslog config, it isn't that complex (at least I think so..) and i'm running out of hair to pull, any input would be GREATLY appreciated. https://gist.github.com/robotman321/217a69d456e543831936b1209b8bf501 -Brad

3 5

Syslog-ng OSE 3.x statistics
by Aguilu, Miguel 07 Jun '18

07 Jun '18

Need toms information how to interpret the syslog-ng-ctl stats result. In particular the period of time for which the output is. The global setting stats_freq is set for every 5 minutes. Any thoughts on what is the statistics period. Anyway to modify it? Thanks Miguel Miguel A Aguilu Netcool Suite SME Walt Disney Account Atos Consulting Mobile: (321) 261-1385 1120 Celebration Blvd Celebration, FL 34747 Miguel.Aguilu(a)Atos.net<mailto:Craig.Elliott@Atos.net> www.atos.net<http://www.atos.net/> [/Users/aguim114/Library/Containers/com.microsoft.Outlook/Data/Library/Caches/Signatures/signature_300359201] This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.

2 1

Re: [syslog-ng] (U) [Non-DoD Source] Re: Rotate syslog-ng log files
by Amin, Jitesh CTR DISA JSP (US) 07 Jun '18

07 Jun '18

CLASSIFICATION: UNCLASSIFIED Hello, So the file rotates now successfully – but what I have noticed is that after the file rotates it collects data for first few minute or so and then it stops collecting data (basically the file size never grows and timestamp never changes to the most latest when I check the file). I do see that syslog process/service is running. If I restart the service/process, it starts collecting data until the file rotation happens. Can you please let me know what would be causing this behavior? Thanks Jitesh Amin CLASSIFICATION: UNCLASSIFIED From: Amin, Jitesh CTR DISA JSP (US) Sent: Tuesday, June 5, 2018 9:59 AM To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: RE: (U) [syslog-ng] [Non-DoD Source] Re: Rotate syslog-ng log files CLASSIFICATION: UNCLASSIFIED OK skipping the {} made it work and I now see a syslog file with timestamp (year-month-day). Does this mean it should rotate to new log file name (tomorrows timestamp) at midnight tonight? OR I need to add syntax so it rolls everyday with new timestamp. Just wanted to confirm. Thanks Jitesh Amin CLASSIFICATION: UNCLASSIFIED From: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> On Behalf Of Scheidler, Balázs Sent: Thursday, May 31, 2018 5:48 AM To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] (U) [Non-DoD Source] Re: Rotate syslog-ng log files All active links contained in this email were disabled. Please verify the identity of the sender, and confirm the authenticity of all links contained within the message prior to copying and pasting the address to a Web browser. _____ I mean syslog-ng 1.6.8 On May 30, 2018 22:54, "Balazs Scheidler" <bazsi77(a)gmail.com < Caution-mailto:bazsi77@gmail.com > > wrote: syslog-ng does have template support, it just doesnt support braces, which came later. Just write $YEAR instead of ${YEAR} On May 30, 2018 09:41, "Gergely Nagy" <algernon(a)balabit.com < Caution-mailto:algernon@balabit.com > > wrote: >>>>> "Amin" == Amin, Jitesh CTR DISA JSP (US) <jitesh.amin.ctr(a)mail.mil < Caution-mailto:jitesh.amin.ctr@mail.mil > > writes: Amin> Let me ask this, with the following config = destination Amin> syslog { file("/var/log/syslog-${YEAR}-${MONTH}-${DAY}.log"); Amin> }; Amin> It created new file and started writing to it (versus creating Amin> new syslog.log). Question, if we plan to accept this for now, Amin> with above config, would it create a new file ever day with Amin> following file names or no it would not work with v1.6.8 With syslog-ng 1.6.8, it would not create a new file every day, and would continue writing to syslog-{YEAR}-{MONTH}-{DAY}.log. With newer versions, it would create files like `syslog-2018-05-30.log`. No `.0`, `.1` or the like would be appended. That's a convention of logrotate. With syslog-ng, you get filenames that match the template, they will have nothing appended or prepended that is not in the filename template. -- |8] ______________________________________________________________________________ Member info: Caution-https://lists.balabit.hu/mailman/listinfo/syslog-ng < Caution-https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: Caution-http://www.balabit.com/support/documentation/?product=syslog-ng < Caution-http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: Caution-http://www.balabit.com/wiki/syslog-ng-faq < Caution-http://www.balabit.com/wiki/syslog-ng-faq > ______________________________________________________________________________ Member info: Caution-https://lists.balabit.hu/mailman/listinfo/syslog-ng < Caution-https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: Caution-http://www.balabit.com/support/documentation/?product=syslog-ng < Caution-http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: Caution-http://www.balabit.com/wiki/syslog-ng-faq < Caution-http://www.balabit.com/wiki/syslog-ng-faq >

4 3

netcat tests
by Eric Benoit 06 Jun '18

06 Jun '18

Hi List, I wanted to send a test message to my syslog-ng server. I tried this: > echo -n "test message" | nc -4u -w1 192.168.100.10 514 But I saw no logs created on the syslog-ng server. I did see the udp package on wireshark. Will syslog-ng except my test message the way I have it formatted and log it, does my message need to be formatted in a specific way for syslog-ng to except and create a log? Also, should I change syslog-ng to TCP instead of UDP to better trouble shoot any network problems I might have pending my netcat test message looks ok for syslog-ng server to except and log? I'm not very good at articulating questions, so I hope what I wrote makes sense if it does not please let me know and I will spend some more time rephrasing my questions. Thank you for any suggestions or comments

2 2

Re: [syslog-ng] (U) [Non-DoD Source] Re: Rotate syslog-ng log files
by Amin, Jitesh CTR DISA JSP (US) 05 Jun '18

05 Jun '18

CLASSIFICATION: UNCLASSIFIED OK skipping the {} made it work and I now see a syslog file with timestamp (year-month-day). Does this mean it should rotate to new log file name (tomorrows timestamp) at midnight tonight? OR I need to add syntax so it rolls everyday with new timestamp. Just wanted to confirm. Thanks Jitesh Amin CLASSIFICATION: UNCLASSIFIED From: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> On Behalf Of Scheidler, Balázs Sent: Thursday, May 31, 2018 5:48 AM To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] (U) [Non-DoD Source] Re: Rotate syslog-ng log files All active links contained in this email were disabled. Please verify the identity of the sender, and confirm the authenticity of all links contained within the message prior to copying and pasting the address to a Web browser. _____ I mean syslog-ng 1.6.8 On May 30, 2018 22:54, "Balazs Scheidler" <bazsi77(a)gmail.com < Caution-mailto:bazsi77@gmail.com > > wrote: syslog-ng does have template support, it just doesnt support braces, which came later. Just write $YEAR instead of ${YEAR} On May 30, 2018 09:41, "Gergely Nagy" <algernon(a)balabit.com < Caution-mailto:algernon@balabit.com > > wrote: >>>>> "Amin" == Amin, Jitesh CTR DISA JSP (US) <jitesh.amin.ctr(a)mail.mil < Caution-mailto:jitesh.amin.ctr@mail.mil > > writes: Amin> Let me ask this, with the following config = destination Amin> syslog { file("/var/log/syslog-${YEAR}-${MONTH}-${DAY}.log"); Amin> }; Amin> It created new file and started writing to it (versus creating Amin> new syslog.log). Question, if we plan to accept this for now, Amin> with above config, would it create a new file ever day with Amin> following file names or no it would not work with v1.6.8 With syslog-ng 1.6.8, it would not create a new file every day, and would continue writing to syslog-{YEAR}-{MONTH}-{DAY}.log. With newer versions, it would create files like `syslog-2018-05-30.log`. No `.0`, `.1` or the like would be appended. That's a convention of logrotate. With syslog-ng, you get filenames that match the template, they will have nothing appended or prepended that is not in the filename template. -- |8] ______________________________________________________________________________ Member info: Caution-https://lists.balabit.hu/mailman/listinfo/syslog-ng < Caution-https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: Caution-http://www.balabit.com/support/documentation/?product=syslog-ng < Caution-http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: Caution-http://www.balabit.com/wiki/syslog-ng-faq < Caution-http://www.balabit.com/wiki/syslog-ng-faq > ______________________________________________________________________________ Member info: Caution-https://lists.balabit.hu/mailman/listinfo/syslog-ng < Caution-https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: Caution-http://www.balabit.com/support/documentation/?product=syslog-ng < Caution-http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: Caution-http://www.balabit.com/wiki/syslog-ng-faq < Caution-http://www.balabit.com/wiki/syslog-ng-faq >

2 1