- syslog-ng - lists.balabit.hu

Issue with Juniper SRX structured stream logs getting parsed by syslog-ng
by Brad Riemann 13 Jun '18

13 Jun '18

Hello, I've been banging my head against the wall for a while with this issue, I have a Juniper SRX that sends traffic logs to syslog-ng (centos 6, 6, version 3.13.2-2), the key being they are structured so the look like the following. [2018-06-12T19:03:54.384014] Incoming log entry; line='<14>1 2018-06-12T19:03:54.125Z corefw-site-0 RT_FLOW - RT_FLOW_SESSION_CLOSE [junos(a)2636.1.1.1.2.40 reason="idle Timeout" source-address="10.1.1.1" source-port="56998" destination-address="0.0.0.0" destination-port="443" service-name="junos-https" nat-source-address="10.100.1.1" nat-source-port="39155" nat-destination-address="0.0.0.0" nat-destination-port="443" src-nat-rule-type="source rule" src-nat-rule-name="test-servers" dst-nat-rule-type="N/A" dst-nat-rule-name="N/A" protocol-id="6" policy-name="allow-outbound" source-zone-name="test-zone" destination-zone-name="untrust" session-id-32="499236" packets-from-client="3" bytes-from-client="180" packets-from-server="0" bytes-from-server="0" elapsed-time="19" application="UNKNOWN" nested-application="UNKNOWN" username="N/A" roles="N/A" packet-incoming-interface="reth0.1" encrypted="UNKNOWN"]' I can confirm that syslog-ng is seeing the whole message as part of the incoming log entry, however it's just not showing up. I've dug through RFC 5424 and these logs DO conform but my confusion still remains. When I check the logs that were written the line looks like the following, which seems to just trunk everything after the RT_FLOW of the message header. 2018-06-12T19:03:54.125+00:00 corefw-site-0 RT_FLOW: I've run syslog-ng in debug mode, which was how I was able to confirm the outgoing message matches the prior line as well as the incoming message matching the expected data from my srx. [2018-06-12T19:03:54.385505] Outgoing message; message='2018-06-12T19:03:54.125+00:00 corefw-site-0 RT_FLOW: \x0a' Here is a snippet of my syslog config, it isn't that complex (at least I think so..) and i'm running out of hair to pull, any input would be GREATLY appreciated. https://gist.github.com/robotman321/217a69d456e543831936b1209b8bf501 -Brad

3 5

Syslog-ng OSE 3.x statistics
by Aguilu, Miguel 07 Jun '18

07 Jun '18

Need toms information how to interpret the syslog-ng-ctl stats result. In particular the period of time for which the output is. The global setting stats_freq is set for every 5 minutes. Any thoughts on what is the statistics period. Anyway to modify it? Thanks Miguel Miguel A Aguilu Netcool Suite SME Walt Disney Account Atos Consulting Mobile: (321) 261-1385 1120 Celebration Blvd Celebration, FL 34747 Miguel.Aguilu(a)Atos.net<mailto:Craig.Elliott@Atos.net> www.atos.net<http://www.atos.net/> [/Users/aguim114/Library/Containers/com.microsoft.Outlook/Data/Library/Caches/Signatures/signature_300359201] This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.

2 1

Re: [syslog-ng] (U) [Non-DoD Source] Re: Rotate syslog-ng log files
by Amin, Jitesh CTR DISA JSP (US) 07 Jun '18

07 Jun '18

CLASSIFICATION: UNCLASSIFIED Hello, So the file rotates now successfully – but what I have noticed is that after the file rotates it collects data for first few minute or so and then it stops collecting data (basically the file size never grows and timestamp never changes to the most latest when I check the file). I do see that syslog process/service is running. If I restart the service/process, it starts collecting data until the file rotation happens. Can you please let me know what would be causing this behavior? Thanks Jitesh Amin CLASSIFICATION: UNCLASSIFIED From: Amin, Jitesh CTR DISA JSP (US) Sent: Tuesday, June 5, 2018 9:59 AM To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: RE: (U) [syslog-ng] [Non-DoD Source] Re: Rotate syslog-ng log files CLASSIFICATION: UNCLASSIFIED OK skipping the {} made it work and I now see a syslog file with timestamp (year-month-day). Does this mean it should rotate to new log file name (tomorrows timestamp) at midnight tonight? OR I need to add syntax so it rolls everyday with new timestamp. Just wanted to confirm. Thanks Jitesh Amin CLASSIFICATION: UNCLASSIFIED From: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> On Behalf Of Scheidler, Balázs Sent: Thursday, May 31, 2018 5:48 AM To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] (U) [Non-DoD Source] Re: Rotate syslog-ng log files All active links contained in this email were disabled. Please verify the identity of the sender, and confirm the authenticity of all links contained within the message prior to copying and pasting the address to a Web browser. _____ I mean syslog-ng 1.6.8 On May 30, 2018 22:54, "Balazs Scheidler" <bazsi77(a)gmail.com < Caution-mailto:bazsi77@gmail.com > > wrote: syslog-ng does have template support, it just doesnt support braces, which came later. Just write $YEAR instead of ${YEAR} On May 30, 2018 09:41, "Gergely Nagy" <algernon(a)balabit.com < Caution-mailto:algernon@balabit.com > > wrote: >>>>> "Amin" == Amin, Jitesh CTR DISA JSP (US) <jitesh.amin.ctr(a)mail.mil < Caution-mailto:jitesh.amin.ctr@mail.mil > > writes: Amin> Let me ask this, with the following config = destination Amin> syslog { file("/var/log/syslog-${YEAR}-${MONTH}-${DAY}.log"); Amin> }; Amin> It created new file and started writing to it (versus creating Amin> new syslog.log). Question, if we plan to accept this for now, Amin> with above config, would it create a new file ever day with Amin> following file names or no it would not work with v1.6.8 With syslog-ng 1.6.8, it would not create a new file every day, and would continue writing to syslog-{YEAR}-{MONTH}-{DAY}.log. With newer versions, it would create files like `syslog-2018-05-30.log`. No `.0`, `.1` or the like would be appended. That's a convention of logrotate. With syslog-ng, you get filenames that match the template, they will have nothing appended or prepended that is not in the filename template. -- |8] ______________________________________________________________________________ Member info: Caution-https://lists.balabit.hu/mailman/listinfo/syslog-ng < Caution-https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: Caution-http://www.balabit.com/support/documentation/?product=syslog-ng < Caution-http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: Caution-http://www.balabit.com/wiki/syslog-ng-faq < Caution-http://www.balabit.com/wiki/syslog-ng-faq > ______________________________________________________________________________ Member info: Caution-https://lists.balabit.hu/mailman/listinfo/syslog-ng < Caution-https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: Caution-http://www.balabit.com/support/documentation/?product=syslog-ng < Caution-http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: Caution-http://www.balabit.com/wiki/syslog-ng-faq < Caution-http://www.balabit.com/wiki/syslog-ng-faq >

4 3

netcat tests
by Eric Benoit 06 Jun '18

06 Jun '18

Hi List, I wanted to send a test message to my syslog-ng server. I tried this: > echo -n "test message" | nc -4u -w1 192.168.100.10 514 But I saw no logs created on the syslog-ng server. I did see the udp package on wireshark. Will syslog-ng except my test message the way I have it formatted and log it, does my message need to be formatted in a specific way for syslog-ng to except and create a log? Also, should I change syslog-ng to TCP instead of UDP to better trouble shoot any network problems I might have pending my netcat test message looks ok for syslog-ng server to except and log? I'm not very good at articulating questions, so I hope what I wrote makes sense if it does not please let me know and I will spend some more time rephrasing my questions. Thank you for any suggestions or comments

2 2

Re: [syslog-ng] (U) [Non-DoD Source] Re: Rotate syslog-ng log files
by Amin, Jitesh CTR DISA JSP (US) 05 Jun '18

05 Jun '18

CLASSIFICATION: UNCLASSIFIED OK skipping the {} made it work and I now see a syslog file with timestamp (year-month-day). Does this mean it should rotate to new log file name (tomorrows timestamp) at midnight tonight? OR I need to add syntax so it rolls everyday with new timestamp. Just wanted to confirm. Thanks Jitesh Amin CLASSIFICATION: UNCLASSIFIED From: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> On Behalf Of Scheidler, Balázs Sent: Thursday, May 31, 2018 5:48 AM To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] (U) [Non-DoD Source] Re: Rotate syslog-ng log files All active links contained in this email were disabled. Please verify the identity of the sender, and confirm the authenticity of all links contained within the message prior to copying and pasting the address to a Web browser. _____ I mean syslog-ng 1.6.8 On May 30, 2018 22:54, "Balazs Scheidler" <bazsi77(a)gmail.com < Caution-mailto:bazsi77@gmail.com > > wrote: syslog-ng does have template support, it just doesnt support braces, which came later. Just write $YEAR instead of ${YEAR} On May 30, 2018 09:41, "Gergely Nagy" <algernon(a)balabit.com < Caution-mailto:algernon@balabit.com > > wrote: >>>>> "Amin" == Amin, Jitesh CTR DISA JSP (US) <jitesh.amin.ctr(a)mail.mil < Caution-mailto:jitesh.amin.ctr@mail.mil > > writes: Amin> Let me ask this, with the following config = destination Amin> syslog { file("/var/log/syslog-${YEAR}-${MONTH}-${DAY}.log"); Amin> }; Amin> It created new file and started writing to it (versus creating Amin> new syslog.log). Question, if we plan to accept this for now, Amin> with above config, would it create a new file ever day with Amin> following file names or no it would not work with v1.6.8 With syslog-ng 1.6.8, it would not create a new file every day, and would continue writing to syslog-{YEAR}-{MONTH}-{DAY}.log. With newer versions, it would create files like `syslog-2018-05-30.log`. No `.0`, `.1` or the like would be appended. That's a convention of logrotate. With syslog-ng, you get filenames that match the template, they will have nothing appended or prepended that is not in the filename template. -- |8] ______________________________________________________________________________ Member info: Caution-https://lists.balabit.hu/mailman/listinfo/syslog-ng < Caution-https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: Caution-http://www.balabit.com/support/documentation/?product=syslog-ng < Caution-http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: Caution-http://www.balabit.com/wiki/syslog-ng-faq < Caution-http://www.balabit.com/wiki/syslog-ng-faq > ______________________________________________________________________________ Member info: Caution-https://lists.balabit.hu/mailman/listinfo/syslog-ng < Caution-https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: Caution-http://www.balabit.com/support/documentation/?product=syslog-ng < Caution-http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: Caution-http://www.balabit.com/wiki/syslog-ng-faq < Caution-http://www.balabit.com/wiki/syslog-ng-faq >

2 1

syslog-ng 3.15.1 on solaris
by Szalai, Attila 31 May '18

31 May '18

Hi, We have a fairly complicated compiling system, so before I start to investigate my compiling issue I would like to ask here. Is there anybody who was able to compile 3.15.1 on solaris? (either x86 or sparc.) For me, it ended to linking process with this: ld: fatal: library -lsyslog-ng: not found collect2: error: ld returned 1 exit status libtool: install: error: relink `modules/diskq/libdisk-buffer.la' with the above command before installing it In linux, the same code compiled/linked fine. Attila Szalai Morgan Stanley | Enterprise Technology & Risk Lechner Odon fasor 8 | Floor 07 Budapest, 1095 Phone: +36 1 881-4366 Attila.Szalai(a)morganstanley.com<mailto:Attila.Szalai@morganstanley.com> Be carbon conscious. Please consider our environment before printing this email. ________________________________ NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent required and/or permitted under applicable law, to monitor electronic communications, including telephone calls with Morgan Stanley personnel. This message is subject to the Morgan Stanley General Disclaimers available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access the links, please notify us by reply message and we will send the contents to you. By communicating with Morgan Stanley you acknowledge that you have read, understand and consent, (where applicable), to the foregoing and the Morgan Stanley General Disclaimers.

4 4

Re: [syslog-ng] (U) [Non-DoD Source] Re: Rotate syslog-ng log files
by Amin, Jitesh CTR DISA JSP (US) 31 May '18

31 May '18

CLASSIFICATION: UNCLASSIFIED Verion 1.6.8 Thanks Jitesh Amin CLASSIFICATION: UNCLASSIFIED -----Original Message----- From: Gergely Nagy <algernon(a)balabit.com> Sent: Wednesday, May 30, 2018 8:47 AM To: Amin, Jitesh CTR DISA JSP (US) <jitesh.amin.ctr(a)mail.mil>; Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] (U) [Non-DoD Source] Re: Rotate syslog-ng log files >>>>> "Amin" == Amin, Jitesh CTR DISA JSP (US) <jitesh.amin.ctr(a)mail.mil> writes: Amin> But now I see 2 log files in /var/log Amin> Syslog.log Amin> syslog-{YEAR}-{MONTH}-{DAY}.log Amin> Not sure why it’s creating syslog-{YEAR}-{MONTH}-{DAY}.log Amin> named file versus actually putting day/month/year in file Amin> name. Anything I missed?? What version of syslog-ng are you running? -- |8]

4 5

Re: [syslog-ng] (U) [Non-DoD Source] Re: Rotate syslog-ng log files
by Amin, Jitesh CTR DISA JSP (US) 30 May '18

30 May '18

CLASSIFICATION: UNCLASSIFIED Hello, So I updated configuration to have following in my conf file: destination syslog { file("/var/log/syslog-${YEAR}-${MONTH}-${DAY}.log"); }; and syslog-ng restarted file, But now I see 2 log files in /var/log Syslog.log syslog-{YEAR}-{MONTH}-{DAY}.log Not sure why it’s creating syslog-{YEAR}-{MONTH}-{DAY}.log named file versus actually putting day/month/year in file name. Anything I missed?? Thanks Jitesh Amin CLASSIFICATION: UNCLASSIFIED From: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> On Behalf Of Scheidler, Balázs Sent: Wednesday, May 30, 2018 8:03 AM To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] (U) [Non-DoD Source] Re: Rotate syslog-ng log files All active links contained in this email were disabled. Please verify the identity of the sender, and confirm the authenticity of all links contained within the message prior to copying and pasting the address to a Web browser. _____ Yep, that would give you one logfile every day. It will not care about whether it reached 500mb though. But on the plus side that gives you a trivial way to identify which file holds a specific date, which is pretty cumbersome otherwise. On May 30, 2018 07:52, "Amin, Jitesh CTR DISA JSP (US)" <jitesh.amin.ctr(a)mail.mil < Caution-mailto:jitesh.amin.ctr@mail.mil > > wrote: CLASSIFICATION: UNCLASSIFIED Hello, So here's what currently is configured in our .conf file: destination syslog { file("/var/log/syslog.log" perm(0644)); }; what you are saying, is if I want logs to rotate on daily basis, I can just make the above line to read as following: destination syslog { file("/var/log/syslog-${YEAR}-${MONTH}-${DAY}.log"); }; Thanks Jitesh Amin CLASSIFICATION: UNCLASSIFIED -----Original Message----- From: Gergely Nagy <algernon(a)balabit.com < Caution-mailto:algernon@balabit.com > > Sent: Wednesday, May 30, 2018 7:25 AM To: Amin, Jitesh CTR DISA JSP (US) <jitesh.amin.ctr(a)mail.mil < Caution-mailto:jitesh.amin.ctr@mail.mil > >; syslog-ng(a)lists.balabit.hu < Caution-mailto:syslog-ng@lists.balabit.hu > Subject: [Non-DoD Source] Re: [syslog-ng] (U) Rotate syslog-ng log files All active links contained in this email were disabled. Please verify the identity of the sender, and confirm the authenticity of all links contained within the message prior to copying and pasting the address to a Web browser. ---- Hi! >>>>> "Amin" == Amin, Jitesh CTR DISA JSP (US) <jitesh.amin.ctr(a)mail.mil < Caution-mailto:jitesh.amin.ctr@mail.mil > > writes: Amin> What we are trying to do is rotate the syslog.log file once it Amin> reaches 500 MB (as well as looking into rotating file every 24 Amin> hours)? Once the log file rotates, it creates a new file named Amin> syslog.log.0 and so on.. syslog-ng does not support size-based rotation, you'll have to configure logrotate for that. See the packaging/debian/syslog-ng-core.syslog-ng.logrotate[1] file for an example. [1]: Caution-Caution-https://github.com/balabit/syslog-ng/blob/master/packaging/… < Caution-https://github.com/balabit/syslog-ng/blob/master/packaging/debian/s… > With logrotate, the most recent logfile syslog-ng writes to will always be syslog.log, and the rotated files will have different names - depending on the logrotate configuration you use. For time-based, daily rotation, you can use templates in the destination filename, such as: destination d_syslog { file("/var/log/syslog-${YEAR}-${MONTH}-${DAY}.log"); }; Hope this helps! -- |8] ______________________________________________________________________________ Member info: Caution-https://lists.balabit.hu/mailman/listinfo/syslog-ng < Caution-https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: Caution-http://www.balabit.com/support/documentation/?product=syslog-ng < Caution-http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: Caution-http://www.balabit.com/wiki/syslog-ng-faq < Caution-http://www.balabit.com/wiki/syslog-ng-faq >

2 1

Re: [syslog-ng] (U) [Non-DoD Source] Re: Rotate syslog-ng log files
by Amin, Jitesh CTR DISA JSP (US) 30 May '18

30 May '18

CLASSIFICATION: UNCLASSIFIED Hello, So here's what currently is configured in our .conf file: destination syslog { file("/var/log/syslog.log" perm(0644)); }; what you are saying, is if I want logs to rotate on daily basis, I can just make the above line to read as following: destination syslog { file("/var/log/syslog-${YEAR}-${MONTH}-${DAY}.log"); }; Thanks Jitesh Amin CLASSIFICATION: UNCLASSIFIED -----Original Message----- From: Gergely Nagy <algernon(a)balabit.com> Sent: Wednesday, May 30, 2018 7:25 AM To: Amin, Jitesh CTR DISA JSP (US) <jitesh.amin.ctr(a)mail.mil>; syslog-ng(a)lists.balabit.hu Subject: [Non-DoD Source] Re: [syslog-ng] (U) Rotate syslog-ng log files All active links contained in this email were disabled. Please verify the identity of the sender, and confirm the authenticity of all links contained within the message prior to copying and pasting the address to a Web browser. ---- Hi! >>>>> "Amin" == Amin, Jitesh CTR DISA JSP (US) <jitesh.amin.ctr(a)mail.mil> writes: Amin> What we are trying to do is rotate the syslog.log file once it Amin> reaches 500 MB (as well as looking into rotating file every 24 Amin> hours)? Once the log file rotates, it creates a new file named Amin> syslog.log.0 and so on.. syslog-ng does not support size-based rotation, you'll have to configure logrotate for that. See the packaging/debian/syslog-ng-core.syslog-ng.logrotate[1] file for an example. [1]: Caution-https://github.com/balabit/syslog-ng/blob/master/packaging/debian/s… With logrotate, the most recent logfile syslog-ng writes to will always be syslog.log, and the rotated files will have different names - depending on the logrotate configuration you use. For time-based, daily rotation, you can use templates in the destination filename, such as: destination d_syslog { file("/var/log/syslog-${YEAR}-${MONTH}-${DAY}.log"); }; Hope this helps! -- |8]

2 1

(U) Rotate syslog-ng log files
by Amin, Jitesh CTR DISA JSP (US) 30 May '18

30 May '18

CLASSIFICATION: UNCLASSIFIED Hello, Currently we have syslog writing a single file (syslog.log) that just grows exponentially? What we are trying to do is rotate the syslog.log file once it reaches 500 MB (as well as looking into rotating file every 24 hours)? Once the log file rotates, it creates a new file named syslog.log.0 and so on.. Can someone please tell me how do I go about doing so and what specific configuration I need to have it in place in syslog-ng.conf file to achieve this? Thanks Jitesh Amin CLASSIFICATION: UNCLASSIFIED

2 1