From sa4.sp-dl at nkn.in Thu Aug 4 13:15:06 2022 From: sa4.sp-dl at nkn.in (Sachchidanand Upadhyay) Date: Thu, 4 Aug 2022 18:45:06 +0530 (IST) Subject: [syslog-ng] drop log lines having keyword "INTERNET" Message-ID: <828065580.773297.1659618906802@nkn.in> Hi, I want to drop all log lines having keyword "INTERNET", I tried following scenario: scenario-1 filter f_log { facility(local3); }; filter f_nointernet { not message("INTERNET"); }; log { source(s_sys); filter(f_nointernet); filter(f_log); destination(d_log); }; scenario-2 filter f_internet { message("INTERNET"); }; filter f_log { facility(local3); and not filter(f_internet); }; log { source(s_sys); filter(f_log); destination(d_log); }; scenario-3 filter f_log { facility(local3); }; filter f_internet { message("INTERNET"); }; filter f_nointernet { not filter(f_internet); }; log { source(s_sys); filter(f_nointernet); filter(f_log); destination(d_log); }; But none of them is working. Please help. Regards, Sachchidanand -------------- next part -------------- An HTML attachment was scrubbed... URL: From francois at fmarier.org Mon Aug 15 03:56:42 2022 From: francois at fmarier.org (Francois Marier) Date: Sun, 14 Aug 2022 20:56:42 -0700 Subject: [syslog-ng] Multiple timezone seen on messages originating from local machine Message-ID: I'm trying to understand timezones in syslog-ng, but I think there's something I'm missing. If I do the following on my OpenWRT machine: /etc/init.d/syslog-ng restart logger TestA I see the following in /var/log/messages: Aug 14 20:39:35 hostname syslog-ng[9860]: syslog-ng shutting down; version='3.37.1' Aug 14 20:39:36 hostname syslog-ng[10024]: syslog-ng starting up; version='3.37.1' Aug 14 20:39:36 hostname syslog-ng[10024]: Syslog connection established; fd='15', server='AF_INET(192.168.1.10:514)', local='AF_INET(0.0.0.0:0)' Aug 15 03:39:49 hostname root: TestA The correct timezone is the one in the first three lines. Other daemon messages are displayed using the incorrect timezone like `logger`. I've attached my syslog-ng.conf file. So my question is why is the time information correct in the first three messages and then incorrect from there on? Francois -- https://fmarier.org/ -------------- next part -------------- ############################################################################# # OpenWrt syslog-ng.conf specific file # which collects all local logs into a single file called /var/log/messages. # More details about these settings can be found here: # https://www.syslog-ng.com/technical-documents/list/syslog-ng-open-source-edition @version: 3.37 @include "scl.conf" options { chain_hostnames(no); # Enable or disable the chained hostname format. create_dirs(yes); keep_hostname(yes); # Enable or disable hostname rewriting. log_fifo_size(256); # The number of messages that the output queue can store. log_msg_size(1024); # Maximum length of a message in bytes. stats_freq(0); # The period between two STATS messages (sent by syslog-ng, containing statistics about dropped logs) in seconds. flush_lines(0); # How many lines are flushed to a destination at a time. use_fqdn(no); # Add Fully Qualified Domain Name instead of short hostname. }; # syslog-ng gets messages from syslog-ng (internal) and from /dev/log source src { internal(); unix-dgram("/dev/log"); }; source net { network_localhost(); }; source s_network { default-network-drivers( # NOTE: TLS support # # the default-network-drivers() source driver opens the TLS # enabled ports as well, however without an actual key/cert # pair they will not operate and syslog-ng would display a # warning at startup. # #tls(key-file("/path/to/ssl-private-key") cert-file("/path/to/ssl-cert")) ); }; source kernel { file("/proc/kmsg" program_override("kernel")); }; destination messages { file("/var/log/messages"); }; log { source(src); source(net); source(kernel); destination(messages); # uncomment this line to open port 514 to receive messages #source(s_network); }; # # Finally, include any user settings last so that s/he can override or # supplement all "canned" settings inherited from the distribution. # @include "/etc/syslog-ng.d/" # Put any customization files in this directory From bazsi77 at gmail.com Mon Aug 15 08:14:53 2022 From: bazsi77 at gmail.com (Balazs Scheidler) Date: Mon, 15 Aug 2022 10:14:53 +0200 Subject: [syslog-ng] Multiple timezone seen on messages originating from local machine In-Reply-To: References: Message-ID: On /dev/log (the usual means of submitting messages) the timestamp is formatted by the syslog() function call, part of libc (probably musl in the case of openwrt). Libc uses /etc/localtime to query the local time zone. On Mon, Aug 15, 2022, 05:56 Francois Marier wrote: > I'm trying to understand timezones in syslog-ng, but I think there's > something I'm missing. > > If I do the following on my OpenWRT machine: > > /etc/init.d/syslog-ng restart > logger TestA > > I see the following in /var/log/messages: > > Aug 14 20:39:35 hostname syslog-ng[9860]: syslog-ng shutting down; > version='3.37.1' > Aug 14 20:39:36 hostname syslog-ng[10024]: syslog-ng starting up; > version='3.37.1' > Aug 14 20:39:36 hostname syslog-ng[10024]: Syslog connection > established; fd='15', server='AF_INET(192.168.1.10:514)', local='AF_INET( > 0.0.0.0:0)' > Aug 15 03:39:49 hostname root: TestA > > The correct timezone is the one in the first three lines. Other daemon > messages are displayed using the incorrect timezone like `logger`. > > I've attached my syslog-ng.conf file. > > So my question is why is the time information correct in the first three > messages and then incorrect from there on? > > Francois > > -- > https://fmarier.org/ > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From syslogng at allycomm.com Mon Aug 15 16:31:26 2022 From: syslogng at allycomm.com (Jeff Kletsky) Date: Mon, 15 Aug 2022 09:31:26 -0700 Subject: [syslog-ng] Multiple timezone seen on messages originating from local machine In-Reply-To: References: Message-ID: As best as I can determine, there is a time zone problem associated with how the messages are generated through some of the channels. https://forum.openwrt.org/t/syslog-ng-timezone-issues/48130 https://forum.openwrt.org/t/utc-timezones-used-for-apps-in-syslog-ng-instead-of-configured-one/45095 https://github.com/openwrt/packages/issues/18633 https://gitlab.nic.cz/turris/os/packages/-/issues/471 Changing how /dev/log is interpreted may either hide or resolve the issue, depending on your perspective: source s_devlog { ??????? unix-dgram("/dev/log", keep-timestamp(no)); }; Jeff On 8/14/22 8:56 PM, Francois Marier wrote: > I'm trying to understand timezones in syslog-ng, but I think there's > something I'm missing. > > If I do the following on my OpenWRT machine: > > /etc/init.d/syslog-ng restart > logger TestA > > I see the following in /var/log/messages: > > Aug 14 20:39:35 hostname syslog-ng[9860]: syslog-ng shutting down; version='3.37.1' > Aug 14 20:39:36 hostname syslog-ng[10024]: syslog-ng starting up; version='3.37.1' > Aug 14 20:39:36 hostname syslog-ng[10024]: Syslog connection established; fd='15', server='AF_INET(192.168.1.10:514)', local='AF_INET(0.0.0.0:0)' > Aug 15 03:39:49 hostname root: TestA > > The correct timezone is the one in the first three lines. Other daemon > messages are displayed using the incorrect timezone like `logger`. > > I've attached my syslog-ng.conf file. > > So my question is why is the time information correct in the first three > messages and then incorrect from there on? > > Francois > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > From bazsi77 at gmail.com Mon Aug 15 16:55:14 2022 From: bazsi77 at gmail.com (Balazs Scheidler) Date: Mon, 15 Aug 2022 18:55:14 +0200 Subject: [syslog-ng] Multiple timezone seen on messages originating from local machine In-Reply-To: References: Message-ID: Hi, Strange indeed. To solve this the logging application and syslog-ng needs to agree on the time-zone being used. syslog-ng uses the system time-zone by default but it can be overridden using the time-zone() option of the unix-dgram() source driver. The applications should also use the system time-zone. Here it's usually the libc (glibc or musl) that generates the timestamp. More specifically the syslog() function. It might happen that: * Some applications bypass the libc api and send its messages directly (util-linux logger does this for instance) * If an application has its own implementation, it may or may not use the system time-zone properly. To debug the problem, try to get access to the raw log data as received by syslog-ng. * It displays the incoming message on the debug level * You could strace the sending app * You could tell syslog-ng to store $RAWMSG using flags(store-raw-message) and then look at its value by using a destination that you use for debugging. In any case, once you recognize what sends the incorrect time-zone, you can probably find a reason why it sends it incorrectly. If you just want to solve this quickly, use keep-timestamp(no), that way syslog-ng will overwrite the message timestamp with the reception time, and that with correct time-zone. As a second alternative, you could use the pretty recent flags(guess-timezone) flag which uses a heuristic to fix up the time-zone, assuming the time between sending and receiving the message is less than 30 seconds. Bazsi On Mon, Aug 15, 2022, 18:31 Jeff Kletsky wrote: > As best as I can determine, there is a time zone problem associated with > how the messages are generated through some of the channels. > > > https://forum.openwrt.org/t/syslog-ng-timezone-issues/48130 > > > https://forum.openwrt.org/t/utc-timezones-used-for-apps-in-syslog-ng-instead-of-configured-one/45095 > > > https://github.com/openwrt/packages/issues/18633 > > https://gitlab.nic.cz/turris/os/packages/-/issues/471 > > > Changing how /dev/log is interpreted may either hide or resolve the > issue, depending on your perspective: > > source s_devlog { > unix-dgram("/dev/log", keep-timestamp(no)); > }; > > > Jeff > > On 8/14/22 8:56 PM, Francois Marier wrote: > > I'm trying to understand timezones in syslog-ng, but I think there's > > something I'm missing. > > > > If I do the following on my OpenWRT machine: > > > > /etc/init.d/syslog-ng restart > > logger TestA > > > > I see the following in /var/log/messages: > > > > Aug 14 20:39:35 hostname syslog-ng[9860]: syslog-ng shutting down; > version='3.37.1' > > Aug 14 20:39:36 hostname syslog-ng[10024]: syslog-ng starting up; > version='3.37.1' > > Aug 14 20:39:36 hostname syslog-ng[10024]: Syslog connection > established; fd='15', server='AF_INET(192.168.1.10:514)', local='AF_INET( > 0.0.0.0:0)' > > Aug 15 03:39:49 hostname root: TestA > > > > The correct timezone is the one in the first three lines. Other daemon > > messages are displayed using the incorrect timezone like `logger`. > > > > I've attached my syslog-ng.conf file. > > > > So my question is why is the time information correct in the first three > > messages and then incorrect from there on? > > > > Francois > > > > > > > ______________________________________________________________________________ > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > -------------- next part -------------- An HTML attachment was scrubbed... URL: From francois at fmarier.org Tue Aug 16 05:38:41 2022 From: francois at fmarier.org (Francois Marier) Date: Mon, 15 Aug 2022 22:38:41 -0700 Subject: [syslog-ng] Multiple timezone seen on messages originating from local machine In-Reply-To: References: Message-ID: Thank you so much Jeff and Balazs! I had not realized that this could simply be an upstream OpenWRT bug (that Turris bug tracker link was very helpful). The `keep-timestamp(no)` work-around sounds perfect since (1) these are all local messages and (2) Turris claims that it has been fixed upstream and so it should eventually arrive on my router, which is in fact a Turris Omnia. I've updated my remote logging blog post [1] to include this work-around. Francois [1] https://feeding.cloud.geek.nz/posts/remote-logging-turris-omnia-router/ -- https://fmarier.org/ From JohnLaw at tfl.gov.uk Fri Aug 19 14:47:42 2022 From: JohnLaw at tfl.gov.uk (John Law) Date: Fri, 19 Aug 2022 14:47:42 +0000 Subject: [syslog-ng] Potential issue with line breaking not working correctly when writing logs to disk Message-ID: Hi, We have a number of network devices sending syslog traffic to syslog-ng OSE 3.25 installed on RHEL 7.9. We are having an issue where multiple events are being written as the same line. The config file is template-function adm1 "${HOST} message-length=$(length \"${MSG}\") ${MESSAGE}\n"; source s_adm1 { udp(ip(0.0.0.0) port(5527) ); tcp(ip(0.0.0.0) port(5527) ); }; destination d_adm1 { file( "/Data/syslog_data/$LOGHOST/$R_YEAR-$R_MONTH-$R_DAY/adm1/$HOST_FROM/$HOST/$FACILITY.local" template("$(adm1-function)\n") ); }; log { source(s_adm1); destination(d_adm1); flags(final); }; A small extract of one of the files is, naturally I have changed the IPs 192.1.1.1/31181->192.1.3.1/135 0x0 source rule r12 N/A N/A 6 ACME-CNB010 ACME CNB 1727847 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN275 <14>Aug 19 11:38:50 sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created 192.6.20.1/47250->192.104.20.1/443 0x0 junos-https 192.6.20.1/47250->192.104.20.1/443 0x0 N/A N/A N/A N/A 6 ACME-BWC002 ACME BWC_EDESIX_AWS 917879 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN275 <14>Aug 19 11:38:50 sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created 192.6.20.1/47252->192.104.20.1/443 0x0 junos-https 192.6.20.1/47252->192.104.20.1/443 0x0 N/A N/A N/A N/A 6 ACME-BWC002 ACME BWC_EDESIX_AWS 595069 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN285 <14>Aug 19 11:38:50 sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created 192.1.57.1/54205->192.1.7.1/137 0x0 junos-nbname 192.1.57.1/54205->192.1.7.1/137 0x0 N/A N/A N/A N/A 17 ACME-COMP007 ACME CCA 1735324 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN288 <14>Aug 19 11:38:50 sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created 192.1.57.1/62486->192.1.7.1/135 0x0 junos-ms-rpc-tcp 192.1.57.1/62486->192.1.7.1/135 0x0 N/A N/A N/A N/A 6 ACME-COMP007 ACME CCA 1027474 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN249 I suspect it might be something to do with the amount of logs that are received in very quick succession, but I have no idea how to address this. One thing I have noticed is that it always appears to be the same device that is causing the issue. When I look at other devices coming in on the same port, they appear to write out correctly. Any suggestion would be really welcome. Thanks John This message has been scanned for malware by Forcepoint. www.forcepoint.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From bazsi77 at gmail.com Sat Aug 20 05:33:15 2022 From: bazsi77 at gmail.com (Balazs Scheidler) Date: Sat, 20 Aug 2022 07:33:15 +0200 Subject: [syslog-ng] Potential issue with line breaking not working correctly when writing logs to disk In-Reply-To: References: Message-ID: Hi, Sorry I have trouble matching up the config with the log output. Please find some comments inline. Bal?zs On Fri, Aug 19, 2022, 16:47 John Law wrote: > Hi, > > We have a number of network devices sending syslog traffic to syslog-ng > OSE 3.25 installed on RHEL 7.9. > > We are having an issue where multiple events are being written as the same > line. > > The config file is > > template-function adm1 "${HOST} message-length=$(length \"${MSG}\") > ${MESSAGE}\n"; > source s_adm1 { > udp(ip(0.0.0.0) port(5527) ); > tcp(ip(0.0.0.0) port(5527) ); > }; > destination d_adm1 { > file( > > "/Data/syslog_data/$LOGHOST/$R_YEAR-$R_MONTH-$R_DAY/adm1/$HOST_FROM/$HOST/$FACILITY.local" > template("$(adm1-function)\n") > You only mention $(adm1) as a template function but this one refers to $(adm1-function) ); > }; > log { source(s_adm1); destination(d_adm1); flags(final); }; > > > A small extract of one of the files is, naturally I have changed the IPs > > > 192.1.1.1/31181->192.1.3.1/135 0x0 source rule r12 N/A N/A 6 ACME-CNB010 > ACME CNB 1727847 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN275 <14>Aug 19 > 11:38:50 sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created > 192.6.20.1/47250->192.104.20.1/443 0x0 junos-https 192.6.20.1/47250-> > 192.104.20.1/443 0x0 N/A N/A N/A N/A 6 ACME-BWC002 ACME BWC_EDESIX_AWS > 917879 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN275 <14>Aug 19 11:38:50 > sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created > Can you show me the lines intermixed here? Again this does not seem to have been formatted with $(adm1) at least as defined above. 192.6.20.1/47252->192.104.20.1/443 0x0 junos-https 192.6.20.1/47252-> > 192.104.20.1/443 0x0 N/A N/A N/A N/A 6 ACME-BWC002 ACME BWC_EDESIX_AWS > 595069 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN285 <14>Aug 19 11:38:50 > sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created > 192.1.57.1/54205->192.1.7.1/137 0x0 junos-nbname 192.1.57.1/54205-> > 192.1.7.1/137 0x0 N/A N/A N/A N/A 17 ACME-COMP007 ACME CCA 1735324 > N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN288 <14>Aug 19 11:38:50 > sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created > 192.1.57.1/62486->192.1.7.1/135 0x0 junos-ms-rpc-tcp 192.1.57.1/62486-> > 192.1.7.1/135 0x0 N/A N/A N/A N/A 6 ACME-COMP007 ACME CCA 1027474 > N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN249 > > I suspect it might be something to do with the amount of logs that are > received in very quick succession, but I have no idea how to address this. > One thing I have noticed is that it always appears to be the same device > that is causing the issue. When I look at other devices coming in on the > same port, they appear to write out correctly. > > Any suggestion would be really welcome. > > Thanks > > John > > > This message has been scanned for malware by Forcepoint. > www.forcepoint.com > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From JohnLaw at tfl.gov.uk Mon Aug 22 12:58:05 2022 From: JohnLaw at tfl.gov.uk (John Law) Date: Mon, 22 Aug 2022 12:58:05 +0000 Subject: [syslog-ng] Potential issue with line breaking not working correctly when writing logs to disk In-Reply-To: References: Message-ID: Thank you Bal?zs, I have done some more digging and I don't believe it is syslog that is the issue, I think the traffic is being forwarded to me from another syslog platform and that is where the issue lies. A packet capture is showing individual events with the correct line breaks, but then it also shows a number of events being sent as one large packet. Unless I can get syslog-ng to linebreak on regex when the data comes in, I need to go back to the source and get them to address this. PS the template mismatch below was a typo. Kind Regards John ________________________________ From: syslog-ng on behalf of Balazs Scheidler Sent: 20 August 2022 06:33 To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Potential issue with line breaking not working correctly when writing logs to disk Hi, Sorry I have trouble matching up the config with the log output. Please find some comments inline. Bal?zs On Fri, Aug 19, 2022, 16:47 John Law > wrote: Hi, We have a number of network devices sending syslog traffic to syslog-ng OSE 3.25 installed on RHEL 7.9. We are having an issue where multiple events are being written as the same line. The config file is template-function adm1 "${HOST} message-length=$(length \"${MSG}\") ${MESSAGE}\n"; source s_adm1 { udp(ip(0.0.0.0) port(5527) ); tcp(ip(0.0.0.0) port(5527) ); }; destination d_adm1 { file( "/Data/syslog_data/$LOGHOST/$R_YEAR-$R_MONTH-$R_DAY/adm1/$HOST_FROM/$HOST/$FACILITY.local" template("$(adm1)\n") You only mention $(adm1) as a template function but this one refers to $(adm1-function) ); }; log { source(s_adm1); destination(d_adm1); flags(final); }; A small extract of one of the files is, naturally I have changed the IPs 192.1.1.1/31181->192.1.3.1/135 0x0 source rule r12 N/A N/A 6 ACME-CNB010 ACME CNB 1727847 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN275 <14>Aug 19 11:38:50 sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created 192.6.20.1/47250->192.104.20.1/443 0x0 junos-https 192.6.20.1/47250->192.104.20.1/443 0x0 N/A N/A N/A N/A 6 ACME-BWC002 ACME BWC_EDESIX_AWS 917879 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN275 <14>Aug 19 11:38:50 sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created Can you show me the lines intermixed here? Again this does not seem to have been formatted with $(adm1) at least as defined above. 192.6.20.1/47252->192.104.20.1/443 0x0 junos-https 192.6.20.1/47252->192.104.20.1/443 0x0 N/A N/A N/A N/A 6 ACME-BWC002 ACME BWC_EDESIX_AWS 595069 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN285 <14>Aug 19 11:38:50 sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created 192.1.57.1/54205->192.1.7.1/137 0x0 junos-nbname 192.1.57.1/54205->192.1.7.1/137 0x0 N/A N/A N/A N/A 17 ACME-COMP007 ACME CCA 1735324 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN288 <14>Aug 19 11:38:50 sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created 192.1.57.1/62486->192.1.7.1/135 0x0 junos-ms-rpc-tcp 192.1.57.1/62486->192.1.7.1/135 0x0 N/A N/A N/A N/A 6 ACME-COMP007 ACME CCA 1027474 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN249 I suspect it might be something to do with the amount of logs that are received in very quick succession, but I have no idea how to address this. One thing I have noticed is that it always appears to be the same device that is causing the issue. When I look at other devices coming in on the same port, they appear to write out correctly. Any suggestion would be really welcome. Thanks John This message has been scanned for malware by Forcepoint. www.forcepoint.com ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq -------------- next part -------------- An HTML attachment was scrubbed... URL: From bazsi77 at gmail.com Mon Aug 22 20:19:46 2022 From: bazsi77 at gmail.com (Balazs Scheidler) Date: Mon, 22 Aug 2022 22:19:46 +0200 Subject: [syslog-ng] Potential issue with line breaking not working correctly when writing logs to disk In-Reply-To: References: Message-ID: Syslog-ng supports breaking up lines based on reg?nyes, but only for files. I could possibly adapt that functionality to network sources of you are willing to try the functionality. But it's best if the source of fixed. On Mon, Aug 22, 2022, 14:58 John Law wrote: > Thank you Bal?zs, > > I have done some more digging and I don't believe it is syslog that is the > issue, I think the traffic is being forwarded to me from another syslog > platform and that is where the issue lies. > > A packet capture is showing individual events with the correct line > breaks, but then it also shows a number of events being sent as one large > packet. Unless I can get syslog-ng to linebreak on regex when the data > comes in, I need to go back to the source and get them to address this. > > PS the template mismatch below was a typo. > > Kind Regards > > John > > ------------------------------ > *From:* syslog-ng on behalf of > Balazs Scheidler > *Sent:* 20 August 2022 06:33 > *To:* Syslog-ng users' and developers' mailing list < > syslog-ng at lists.balabit.hu> > *Subject:* Re: [syslog-ng] Potential issue with line breaking not working > correctly when writing logs to disk > > Hi, > > Sorry I have trouble matching up the config with the log output. Please > find some comments inline. > > Bal?zs > > On Fri, Aug 19, 2022, 16:47 John Law wrote: > > Hi, > > We have a number of network devices sending syslog traffic to syslog-ng > OSE 3.25 installed on RHEL 7.9. > > We are having an issue where multiple events are being written as the same > line. > > The config file is > > template-function adm1 "${HOST} message-length=$(length \"${MSG}\") > ${MESSAGE}\n"; > source s_adm1 { > udp(ip(0.0.0.0) port(5527) ); > tcp(ip(0.0.0.0) port(5527) ); > }; > destination d_adm1 { > file( > > "/Data/syslog_data/$LOGHOST/$R_YEAR-$R_MONTH-$R_DAY/adm1/$HOST_FROM/$HOST/$FACILITY.local" > template("$(adm1)\n") > > > You only mention $(adm1) as a template function but this one refers to > $(adm1-function) > > ); > }; > log { source(s_adm1); destination(d_adm1); flags(final); }; > > > A small extract of one of the files is, naturally I have changed the IPs > > > 192.1.1.1/31181- > > >192.1.3.1/135 > > 0x0 source rule r12 N/A N/A 6 ACME-CNB010 ACME CNB 1727847 N/A(N/A) > reth3.860 UNKNOWN UNKNOWN UNKNOWN275 <14>Aug 19 11:38:50 sunny-fwl29 > RT_FLOW: RT_FLOW_SESSION_CREATE: session created 192.6.20.1/47250- > > >192.104.20.1/443 > > 0x0 junos-https 192.6.20.1/47250- > > >192.104.20.1/443 > > 0x0 N/A N/A N/A N/A 6 ACME-BWC002 ACME BWC_EDESIX_AWS 917879 N/A(N/A) > reth3.860 UNKNOWN UNKNOWN UNKNOWN275 <14>Aug 19 11:38:50 sunny-fwl29 > RT_FLOW: RT_FLOW_SESSION_CREATE: session created > > > Can you show me the lines intermixed here? Again this does not seem to > have been formatted with $(adm1) at least as defined above. > > > 192.6.20.1/47252- > > >192.104.20.1/443 > > 0x0 junos-https 192.6.20.1/47252- > > >192.104.20.1/443 > > 0x0 N/A N/A N/A N/A 6 ACME-BWC002 ACME BWC_EDESIX_AWS 595069 N/A(N/A) > reth3.860 UNKNOWN UNKNOWN UNKNOWN285 <14>Aug 19 11:38:50 sunny-fwl29 > RT_FLOW: RT_FLOW_SESSION_CREATE: session created 192.1.57.1/54205- > > >192.1.7.1/137 > > 0x0 junos-nbname 192.1.57.1/54205- > > >192.1.7.1/137 > > 0x0 N/A N/A N/A N/A 17 ACME-COMP007 ACME CCA 1735324 N/A(N/A) reth3.860 > UNKNOWN UNKNOWN UNKNOWN288 <14>Aug 19 11:38:50 sunny-fwl29 RT_FLOW: > RT_FLOW_SESSION_CREATE: session created 192.1.57.1/62486- > > >192.1.7.1/135 > > 0x0 junos-ms-rpc-tcp 192.1.57.1/62486- > > >192.1.7.1/135 > > 0x0 N/A N/A N/A N/A 6 ACME-COMP007 ACME CCA 1027474 N/A(N/A) reth3.860 > UNKNOWN UNKNOWN UNKNOWN249 > > I suspect it might be something to do with the amount of logs that are > received in very quick succession, but I have no idea how to address this. > One thing I have noticed is that it always appears to be the same device > that is causing the issue. When I look at other devices coming in on the > same port, they appear to write out correctly. > > Any suggestion would be really welcome. > > Thanks > > John > > > This message has been scanned for malware by Forcepoint. > www.forcepoint.com > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From JohnLaw at tfl.gov.uk Tue Aug 23 06:19:02 2022 From: JohnLaw at tfl.gov.uk (John Law) Date: Tue, 23 Aug 2022 06:19:02 +0000 Subject: [syslog-ng] Potential issue with line breaking not working correctly when writing logs to disk In-Reply-To: References: Message-ID: Thanks Balazs, I have gone back to the source to get them to address. Kind Regards John ________________________________ From: syslog-ng on behalf of Balazs Scheidler Sent: 22 August 2022 21:19 To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Potential issue with line breaking not working correctly when writing logs to disk Syslog-ng supports breaking up lines based on reg?nyes, but only for files. I could possibly adapt that functionality to network sources of you are willing to try the functionality. But it's best if the source of fixed. On Mon, Aug 22, 2022, 14:58 John Law > wrote: Thank you Bal?zs, I have done some more digging and I don't believe it is syslog that is the issue, I think the traffic is being forwarded to me from another syslog platform and that is where the issue lies. A packet capture is showing individual events with the correct line breaks, but then it also shows a number of events being sent as one large packet. Unless I can get syslog-ng to linebreak on regex when the data comes in, I need to go back to the source and get them to address this. PS the template mismatch below was a typo. Kind Regards John ________________________________ From: syslog-ng > on behalf of Balazs Scheidler > Sent: 20 August 2022 06:33 To: Syslog-ng users' and developers' mailing list > Subject: Re: [syslog-ng] Potential issue with line breaking not working correctly when writing logs to disk Hi, Sorry I have trouble matching up the config with the log output. Please find some comments inline. Bal?zs On Fri, Aug 19, 2022, 16:47 John Law > wrote: Hi, We have a number of network devices sending syslog traffic to syslog-ng OSE 3.25 installed on RHEL 7.9. We are having an issue where multiple events are being written as the same line. The config file is template-function adm1 "${HOST} message-length=$(length \"${MSG}\") ${MESSAGE}\n"; source s_adm1 { udp(ip(0.0.0.0) port(5527) ); tcp(ip(0.0.0.0) port(5527) ); }; destination d_adm1 { file( "/Data/syslog_data/$LOGHOST/$R_YEAR-$R_MONTH-$R_DAY/adm1/$HOST_FROM/$HOST/$FACILITY.local" template("$(adm1)\n") You only mention $(adm1) as a template function but this one refers to $(adm1-function) ); }; log { source(s_adm1); destination(d_adm1); flags(final); }; A small extract of one of the files is, naturally I have changed the IPs 192.1.1.1/31181->192.1.3.1/135 0x0 source rule r12 N/A N/A 6 ACME-CNB010 ACME CNB 1727847 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN275 <14>Aug 19 11:38:50 sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created 192.6.20.1/47250->192.104.20.1/443 0x0 junos-https 192.6.20.1/47250->192.104.20.1/443 0x0 N/A N/A N/A N/A 6 ACME-BWC002 ACME BWC_EDESIX_AWS 917879 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN275 <14>Aug 19 11:38:50 sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created Can you show me the lines intermixed here? Again this does not seem to have been formatted with $(adm1) at least as defined above. 192.6.20.1/47252->192.104.20.1/443 0x0 junos-https 192.6.20.1/47252->192.104.20.1/443 0x0 N/A N/A N/A N/A 6 ACME-BWC002 ACME BWC_EDESIX_AWS 595069 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN285 <14>Aug 19 11:38:50 sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created 192.1.57.1/54205->192.1.7.1/137 0x0 junos-nbname 192.1.57.1/54205->192.1.7.1/137 0x0 N/A N/A N/A N/A 17 ACME-COMP007 ACME CCA 1735324 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN288 <14>Aug 19 11:38:50 sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created 192.1.57.1/62486->192.1.7.1/135 0x0 junos-ms-rpc-tcp 192.1.57.1/62486->192.1.7.1/135 0x0 N/A N/A N/A N/A 6 ACME-COMP007 ACME CCA 1027474 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN249 I suspect it might be something to do with the amount of logs that are received in very quick succession, but I have no idea how to address this. One thing I have noticed is that it always appears to be the same device that is causing the issue. When I look at other devices coming in on the same port, they appear to write out correctly. Any suggestion would be really welcome. Thanks John This message has been scanned for malware by Forcepoint. www.forcepoint.com ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq -------------- next part -------------- An HTML attachment was scrubbed... URL: From Laszlo.Varady at oneidentity.com Mon Aug 29 11:51:07 2022 From: Laszlo.Varady at oneidentity.com (Laszlo Varady (lvarady)) Date: Mon, 29 Aug 2022 11:51:07 +0000 Subject: [syslog-ng] syslog-ng-3.38.1 Message-ID: 3.38.1 Highlights Sneak peek into syslog-ng v4.0 syslog-ng v4.0 is right around the corner. This release (v3.38.1) contains all major changes, however, they are currently all hidden behind a feature flag. To enable and try those features, you need to specify @version: 4.0 at the top of the configuration file. You can find out more about the 4.0 changes and features here. Read our practical introduction to typing at syslog-ng-future.blog. Features * grouping-by(): added inject-mode(aggregate-only) This inject mode will drop individual messages that make up the correlation context (key() groups) and would only yield the aggregate messages (e.g. the results of the correlation). (#3998) * add-contextual-data(): add support for type propagation, e.g. set the type of name-value pairs as they are created/updated to the value returned by the template expression that we use to set the value. The 3rd column in the CSV file (e.g. the template expression) now supports specifying a type-hint, in the format of "type-hint(template-expr)". Example line in the CSV database: selector-value,name-value-pair-to-be-created,list(foo,bar,baz) (#4051) * $(format-json): add --key-delimiter option to reconstruct JSON objects using an alternative structure separator, that was created using the key-delimiter() option of json-parser(). (#4093) * json-parser(): add key-delimiter() option to extract JSON structure members into name-value pairs, so that the names are flattened using the character specified, instead of dot. Example: Input: {"foo":{"key":"value"}} Using json-parser() without key-delimiter() this is extracted to: foo.key="value" Using json-parser(key-delimiter("~")) this is extracted to: foo~key="value" This feature is useful in case the JSON keys contain dots themselves, in those cases the syslog-ng representation is ambigious. (#4093) Bugfixes * Fixed buffer handling of syslog and timestamp parsers Multiple buffer out-of-bounds issues have been fixed, which could cause hangs, high CPU usage, or other undefined behavior. (#4110) * Fixed building with LibreSSL (#4081) * network(): Fixed a bug, where syslog-ng halted the input instead of skipping a character in case of a character conversion error. (#4084) * redis(): Fixed bug where using redis driver without the batch-lines option caused program crash. (#4114) * pdbtool: fix a SIGABRT on FreeBSD that was triggered right before pdbtool exits. Apart from being an ugly crash that produces a core file, functionally the tool behaved correctly and this case does not affect syslog-ng itself. (#4037) * regexp-parser(): due to a change introduced in 3.37, named capture groups are stored indirectly in the LogMessage to avoid copying of the value. In this case the name-value pair created with the regexp is only stored as a reference (name + length of the original value), which improves performance and makes such name-value pairs use less memory. One omission in the original change in 3.37 is that syslog-ng does not allow builtin values to be stored indirectly (e.g. $MESSAGE and a few of others) and this case causes an assertion to fail and syslog-ng to crash with a SIGABRT. This abort is now fixed. Here's a sample config that reproduces the issue: regexp-parser(patterns('(?.*)')); (#4043) * set-tag: fix cloning issue when string literal were used (see #4062) (#4065) * add-contextual-data(): fix high memory usage when using large CSV files (#4067) Other changes * The json-c library is no longer bundled in the syslog-ng source tarball Since all known OS package managers provide json-c packages nowadays, the json-c submodule has been removed from the source tarball. The --with-jsonc=internal option of the configure script has been removed accordingly, system libraries will be used instead. For special cases, the JSON support can be disabled by specifying --with-jsonc=no. (#4078) * platforms: Dropped support for ubuntu-impish as it became EOL (#4088) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Alvin ?ipraga, Andras Mitzki, Attila Szakacs, Balazs Scheidler, B?lint Horv?th, Daniel Klauer, Fabrice Fontaine, Gabor Nagy, HenryTheSir, L?szl? V?rady, Parrag Szil?rd, Peter Kokai, Shikhar Vashistha, Szil?rd Parrag, Vivin Peris Installation from binaries: https://github.com/syslog-ng/syslog-ng#installation-from-binaries -------------- next part -------------- An HTML attachment was scrubbed... URL: