- syslog-ng - lists.balabit.hu

Filtering out messages before any other action
by Faine, Mark R. (MSFC-IS40)[NICS] 27 Mar '19

27 Mar '19

I have the following as the first log path in my configuration, however, My understanding is that this should match any message with the text 'type=traffic' AND any of the listed policyid=## or a message with the text 'action=timeout'. Flags(final) indicates that it should not be further processed by any other log path. However, I'm still seeing messages that should be discarded, am I doing something wrong? log { if { filter { message('type=traffic') }; filter { ( message('policyid=35 ') or message('policyid=37 ') or message('policyid=38 ') or message('policyid=40_D ') or message('policyid=5 ') or message('policyid=10 ') or message('policyid=12 ') or message('policyid=50 ') or message('policyid=3 ') or message('action=timeout ') ) }; destination { file("/dev/null"); }; }; flags(final); }; Thanks, -Mark

2 2

syslog-ng.conf filter not working
by Lin, Victor 27 Mar '19

27 Mar '19

Dear All, I am trying to forwarding all cisco IOS and Nexus to remote server Here is from my syslog-ng.conf ********************* # Syslog collection for all devices source s_network { network( transport("udp") port(514) flags(syslog_protocol) keep_hostname(yes) keep_timestamp(yes) use_dns(no) use_fqdn(no) ); }; destination d_all_logs { file("/app/syslog-ng/My_custom/My_output/all_devices.log"); }; #All syslogs log { source(s_network); destination(d_all_logs); }; ***************************** #Cisco to elastic Mar.22.2019 destination d_cisco_logs { file("/app/syslog-ng/My_custom/My_output/cisco.log"); network("10.20.30.44" port(2514) transport(udp) spoof_source(yes)); }; ***************************************** #Cisco logs to elastic Mar.22.2019 log { source(s_network); filter(f_cisco_message); destination(d_cisco_logs); }; *********************************** #Cisco to elastic Mar.22.2019 filter f_cisco_message { match ("Cisco IOS", value ("MESSAGE")); or match ("Cisco Nexus", value ("MESSAGE")); }; But looks like cisco.log is never have any data inside. Below is from Could you please review my config and advice ? Thank you so much for your reply in advance! VL _______________________________________________________________________ If you received this email in error, please advise the sender (by return email or otherwise) immediately. You have consented to receive the attached electronically at the above-noted email address; please retain a copy of this confirmation for future reference. Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur immédiatement, par retour de courriel ou par un autre moyen. Vous avez accepté de recevoir le(s) document(s) ci-joint(s) par voie électronique à l'adresse courriel indiquée ci-dessus; veuillez conserver une copie de cette confirmation pour les fins de reference future.

2 1

Best way to manage apache's access logs
by Nik Ambrosch 26 Mar '19

26 Mar '19

How are people managing their apache access logs using syslog-ng these days - still just piping through logger? I'm looking to take over logging entirely using syslog-ng, not just reading files from disk and sending to ES or something. In the past I've used cronolog, which works fine, but I'd love the flexibility of sending the logs through syslog-ng.

3 4

Setting and using variables
by Faine, Mark R. (MSFC-IS40)[NICS] 25 Mar '19

25 Mar '19

Is there a way to set variables in syslog-ng? I have a log path with about 20 if/else branches and each one does a unnamed destination for that branch: log { source(pan_splunk); if ( tags('mytag') ) { destination { file("/var/log/remote/backup/$HOST/asa/${HOST}_asa.log" create-dirs(yes) dir-owner("splunk") dir-group("splunk") dir-perm(0750)); }; } elif ( message('something else') ) { destination { file("/var/log/remote/backup/$HOST/pubfw/${HOST}_pubfw.log" create-dirs(yes) dir-owner("splunk") dir-group("splunk") dir-perm(0750)); }; } elif { filter { message('foo') or message('bar') or message('baz') or ... I'd need to introduce another directory level as a variable and I'd also like to change an existing part of the path to a variable so that then I could then do something like this: if ( tags('mytag') ) { app = asa location = msfc elif ... and at the end I could then just do a single destination that had a file path with the variables file("/var/log/remote/backup/$location/$HOST/$app/${HOST}_$app.log" Thanks, -Mark

3 11

seems like program filter is broken
by Stanislav 22 Mar '19

22 Mar '19

Greetings, I'm getting this issue after my last package upgrade ====================================== Name : syslog-ng Version : 3.20.1 Installed on : Mon Mar 11 23:27:29 2019 EET Origin : sysutils/syslog-ng Architecture : FreeBSD:12:amd64 Prefix : /usr/local Categories : sysutils Licenses : Maintainer : cy(a)FreeBSD.org WWW : http://www.syslog-ng.org/ Comment : Powerful syslogd replacement Options : AMQP : off CURL : off DOCS : on GEOIP2 : off IPV6 : off JAVA : off JAVA_MOD : off JSON : on MONGO : off PYTHON : off REDIS : off RIEMANN : off SMTP : off SPOOF : off SQL : off TCP_WRAPPERS : off ====================================== I have following configuration: options { chain_hostnames(off); flush_lines(0); threaded(yes); create_dirs(yes); }; source local { internal(); unix-dgram( "/var/run/log" owner(root) group(wheel) perm(0666) ); unix-dgram( "/var/run/logpriv" owner(root) group(wheel) perm(0600) ); file( "/dev/klog" program_override("kernel") ); }; ... destination all { file("/var/log/all.log"); }; destination maillog_mda { file("/var/log/maillog-mda"); }; ... filter p_mail_imap { program("dovecot"); }; ... log { source(local); destination(all); }; log { source(local); filter(p_mail_imap); destination(maillog_mda); }; ====================================== # ps auxww|grep dovecot root 9648 0.0 0.1 13268 4196 - Is 00:46 0:00.04 /usr/local/sbin/dovecot -c /usr/local/etc/dovecot/dovecot.conf dovecot 9651 0.0 0.0 12724 3784 - I 00:46 0:00.01 anvil: [2 connections] (anvil) root 15259 0.0 0.0 12796 4168 - I 01:42 0:00.00 dovecot/log root 16126 0.0 0.1 13744 5020 - I 01:52 0:00.02 dovecot/config dovecot 16127 0.0 0.0 12724 4180 - I 01:52 0:00.01 stats: [3 connections] (stats) dovecot 17328 0.0 0.1 21284 12276 - I 02:05 0:00.01 auth: [0 wait, 0 passdb, 0 userdb] (auth) ====================================== # syslog-ng -s # echo $? 0 ====================================== I'm getting logs from dovecot program to /var/log/all.log but not /var/log/maillog-mda . As I mentioned before it was working on previous version of syslog-ng . Does anybody have this issue? Just me, lucky?

3 8

use loggen to insert content of a file into local log file
by Lin, Victor 19 Mar '19

19 Mar '19

Hello Fabien, Udp 5514 be listening by syslog-ng and write the info into /app/syslog-ng/output/AAA.log ****************************************************************** Here is from syslog-ng.conf source s_AAA_network { network( transport("udp") port(5514) flags(syslog_protocol) keep_hostname(yes) keep_timestamp(yes) use_dns(no) use_fqdn(no) tags("cppm") ); }; destination d_cppm_logs { file("/app/syslog-ng/output/AAA.log"); network("192.168.22. 39" port(5514) transport(udp) spoof_source(yes)); }; log { source(s_AAA_network); destination(d_AAA_Audit_logs); }; I can see both AAA.log and remote server 192.168.22. 39 can receive the syslog messages. ************************************************************************ When I run a test command below, I got the following output /app/syslog-ng/bin/loggen -D localhost 5514 "test test test" count=1864, rate = 963.75 msg/sec count=2348, rate = 967.19 msg/sec count=2830, rate = 963.87 msg/sec count=3314, rate = 967.87 msg/sec count=3796, rate = 963.87 msg/sec count=4280, rate = 967.41 msg/sec count=4762, rate = 963.75 msg/sec count=5246, rate = 967.87 msg/sec count=5728, rate = 963.87 msg/sec count=6212, rate = 967.87 msg/sec count=6694, rate = 963.71 msg/sec count=7177, rate = 965.46 msg/sec count=7660, rate = 965.70 msg/sec count=8142, rate = 963.74 msg/sec count=8626, rate = 967.71 msg/sec count=9108, rate = 963.85 msg/sec count=9590, rate = 963.84 msg/sec count=10074, rate = 966.30 msg/sec count=10554, rate = 959.80 msg/sec average rate = 1055.01 msg/sec, count=10554, time=10.0037, (average) msg size=256, bandwidth=263.75 kB/sec but I didn't see "test test test" been writen into /app/syslog-ng/output/AAA.log Do I using the correct format of loggen command? Thank you very much! VL -----Original Message----- From: syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of syslog-ng-request(a)lists.balabit.hu Sent: 2019, March, 07 7:00 AM To: syslog-ng(a)lists.balabit.hu Subject: syslog-ng Digest, Vol 167, Issue 15 Send syslog-ng mailing list submissions to syslog-ng(a)lists.balabit.hu To subscribe or unsubscribe via the World Wide Web, visit https://lists.balabit.hu/mailman/listinfo/syslog-ng or, via email, send a message with subject or body 'help' to syslog-ng-request(a)lists.balabit.hu You can reach the person managing the list at syslog-ng-owner(a)lists.balabit.hu When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..." Today's Topics: 1. Re: use loggen to insert content of a file into local log file (Fabien Wernli) ---------------------------------------------------------------------- Message: 1 Date: Thu, 7 Mar 2019 09:03:34 +0100 From: Fabien Wernli <wernli(a)in2p3.fr> To: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] use loggen to insert content of a file into local log file Message-ID: <20190307080334.n3xc2q5nq2kfr7wv(a)ccfawe.in2p3.fr> Content-Type: text/plain; charset=iso-8859-1 On Wed, Mar 06, 2019 at 01:35:02PM +0000, Lin, Victor wrote: > Could anyone can give an example of using loggen to insert content of a file into a local log file ? you could use loggen to send to a specific port, on which syslog-ng would be listening, then create a log statement sendîng to that file, e.g. off the top of my head: log { source { syslog( port(1234) ); }; destination { file( '/tmp/loggen.log' ); }; }; loggen -D "blah blah blah" localhost 1234 ------------------------------ Subject: Digest Footer _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng ------------------------------ End of syslog-ng Digest, Vol 167, Issue 15 ****************************************** _______________________________________________________________________ If you received this email in error, please advise the sender (by return email or otherwise) immediately. You have consented to receive the attached electronically at the above-noted email address; please retain a copy of this confirmation for future reference. Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur immédiatement, par retour de courriel ou par un autre moyen. Vous avez accepté de recevoir le(s) document(s) ci-joint(s) par voie électronique à l'adresse courriel indiquée ci-dessus; veuillez conserver une copie de cette confirmation pour les fins de reference future.

2 1

Insider 2019-03: syslog-ng 3.20.1; fail2ban; HTTP source; container tools; UDP; Conferences & Webinars
by Czanik, Péter 14 Mar '19

14 Mar '19

Dear syslog-ng users, This is the 73rd issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng 3.20.1 released ------------------------- Version 3.20.1 of syslog-ng was released. It adds persist-tool to manipulate persists files, a collectd destination, parsers for Netskope and Websense logs, and list support to JSON and XML. Naturally, bug fixes and many more changes were under the hood as well. There are some developer facing improvements, too, such as the fact that Python developers can now generate internal() log messages. For a complete list of changes with links to individual pull requests describing the changes in more detail, check https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.20.1 Documentation for most of the new features is now available online at https://www.syslog-ng.com/technical-documents/list/syslog-ng-open-source-ed… Visualizing Fail2ban logs in Kibana ----------------------------------- In his previous syslog-ng blog post, Balage wrote about how you can enrich Fail2ban logs with GeoIP metadata and other data parsed from the logs. In this post he will show you how you can use syslog-ng to send the logs into Elasticsearch and how visualizing Fail2ban logs in Kibana can show you where the failed login attempts come from. https://balagetech.com/visualizing-fail2ban-logs-in-kibana/ Creating an HTTP source for syslog-ng in Python ----------------------------------------------- HTTP is quickly becoming the universal transport protocol of the Internet. Nowadays even DNS over HTTPS implementations are available. There is no HTTP source implemented in C for syslog-ng, but starting with syslog-ng version 3.18, you can write new source drivers for syslog-ng in Python. While performance is not as good as it would be using C, you gain flexibility and ease of implementation by using Python. From this blog post you can learn how to create a basic HTTP source for syslog-ng in Python. https://www.syslog-ng.com/community/b/blog/posts/creating-an-http-source-fo… Building and running a syslog-ng container using the latest tools ----------------------------------------------------------- >From this blog you can learn the basics of skopeo, buildah and podman through the examples of a syslog-ng container. These tools come from Red Hat, replacing and extending functionality of the docker daemon without the need to run a “big fat daemon” on the hosts. The architectural redesign is closer to the UNIX philosophy and here is no single point of failure. https://www.syslog-ng.com/community/b/blog/posts/building-and-running-a-sys… Improved log collection over UDP -------------------------------- “I'd tell you the joke about UDP, but you might not get it.” This old joke perfectly summarizes UDP. There is no guarantee (frankly, not even a real effort) that data sent over UDP ever reaches the receiving end. Still, a surprisingly large number of syslog(-ng) users still keep using UDP as their transport protocol. The introduction of the so-reuseport() option for the UDP source in syslog-ng 3.19 is for those people who, for policy or other reasons, can not switch to TCP logging but still want to make UDP log reception as reliable as possible. https://www.syslog-ng.com/community/b/blog/posts/improved-log-collection-ov… CONFERENCES * DevOps Pro talk: Syslog-ng for DevOps: Customized Logging with Python ( https://devopspro.lt/peter-czanik/ ) WEBINARS You can watch our past webinars: * Got a Hungry Splunk? Feed it Smartly with syslog-ng: https://www.brighttalk.com/webcast/16207/350610 Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

Trigger dynamic action in syslog-ng
by Jim Hendrick 08 Mar '19

08 Mar '19

I was wondering if anyone has used syslog-ng to trigger some dynamic action based on logs. For example, if a certain threshold of messages happens in a time window, send an alert. LIke suppress () but more general actions. Or if a specific event happens, send *.debug from that system for 5 minutes. Or run a program to collect system data and send it along based on some condition. Not thinking SIEM functionality here, but maybe allow the log servers to be more dynamic around what actions they take for basic things. Thoughts? Thanks. Jim

3 3

Problems parsing Cisco syslogs
by N. Max Pierson 08 Mar '19

08 Mar '19

Hi List, I have been trying to get something in place that can parse syslogs from various Cisco devices. The message format is almost the same with a few exceptions. Here is what I have tried and it works but now it has created another problem I do not know how to troubleshoot. So that I could see exactly what was being parsed, I disabled the default parsing using the below. source s_network { udp(ip(0.0.0.0) port(514) flags(no-parse)); }; rewrite r_cisco{ subst('^<\d+>(\d+:|:)\s+(\.\w+|\w+)\s+\d+\s+\d+\s\d+:\d+:\d+:\s|^<\d+>:\s+\d+\s+\w+\s+\d+\s+\d+:\d+:\d+\s\w+:\s|^<\d+>(\d+:|:)\s', "", value("MESSAGE"), type("pcre"), flags("ignore-case")); }; destination d_mysql { sql(type(mysql) host("127.0.0.1") username("syslog-ng") password("password") database("syslog") table("messages_${HOST}") columns("datetime datetime", "host varchar(50)", "level varchar(10)", "message text") values("${R_YEAR}-${R_MONTH}-${R_DAY} ${R_HOUR}:${R_MIN}:${R_SEC}", "${HOST}", "${LEVEL}", "${MESSAGE}") indexes("datetime", "level") ); }; log { source(s_network); rewrite(r_cisco); destination(d_mysql); }; This works perfectly as it formats the message as I want and covers IOS and NX-OS devices. The problem is when I turned off the default parser, now all of my logs show "notice" in the $LEVEL macro and doesn't reflect the real message header level. The $HOST macro still works fine however. Is this the expected behavior that the message header fields are not parsed as well as the $MESSAGE itself not being parsed? How can map the header level field properly to the $LEVEL marco if I disable the default parser? Regards, Max

7 20

use loggen to insert content of a file into local log file
by Lin, Victor 07 Mar '19

07 Mar '19

Dear All, Could anyone can give an example of using loggen to insert content of a file into a local log file ? Thank you very much for you great help! VL _______________________________________________________________________ If you received this email in error, please advise the sender (by return email or otherwise) immediately. You have consented to receive the attached electronically at the above-noted email address; please retain a copy of this confirmation for future reference. Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur immédiatement, par retour de courriel ou par un autre moyen. Vous avez accepté de recevoir le(s) document(s) ci-joint(s) par voie électronique à l'adresse courriel indiquée ci-dessus; veuillez conserver une copie de cette confirmation pour les fins de reference future.

2 1