- syslog-ng - lists.balabit.hu

Issue with timestamp
by Andrew 18 Jun '17

18 Jun '17

Hi I am new to syslog-ng and logging in general so apologies but I am having trouble understanding timestamps. I am trying to extract the timestamp inside the message and create a field called 'timestamp' for example. The message looks like this: "MESSAGE":"06:17-16:21:21 fw-1-1 ulogd[9300]: id=\"2001\" severity=\"info\" sys=\"SecureNet\" sub=\"packetfilter\" name=\"Packet dropped\" action=\"drop\" fwrule=\"60002\" initf=\"eth0\" outitf=\"eth1\" srcmac=\"4c:00:82:8a:f4:11\" dstmac=\"00:1a:8c:f0:3c:40\" srcip=\"1.1.1.1\" dstip=\"2.2.2.2\" proto=\"17\" length=\"143\" tos=\"0x00\" prec=\"0x00\" ttl=\"125\" srcport=\"51941\" dstport=\"161\" How can I create a field called 'timestamp' which contains the timestamp in the message but also adds the year to it and puts it into a better format? I am also using the kv-parser and json output looks good I have pairs of everything from the message but I also need a pair called timestamp="Jun 16 2017 16:21:21' for example. Hope I'm making sense here. Thanks for any help.

4 5

documentation for 3.10 available
by Czanik, Péter 16 Jun '17

16 Jun '17

Hi, Documentation for the upcoming syslog-ng 3.10 version is now available. If you want to check out what to expect in the next release, the summary of changes is at https://www.balabit.com/documents/syslog-ng-ose-3.10-guides/en/syslog-ng-os… If you also want to test the new features, I have RPM packages available compiled from git head snapshots. Installation instructions are available at https://www.balabit.com/blog/rpm-packages-from-syslog-ng-git-head/ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

Insider 2017-06: HEPIX; Python parser; Turris; openSUSE; upcoming events;
by Czanik, Péter 15 Jun '17

15 Jun '17

Dear syslog-ng users, This is the 59th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS HEPIX and syslog-ng ------------------- One of the most important discoveries of this decade was the Higgs boson. But researchers at High Energy Physics and Nuclear Physics laboratories and institutes would have been unable to find the Higgs boson without the IT staff maintaining the computer infrastructure collecting and analyzing the massive amount of data generated during their experiments. HEPIX is a community that brings together these IT guys twice a year from around the world. This spring their event was hosted by the Wigner Research Centre for Physics in Budapest, where I also talked to faxmodem. He is both a HEPIX member and one of the most active syslog-ng community members: https://www.balabit.com/blog/helped-turning-higgs-boson-theory-reality/ Process trade data using Python ------------------------------- Upcoming syslog-ng Open Source and Premium Edition releases will feature a Python parser. Learn how to utilize this parser from our webinar. In the demo FIX data is transformed into easy-to-use name-value pairs, but the solution itself is not limited to trade data. You can access the recording of the webinar at https://pages2.balabit.com/fix-data-parser/ (requires registration). syslog-ng updated on Turris Omnia --------------------------------- The release candidate of version 3.7 of Turris OS – the OpenWRT-based operating system of Turris Omnia routers – is now available. Among many other changes, this updates syslog-ng from version 3.0 to 3.9. In other words, it adds about seven years’ worth of new syslog-ng features, including new parsers, filters, formatting options, destinations, and performance enhancements. https://www.balabit.com/blog/using-the-latest-syslog-ng-on-turris-omnia/ openSUSE conference 2017 ------------------------- Being a long-time openSUSE user, I visit the openSUSE conference not only to present on one of its components – syslog-ng – but also to meet friends and learn about new technologies and the plans for the upcoming year. Some talks, like those about Package Hub, were very interesting and important also from a syslog-ng perspective. Of course, I also joined a few talks for my personal interest, like the one on the new ARM devices supported by openSUSE. Read more about my experiences at https://www.balabit.com/blog/friends-syslog-ng-package-hub-arm-opensuse-con… UPCOMING EVENTS The next event where you can learn about syslog-ng: Libre Software Meeting (July 1-7): https://2017.rmll.info/en/ Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

Changing message priority
by To Mitz 15 Jun '17

15 Jun '17

Hi, I am trying to use rewrite to change a message's PRIORITY. I am able to change the HOST to "info" using: rewrite r_rewrite_set{set("info", value("HOST") condition(filter(f_logstash)));}; But, when I try setting the PRIORITY to "info", using this: rewrite r_rewrite_set{set("info", value("PRIORITY") condition(filter(f_logstash)));}; But, when I try setting the PRIORITY to "info", using this: Error parsing rewrite expression, syntax error, unexpected LL_NUMBER, expecting LL_IDENTIFIER or LL_STRING in /etc/syslog-ng/f_messages.conf at line 8, column 27:included from /etc/syslog-ng.conf line 18, column 34rewrite r_rewrite_set{set(3, value("PRIORITY") condition(filter(f_logstash)));};^ Please advise on how I can fix this. Thanks.

3 2

date-parser not found
by Sandbox 13 Jun '17

13 Jun '17

Hi, I would like to use the date-parser. When i using the below parse it says "date-parser not found": parser haproxy_date { date-parser( format("%d/%m/%Y:%H:%M:%S %z") template("$(substr ${MSG} 0 27)") );. }; Plugin module not found in 'module-path'; module-path='/usr/lib/syslog-ng/3.5.6', module='date-parser' Error parsing parser expression, parser plugin date-parser not found in /etc/syslog-ng/conf.d/20-haproxy.conf at line 26, column 5: included from /etc/syslog-ng/syslog-ng.conf line 165, column 1 date-parser( ^^^^^^^^^^^ It's weird, because when i try to use the date-parser module in the log section: log { source(s_haproxy); filter(f_haproxy_info); date-parser(format("%d/%m/%Y:%H:%M:%S %z") template("$(substr ${MSG} 0 27)") ); destination(d_haproxy_info); }; it says: Error parsing log, syntax error, unexpected LL_IDENTIFIER, expecting '}' in /etc/syslog-ng/conf.d/20-haproxy.conf at line 37, column 5: included from /etc/syslog-ng/syslog-ng.conf line 165, column 1 date-parser(format("%d/%m/%Y:%H:%M:%S %z") template("$(substr ${MSG} 0 27)") ); ^^^^^^^^^^^ In OSE version the date-parser module is not available, or i'm missing something? Thanks, Robert

3 3

testing new features, poll
by Czanik, Péter 13 Jun '17

13 Jun '17

Hi, syslog-ng 3.10 is coming soon. If you would like to test it and use one of the RPM-based distributions, there are some ready to use packages to get you started: https://www.balabit.com/blog/rpm-packages-from-syslog-ng-git-head/ I'd also appreciate if you could answer my quick poll on twitter which is your most waited feature in the upcoming syslog-ng version: https://twitter.com/PCzanik/status/874527761532104704 Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

pdbtool Segmentation fault
by Evan Rempel 10 Jun '17

10 Jun '17

I am running a relatively recent githead (April 17) of 3.9.1. I have a single pattern that results in a segmentation fault when running pdbtool test --validate patterndb.xml patterndb can be found at http://web.uvic.ca/~erempel/syslog-ng/patterndb.xml As it stands now I realize that the pattern does NOT match the test, but it should not segfault. A single space removed from the pattern makes the test succeed. The working patterndb is at http://web.uvic.ca/~erempel/syslog-ng/patterndb2.xml Any help would be appreciated.

1 0

Multiple dates
by Sandbox 09 Jun '17

09 Jun '17

Hi, I just started to test, learn etc syslog-ng, my server configuration is really basic: Q: Can i filter (and mark them on client) the incoming logs, so i dont have to open multiple ports for different logs? #-----------------------------------Source--------------------------------- source s_network_testweb01_access { tcp(ip(192.168.56.48) port("16601")); }; source s_network_testweb01_error { tcp(ip(192.168.56.48) port("16602")); }; source s_network_testweb01_referer { tcp(ip(192.168.56.48) port("16603")); }; #------------------------------------Destination-------------------------- destination d_apache_access_testweb01 { file("/var/log/syslog-ng/apache2/$FULLHOST.access" owner("root") group("adm") perm(0640) create_dirs(yes) dir_owner("root") dir_group("adm") persist-name("testweb_access")); .... file("/var/log/archive/$R_YEAR/apache/$R_MONTH/$FULLHOST.access.$R_DAY" owner("root") group("adm") perm(0640) create_dirs(yes) dir_owner("root") dir_group("adm") persist-name("testweb_access_archive")); }; destination d_apache_error_testweb01 { file("/var/log/syslog-ng/apache2/$FULLHOST.error" owner("root") group("adm") perm(0640) create_dirs(yes) dir_owner("root") dir_group("adm") persist-name("testweb_error")); .... file("/var/log/archive/$R_YEAR/apache/$R_MONTH/$FULLHOST.error.$R_DAY" owner("root") group("adm") perm(0640) create_dirs(yes) dir_owner("root") dir_group("adm") persist-name("testweb_error_archive")); }; destination d_apache_referer_testweb01 { file("/var/log/syslog-ng/apache2/$FULLHOST.referer" owner("root") group("adm") perm(0640) create_dirs(yes) dir_owner("root") dir_group("adm") persist-name("testweb_referer")); .... file("/var/log/archive/$R_YEAR/apache/$R_MONTH/$FULLHOST.referer.$R_DAY" owner("root") group("adm") perm(0640) create_dirs(yes) dir_owner("root") dir_group("adm") persist-name("testweb_referer_archive")); }; #---------------------Connections-------------------------- log { source(s_network_testweb01_access); destination(d_apache_access_testweb01); }; log { source(s_network_testweb01_error); destination(d_apache_error_testweb01); }; log { source(s_network_testweb01_referer); destination(d_apache_referer_testweb01); }; The stored log: Q: Why does it store the date 3 times in every logs? Jun 8 08:20:11 192.168.7.30 133 <13>1 2017-06-08T08:20:11+02:00 testweb01 - - - [meta sequenceId="24"] :1 - - [08/Jun/2017:08:20:10 +0200] "GET / HTTP/1.1" 200 3004 Also my client configuration: #Log sources source s_apache_error { file("/var/log/apache2/error_testweb_log" follow-freq(1)); }; source s_apache_access { file("/var/log/apache2/access_testweb_log" follow-freq(1)); }; source s_apache_referer { file("/var/log/apache2/www_referer_log" follow-freq(1)); }; #LOG Destination destination d_network_access { syslog("192.168.56.48" transport("tcp") ip-protocol(4) port("16601") persist-name("testweb_access")); }; destination d_network_error { syslog("192.168.56.48" transport("tcp") ip-protocol(4) port("16602") persist-name("testweb_error")); }; destination d_network_referer { syslog("192.168.56.48" transport("tcp") ip-protocol(4) port("16603") persist-name("testweb_referer")); }; #LOG connections log { source(s_apache_access); destination(d_network_access); }; log { source(s_apache_error); destination(d_network_error); }; log { source(s_apache_referer); destination(d_network_referer); }; Thanks, Robert

2 4

very basic router
by Thufir Hawat 09 Jun '17

09 Jun '17

for this router: http://www.netgear.com/support/product/WNR1000v2 I don't see any option in the configuration for using syslog. Is there a work-around? The only way to grab logs is to either copy/paste or e-mail -- but the e-mail uses port 25, which is blocked. thanks, Thufir

2 4

admin guide in PDF
by Thufir Hawat 09 Jun '17

09 Jun '17

I can't seem to find the admin guide as a PDF. It's just html? thanks, Thufir

2 2