January 2014 - syslog-ng - lists.balabit.hu

Issue with pdbtool patterndb and percent symbols
by Mark Shetka 29 Jan '14

29 Jan '14

I am setting up some patterns to parse Cisco syslog messages. I noticed that pdbtool will not complete if I have a "%F" anywhere in the string. Example log message: %FWSM-1-109006: Authentication failed for user 'test' from 131.212.1.1/43250to 10.1.1.1/22 on interface management This does not complete: pdbtool match -p cisco.xml -M "%FWSM-1-109006: Authentication failed for user 'test' from 131.212.1.1/43250 to 10.1.1.1/22 on interface management" Nor does simply %F: pdbtool match -p cisco.xml -M "%F" It is fine without the %: pdbtool match -p cisco.xml -M "FWSM-1-109006: Authentication failed for user 'test' from 131.212.1.1/43250 to 10.1.1.1/22 on interface management" MESSAGE=FWSM-1-109006: Authentication failed for user 'test' from 131.212.1.1/43250 to 10.1.1.1/22 on interface management .classifier.class=login .classifier.rule_id=5cfbcb23-cfe4-4120-85c1-918df65c0edc usracct.username=test usracct.device=131.212.1.1 usracct.service=22 usracct.type=login usracct.sessionid= usracct.application= secevt.verdict=REJECT TAGS=.classifier.login,usracct,secevt It also seems to have issues with "%S", although not quite in the same way. Any ideas what could be causing this? Mark -- Mark Shetka Information Technology Systems & Services University of Minnesota - Duluth (218) 726-7682

1 0

[Bug 268] New: filter on tag does partial match
by bugzilla＠bugzilla.balabit.com 27 Jan '14

27 Jan '14

https://bugzilla.balabit.com/show_bug.cgi?id=268 Summary: filter on tag does partial match Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 According to the documentation ------- tags() Synopsis: tag Description: Select messages labeled with the specified tag. Every message automatically has the tag of its source in .source.<id_of_the_source_statement> format. This option is available only in syslog-ng 3.1 and later. ------------ I interpret this to mean that an exact and full match of the tag is required. I have a tag on a message of "collection.required" and a filter filter f_collection { tags("collection"); }; this filter matches the messages with the tag "collection.required" which I think is a bug. I expect this filter on only match tags that exactly match "collection". -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 4

syslog-ng and OS library path?
by Matt Zagrabelny 25 Jan '14

25 Jan '14

Hello syslog-ng users! I might be getting a little out of the scope of syslog-ng usage here, but I thought I'd start here and work my way elsewhere. I've got a Debian unstable workstation that I do some syslog-ng testing on. I recently started using the sql destination driver and am hitting an issue on my unstable workstation, but not on a Debian "old-stable" squeeze system. # syslog-ng -F -e Log pattern database reloaded; file='/var/lib/syslog-ng/patterndb.xml', version='3', pub_date='2013-06-27' Unable to initialize database access (DBI); rc='-1', error='No such file or directory (2)' Error initializing message pipeline; Which I run under strace and get: openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/dbd", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory) The location looks okay for a Debian multiarch system, but according to dpkg the library isn't under the multiarch library path: $ dpkg -L libdbd-pgsql | grep x86_64 $ dpkg -L libdbd-pgsql | grep '^/usr/lib' /usr/lib /usr/lib/dbd /usr/lib/dbd/libdbdpgsql.la /usr/lib/dbd/libdbdpgsql.so I am under the impression that /usr/lib is still searched by the system loader. Why is syslog-ng not able to find the library file. What am I missing? Thanks for any suggestions or hints. -mz

3 3

[ANN]: Incubator news, Debian/Ubuntu packages
by Gergely Nagy 24 Jan '14

24 Jan '14

Hi! I have recently released version 0.1.1 of the syslog-ng Incubator project[1], which includes a brand new destination: Lua. With this, you can write destination plugins without having to touch C, or compile anything. It does have its limitations and issues still, but it is usable to the point that an example ElasticSearch plugin is included with the sources. I also made Debian and Ubuntu packages[2] out of the new release, which includes all modules (the new Lua one, and the ones I described on my blog[3] some time ago). However, unlike the syslog-ng packages, these are only available for Ubuntu Precise and newer, or Debian Wheezy and newer. Ubuntu Lucid and Debian Squeeze are not supported, and will not be. If you're using syslog-ng 3.5 packages, the new modules will automatically appear, as they are put into the same repository (they will not be automatically installed, however, not even when using the syslog-ng meta-package). Also, while I was there, I disabled Ubuntu Raring in the repository: it will be reaching its end of life this month, so no new updates will be made (and the Incubator is not available for it, either). It will stay in the repository for a while, though. [1]: https://github.com/balabit/syslog-ng-incubator [2]: http://asylum.madhouse-project.org/projects/debian/ [3]: http://asylum.madhouse-project.org/blog/2013/12/29/the-incubator/ -- |8]

1 0

Re: [syslog-ng] how to use macros from patterndb in syslog-ng.conf
by Jim Hendrick 10 Jan '14

10 Jan '14

Thanks. Seems obvious now. The pdbtool test *was* parsing but syslog-ng was not... Again *doh* :-) Sent from my Verizon Wireless 4G LTE Smartphone -------- Original message -------- From: Evan Rempel <erempel(a)uvic.ca> Date: 01/10/2014 2:03 PM (GMT-05:00) To: syslog-ng(a)lists.balabit.hu Subject: Re: [syslog-ng] how to use macros from patterndb in syslog-ng.conf With no-parse you will not get a program name, which will prevent the pattern from matching in patterndb Evan. On 01/10/2014 10:59 AM, jrhendri(a)roadrunner.com wrote: > DOH!!! > quoth the bard Homer Simpson... > > in my testing, I was appending syslog messages to "testfile" like this: > head -10 user.info.2014.01.08 > testfile > > and using "flags(no-parse)" on that file source which looks like this: > > [n0142566@VDDP13E-0D6B677 ~]$ cat testfile > Jan 8 20:00:00 10.192.225.12 RT_FLOW: RT_FLOW_SESSION_CREATE: session created 146.150.19.13/46188->10.178.128.38/11000 None 146.150.19.13/46188->10.178.128.38/11000 None None 6 3798 Int_NonProd_CORE_Liberty Int_NonProd_CORE_Secure 180869281 N/A(N/A) reth0.3931 UNKNOWN UNKNOWN UNKNOWN > Jan 8 20:00:00 10.192.225.12 RT_FLOW: RT_FLOW_SESSION_CLOSE: session closed TCP RST: 10.178.200.127/43833->10.181.72.46/2059 junos-tcp-any 10.178.200.127/43833->10.181.72.46/2059 None None 6 3384 Int_NonProd_CORE_Secure Int_NonProd_CORE_Liberty 180781521 2(100) 1(60) 3 UNKNOWN UNKNOWN N/A(N/A) reth0.3930 UNKNOWN > [n0142566@VDDP13E-0D6B677 ~]$ > > > > not quite sure how / why - but that was breaking this even though this worked: > > [n0142566@VDDP13E-0D6B677 ~]$ pdbtool match -f testfile -p ./log/juniper_db.xml > HOST=10.192.225.12 > MESSAGE=RT_FLOW_SESSION_CREATE: session created 146.150.19.13/46188->10.178.128.38/11000 None 146.150.19.13/46188->10.178.128.38/11000 None None 6 3798 Int_NonProd_CORE_Liberty Int_NonProd_CORE_Secure 180869281 N/A(N/A) reth0.3931 UNKNOWN UNKNOWN UNKNOWN > PROGRAM=RT_FLOW > LEGACY_MSGHDR=RT_FLOW: > .classifier.class=system > .classifier.rule_id=2a270520-2ff7-7048-a088-b03d5b3b5f7d > J.MSG.SRC=146.150.19.13 > J.MSG.SPORT=46188 > J.MSG.DST=10.178.128.38 > J.MSG.DPORT=11000 > TAGS=.classifier.system > > HOST=10.192.225.12 > MESSAGE=RT_FLOW_SESSION_CLOSE: session closed TCP RST: 10.178.200.127/43833->10.181.72.46/2059 junos-tcp-any 10.178.200.127/43833->10.181.72.46/2059 None None 6 3384 Int_NonProd_CORE_Secure Int_NonProd_CORE_Liberty 180781521 2(100) 1(60) 3 UNKNOWN UNKNOWN N/A(N/A) reth0.3930 UNKNOWN > PROGRAM=RT_FLOW > LEGACY_MSGHDR=RT_FLOW: > .classifier.class=system > .classifier.rule_id=2a270520-2ff7-7048-a088-b03d5b3b5f7d > J.MSG.CLOSE.REASON=TCP RST > J.MSG.SRC=10.178.200.127 > J.MSG.SPORT=43833 > J.MSG.DST=10.181.72.46 > J.MSG.DPORT=2059 > TAGS=.classifier.system > > > After I took out flags(no-parse) on the file source, it works as (I) expected ... > > I am still curious what I overlooked - misunderstood. But it appears to be related to how the messages were parsed on the source end. > > Thanks again! > Jim > > > > > ---- jrhendri(a)roadrunner.com wrote: >> this seems like a simple thing to do, but a few days of searching and some hours of testing has not shown me the answer. >> >> Essentially I want to parse log events and output select fields, so I have created a basic patterndb xml file. >> >> <patterndb version='3' pub_date='2014-01-09'> >> <ruleset name='patternize' id='6cb77f11-6c9b-ee4e-9f62-b97224d4384c'> >> <rules> >> <rule id='2a270520-2ff7-7048-a088-b03d5b3b5f7d' class='system' provider='patternize'> >>  >> <patterns> >> <pattern>RT_FLOW_SESSION_CLOSE: session closed @ESTRING:J.MSG.CLOSE.REASON::@ @IPv4:J.MSG.SRC@/@NUMBER:J.MSG.SPORT@->@IPv4:J.MSG.DST@/@NUMBER:J.MSG.DPORT@ </pattern> >> <pattern>RT_FLOW_SESSION_CREATE: session created @IPv4:J.MSG.SRC@/@NUMBER:J.MSG.SPORT@->@IPv4:J.MSG.DST@/@NUMBER:J.MSG.DPORT@ </pattern> >> </patterns> >> </rule> >> </rules> >> </ruleset> >> </patterndb> >> >> This tests fine using pdbtool on sample data in a file. (cut from a real syslog file of logs) >> Inside syslog-ng.conf I want to use the parsed values as a template: >> source s_testfile { >> file("/home/n0142566/testfile" >> flags(no-parse) ); >> }; >> >> filter f_juniper_session_create { >> match("RT_FLOW_SESSION_CREATE" value( "MESSAGE" ) ); >> }; >> filter f_juniper_session_close { >> match("RT_FLOW_SESSION_CLOSE" value( "MESSAGE" ) ); >> }; >> >> parser juniper_db { >> db-parser ( >> file("/home/n0142566/log/juniper_db.xml") >> ); >> }; >> >> destination d_local_create { >> file("/home/n0142566/log/messages-create-$HOST" >> template("${J.MSG.SRC}, ${J.MSG.SPORT}, ${J.MSG.DST}, ${J.MSG.DPORT}\n") ); >> }; >> >> destination d_local_close { >> file("/home/n0142566/log/messages-close-$HOST" >> #!# template("${J.MSG.SRC}, ${J.MSG.SPORT}, ${J.MSG.DST}, ${J.MSG.DPORT}, ${J.MSG.CLOSE.REASON}\n") ); >> template("${J.MSG}\n") ); >> }; >> >> >> log { >> source(s_testfile); >> filter(f_juniper_session_create); >> parser(juniper_db); >> destination(d_local_create); >> }; >> log { >> source(s_testfile); >> filter(f_juniper_session_close); >> parser(juniper_db); >> destination(d_local_close); >> }; >> >> >> But when I run syslog-ng and append events to "testfile" the output is simply comma separated blanks :-( >> >> I am quite sure I am missing something *extremely* basic - but at this point I thought I would ask clearer minds for help!! >> >> Thanks, ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

how to use macros from patterndb in syslog-ng.conf
by jrhendri＠roadrunner.com 10 Jan '14

10 Jan '14

this seems like a simple thing to do, but a few days of searching and some hours of testing has not shown me the answer. Essentially I want to parse log events and output select fields, so I have created a basic patterndb xml file. <patterndb version='3' pub_date='2014-01-09'> <ruleset name='patternize' id='6cb77f11-6c9b-ee4e-9f62-b97224d4384c'> <rules> <rule id='2a270520-2ff7-7048-a088-b03d5b3b5f7d' class='system' provider='patternize'>  <patterns> <pattern>RT_FLOW_SESSION_CLOSE: session closed @ESTRING:J.MSG.CLOSE.REASON::@ @IPv4:J.MSG.SRC@/@NUMBER:J.MSG.SPORT@->@IPv4:J.MSG.DST@/@NUMBER:J.MSG.DPORT@ </pattern> <pattern>RT_FLOW_SESSION_CREATE: session created @IPv4:J.MSG.SRC@/@NUMBER:J.MSG.SPORT@->@IPv4:J.MSG.DST@/@NUMBER:J.MSG.DPORT@ </pattern> </patterns> </rule> </rules> </ruleset> </patterndb> This tests fine using pdbtool on sample data in a file. (cut from a real syslog file of logs) Inside syslog-ng.conf I want to use the parsed values as a template: source s_testfile { file("/home/n0142566/testfile" flags(no-parse) ); }; filter f_juniper_session_create { match("RT_FLOW_SESSION_CREATE" value( "MESSAGE" ) ); }; filter f_juniper_session_close { match("RT_FLOW_SESSION_CLOSE" value( "MESSAGE" ) ); }; parser juniper_db { db-parser ( file("/home/n0142566/log/juniper_db.xml") ); }; destination d_local_create { file("/home/n0142566/log/messages-create-$HOST" template("${J.MSG.SRC}, ${J.MSG.SPORT}, ${J.MSG.DST}, ${J.MSG.DPORT}\n") ); }; destination d_local_close { file("/home/n0142566/log/messages-close-$HOST" #!# template("${J.MSG.SRC}, ${J.MSG.SPORT}, ${J.MSG.DST}, ${J.MSG.DPORT}, ${J.MSG.CLOSE.REASON}\n") ); template("${J.MSG}\n") ); }; log { source(s_testfile); filter(f_juniper_session_create); parser(juniper_db); destination(d_local_create); }; log { source(s_testfile); filter(f_juniper_session_close); parser(juniper_db); destination(d_local_close); }; But when I run syslog-ng and append events to "testfile" the output is simply comma separated blanks :-( I am quite sure I am missing something *extremely* basic - but at this point I thought I would ask clearer minds for help!! Thanks, Jim

2 2

insider 2014-01: Conferences; 3.6 pre-alpha ; incubator ; PCI-DSS
by Peter Czanik 09 Jan '14

09 Jan '14

Dear syslog-ng users, This is the 30th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. FEATURED NEWS syslog-ng at conferences ------------------------ This year, we plan to present syslog-ng in many conferences, or just participate conferences where many syslog-ng users are expected to show up. If you want to meet someone from the syslog-ng team, come to one of our presentations or drop an e-mail so we can find each other. We are also very interested to hear suggestions, where syslog-ng should be presented! Here is a list of confirmed events, which will be updated regularly in later newsletters: FOSDEM, 1-2 February, Brussels, Belgium: https://fosdem.org/2014/, giving a talk titled: “Babelfish for DevOps: syslog-ng” Fedora, JBoss and RedHat developers conference, 7-9 February, Brno, Czech Republic: http://devconf.cz/, Participating only Open Source Data Center Conference, 8-10 April, Berlin, Germany: http://www.netways.de/osdc/, Giving a talk titled "Monitoring with syslog-ng, Riemann and Kibana" Check out syslog-ng 3.6 pre-alpha! ---------------------------------- While syslog-ng 3.6 still did not even reach an alpha release, it received already a lot of development. So, while it might still eat your logs for lunch, those who are interested in where syslog-ng is heading should check out syslog-ng 3.6 from git. Unlike previous syslog-ng versions, it is developed in a unified git repository without a version string attached: https://github.com/balabit/syslog-ng While most changes are under the hood, there are also some user visible features like pseudofile destination: https://github.com/balabit/syslog-ng/commit/be381a4e3f3eca0695f5976678ea26c… Nodejs support was also added tosyslog-ng: use the widespread winston logging API, and syslog-ng will process its JSON formatted messages: https://bazsi.blogs.balabit.com/2014/01/nodejs-support-in-syslog-ng/ And if you look at the stats at http://www.ohloh.net/p/syslog-ng you can see a healthy growth of the code base and in the number of contributors. Thank you for your support! syslog-ng incubator ------------------- The syslog-ng incubator is a collection of tools and modules which are not (yet) part of the official repository. It has some very interesting code in it, like a riemann or an RSS destination, but until now it was completely undocumented. Not any more: http://asylum.madhouse-project.org/blog/2013/12/29/the-incubator/ The Incubator also includes a Lua destination, which makes it possible to write simple destination drivers without a line of C. It is still a work in progress, but is an important step into writing modules in other languages. And to make your life easier packages are available in Debian testing, Ubuntu Trustyand for openSUSE in the 3^rd party repositories ( http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… ). PCI DSS 3.0 Continues to Emphasize the Importance of Log Management ------------------------------------------------------------------- The Payment Card Industry Security Standards Council recently released the Data Security Standard 3.0, three years after the prior version. As one of the most important international data security standards, the latest release was eagerly awaited by IT security practitioners. Clarifications make up the bulk of the changes but the standards council changed most of the 12 major requirements to include modified or additional sub-requirements. With PCI DSS 3.0 the standards council has reiterated that log management is a critical part of security best practices. You can read more about it at https://jluby.blogs.balabit.com/2013/12/09/pci-dss-3-0-continues-to-emphasi… SHORT NEWS: * Testing syslog-ng PE on Windows was never so easy: Windows Quick Start Guide for syslog-ng Premium Edition: http://czanik.blogs.balabit.com/2013/12/syslog-ng-on-windows-quick-start-gu… * TLS vulnerabilities in log traffic: syslog-ng vs. BEAST, CRIME and BREACH:http://gyp.blogs.balabit.com/2013/12/syslog-ng-vs-beast-crime-and-br… * Top 6 SANS Essential Categories of Log Reports 2013: http://czanik.blogs.balabit.com/2013/12/top-6-sans-essential-categories-of-… * Shortening Breach Resolution Times with Distributed Log Management: http://jluby.blogs.balabit.com/2013/12/30/shortening-breach-resolution-time… NEW RELEASES * syslog-ng 3.5.3: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-December/000177.… * syslog-ng 3.4.7: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-December/000178.… Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0