- syslog-ng - lists.balabit.hu

patterndb: collect login/logout samples
by Balazs Scheidler 28 Jul '10

28 Jul '10

Hi, After getting the generic patterndb policy into shape, I'd like to start collecting log samples, preferably in a domain that is useful for everyone. My target is at first is login/logout/login failure events. I'd start with a generic Linux installation and try to cover all applications that perform authentication. As a starter, I've commited access/sshd.pdb, containing three rules for OpenSSH login/logout/login failure events. I'd head towards standard services, ftp, pop3 and imap authentication, using their "default" implementation in Ubuntu/Debian. (if there's no default, I'll just pick one at random). If any of you can collect these 3 samples of any of the applications that you run daily on your system and submit them here, it'd be tremendous use and would be appreciated. The format of the submission would be preferred in patterndb format (see the ssh sample I've just pushed), but if you are afraid of that, even simple samples would be useful, I'll do the markup myself. -- Bazsi

9 27

Small problem with pdbtool and the parser ?
by Ilas, Yann 28 Jul '10

28 Jul '10

Hello, I have a small problem with pdbtool. When I try to match this message "> Message message n°2 message 255", I obtain this error : "Error parsing command line arguments: Invalid byte sequence in conversion input". The problem seems to be the "°" which is bad interpreted by pdbtool. But the syslog-ng application has not that problem. Here is the xml file I used : <?xml version='1.0' encoding='UTF-8'?> <patterndb version='3' pub_date='2010-07-28'> <ruleset name='test' id='ab-cd-123456789'> <pattern>test</pattern> <rules> <rule provider='yann' id='xxxxx:syslog:abcdef:1234567890:id001' class='system'> <patterns> <pattern>> Message message n°@NUMBER@ message @NUMBER@</pattern> <values> <value name=".classifier.facility">local5</value> <value name=".classifier.severity">notice</value> <value name=".classifier.priority">173</value> </values> </patterns> </rule> </rules> </ruleset> </patterndb> If I use the pattern "<pattern>> Message message n°@NUMBER@ message @NUMBER@</pattern>", I obtain : $ /opt/syslog-ng/bin/pdbtool match -p patterndb.xml -P test -M '> Message message n°2 message 255' Error parsing command line arguments: Invalid byte sequence in conversion input => It's not OK If I use the pattern "<pattern>> Message message nx@NUMBER@ message @NUMBER@</pattern>", I obtain : (the diffence is the charactere "x" after the " n") $ /opt/syslog-ng/bin/pdbtool match -p patterndb.xml -P test -M '> Message message nx2 message 255' MESSAGE=> Message message nx2 message 255 PROGRAM=test .classifier.class=system .classifier.rule_id=xxxxx:syslog:abcdef:1234567890:id001 .classifier.facility=local5 .classifier.severity=notice .classifier.priority=173 => It's OK With syslog-ng and the pattern "<pattern>> Message message n°@NUMBER@ message @NUMBER@</pattern>", I tried to send the same message (with the "°") by using "nc" and it works, ie the message is well interpreted by the application : $ LANG=C date +"<70>%b %d %R:%S server.domain test[1234]: > Message message n°123 message 456" | nc -4 127.0.0.1 514 Why pdbtool can't parse that string ? Regards, Yann I.

2 1

sql() and pgsql
by John Kristoff 28 Jul '10

28 Jul '10

Hello, First thanks to Bazsi and co. for years of great work on syslog-ng. I've used it off and on for some time and have always found it to be well implemented and supported. I've experimented a bit with rsyslog since some distros have been making it their new default and found syslog-ng to be easier to work with and more reliable for my tasks. I'm been playing around with v3.0.8 and the sql() / pgsql output driver to insert log data into a specific table on a specific schema. I couldn't figure out how to specify the schema and maybe its just not supoprted. For instance, if the following is defined: database: foo schema: bar table: baz but the PostgreSQL search path for the user doesn't include bar, then syslog-ng will attempt to create a table in the default schema (usually 'public'). I tried setting the table name to "bar.baz", but that just created a table named bar_baz in the default schema. I did a quick check in the source code and didn't see where this was occurring, maybe its a function of a third party library? If schema support was available that would be nice. For posterity sake and maybe a FAQ, the work around is to alter the search_path for the user (e.g. ALTER ROLE user SET search_path = bar,public;). On a related note, it might be nice to have an option to avoid creating a table if it does not already exist. If nothing else, this helps limit the permissions granted to the syslog-ng db user, which would be appreciated. Likewise, it might also be nice to have an option that can disable checking if the table exists (the "SELECT * FROM %s WHERE 0=1" query) to further lock down user permissions. John

3 3

config file parsing bugs in syslog-ng 3.0.7
by Matthew Hall 28 Jul '10

28 Jul '10

Hello list, In the course of my work I discovered two small defects in syslog-ng syslog-ng 3.0.7. Since I am not familiar enough with the code to patch these I am trying to do the best I can by reporting them to the experts. There are workarounds for these. I am in the process of doing some other work on the product which I would like to contribute back in the near future after I figure out how the open-source contribution process works at this new company I joined. Thank you for providing this product. I am impressed with the documentation and all of the capabilities and I look forward to becoming more expert with it over the coming few months. Best Regards, Matthew Hall. 1) specifying the flags directive in a log stanza other than in the last position causes a syntax error This specific format works: log { filter(filter_local1); destination(dst_local1); parser(parse_switch); flags(final); }; Any format like this raises an error: log { filter(filter_local1); destination(dst_local1); flags(final); parser(parse_switch); }; Is there some reason why or just a parser glitch? 2) using a version directive newer than the daemon version has unexpected behavior If I feed 3.0 a file marked with @version: 3.1 tag, I get this error: Configuration file has no version number, assuming syslog-ng 2.1 format. Please add @version: maj.min to the beginning of the file; I think it does not make sense to fall back to the oldest format if the file has a newer version tag. Would it be better to try to process the file as the newest version with a warning, or print an error that the file is too new for 3.0 to read?

2 1

Pipe() in syslog-ng ---- HELP
by Khaleelah Peerbocus 28 Jul '10

28 Jul '10

Hi How can I avoid line breaks in syslog-ng when it is configured using pipe destination d_mssql { pipe("/tmp/mssql.syslog-ng.pipe" flags(no-multi-line) template("INSERT INTO logs(datetime,host,program,pid,message)VALUES('$R_DATE','$HOST','$PROGRAM',' $PID','$MSGONLY');") template-escape(yes)); }; log { source(src); filter(f_debug); destination(d_mssql); }; Then I have a script reading the pipe if [ -e /tmp/mssql.syslog-ng.pipe ]; then while [ -e /tmp/mssql.syslog-ng.pipe ] do isql -v MSTEST "sa" "password" < /tmp/mssql.syslog-ng.pipe done else mkfifo /tmp/mssql.syslog-ng.pipe fi Problem is the sql query is getting broken down in 3 lines KP

2 12

prevent syslog-ng to collect syslogd messages
by Hoenig, Stefan, VF-Group 27 Jul '10

27 Jul '10

Hi Guys, we use syslog-ng to collect application logs and still use the standard syslogd for system logging. The application logs were defined in an include config file so the application teams are able to configure them as they need it. The problem is that we want to prevent that they can also configure systemlogging sources in their config like "/proc/kmsg" or "dev/log" on Linux. Is there a possibility to overwrite such sources in the global config or better a global parameter that avoids such configurations? Thanks in advance.... With best regards / Mit freundlichen Grüßen Stefan Hoenig

2 1

Question about QSTRING and "<>" separators
by Ilas, Yann 26 Jul '10

26 Jul '10

Hello, I have another question (the last of the day ?) about db parser. In fact, my question is about the QSTRING. I'm not sur to understand how to use it. I follow the example given here (http://marci.blogs.balabit.com/2009/04/intorduction-to-parser-in-syslog -ng-db.html) but it doesn't seem to work. Here is my xml file : <?xml version='1.0' encoding='UTF-8'?> <patterndb version='3' pub_date='2010-07-23'> <ruleset name='test' id='yann-test'> <description>Programme : test</description> <pattern>test</pattern> <rules> <rule provider='yann' id='123456789' class='system'> <patterns> <pattern>@QSTRING:message:<>@ Message message message</pattern> </patterns> </rule> </rules> </ruleset> </patterndb> I try to test a message by using the pdbtool : $ /opt/syslog-ng/bin/pdbtool match -p /tmp/syslog-ng.xml -P test -M "<123> Message message message" Key contains '@' without escaping; key='@QSTRING:message:', value='123456789' Error parsing pattern database file; filename='/tmp/syslog-ng.xml', error='Error on line 9 char 38: \'>\' is not a valid character following a \'<\' character; it may not begin an element name' ...and I obtain this error. What's wrong with the message or the use of QSTRING ? I replaced '<>' by "'" (simple quote) and it works... Regards, Yann I.

2 1

Pattern database and '<'
by Ilas, Yann 26 Jul '10

26 Jul '10

Hello, Currently, I'm using syslog-ng version 3.1.1. I would like to parse that kind of message by using pattern database. Here is the message : "<<<<< Message message message". So I created that xml file : <?xml version='1.0' encoding='UTF-8'?> <patterndb version='3' pub_date='2010-07-22'> <ruleset name='test' id='yann-test-9999999999'> <description>Programme : test</description> <pattern>test</pattern> <rules> <rule provider='yann' id='ss-test:syslog:dest:123123123:id005' class='system'> <patterns> <pattern><<<<< Message message message</pattern> <values> <value name=".classifier.facility">local0</value> <value name=".classifier.severity">notice</value> <value name=".classifier.priority">133</value> </values> </patterns> </rule> </rules> </ruleset> </patterndb> I obtain that message when I try to parse with the pdbtool application : $ pdbtool match -p syslog-ng.xml -P test -M "<<<<< Message message message" Error parsing pattern database file; filename='/tmp/syslog-ng.xml', error='Error on line 9 char 22: \'<\' is not a valid character following a \'<\' character; it may not begin an element name' Into the xml file, I replaced '<<<<<' with @STRING::@ or @ESTRING...@ but it doesn't work. Is it possible to match that kind of message ? Regards, Yann

3 2

How to deal with duplicate log entries
by Chuck 26 Jul '10

26 Jul '10

I have the following log statgements.. (Im basically trying to mirror what solaris 10's default syslogl.conf does): log { source (s_streams); filter (f_emerg); destination (l_messages); }; log { source (s_streams); filter (f_err); destination (l_messages); }; log { source (s_streams); filter (f_kern); filter (f_debug); destination (l_messages); }; log { source (s_streams); filter (f_daemon); filter (f_notice); destination (l_messages); }; log { source (s_streams); filter (f_mail); filter (f_crit); destination (l_messages); }; log { source (s_streams); filter (f_mail); filter (f_debug); destination (l_syslog); }; log { source (s_streams); filter (f_auth); filter (f_info); destination (l_authlog); }; # sshd logging However when sending to daemon.err I get duplicate messages... Should I change line 2 to this: log { source (s_streams); filter (f_err); filter (not f_daemon) destination (l_messages); }; Or should I hard-code every facility like so: log { source (s_streams); filter (f_err); filter (f_daemon) destination (l_messages); }; log { source (s_streams); filter (f_err); filter (f_mail) destination (l_messages); }; log { source (s_streams); filter (f_err); filter (f_auth) destination (l_messages); }; ... and so on? Thx,.CC

2 1

Is logger an effective test tool on Solaris 10?
by Chuck 26 Jul '10

26 Jul '10

I am using the following script to test syslog-ng. (Running this from the same machine as the syslog-ng server): #!/bin/bash d=`date '+%y%m%d_%H%M%S'` logger -p kern.debug "__kern.debug__ $d" sleep 1; d=`date '+%y%m%d_%H%M%S'` logger -p kern.crit "__kern.crit__ $d" sleep 1; d=`date '+%y%m%d_%H%M%S'` logger -p kern.info "__kern.info__ $d" sleep 1; However, here is how the messages are being seen within syslog-ng: Jul 21 10:30:35 log01 cars: [ID 702911 user.debug] __kern.debug__ 100721_103035 Jul 21 10:30:36 log01 cars: [ID 702911 user.crit] __kern.crit__ 100721_103036 Jul 21 10:30:37 log01 cars: [ID 702911 user.info] __kern.info__ 100721_103037 The facility is being seen as user and not kern... Anyone have any ideas? Thx,

4 3