July 2010 - syslog-ng - lists.balabit.hu

Question about QSTRING and "<>" separators
by Ilas, Yann 26 Jul '10

26 Jul '10

Hello, I have another question (the last of the day ?) about db parser. In fact, my question is about the QSTRING. I'm not sur to understand how to use it. I follow the example given here (http://marci.blogs.balabit.com/2009/04/intorduction-to-parser-in-syslog -ng-db.html) but it doesn't seem to work. Here is my xml file : <?xml version='1.0' encoding='UTF-8'?> <patterndb version='3' pub_date='2010-07-23'> <ruleset name='test' id='yann-test'> <description>Programme : test</description> <pattern>test</pattern> <rules> <rule provider='yann' id='123456789' class='system'> <patterns> <pattern>@QSTRING:message:<>@ Message message message</pattern> </patterns> </rule> </rules> </ruleset> </patterndb> I try to test a message by using the pdbtool : $ /opt/syslog-ng/bin/pdbtool match -p /tmp/syslog-ng.xml -P test -M "<123> Message message message" Key contains '@' without escaping; key='@QSTRING:message:', value='123456789' Error parsing pattern database file; filename='/tmp/syslog-ng.xml', error='Error on line 9 char 38: \'>\' is not a valid character following a \'<\' character; it may not begin an element name' ...and I obtain this error. What's wrong with the message or the use of QSTRING ? I replaced '<>' by "'" (simple quote) and it works... Regards, Yann I.

2 1

Pattern database and '<'
by Ilas, Yann 26 Jul '10

26 Jul '10

Hello, Currently, I'm using syslog-ng version 3.1.1. I would like to parse that kind of message by using pattern database. Here is the message : "<<<<< Message message message". So I created that xml file : <?xml version='1.0' encoding='UTF-8'?> <patterndb version='3' pub_date='2010-07-22'> <ruleset name='test' id='yann-test-9999999999'> <description>Programme : test</description> <pattern>test</pattern> <rules> <rule provider='yann' id='ss-test:syslog:dest:123123123:id005' class='system'> <patterns> <pattern><<<<< Message message message</pattern> <values> <value name=".classifier.facility">local0</value> <value name=".classifier.severity">notice</value> <value name=".classifier.priority">133</value> </values> </patterns> </rule> </rules> </ruleset> </patterndb> I obtain that message when I try to parse with the pdbtool application : $ pdbtool match -p syslog-ng.xml -P test -M "<<<<< Message message message" Error parsing pattern database file; filename='/tmp/syslog-ng.xml', error='Error on line 9 char 22: \'<\' is not a valid character following a \'<\' character; it may not begin an element name' Into the xml file, I replaced '<<<<<' with @STRING::@ or @ESTRING...@ but it doesn't work. Is it possible to match that kind of message ? Regards, Yann

3 2

How to deal with duplicate log entries
by Chuck 26 Jul '10

26 Jul '10

I have the following log statgements.. (Im basically trying to mirror what solaris 10's default syslogl.conf does): log { source (s_streams); filter (f_emerg); destination (l_messages); }; log { source (s_streams); filter (f_err); destination (l_messages); }; log { source (s_streams); filter (f_kern); filter (f_debug); destination (l_messages); }; log { source (s_streams); filter (f_daemon); filter (f_notice); destination (l_messages); }; log { source (s_streams); filter (f_mail); filter (f_crit); destination (l_messages); }; log { source (s_streams); filter (f_mail); filter (f_debug); destination (l_syslog); }; log { source (s_streams); filter (f_auth); filter (f_info); destination (l_authlog); }; # sshd logging However when sending to daemon.err I get duplicate messages... Should I change line 2 to this: log { source (s_streams); filter (f_err); filter (not f_daemon) destination (l_messages); }; Or should I hard-code every facility like so: log { source (s_streams); filter (f_err); filter (f_daemon) destination (l_messages); }; log { source (s_streams); filter (f_err); filter (f_mail) destination (l_messages); }; log { source (s_streams); filter (f_err); filter (f_auth) destination (l_messages); }; ... and so on? Thx,.CC

2 1

Is logger an effective test tool on Solaris 10?
by Chuck 26 Jul '10

26 Jul '10

I am using the following script to test syslog-ng. (Running this from the same machine as the syslog-ng server): #!/bin/bash d=`date '+%y%m%d_%H%M%S'` logger -p kern.debug "__kern.debug__ $d" sleep 1; d=`date '+%y%m%d_%H%M%S'` logger -p kern.crit "__kern.crit__ $d" sleep 1; d=`date '+%y%m%d_%H%M%S'` logger -p kern.info "__kern.info__ $d" sleep 1; However, here is how the messages are being seen within syslog-ng: Jul 21 10:30:35 log01 cars: [ID 702911 user.debug] __kern.debug__ 100721_103035 Jul 21 10:30:36 log01 cars: [ID 702911 user.crit] __kern.crit__ 100721_103036 Jul 21 10:30:37 log01 cars: [ID 702911 user.info] __kern.info__ 100721_103037 The facility is being seen as user and not kern... Anyone have any ideas? Thx,

4 3

Messages Not Getting Logged
by Chuck 26 Jul '10

26 Jul '10

Hello, I am using the sunfreeware build of syslog-ng 3.04 on a Solaris 10 Update 8 system: syslog-ng 3.0.4 Revision: ssh+git://bazsi@git.balabit //var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.0#master#1b5d618e301ad94aa20e692ffba16469dece8d10 Compile-Date: Sep 2 2009 05:14:23 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: on Enable-Sun-Door: on Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: on My internal() source is working but the sun-streams source is not. Here are my options: options { use_fqdn(no); flush_lines(0); dir_perm(0755); dir_group(sysadmin); dir_owner(root); perm(0644); stats_freq(300); use_dns(no); create_dirs(yes); time_reopen(10); }; Here are my sources: source s_internal { internal(); }; source s_udp { udp(); }; source s_tcp { tcp(); }; source s_streams { sun-streams ("/dev/log" door("/etc/.syslog_door")); }; Here are my filters so far: # Level Filters filter f_emerg { level (emerg); }; filter f_alert { level (alert .. emerg); }; filter f_crit { level (crit .. emerg); }; filter f_err { level (err .. emerg); }; filter f_warning { level (warning .. emerg); }; filter f_notice { level (notice .. emerg); }; filter f_info { level (info .. emerg); }; filter f_debug { level (debug .. emerg); }; # Facility Filters filter f_kern { facility (kern); }; filter f_user { facility (user); }; filter f_mail { facility (mail); }; filter f_daemon { facility (daemon); }; filter f_auth { facility (auth); }; filter f_syslog { facility (syslog); }; filter f_lpr { facility (lpr); }; filter f_news { facility (news); }; filter f_uucp { facility (uucp); }; filter f_cron { facility (cron); }; filter f_local0 { facility (local0); }; filter f_local1 { facility (local1); }; filter f_local2 { facility (local2); }; filter f_local3 { facility (local3); }; filter f_local4 { facility (local4); }; filter f_local5 { facility (local5); }; filter f_local6 { facility (local6); }; filter f_local7 { facility (local7); }; Here are my destinations so far: # Destinations: local files, the console, and the client files destination l_internal { file ("/var/adm/syslog-ng"); }; destination l_authlog { file ("/var/log/authlog"); }; destination l_messages { file ("/var/log/messages"); }; destination l_maillog { file ("/var/log/maillog"); }; destination l_ipflog { file ("/var/log/ipflog"); }; destination l_imaplog { file ("/var/log/imaplog"); }; destination l_syslog { file ("/var/log/syslog"); }; destination l_console { file ("/dev/console"); }; Here are my log statements: log { source (s_internal); destination (l_internal); }; # <==== this one is working log { source (s_streams); filter (f_kern); filter (f_debug); destination (l_messages); }; #<====== this on is not working I have tried the following logger tests and am not getting anything logged: logger -p kern.debug "some message" logger -p kern.crit "some message" logger -p kern.info "some message" I have restarted syslog-ng and ensured that it is using my most recent syslog-ng.conf file. Thanks for any help, CC

4 7

Quotes and parser with pattern db in syslog-ng 3.1.1
by Ilas, Yann 26 Jul '10

26 Jul '10

I have a question about the parser using the db_parser and xml file. I'm currently using the version 3.1.1 of syslog-ng. Here is the xml file named "syslog-ng.xml" : <?xml version='1.0' encoding='UTF-8'?> <patterndb version='3' pub_date='2010-07-21'> <ruleset name='syslog-ng' id='syslog-ng-01'> <pattern>syslog-ng</pattern> <rules> <rule provider='yann' id='syslog:server:0123456789:id001' class='system'> <patterns> <pattern>syslog-ng shutting down; version='@STRING::@'</pattern> <examples> <example> <test_message>syslog-ng shutting down; version='3.1.1'</test_message> </example> </examples> <values> <value name=".classifier.facility">syslog</value> <value name=".classifier.severity">notice</value> <value name=".classifier.priority">45</value> </values> </patterns> </rule> </rules> </ruleset> </patterndb> I would like to test that configuration with the following message "syslog-ng shutting down; version='3.1.1'" by using pdbtool. # /opt/syslog-ng/bin/pdbtool match -p /tmp/syslog-ng.xml -P syslog-ng -M "syslog-ng shutting down; version='3.1.1'" MESSAGE=syslog-ng shutting down; version='3.1.1' PROGRAM=syslog-ng .classifier.class=unknown The result is odd because my message seems to be right... I suspect the quotes so I remove the quotes in the xml file. Let's try again with the same message without quotes : (...) <pattern>syslog-ng shutting down; version=@STRING::@</pattern> (...) Test: # /opt/syslog-ng/bin/pdbtool match -p /tmp/syslog-ng.xml -P syslog-ng -M "syslog-ng shutting down; version=3.1.1" MESSAGE=syslog-ng shutting down; version=3.1.1 PROGRAM=syslog-ng .classifier.class=system .classifier.rule_id=syslog:server:0123456789:id001 .classifier.facility=syslog .classifier.severity=notice .classifier.priority=45 It works. What's wrong with my message ? I tried to remove the second quote in the xml file like this : (...) <pattern>syslog-ng shutting down; version='@STRING::@</pattern> (...) Test : # /opt/syslog-ng/bin/pdbtool match -p /tmp/syslog-ng.xml -P syslog-ng -M "syslog-ng shutting down; version='3.1.1" MESSAGE=syslog-ng shutting down; version='3.1.1 PROGRAM=syslog-ng .classifier.class=system .classifier.rule_id=syslog:server:0123456789:id001 .classifier.facility=syslog .classifier.severity=notice .classifier.priority=45 I have the same result if I changed my xml file like this : (...) <pattern>syslog-ng shutting down; version=''@STRING::@</pattern> (...) Test : # /opt/syslog-ng/bin/pdbtool match -p /tmp/syslog-ng.xml -P syslog-ng -M "syslog-ng shutting down; version=''3.1.1" MESSAGE=syslog-ng shutting down; version=''3.1.1 PROGRAM=syslog-ng .classifier.class=system .classifier.rule_id=syslog:server:0123456789:id001 .classifier.facility=syslog .classifier.severity=notice .classifier.priority=45 Can't I have a message like this one 'texttexttext' ? Or did I miss something ? Last test : I change the xml file like this : (...) <pattern>syslog-ng shutting down; version='3.1.1'</pattern> (...) => I remove the @STRING::@ and the pdbtool returned the .classifier.facility, .classifier.severity, etc. May I use @STRING::@ between two simple quotes ? Regards, Yann I.

2 1

Rewrite Hostname Field of Syslog Header
by lecalcot 26 Jul '10

26 Jul '10

Hi, I¹m wondering if syslog-ng is capable of rewriting the hostname field in the header of syslogs as they are forwarded to a remote loghost. Is this possible? Thanks, Lee

5 4

Re: [syslog-ng] Pattern database and '<'
by Ilas, Yann 23 Jul '10

23 Jul '10

>> On 2010-07-23, Ilas, Yann wrote: >> >> Currently, I'm using syslog-ng version 3.1.1. >> I would like to parse that kind of message by using pattern database. >> Here is the message : "<<<<< Message message message". >> So I created that xml file : >>[...] >> <pattern><<<<< Message message message</pattern> >[...] >> I obtain that message when I try to parse with the pdbtool application : >> $ pdbtool match -p syslog-ng.xml -P test -M "<<<<< Message message >> message" >> Error parsing pattern database file; filename='/tmp/syslog-ng.xml', >> error='Error on line 9 char 22: \'<\' is not a valid character following >> a \'<\' character; it may not begin an element name' >> >> Into the xml file, I replaced '<<<<<' with @STRING::@ or @ESTRING...@ >> but it doesn't work. Is it possible to match that kind of message ? > > Characters < and > are special to XML notation. Try using < and > > instead. > > > HTH > > -- > Jakub Jankowski|shasta@toxcorp.com|http://toxcorp.com/ > GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D Thanks ! It works. Regards, Yann I.

1 0

Syslog-ng Server Setup
by Sam Oehlert 22 Jul '10

22 Jul '10

I am attempting to set up a test syslog-ng server in a VM farm. I run CentOS 5.5 and Syslog-ng OSE 2.1.4. My VMs can ping each other and ssh to and from each other. The firewalls are off. Yet I still can only locally log for both server and client, the server doesn't see my clients logs at all. Attached are my server and client configuration files. Thanks for the help, Sam

2 1

More Solaris 10 Woes - tcp/udp issues
by Chuck 21 Jul '10

21 Jul '10

I can't get syslog-ng to listen on port 514 using the tcp or udp sources. First, I have defined entries in /etc/services as follows: syslog-ng-udp 514/udp syslog syslog-ng-tcp 514/tcp syslog I have of course disabled the default system-log service. The following two I can't get to work/listen: (Verifying this with netstat -a) source s_udp { udp(port(514)); }; source s_tcp { tcp(port(514)); }; I can get it to listen with the following: source s_syslog { syslog( transport("udp") port(514) ); }; OR source s_syslog { syslog( transport("tcp") port(514) ); }; # netstat -a UDP: IPv4 Local Address Remote Address State -------------------- -------------------- ---------- *.syslog-ng-udp Idle However, syslog-ng doesn't seem to be doing anything with this. I can see the syslog message when snooping the interface on my syslog-ng server: root@log01:~# snoop -d nge0 udp port 514 Using device nge0 (promiscuous mode) ds01 -> log01 SYSLOG C port=32947 daemon.crit: <26>Jul 21 13:22:08 ds01 -> log01 SYSLOG C port=32947 daemon.crit: <26>Jul 21 13:23:11 ds01 -> log01 SYSLOG C port=32947 daemon.crit: <26>Jul 21 13:23:17 Here is the catch all log statement I am using: destination r_messages { file ("/var/adm/messages_test"); }; log { source (s_syslog); destination (r_messages); }; Anyone have any ideas? Thx, CC

2 1