- syslog-ng - lists.balabit.hu

Messages Not Getting Logged
by Chuck 26 Jul '10

26 Jul '10

Hello, I am using the sunfreeware build of syslog-ng 3.04 on a Solaris 10 Update 8 system: syslog-ng 3.0.4 Revision: ssh+git://bazsi@git.balabit //var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.0#master#1b5d618e301ad94aa20e692ffba16469dece8d10 Compile-Date: Sep 2 2009 05:14:23 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: on Enable-Sun-Door: on Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: on My internal() source is working but the sun-streams source is not. Here are my options: options { use_fqdn(no); flush_lines(0); dir_perm(0755); dir_group(sysadmin); dir_owner(root); perm(0644); stats_freq(300); use_dns(no); create_dirs(yes); time_reopen(10); }; Here are my sources: source s_internal { internal(); }; source s_udp { udp(); }; source s_tcp { tcp(); }; source s_streams { sun-streams ("/dev/log" door("/etc/.syslog_door")); }; Here are my filters so far: # Level Filters filter f_emerg { level (emerg); }; filter f_alert { level (alert .. emerg); }; filter f_crit { level (crit .. emerg); }; filter f_err { level (err .. emerg); }; filter f_warning { level (warning .. emerg); }; filter f_notice { level (notice .. emerg); }; filter f_info { level (info .. emerg); }; filter f_debug { level (debug .. emerg); }; # Facility Filters filter f_kern { facility (kern); }; filter f_user { facility (user); }; filter f_mail { facility (mail); }; filter f_daemon { facility (daemon); }; filter f_auth { facility (auth); }; filter f_syslog { facility (syslog); }; filter f_lpr { facility (lpr); }; filter f_news { facility (news); }; filter f_uucp { facility (uucp); }; filter f_cron { facility (cron); }; filter f_local0 { facility (local0); }; filter f_local1 { facility (local1); }; filter f_local2 { facility (local2); }; filter f_local3 { facility (local3); }; filter f_local4 { facility (local4); }; filter f_local5 { facility (local5); }; filter f_local6 { facility (local6); }; filter f_local7 { facility (local7); }; Here are my destinations so far: # Destinations: local files, the console, and the client files destination l_internal { file ("/var/adm/syslog-ng"); }; destination l_authlog { file ("/var/log/authlog"); }; destination l_messages { file ("/var/log/messages"); }; destination l_maillog { file ("/var/log/maillog"); }; destination l_ipflog { file ("/var/log/ipflog"); }; destination l_imaplog { file ("/var/log/imaplog"); }; destination l_syslog { file ("/var/log/syslog"); }; destination l_console { file ("/dev/console"); }; Here are my log statements: log { source (s_internal); destination (l_internal); }; # <==== this one is working log { source (s_streams); filter (f_kern); filter (f_debug); destination (l_messages); }; #<====== this on is not working I have tried the following logger tests and am not getting anything logged: logger -p kern.debug "some message" logger -p kern.crit "some message" logger -p kern.info "some message" I have restarted syslog-ng and ensured that it is using my most recent syslog-ng.conf file. Thanks for any help, CC

4 7

Quotes and parser with pattern db in syslog-ng 3.1.1
by Ilas, Yann 26 Jul '10

26 Jul '10

I have a question about the parser using the db_parser and xml file. I'm currently using the version 3.1.1 of syslog-ng. Here is the xml file named "syslog-ng.xml" : <?xml version='1.0' encoding='UTF-8'?> <patterndb version='3' pub_date='2010-07-21'> <ruleset name='syslog-ng' id='syslog-ng-01'> <pattern>syslog-ng</pattern> <rules> <rule provider='yann' id='syslog:server:0123456789:id001' class='system'> <patterns> <pattern>syslog-ng shutting down; version='@STRING::@'</pattern> <examples> <example> <test_message>syslog-ng shutting down; version='3.1.1'</test_message> </example> </examples> <values> <value name=".classifier.facility">syslog</value> <value name=".classifier.severity">notice</value> <value name=".classifier.priority">45</value> </values> </patterns> </rule> </rules> </ruleset> </patterndb> I would like to test that configuration with the following message "syslog-ng shutting down; version='3.1.1'" by using pdbtool. # /opt/syslog-ng/bin/pdbtool match -p /tmp/syslog-ng.xml -P syslog-ng -M "syslog-ng shutting down; version='3.1.1'" MESSAGE=syslog-ng shutting down; version='3.1.1' PROGRAM=syslog-ng .classifier.class=unknown The result is odd because my message seems to be right... I suspect the quotes so I remove the quotes in the xml file. Let's try again with the same message without quotes : (...) <pattern>syslog-ng shutting down; version=@STRING::@</pattern> (...) Test: # /opt/syslog-ng/bin/pdbtool match -p /tmp/syslog-ng.xml -P syslog-ng -M "syslog-ng shutting down; version=3.1.1" MESSAGE=syslog-ng shutting down; version=3.1.1 PROGRAM=syslog-ng .classifier.class=system .classifier.rule_id=syslog:server:0123456789:id001 .classifier.facility=syslog .classifier.severity=notice .classifier.priority=45 It works. What's wrong with my message ? I tried to remove the second quote in the xml file like this : (...) <pattern>syslog-ng shutting down; version='@STRING::@</pattern> (...) Test : # /opt/syslog-ng/bin/pdbtool match -p /tmp/syslog-ng.xml -P syslog-ng -M "syslog-ng shutting down; version='3.1.1" MESSAGE=syslog-ng shutting down; version='3.1.1 PROGRAM=syslog-ng .classifier.class=system .classifier.rule_id=syslog:server:0123456789:id001 .classifier.facility=syslog .classifier.severity=notice .classifier.priority=45 I have the same result if I changed my xml file like this : (...) <pattern>syslog-ng shutting down; version=''@STRING::@</pattern> (...) Test : # /opt/syslog-ng/bin/pdbtool match -p /tmp/syslog-ng.xml -P syslog-ng -M "syslog-ng shutting down; version=''3.1.1" MESSAGE=syslog-ng shutting down; version=''3.1.1 PROGRAM=syslog-ng .classifier.class=system .classifier.rule_id=syslog:server:0123456789:id001 .classifier.facility=syslog .classifier.severity=notice .classifier.priority=45 Can't I have a message like this one 'texttexttext' ? Or did I miss something ? Last test : I change the xml file like this : (...) <pattern>syslog-ng shutting down; version='3.1.1'</pattern> (...) => I remove the @STRING::@ and the pdbtool returned the .classifier.facility, .classifier.severity, etc. May I use @STRING::@ between two simple quotes ? Regards, Yann I.

2 1

Rewrite Hostname Field of Syslog Header
by lecalcot 26 Jul '10

26 Jul '10

Hi, I¹m wondering if syslog-ng is capable of rewriting the hostname field in the header of syslogs as they are forwarded to a remote loghost. Is this possible? Thanks, Lee

5 4

Re: [syslog-ng] Pattern database and '<'
by Ilas, Yann 23 Jul '10

23 Jul '10

>> On 2010-07-23, Ilas, Yann wrote: >> >> Currently, I'm using syslog-ng version 3.1.1. >> I would like to parse that kind of message by using pattern database. >> Here is the message : "<<<<< Message message message". >> So I created that xml file : >>[...] >> <pattern><<<<< Message message message</pattern> >[...] >> I obtain that message when I try to parse with the pdbtool application : >> $ pdbtool match -p syslog-ng.xml -P test -M "<<<<< Message message >> message" >> Error parsing pattern database file; filename='/tmp/syslog-ng.xml', >> error='Error on line 9 char 22: \'<\' is not a valid character following >> a \'<\' character; it may not begin an element name' >> >> Into the xml file, I replaced '<<<<<' with @STRING::@ or @ESTRING...@ >> but it doesn't work. Is it possible to match that kind of message ? > > Characters < and > are special to XML notation. Try using < and > > instead. > > > HTH > > -- > Jakub Jankowski|shasta@toxcorp.com|http://toxcorp.com/ > GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D Thanks ! It works. Regards, Yann I.

1 0

Syslog-ng Server Setup
by Sam Oehlert 22 Jul '10

22 Jul '10

I am attempting to set up a test syslog-ng server in a VM farm. I run CentOS 5.5 and Syslog-ng OSE 2.1.4. My VMs can ping each other and ssh to and from each other. The firewalls are off. Yet I still can only locally log for both server and client, the server doesn't see my clients logs at all. Attached are my server and client configuration files. Thanks for the help, Sam

2 1

More Solaris 10 Woes - tcp/udp issues
by Chuck 21 Jul '10

21 Jul '10

I can't get syslog-ng to listen on port 514 using the tcp or udp sources. First, I have defined entries in /etc/services as follows: syslog-ng-udp 514/udp syslog syslog-ng-tcp 514/tcp syslog I have of course disabled the default system-log service. The following two I can't get to work/listen: (Verifying this with netstat -a) source s_udp { udp(port(514)); }; source s_tcp { tcp(port(514)); }; I can get it to listen with the following: source s_syslog { syslog( transport("udp") port(514) ); }; OR source s_syslog { syslog( transport("tcp") port(514) ); }; # netstat -a UDP: IPv4 Local Address Remote Address State -------------------- -------------------- ---------- *.syslog-ng-udp Idle However, syslog-ng doesn't seem to be doing anything with this. I can see the syslog message when snooping the interface on my syslog-ng server: root@log01:~# snoop -d nge0 udp port 514 Using device nge0 (promiscuous mode) ds01 -> log01 SYSLOG C port=32947 daemon.crit: <26>Jul 21 13:22:08 ds01 -> log01 SYSLOG C port=32947 daemon.crit: <26>Jul 21 13:23:11 ds01 -> log01 SYSLOG C port=32947 daemon.crit: <26>Jul 21 13:23:17 Here is the catch all log statement I am using: destination r_messages { file ("/var/adm/messages_test"); }; log { source (s_syslog); destination (r_messages); }; Anyone have any ideas? Thx, CC

2 1

Re: [syslog-ng] Quotes and parser with pattern db in syslog-ng 3.1.1
by Fekete Róbert 21 Jul '10

21 Jul '10

Hi, it seems to be a bit odd that your original pattern is not working, it might be some quirk with the STRING parser. However, using a QSTRING parser should be better in your case, as it is generally faster than the STRING parser, and more suitable for this message part, like this: <pattern>syslog-ng shutting down; version=@QSTRING::'@</pattern> Regards, Robert On Wednesday, July 21, 2010 18:07 CEST, "Ilas, Yann" <yann.ilas(a)eads.com> wrote: > I have a question about the parser using the db_parser and xml file. I'm > currently using the version 3.1.1 of syslog-ng. > > Here is the xml file named "syslog-ng.xml" : > <?xml version='1.0' encoding='UTF-8'?> > <patterndb version='3' pub_date='2010-07-21'> > <ruleset name='syslog-ng' id='syslog-ng-01'> > <pattern>syslog-ng</pattern> > <rules> > <rule provider='yann' id='syslog:server:0123456789:id001' > class='system'> > <patterns> > <pattern>syslog-ng shutting down; > version='@STRING::@'</pattern> > <examples> > <example> > <test_message>syslog-ng shutting down; > version='3.1.1'</test_message> > </example> > </examples> > <values> > <value name=".classifier.facility">syslog</value> > <value name=".classifier.severity">notice</value> > <value name=".classifier.priority">45</value> > </values> > </patterns> > </rule> > </rules> > </ruleset> > </patterndb> > > I would like to test that configuration with the following message > "syslog-ng shutting down; version='3.1.1'" by using pdbtool. > > # /opt/syslog-ng/bin/pdbtool match -p /tmp/syslog-ng.xml -P syslog-ng -M > "syslog-ng shutting down; version='3.1.1'" > MESSAGE=syslog-ng shutting down; version='3.1.1' > PROGRAM=syslog-ng > .classifier.class=unknown > > The result is odd because my message seems to be right... > I suspect the quotes so I remove the quotes in the xml file. > > Let's try again with the same message without quotes : > (...) > <pattern>syslog-ng shutting down; version=@STRING::@</pattern> > (...) > > Test: > # /opt/syslog-ng/bin/pdbtool match -p /tmp/syslog-ng.xml -P syslog-ng -M > "syslog-ng shutting down; version=3.1.1" > MESSAGE=syslog-ng shutting down; version=3.1.1 > PROGRAM=syslog-ng > .classifier.class=system > .classifier.rule_id=syslog:server:0123456789:id001 > .classifier.facility=syslog > .classifier.severity=notice > .classifier.priority=45 > > It works. > > What's wrong with my message ? > I tried to remove the second quote in the xml file like this : > (...) > <pattern>syslog-ng shutting down; version='@STRING::@</pattern> > (...) > > Test : > # /opt/syslog-ng/bin/pdbtool match -p /tmp/syslog-ng.xml -P syslog-ng -M > "syslog-ng shutting down; version='3.1.1" > MESSAGE=syslog-ng shutting down; version='3.1.1 > PROGRAM=syslog-ng > .classifier.class=system > .classifier.rule_id=syslog:server:0123456789:id001 > .classifier.facility=syslog > .classifier.severity=notice > .classifier.priority=45 > > I have the same result if I changed my xml file like this : > (...) > <pattern>syslog-ng shutting down; version=''@STRING::@</pattern> > (...) > > Test : > # /opt/syslog-ng/bin/pdbtool match -p /tmp/syslog-ng.xml -P syslog-ng -M > "syslog-ng shutting down; version=''3.1.1" > MESSAGE=syslog-ng shutting down; version=''3.1.1 > PROGRAM=syslog-ng > .classifier.class=system > .classifier.rule_id=syslog:server:0123456789:id001 > .classifier.facility=syslog > .classifier.severity=notice > .classifier.priority=45 > > Can't I have a message like this one 'texttexttext' ? Or did I miss > something ? > > Last test : I change the xml file like this : > (...) > <pattern>syslog-ng shutting down; version='3.1.1'</pattern> > (...) > => I remove the @STRING::@ and the pdbtool returned the > .classifier.facility, .classifier.severity, etc. > > May I use @STRING::@ between two simple quotes ? > > > Regards, > > Yann I.

1 0

Re: [syslog-ng] How to deal with duplicate log entries
by chris packham 21 Jul '10

21 Jul '10

Hi, I don't know what solaris does, but have you tried a simple config like this? log { source (s_streams); filter (f_mail); filter (f_debug); destination (l_syslog); }; log { source (s_streams); filter (f_auth); filter (f_info); destination (l_authlog); }; log { source (s_streams); destination (l_messages); }; If there are specific things you don't want in l_messages you can filter them out (as opposed to filtering stuff in). The right approach depends on how many messages you want (or don't want) in l_messages. Another trick I've seen to squash messages is using a destination that equates to '/dev/null' and the flags(final); setting to ditch messages that you don't want. >>> Chuck <chuck.carson(a)gmail.com> 07/21/10 12:01 PM >>> I have the following log statgements.. (Im basically trying to mirror what solaris 10's default syslogl.conf does): log { source (s_streams); filter (f_emerg); destination (l_messages); }; log { source (s_streams); filter (f_err); destination (l_messages); }; log { source (s_streams); filter (f_kern); filter (f_debug); destination (l_messages); }; log { source (s_streams); filter (f_daemon); filter (f_notice); destination (l_messages); }; log { source (s_streams); filter (f_mail); filter (f_crit); destination (l_messages); }; log { source (s_streams); filter (f_mail); filter (f_debug); destination (l_syslog); }; log { source (s_streams); filter (f_auth); filter (f_info); destination (l_authlog); }; # sshd logging However when sending to daemon.err I get duplicate messages... Should I change line 2 to this: log { source (s_streams); filter (f_err); filter (not f_daemon) destination (l_messages); }; Or should I hard-code every facility like so: log { source (s_streams); filter (f_err); filter (f_daemon) destination (l_messages); }; log { source (s_streams); filter (f_err); filter (f_mail) destination (l_messages); }; log { source (s_streams); filter (f_err); filter (f_auth) destination (l_messages); }; ... and so on? Thx,.CC

1 0

patterndb: grep on steroids
by Balazs Scheidler 20 Jul '10

20 Jul '10

Hi, This is a repost of my blog post at http://bazsi.blogs.balabit.com/ regarding patterndb and syslog-ng OSE 3.2. I felt it was important enough to crosspost it here. You may have heard of my last project to collect log samples from various applications, in order to convert log data from free-form human readable strings into structured information. The first round to collect login/logout messages from sshd is now complete. You could ask: ok, but what is the immediate benefit? You supposedly have a lot of unprocessed log files, and syslog-ng's db-parser() has not been used to process them, thus they are stored as good-old plain text files. I spent a couple of hours to add a "grep"-like functionality to pdbtool which makes it easy to process already existing log files, giving you immediate benefit for each and every sample added to patterndb. For example, if you are interested in login failure events, you could say: zcat logfile.gz | pdbtool match -p access/sshd.pdb \ --file - \ --filter 'tags("usracct") and match('REJECT' type(string) value("secevt.verdict"));' \ --template '${usracct.type},${secevt.verdict},${usracct.username}\n' What the command above does is the following: * reads a compressed logfile from logfile.gz * tells pdbtool to use access/sshd.pdb (in the patterndb git repo) as its pattern database file * tells pdbtool to read its stdin as a logfile, and * apply the db-parser() for each log message * apply the syslog-ng filter specified above * and print matching messages using the template also specified above As a combination, it results in a CSV file, containing login failure records found in the logfile. Also please note that as long there's a pattern in the pdb file, it doesn't really matter how that originally looked like, the fact that ssh can use 3-5 different messages for the same meaning is hidden nicely under the hood. And imagine we'd have patterns for all common applications running on our computers: this would mean that the same command above would produce login-failure reports independently from the application/OS combination being used. Try that with grep. :) This pdbtool is in the OSE 3.2 tree, clone the tree from: git://git.balabit.hu/bazsi/syslog-ng-3.2.git If you want to experiment with the OSE 3.2 tree, please read the OSE 3.2 caveats post at: http://bazsi.blogs.balabit.com/2010/07/syslog-ng-ose-32-caveats.html -- Bazsi

1 0

configuring different sources on the syslog-ng server
by Khaleelah Peerbocus 20 Jul '10

20 Jul '10

Hello I would like to get some clarification on the source tag in both syslog client and server. I have different clients hosting apache / weblogic My Syslog-ng Server has ip 192.x.x.x Apache (client) has ip 192.x.x.1 Weblogic (client) has ip 192.x.x.2 In all the clients syslog configuration files, i have added destination apacheaccesslog { tcp("192.x.x.x " port(3331)); }; destination weblogiclog { tcp("192.x.x.x " port(3332)); }; My issue is in the syslog-ng server configuration file, if i add source apache_access {tcp(ip(192.x.x.x) port(3331)); }; Should the ip be the server ip or the client ip (in this example, i have put the syslog-ng server ip). Does the port number differentiate the different sources we are receiving the log files ??? logo-emailsignature.jpgKhaleelah Peerbocus - Systems Analyst Mauritius Cargo Community Services Business Registration No.: C08077158 VAT Registration No.: VAT20427044 1st Floor, Trade and Marketing Centre, Mer Rouge Port-Louis, Mauritius Tel: +230 206 2970 - Cell: +230 498 7897 Fax: +230 216 8858 Email: <mailto:HDagent6@maccs.mu> systems2(a)maccs.mu - Website: <http://www.maccs.mu/> http://www.maccs.mu

3 6