- syslog-ng - lists.balabit.hu

Is logger an effective test tool on Solaris 10?
by Chuck 26 Jul '10

26 Jul '10

I am using the following script to test syslog-ng. (Running this from the same machine as the syslog-ng server): #!/bin/bash d=`date '+%y%m%d_%H%M%S'` logger -p kern.debug "__kern.debug__ $d" sleep 1; d=`date '+%y%m%d_%H%M%S'` logger -p kern.crit "__kern.crit__ $d" sleep 1; d=`date '+%y%m%d_%H%M%S'` logger -p kern.info "__kern.info__ $d" sleep 1; However, here is how the messages are being seen within syslog-ng: Jul 21 10:30:35 log01 cars: [ID 702911 user.debug] __kern.debug__ 100721_103035 Jul 21 10:30:36 log01 cars: [ID 702911 user.crit] __kern.crit__ 100721_103036 Jul 21 10:30:37 log01 cars: [ID 702911 user.info] __kern.info__ 100721_103037 The facility is being seen as user and not kern... Anyone have any ideas? Thx,

4 3

Messages Not Getting Logged
by Chuck 26 Jul '10

26 Jul '10

Hello, I am using the sunfreeware build of syslog-ng 3.04 on a Solaris 10 Update 8 system: syslog-ng 3.0.4 Revision: ssh+git://bazsi@git.balabit //var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.0#master#1b5d618e301ad94aa20e692ffba16469dece8d10 Compile-Date: Sep 2 2009 05:14:23 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: on Enable-Sun-Door: on Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: on My internal() source is working but the sun-streams source is not. Here are my options: options { use_fqdn(no); flush_lines(0); dir_perm(0755); dir_group(sysadmin); dir_owner(root); perm(0644); stats_freq(300); use_dns(no); create_dirs(yes); time_reopen(10); }; Here are my sources: source s_internal { internal(); }; source s_udp { udp(); }; source s_tcp { tcp(); }; source s_streams { sun-streams ("/dev/log" door("/etc/.syslog_door")); }; Here are my filters so far: # Level Filters filter f_emerg { level (emerg); }; filter f_alert { level (alert .. emerg); }; filter f_crit { level (crit .. emerg); }; filter f_err { level (err .. emerg); }; filter f_warning { level (warning .. emerg); }; filter f_notice { level (notice .. emerg); }; filter f_info { level (info .. emerg); }; filter f_debug { level (debug .. emerg); }; # Facility Filters filter f_kern { facility (kern); }; filter f_user { facility (user); }; filter f_mail { facility (mail); }; filter f_daemon { facility (daemon); }; filter f_auth { facility (auth); }; filter f_syslog { facility (syslog); }; filter f_lpr { facility (lpr); }; filter f_news { facility (news); }; filter f_uucp { facility (uucp); }; filter f_cron { facility (cron); }; filter f_local0 { facility (local0); }; filter f_local1 { facility (local1); }; filter f_local2 { facility (local2); }; filter f_local3 { facility (local3); }; filter f_local4 { facility (local4); }; filter f_local5 { facility (local5); }; filter f_local6 { facility (local6); }; filter f_local7 { facility (local7); }; Here are my destinations so far: # Destinations: local files, the console, and the client files destination l_internal { file ("/var/adm/syslog-ng"); }; destination l_authlog { file ("/var/log/authlog"); }; destination l_messages { file ("/var/log/messages"); }; destination l_maillog { file ("/var/log/maillog"); }; destination l_ipflog { file ("/var/log/ipflog"); }; destination l_imaplog { file ("/var/log/imaplog"); }; destination l_syslog { file ("/var/log/syslog"); }; destination l_console { file ("/dev/console"); }; Here are my log statements: log { source (s_internal); destination (l_internal); }; # <==== this one is working log { source (s_streams); filter (f_kern); filter (f_debug); destination (l_messages); }; #<====== this on is not working I have tried the following logger tests and am not getting anything logged: logger -p kern.debug "some message" logger -p kern.crit "some message" logger -p kern.info "some message" I have restarted syslog-ng and ensured that it is using my most recent syslog-ng.conf file. Thanks for any help, CC

4 7

Quotes and parser with pattern db in syslog-ng 3.1.1
by Ilas, Yann 26 Jul '10

26 Jul '10

I have a question about the parser using the db_parser and xml file. I'm currently using the version 3.1.1 of syslog-ng. Here is the xml file named "syslog-ng.xml" : <?xml version='1.0' encoding='UTF-8'?> <patterndb version='3' pub_date='2010-07-21'> <ruleset name='syslog-ng' id='syslog-ng-01'> <pattern>syslog-ng</pattern> <rules> <rule provider='yann' id='syslog:server:0123456789:id001' class='system'> <patterns> <pattern>syslog-ng shutting down; version='@STRING::@'</pattern> <examples> <example> <test_message>syslog-ng shutting down; version='3.1.1'</test_message> </example> </examples> <values> <value name=".classifier.facility">syslog</value> <value name=".classifier.severity">notice</value> <value name=".classifier.priority">45</value> </values> </patterns> </rule> </rules> </ruleset> </patterndb> I would like to test that configuration with the following message "syslog-ng shutting down; version='3.1.1'" by using pdbtool. # /opt/syslog-ng/bin/pdbtool match -p /tmp/syslog-ng.xml -P syslog-ng -M "syslog-ng shutting down; version='3.1.1'" MESSAGE=syslog-ng shutting down; version='3.1.1' PROGRAM=syslog-ng .classifier.class=unknown The result is odd because my message seems to be right... I suspect the quotes so I remove the quotes in the xml file. Let's try again with the same message without quotes : (...) <pattern>syslog-ng shutting down; version=@STRING::@</pattern> (...) Test: # /opt/syslog-ng/bin/pdbtool match -p /tmp/syslog-ng.xml -P syslog-ng -M "syslog-ng shutting down; version=3.1.1" MESSAGE=syslog-ng shutting down; version=3.1.1 PROGRAM=syslog-ng .classifier.class=system .classifier.rule_id=syslog:server:0123456789:id001 .classifier.facility=syslog .classifier.severity=notice .classifier.priority=45 It works. What's wrong with my message ? I tried to remove the second quote in the xml file like this : (...) <pattern>syslog-ng shutting down; version='@STRING::@</pattern> (...) Test : # /opt/syslog-ng/bin/pdbtool match -p /tmp/syslog-ng.xml -P syslog-ng -M "syslog-ng shutting down; version='3.1.1" MESSAGE=syslog-ng shutting down; version='3.1.1 PROGRAM=syslog-ng .classifier.class=system .classifier.rule_id=syslog:server:0123456789:id001 .classifier.facility=syslog .classifier.severity=notice .classifier.priority=45 I have the same result if I changed my xml file like this : (...) <pattern>syslog-ng shutting down; version=''@STRING::@</pattern> (...) Test : # /opt/syslog-ng/bin/pdbtool match -p /tmp/syslog-ng.xml -P syslog-ng -M "syslog-ng shutting down; version=''3.1.1" MESSAGE=syslog-ng shutting down; version=''3.1.1 PROGRAM=syslog-ng .classifier.class=system .classifier.rule_id=syslog:server:0123456789:id001 .classifier.facility=syslog .classifier.severity=notice .classifier.priority=45 Can't I have a message like this one 'texttexttext' ? Or did I miss something ? Last test : I change the xml file like this : (...) <pattern>syslog-ng shutting down; version='3.1.1'</pattern> (...) => I remove the @STRING::@ and the pdbtool returned the .classifier.facility, .classifier.severity, etc. May I use @STRING::@ between two simple quotes ? Regards, Yann I.

2 1

Rewrite Hostname Field of Syslog Header
by lecalcot 26 Jul '10

26 Jul '10

Hi, I¹m wondering if syslog-ng is capable of rewriting the hostname field in the header of syslogs as they are forwarded to a remote loghost. Is this possible? Thanks, Lee

5 4

Re: [syslog-ng] Pattern database and '<'
by Ilas, Yann 23 Jul '10

23 Jul '10

>> On 2010-07-23, Ilas, Yann wrote: >> >> Currently, I'm using syslog-ng version 3.1.1. >> I would like to parse that kind of message by using pattern database. >> Here is the message : "<<<<< Message message message". >> So I created that xml file : >>[...] >> <pattern><<<<< Message message message</pattern> >[...] >> I obtain that message when I try to parse with the pdbtool application : >> $ pdbtool match -p syslog-ng.xml -P test -M "<<<<< Message message >> message" >> Error parsing pattern database file; filename='/tmp/syslog-ng.xml', >> error='Error on line 9 char 22: \'<\' is not a valid character following >> a \'<\' character; it may not begin an element name' >> >> Into the xml file, I replaced '<<<<<' with @STRING::@ or @ESTRING...@ >> but it doesn't work. Is it possible to match that kind of message ? > > Characters < and > are special to XML notation. Try using < and > > instead. > > > HTH > > -- > Jakub Jankowski|shasta@toxcorp.com|http://toxcorp.com/ > GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D Thanks ! It works. Regards, Yann I.

1 0

Syslog-ng Server Setup
by Sam Oehlert 22 Jul '10

22 Jul '10

I am attempting to set up a test syslog-ng server in a VM farm. I run CentOS 5.5 and Syslog-ng OSE 2.1.4. My VMs can ping each other and ssh to and from each other. The firewalls are off. Yet I still can only locally log for both server and client, the server doesn't see my clients logs at all. Attached are my server and client configuration files. Thanks for the help, Sam

2 1

More Solaris 10 Woes - tcp/udp issues
by Chuck 21 Jul '10

21 Jul '10

I can't get syslog-ng to listen on port 514 using the tcp or udp sources. First, I have defined entries in /etc/services as follows: syslog-ng-udp 514/udp syslog syslog-ng-tcp 514/tcp syslog I have of course disabled the default system-log service. The following two I can't get to work/listen: (Verifying this with netstat -a) source s_udp { udp(port(514)); }; source s_tcp { tcp(port(514)); }; I can get it to listen with the following: source s_syslog { syslog( transport("udp") port(514) ); }; OR source s_syslog { syslog( transport("tcp") port(514) ); }; # netstat -a UDP: IPv4 Local Address Remote Address State -------------------- -------------------- ---------- *.syslog-ng-udp Idle However, syslog-ng doesn't seem to be doing anything with this. I can see the syslog message when snooping the interface on my syslog-ng server: root@log01:~# snoop -d nge0 udp port 514 Using device nge0 (promiscuous mode) ds01 -> log01 SYSLOG C port=32947 daemon.crit: <26>Jul 21 13:22:08 ds01 -> log01 SYSLOG C port=32947 daemon.crit: <26>Jul 21 13:23:11 ds01 -> log01 SYSLOG C port=32947 daemon.crit: <26>Jul 21 13:23:17 Here is the catch all log statement I am using: destination r_messages { file ("/var/adm/messages_test"); }; log { source (s_syslog); destination (r_messages); }; Anyone have any ideas? Thx, CC

2 1

Re: [syslog-ng] Quotes and parser with pattern db in syslog-ng 3.1.1
by Fekete Róbert 21 Jul '10

21 Jul '10

Hi, it seems to be a bit odd that your original pattern is not working, it might be some quirk with the STRING parser. However, using a QSTRING parser should be better in your case, as it is generally faster than the STRING parser, and more suitable for this message part, like this: <pattern>syslog-ng shutting down; version=@QSTRING::'@</pattern> Regards, Robert On Wednesday, July 21, 2010 18:07 CEST, "Ilas, Yann" <yann.ilas(a)eads.com> wrote: > I have a question about the parser using the db_parser and xml file. I'm > currently using the version 3.1.1 of syslog-ng. > > Here is the xml file named "syslog-ng.xml" : > <?xml version='1.0' encoding='UTF-8'?> > <patterndb version='3' pub_date='2010-07-21'> > <ruleset name='syslog-ng' id='syslog-ng-01'> > <pattern>syslog-ng</pattern> > <rules> > <rule provider='yann' id='syslog:server:0123456789:id001' > class='system'> > <patterns> > <pattern>syslog-ng shutting down; > version='@STRING::@'</pattern> > <examples> > <example> > <test_message>syslog-ng shutting down; > version='3.1.1'</test_message> > </example> > </examples> > <values> > <value name=".classifier.facility">syslog</value> > <value name=".classifier.severity">notice</value> > <value name=".classifier.priority">45</value> > </values> > </patterns> > </rule> > </rules> > </ruleset> > </patterndb> > > I would like to test that configuration with the following message > "syslog-ng shutting down; version='3.1.1'" by using pdbtool. > > # /opt/syslog-ng/bin/pdbtool match -p /tmp/syslog-ng.xml -P syslog-ng -M > "syslog-ng shutting down; version='3.1.1'" > MESSAGE=syslog-ng shutting down; version='3.1.1' > PROGRAM=syslog-ng > .classifier.class=unknown > > The result is odd because my message seems to be right... > I suspect the quotes so I remove the quotes in the xml file. > > Let's try again with the same message without quotes : > (...) > <pattern>syslog-ng shutting down; version=@STRING::@</pattern> > (...) > > Test: > # /opt/syslog-ng/bin/pdbtool match -p /tmp/syslog-ng.xml -P syslog-ng -M > "syslog-ng shutting down; version=3.1.1" > MESSAGE=syslog-ng shutting down; version=3.1.1 > PROGRAM=syslog-ng > .classifier.class=system > .classifier.rule_id=syslog:server:0123456789:id001 > .classifier.facility=syslog > .classifier.severity=notice > .classifier.priority=45 > > It works. > > What's wrong with my message ? > I tried to remove the second quote in the xml file like this : > (...) > <pattern>syslog-ng shutting down; version='@STRING::@</pattern> > (...) > > Test : > # /opt/syslog-ng/bin/pdbtool match -p /tmp/syslog-ng.xml -P syslog-ng -M > "syslog-ng shutting down; version='3.1.1" > MESSAGE=syslog-ng shutting down; version='3.1.1 > PROGRAM=syslog-ng > .classifier.class=system > .classifier.rule_id=syslog:server:0123456789:id001 > .classifier.facility=syslog > .classifier.severity=notice > .classifier.priority=45 > > I have the same result if I changed my xml file like this : > (...) > <pattern>syslog-ng shutting down; version=''@STRING::@</pattern> > (...) > > Test : > # /opt/syslog-ng/bin/pdbtool match -p /tmp/syslog-ng.xml -P syslog-ng -M > "syslog-ng shutting down; version=''3.1.1" > MESSAGE=syslog-ng shutting down; version=''3.1.1 > PROGRAM=syslog-ng > .classifier.class=system > .classifier.rule_id=syslog:server:0123456789:id001 > .classifier.facility=syslog > .classifier.severity=notice > .classifier.priority=45 > > Can't I have a message like this one 'texttexttext' ? Or did I miss > something ? > > Last test : I change the xml file like this : > (...) > <pattern>syslog-ng shutting down; version='3.1.1'</pattern> > (...) > => I remove the @STRING::@ and the pdbtool returned the > .classifier.facility, .classifier.severity, etc. > > May I use @STRING::@ between two simple quotes ? > > > Regards, > > Yann I.

1 0

Re: [syslog-ng] How to deal with duplicate log entries
by chris packham 21 Jul '10

21 Jul '10

Hi, I don't know what solaris does, but have you tried a simple config like this? log { source (s_streams); filter (f_mail); filter (f_debug); destination (l_syslog); }; log { source (s_streams); filter (f_auth); filter (f_info); destination (l_authlog); }; log { source (s_streams); destination (l_messages); }; If there are specific things you don't want in l_messages you can filter them out (as opposed to filtering stuff in). The right approach depends on how many messages you want (or don't want) in l_messages. Another trick I've seen to squash messages is using a destination that equates to '/dev/null' and the flags(final); setting to ditch messages that you don't want. >>> Chuck <chuck.carson(a)gmail.com> 07/21/10 12:01 PM >>> I have the following log statgements.. (Im basically trying to mirror what solaris 10's default syslogl.conf does): log { source (s_streams); filter (f_emerg); destination (l_messages); }; log { source (s_streams); filter (f_err); destination (l_messages); }; log { source (s_streams); filter (f_kern); filter (f_debug); destination (l_messages); }; log { source (s_streams); filter (f_daemon); filter (f_notice); destination (l_messages); }; log { source (s_streams); filter (f_mail); filter (f_crit); destination (l_messages); }; log { source (s_streams); filter (f_mail); filter (f_debug); destination (l_syslog); }; log { source (s_streams); filter (f_auth); filter (f_info); destination (l_authlog); }; # sshd logging However when sending to daemon.err I get duplicate messages... Should I change line 2 to this: log { source (s_streams); filter (f_err); filter (not f_daemon) destination (l_messages); }; Or should I hard-code every facility like so: log { source (s_streams); filter (f_err); filter (f_daemon) destination (l_messages); }; log { source (s_streams); filter (f_err); filter (f_mail) destination (l_messages); }; log { source (s_streams); filter (f_err); filter (f_auth) destination (l_messages); }; ... and so on? Thx,.CC

1 0

patterndb: grep on steroids
by Balazs Scheidler 20 Jul '10

20 Jul '10

Hi, This is a repost of my blog post at http://bazsi.blogs.balabit.com/ regarding patterndb and syslog-ng OSE 3.2. I felt it was important enough to crosspost it here. You may have heard of my last project to collect log samples from various applications, in order to convert log data from free-form human readable strings into structured information. The first round to collect login/logout messages from sshd is now complete. You could ask: ok, but what is the immediate benefit? You supposedly have a lot of unprocessed log files, and syslog-ng's db-parser() has not been used to process them, thus they are stored as good-old plain text files. I spent a couple of hours to add a "grep"-like functionality to pdbtool which makes it easy to process already existing log files, giving you immediate benefit for each and every sample added to patterndb. For example, if you are interested in login failure events, you could say: zcat logfile.gz | pdbtool match -p access/sshd.pdb \ --file - \ --filter 'tags("usracct") and match('REJECT' type(string) value("secevt.verdict"));' \ --template '${usracct.type},${secevt.verdict},${usracct.username}\n' What the command above does is the following: * reads a compressed logfile from logfile.gz * tells pdbtool to use access/sshd.pdb (in the patterndb git repo) as its pattern database file * tells pdbtool to read its stdin as a logfile, and * apply the db-parser() for each log message * apply the syslog-ng filter specified above * and print matching messages using the template also specified above As a combination, it results in a CSV file, containing login failure records found in the logfile. Also please note that as long there's a pattern in the pdb file, it doesn't really matter how that originally looked like, the fact that ssh can use 3-5 different messages for the same meaning is hidden nicely under the hood. And imagine we'd have patterns for all common applications running on our computers: this would mean that the same command above would produce login-failure reports independently from the application/OS combination being used. Try that with grep. :) This pdbtool is in the OSE 3.2 tree, clone the tree from: git://git.balabit.hu/bazsi/syslog-ng-3.2.git If you want to experiment with the OSE 3.2 tree, please read the OSE 3.2 caveats post at: http://bazsi.blogs.balabit.com/2010/07/syslog-ng-ose-32-caveats.html -- Bazsi

1 0

configuring different sources on the syslog-ng server
by Khaleelah Peerbocus 20 Jul '10

20 Jul '10

Hello I would like to get some clarification on the source tag in both syslog client and server. I have different clients hosting apache / weblogic My Syslog-ng Server has ip 192.x.x.x Apache (client) has ip 192.x.x.1 Weblogic (client) has ip 192.x.x.2 In all the clients syslog configuration files, i have added destination apacheaccesslog { tcp("192.x.x.x " port(3331)); }; destination weblogiclog { tcp("192.x.x.x " port(3332)); }; My issue is in the syslog-ng server configuration file, if i add source apache_access {tcp(ip(192.x.x.x) port(3331)); }; Should the ip be the server ip or the client ip (in this example, i have put the syslog-ng server ip). Does the port number differentiate the different sources we are receiving the log files ??? logo-emailsignature.jpgKhaleelah Peerbocus - Systems Analyst Mauritius Cargo Community Services Business Registration No.: C08077158 VAT Registration No.: VAT20427044 1st Floor, Trade and Marketing Centre, Mer Rouge Port-Louis, Mauritius Tel: +230 206 2970 - Cell: +230 498 7897 Fax: +230 216 8858 Email: <mailto:HDagent6@maccs.mu> systems2(a)maccs.mu - Website: <http://www.maccs.mu/> http://www.maccs.mu

3 6

[announcement] syslog-ng contributions redefined
by Balazs Scheidler 14 Jul '10

14 Jul '10

Dear syslog-ng users, This is a repost of my blog post at http://bazsi.blogs.balabit.com/ which I felt was important enough to repeat here. Links may not be properly reproduced, so please follow the link above if you'd like to browse the references as well. syslog-ng has been around for about 12 years now, but I think the biggest change in the project's life is imminent: with the upcoming release of syslog-ng OSE 3.2, syslog-ng will become an independent entity. Until now, syslog-ng was primarily maintained & developed by BalaBit, copyrights needed to be reassigned in order to grant BalaBit special privileges. BalaBit used her privileges to create a dual-licensed fork of syslog-ng, named "syslog-ng Premium Edition". The value we offer over the Open Source Edition of syslog-ng are things that larger enterprises require: * support on a large number of UNIX platforms (27 as of 3.1), * smaller and larger feature differences (like the encrypted/digitally signed logfile feature) * better test coverage and release management * longer term support Although perfectly legal, this business model was not welcome in various Free Software communities, and has caused friction and harm, because BalaBit has enjoyed a privilege that no others could get. We plan to fix this situation and we've worked hard in the past months to make this possible. We're letting syslog-ng go: no signed Contributory License Agreements will be required in order to contribute to syslog-ng in the future. This is not just a matter of policy: while BalaBit wants to be a true citizen of the Free Software world, we also need to ensure the continued revenue stream that the Premium Edition provides. The adjusted business model allows us to deliver what our customers need, and at the same time make it possible for anyone else in the community to have the same privileges as BalaBit has. The syslog-ng changes in 3.2 that I feel are most important are detailed below. Plugins syslog-ng was transformed from being a monolithic executable, to a core and a set of plugins. Plugins range from source and destination drivers, filters, parsers, rewrite objects, anything can be extended with the use of a relatively simple plugin. And this change didn't transform the configuration file format, it remained the same as before: readable and flexible. If you don't want to look under the hood, no functionality has changed, except plugins are loaded at runtime, and as a side effect, the syntax error reports are way better. The core is licensed under the LGPL and plugins are licensed under the GPL. This legal framework allows BalaBit to deliver value to its customers in non-free plugins and syslog-ng related services. syslog-ng Configuration Library syslog-ng is an infrastructure element: very flexible, but sometimes intimidating. For example, having to specify the pad_size() option for an HP-UX /dev/log device explicitly makes syslog-ng flexible, but not very user-friendly. It is common to share working configuration snippets on the mailing list or elsewhere in the syslog-ng community. We try to concentrate this effort with the creation of the syslog-ng Configuration Library (aka SCL). This library is a set of config snippets that can be included into a syslog-ng configuration file, using proper defaults but still allowing customization. For example: we've created a source driver named "system()" which automatically expands to the local log devices as needed by the current Operating System syslog-ng runs on, but it is also possible to create application specific configuration snippets (for example apache source?) and ship it as a ready to use config block. See this post for more information. This makes it possible to create a configuration file that will run on each of your platforms. In fact, we did that too: syslog-ng now comes with a default configuration file. Support for non-syslog messages Since syslog-ng is now plugin based and even the "syslog" message format as such is a plugin, it is now quite easy to add support for non-syslog message sources. As an example, we've added a plugin to parse Linux process accounting records, which makes it trivial to collect this data as well and possibly use it as a source of information when correllating data. Future plugins like creating a MIB aware SNMP listener, or possibly processing netflow data, but I'd like to create a generic SQL source as well. Support for these formats doesn't mean that syslog-ng would transform them into textual messages: proper name-value structure is kept and it is possible to put them into nicely structured SQL tables. Of course if all you want to transform them to text for readability, that's possible too with the use of a template(). patterndb project Like I announced before, we're starting a parallel project to create a set of message patterns directly usable with syslog-ng's db-parser(). Current state The changes are happening in the syslog-ng 3.2 repository at: http://git.balabit.hu/ We have created a source-only 3.2alpha1 release of the current state. It runs our automated test harness on Linux, but of course it is not yet recommended in production environments. The release can be downloaded from www.balabit.com. There were significant changes in how syslog-ng is compiled, thus it is expected that we have build issues on non-Linux systems. We expect to address these issues as they are found and create a 3.2beta1 release, for a longer beta testing. Future Parallel to fixing up the remaining issues on syslog-ng 3.2, we're going to open new ways to improve syslog-ng in the 3.3 branch. The most important item on our product backlog is to address the scalability problems and improve performance of syslog-ng. Summary With the upcoming changes in syslog-ng OSE 3.2 possible contributions will be greatly expanded: * write a pattern for your favourite application and process log data faster + easier * write an SCL configuration snippet, make your application easier to integrate with syslog-ng * write a plugin for your favourite NoSQL database, * write a plugin for a transformation that syslog-ng is not capable of doing right now (what about facility / severity rewrite rules? easy as a piece of cake) * write a plugin for things that you do with an external script * or contribute to the core of syslog-ng. And all this without having to sign paperwork... -- Bazsi

1 0

monitoring SQL-Server-ERRORLOG
by Schöppel Marcus 14 Jul '10

14 Jul '10

Hi all@syslog-ng-mailing-list! I'm trying to monitor the MSSQLServer ERROR-logfile but I get a logentry for every single letter that is added to the logfile: -> May 18 13:57:40 test mssql: L May 18 13:57:40 test mssql: o May 18 13:57:40 test mssql: g May 18 13:57:40 test mssql: i May 18 13:57:40 test mssql: n May 18 13:57:40 test mssql: May 18 13:57:40 test mssql: f May 18 13:57:40 test mssql: a May 18 13:57:40 test mssql: i May 18 13:57:40 test mssql: l May 18 13:57:40 test mssql: e May 18 13:57:40 test mssql: d <- This seems to stem from the encoding of the file (UTF-16LE) because with other logfiles this problem doesn't arise. I suppose it would help if the "encoding"-Parameter of the file-source would work but syslog-ng/cygwin (Version 3.0.1) won't start when I add it no matter which encoding I use. I tried this workaround-cronjob: - copy file somewhere - convert to UTF-8 using iconv - give the same timestamp to the converted logfile so that syslog-ng doesn't transfer the whole file again - monitoring the converted file with syslog-ng This doesn't work completely correct as despite assigning the same timestamp the whole file is sometimes transferred again (and not just the new entries). Is this problem (following this UTF-16LE-Windowsfile (easily)) not resolvable by using the cygwin-version of syslog-ng?? Or am I maybe using a wrong encoding spelling? Does it work using the commercial windows-version of the agent?

2 1

[announce] patterndb project
by Balazs Scheidler 13 Jul '10

13 Jul '10

Hi, By now probably most of you know about patterndb, a powerful framework in syslog-ng that lets you extract structured information from log messages and perform classification at a high speed: http://www.balabit.com/dl/html/syslog-ng-ose-v3.1-guide-admin-en.html/conce… Until now, syslog-ng offered the feature, but no release-quality patterns were produced by the syslog-ng developers. Some samples based on the logcheck database were created, but otherwise every syslog-ng user had to create her samples manually, possibly repeating work performed by others. Since this calls out to be a community project, I'm hereby starting one. Goals ===== Create release-quality pattern databases that can simply be deployed to an existing syslog-ng installation. The goal of the patterns is to extract structured information from the free-form syslog messages, e.g. create name-value pairs based on the syslog message. Since the key factor when doing something like this is the naming of fields, we're going to create our generic naming guidelines that can be applied to any application in the industry. It is not our goal to implement correllation or any other advanced form of analysis, although we feel that with the results of this project, event correllation and analysis can be performed much easier than without it. Related projects ================ I know there are other efforts in the field, why not simply join them? CEF - is the log message format for a proprietary log analysis engine, primarily meant to be used to hold IP security device logs (firewalls, IPSs, virus gateways etc). The patterndb project aims to create patterns for a wider range of device logs and be more generic in the approach. On the other hand we feel that it might be useful to create a solution for converting db-parser output to the CEF format. CEE - Common Event Expression project by Mitre has a focus on creating a nv pair dictionary for all kinds of devices/log messages out there. Although I might be missing something, but I didn't find the concrete results so far, apart from a nicely looking white paper. If the CEE delivers something, then patterndb would probably adapt the naming/taxonomy structure. But I guess not all devices will start logging in the new shiny format, thus the existing devices would need their logs converted, so the patterndb work wouldn't be wasted. Infrastructure ============== Our original patterndb related plans were to create an easy to use web based interface for editing patterns, but since that project is progressing slowly, I'm calling for a minimalist approach: git based version control of simple plain text files. Of course once the nice web based interface is finished, we're going to be ready to use it. First steps =========== I have created a git repository at: http://git.balabit.hu/bazsi/syslog-ng-patterndb.git This contains the initial version of the naming policy document and a simple schema for SIEM-style and a user login-logout naming schema. If you are interested please read the file README.txt in the git archive, or if you prefer a web browser, use this link: http://git.balabit.hu/?p=bazsi/syslog-ng-patterndb.git;a=blob;f=README.txt;… Licensing ========= I do not have a decision yet, but for sure this is going to use one of the open source licenses or Creative Commons. Let me know if you have a preference in this area. Getting involved ================ Join the syslog-ng mailing list, a start discussing! If you have existing patterns, great. If you don't, it is not late to join. http://lists.balabit.hu/mailman/listinfo/syslog-ng -- Bazsi

3 20

Syslog-ng 3.0.7 fails to insert into microsoft sql server
by Khaleelah Peerbocus 13 Jul '10

13 Jul '10

Dear All, Im currently configuring syslog-ng on SUSE Linux 10 SP2 to communicate with microsoft sql server 2005. I have installed freeTds 8.0. Tested TSQL ..works fine ISQL works fine. when testing with syslog-ng Conf file destination d_mssql { sql(type(mssql) host("ip address") port("1433") username("sa") password("password") database("syslogng") table("test1") columns("datetime varchar(16)","host varchar(32)" ,"program varchar(32)","pid varchar(8)","message varchar(4096)") values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY") indexes() ); }; log { source(src); destination(d_mssql); }; I get the following error Note I created the table test1 manually in SQL server Code: Jul 12 09:57:06 syslogng syslog-ng[25034]: Error running SQL query; type='freetds', host='ip address', port='1433', user='sa', database='syslogng', error='', query='INSERT INTO test1 (datetime, host, program, pid, message) VALUES (\'Jul 12 08:59:46\', \'syslogng\', \'agetty\', \'25154\', \'/dev/xvc0: No such file or directory\')' Jul 12 09:57:36 syslogng syslog-ng[25034]: Error opening pipe, underlying file is not a FIFO, it should be used by file(); filename='/dev/tty10' Jul 12 09:57:36 syslogng syslog-ng[25034]: Error opening file for writing; filename='/dev/tty10', error='Illegal seek (29)' Jul 12 09:59:06 syslogng syslog-ng[25034]: Error running SQL query; type='freetds', host='ip address', port='1433', user='sa', database='syslogng', error='', query='INSERT INTO test1 (datetime, host, program, pid, message) VALUES (\'Jul 12 08:59:46\', \'syslogng\', \'agetty\', \'25154\', \'/dev/xvc0: No such file or directory\')' Jul 12 09:59:06 syslogng syslog-ng[25034]: Error inserting message into database. It is tried to insert 3 times, but it's failed. Message is dropped; Grateful if someone can help me figure out the problem Regards KP

2 4

ts_format(iso); frac_digits(6); == 000000+00:00 always ?
by Daniel Maher 13 Jul '10

13 Jul '10

Hello, I wish to enable the ISO timestamp format, and to have a granularity of six digits. I have enabled the following in my global options : frac_digits (6); ts_format(iso); The timestamp format is enabled as expected, but the fractional digits always appear as straight zeros. For example : 2010-07-12T16:49:27.000000+00:00 s_sys@srv-01 postfix/qmgr[2851]: 2EF1B1F9034E: removed Is there a way to enable the desired millisecond timestamps ? Details : $ uname -s -r -v -p -o Linux 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010 x86_64 GNU/Linux $ cat /etc/redhat-release CentOS release 5.5 (Final) $ rpm -qa | grep syslog-ng syslog-ng-2.1.4-1.el5 Thank you, and have an excellent day. -- Daniel Maher <dma PLUS syslogng AT witbe DOT net>

3 3

Syslog messages being received as kern/emerg MARK
by Lee, Steve 13 Jul '10

13 Jul '10

Good day, We are evaluating syslog-ng Premium using the Windows Agent sending syslogs back to a client on Linux. Everything was working fine for about a week. Now anytime a message is sent from the agent, it shows up on the receiving end as "[kern] [emerg] Jul 9 10:32:38 <IP Address> <server name> --- MARK ---", where MARK is actually in the message field. The messages should be coming in as local6/notice instead of kern/emerg. Everything seems to be setup correctly on the Windows agent. We have tried restarting the agent to no avail. The windows server shows nothing in the event log for the time that the MARK message comes across. If anyone has any insight into this problem, please let me know. Thanks. Steve ------------- Steve Lee Technical Operations Center University Technology Services Emory University ------------- ________________________________ This e-mail message (including any attachments) is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message (including any attachments) is strictly prohibited. If you have received this message in error, please contact the sender by reply e-mail message and destroy all copies of the original message (including attachments).

2 1

cannot find -lglib-2.0 while cross compiling
by Hasan Ümit Ezerçe 10 Jul '10

10 Jul '10

Hi, I need to run syslog-ng on an itanium RHEL 4.7 server but cannot find any RPM's for ia64 arch. So decided to cross compile it. I am trying to build it on a centos 4.7 VM and get the following error: ia64-unknown-linux-gnu-gcc -Os -Wall -o syslog-ng main.o libsyslog-ng.a -lnsl -lrt -Wl,-Bstatic -lglib-2.0 -levtlog -lnet -Wl,-Bdynamic -ldl /opt/crosstool/gcc-3.4.5-glibc-2.3.6/ia64-unknown-linux-gnu/lib/gcc/ia64-unknown-linux-gnu/3.4.5/../../../../ia64-unknown-linux-gnu/bin/ld: cannot find -lglib-2.0 collect2: ld returned 1 exit status However it seems that I already have glib-2.0 libs installed: $ pkg-config glib-2.0 --cflags --libs -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -lglib-2.0 Can you tell what am I missing here? Best regards.

3 2

syslog-ng 3.0.8 has been released
by devel＠balabit.hu 09 Jul '10

09 Jul '10

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.0.8 SUMMARY : new stable release DATE : Jul 9, 2010 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.0.8) has been released. For latest fixes in the 3.0.x branch you are recommended to upgrade to this version. CHANGES: 3.0.8 Fri, 09 Jul 2010 09:27:37 +0200 Bugfixes: * The flush_timeout() fix in 3.0.7 was not working properly for file() destinations. This has been fixed. * Fixed the use of the greedy and drop-invalid flags for csv-parser. * Fixed a possible but rare memory leak in the handling of message suppression. DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system…

1 0

Syslog-NG to parse HTTP query strings?
by Steve Barnes 09 Jul '10

09 Jul '10

Hello Is Syslog-NG capable of parsing HTTP query string key=value pairs? I've read through both the CSV and pattern DB parser documentation and it seems they both presume a consistent ordering in the $MSG field. Neither seem appropriate when if the order of query string pairs changes, such as: http://somehost.com/somefile.cgi?keyA=valueA&keyB=valueB or http://somehost.com/somefile.cgi?keyB=valueB&keyA=valueA Basically, I want to grab keyA's value irrespective of which order it appears and use it as a value in an SQL insert statement. Is this possible with Syslog-NG's native parsers or am I going to have to handle this in a Perl script/program? Cheers Steve

2 3

[Bug 90] New: csv-parser's flags(drop-invalid) doesn' t work with escape-{backslash, double-char}
by bugzilla＠bugzilla.balabit.com 09 Jul '10

09 Jul '10

https://bugzilla.balabit.com/show_bug.cgi?id=90 Summary: csv-parser's flags(drop-invalid) doesn't work with escape-{backslash,double-char} Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: andrewn(a)locus.net Type of the Report: bug Estimated Hours: 0.0 Created an attachment (id=19) --> (https://bugzilla.balabit.com/attachment.cgi?id=19) Possible fix The example parser p_apache in syslog-ng-v3.0-guide-admin-en.pdf on numbered page 63 does not work when drop-invalid is added. The 2 main loops of log_csv_parser_process seem to treat cur_column and src differently upon termination. The patch attached attempts to fix that. It also checks to see that the whole string was parsed. I don't know if this is the correct fix, but so far it seems to Work For Me (tm). -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

[Bug 86] New: flush_timeout has no effect
by bugzilla＠bugzilla.balabit.com 09 Jul '10

09 Jul '10

https://bugzilla.balabit.com/show_bug.cgi?id=86 Summary: flush_timeout has no effect Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 After upgrading to version 3.0.7 OSE the flush_timeout option has no effect for file destinations. The messages are buffered until flush_lines is reached and then they are flushed. TCP destinations are no affected by this issue. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 7

Files still opened after time_reap
by Erwan Miran 09 Jul '10

09 Jul '10

Hi, I'm using syslog-ng 2.0.0 (yes I know...) on a debian loghost. Files generated via df_monitor (see configuration extract below) are deleted by the monitoring process about 4 min after having been written by syslog-ng. My problem is that these files remain opened (lsof | grep deleted) by syslog-ng after deleted whereas time_reap option is set to 30s. I googled the issue and just found https://lists.balabit.hu/pipermail/syslog-ng/2003-January/004416.html which is not my exact case (macros are actually used in filenames). I didn't find any bugfix/feature on time_reap in the changelog of subsequent versions ... Thanks in advance for your help Regards Erwan options { chain_hostnames(0); time_reopen(10); time_reap(30); log_fifo_size(8192); create_dirs(yes); group(adm); perm(0660); dir_group(adm); dir_perm(0755); use_dns(no); log_msg_size(4096); use_fqdn(yes); keep_hostname(yes); use_time_recvd(no); }; source s_machines { tcp(ip("890.123.456.789") port(514) max_connections(100)); }; template tpl_monitor { template("$S_YEAR-$S_MONTH-$S_DAY $S_HOUR:$S_MIN:$S_SEC $FULLHOST ==$LEVEL== [$PROGRAM] $MESSAGE\n"); template_escape(no); }; destination df_monitor { file("/var/log/monitor/$R_YEAR-$R_MONTH-$R_DAY-$R_HOUR-$R_MIN" template(tpl_monitor) group(monitor) dir_perm(0775) dir_group(monitor) owner(loghost-maintainer)); }; destination dh_backup-loghost { tcp("123.456.789.890" port(514)); }; log{ source(s_machines); destination(df_monitor); destination(dh_backup-loghost); };

2 3

file() source driver on regular file
by Risto Vaarandi 07 Jul '10

07 Jul '10

hi all, is file() driver intended for reading kernel messages only on Linux, or does it work for regular files also? I've tried to get the following simple configuration working, but without success: source textfile { file("/home/risto/test.log" flags(no-parse) follow-freq(1)); }; destination testlog { file("/var/log/test.log"); }; log { source(textfile); destination(testlog); }; The source definition is identical to the example from the syslog-ng administrator guide (I've played with the options, but the result is the same). When syslog-ng starts up, it produces the following error messages: Follow-mode file source not found, deferring open; filename='/home/risto/test.log' The current log file has a mismatching size/inode information, restarting from the beginning; filename='/home/risto/test.log' If /home/risto/test.log does not exist, only the first message is produced. Is reading from regular files only supported by PE? (I am using the 3.0.7 open source version of syslog-ng.) kind regards, risto

2 1

The latest version of eventlog?
by Sergei Zhirikov 05 Jul '10

05 Jul '10

Hi, I'm wondering what is the latest version of the eventlog library? If I look here: http://www.balabit.com/downloads/files/eventlog/0.2/ then the latest seems to be 0.2.9. Although on my other computer I have 0.2.10 installed (built from source), but I can't remember where I got it from. Is there something missing on the download page? -- Thanks and regards, Sergei.

2 2