July 2010 - syslog-ng - lists.balabit.hu

Re: [syslog-ng] Quotes and parser with pattern db in syslog-ng 3.1.1
by Fekete Róbert 21 Jul '10

21 Jul '10

Hi, it seems to be a bit odd that your original pattern is not working, it might be some quirk with the STRING parser. However, using a QSTRING parser should be better in your case, as it is generally faster than the STRING parser, and more suitable for this message part, like this: <pattern>syslog-ng shutting down; version=@QSTRING::'@</pattern> Regards, Robert On Wednesday, July 21, 2010 18:07 CEST, "Ilas, Yann" <yann.ilas(a)eads.com> wrote: > I have a question about the parser using the db_parser and xml file. I'm > currently using the version 3.1.1 of syslog-ng. > > Here is the xml file named "syslog-ng.xml" : > <?xml version='1.0' encoding='UTF-8'?> > <patterndb version='3' pub_date='2010-07-21'> > <ruleset name='syslog-ng' id='syslog-ng-01'> > <pattern>syslog-ng</pattern> > <rules> > <rule provider='yann' id='syslog:server:0123456789:id001' > class='system'> > <patterns> > <pattern>syslog-ng shutting down; > version='@STRING::@'</pattern> > <examples> > <example> > <test_message>syslog-ng shutting down; > version='3.1.1'</test_message> > </example> > </examples> > <values> > <value name=".classifier.facility">syslog</value> > <value name=".classifier.severity">notice</value> > <value name=".classifier.priority">45</value> > </values> > </patterns> > </rule> > </rules> > </ruleset> > </patterndb> > > I would like to test that configuration with the following message > "syslog-ng shutting down; version='3.1.1'" by using pdbtool. > > # /opt/syslog-ng/bin/pdbtool match -p /tmp/syslog-ng.xml -P syslog-ng -M > "syslog-ng shutting down; version='3.1.1'" > MESSAGE=syslog-ng shutting down; version='3.1.1' > PROGRAM=syslog-ng > .classifier.class=unknown > > The result is odd because my message seems to be right... > I suspect the quotes so I remove the quotes in the xml file. > > Let's try again with the same message without quotes : > (...) > <pattern>syslog-ng shutting down; version=@STRING::@</pattern> > (...) > > Test: > # /opt/syslog-ng/bin/pdbtool match -p /tmp/syslog-ng.xml -P syslog-ng -M > "syslog-ng shutting down; version=3.1.1" > MESSAGE=syslog-ng shutting down; version=3.1.1 > PROGRAM=syslog-ng > .classifier.class=system > .classifier.rule_id=syslog:server:0123456789:id001 > .classifier.facility=syslog > .classifier.severity=notice > .classifier.priority=45 > > It works. > > What's wrong with my message ? > I tried to remove the second quote in the xml file like this : > (...) > <pattern>syslog-ng shutting down; version='@STRING::@</pattern> > (...) > > Test : > # /opt/syslog-ng/bin/pdbtool match -p /tmp/syslog-ng.xml -P syslog-ng -M > "syslog-ng shutting down; version='3.1.1" > MESSAGE=syslog-ng shutting down; version='3.1.1 > PROGRAM=syslog-ng > .classifier.class=system > .classifier.rule_id=syslog:server:0123456789:id001 > .classifier.facility=syslog > .classifier.severity=notice > .classifier.priority=45 > > I have the same result if I changed my xml file like this : > (...) > <pattern>syslog-ng shutting down; version=''@STRING::@</pattern> > (...) > > Test : > # /opt/syslog-ng/bin/pdbtool match -p /tmp/syslog-ng.xml -P syslog-ng -M > "syslog-ng shutting down; version=''3.1.1" > MESSAGE=syslog-ng shutting down; version=''3.1.1 > PROGRAM=syslog-ng > .classifier.class=system > .classifier.rule_id=syslog:server:0123456789:id001 > .classifier.facility=syslog > .classifier.severity=notice > .classifier.priority=45 > > Can't I have a message like this one 'texttexttext' ? Or did I miss > something ? > > Last test : I change the xml file like this : > (...) > <pattern>syslog-ng shutting down; version='3.1.1'</pattern> > (...) > => I remove the @STRING::@ and the pdbtool returned the > .classifier.facility, .classifier.severity, etc. > > May I use @STRING::@ between two simple quotes ? > > > Regards, > > Yann I.

1 0

Re: [syslog-ng] How to deal with duplicate log entries
by chris packham 21 Jul '10

21 Jul '10

Hi, I don't know what solaris does, but have you tried a simple config like this? log { source (s_streams); filter (f_mail); filter (f_debug); destination (l_syslog); }; log { source (s_streams); filter (f_auth); filter (f_info); destination (l_authlog); }; log { source (s_streams); destination (l_messages); }; If there are specific things you don't want in l_messages you can filter them out (as opposed to filtering stuff in). The right approach depends on how many messages you want (or don't want) in l_messages. Another trick I've seen to squash messages is using a destination that equates to '/dev/null' and the flags(final); setting to ditch messages that you don't want. >>> Chuck <chuck.carson(a)gmail.com> 07/21/10 12:01 PM >>> I have the following log statgements.. (Im basically trying to mirror what solaris 10's default syslogl.conf does): log { source (s_streams); filter (f_emerg); destination (l_messages); }; log { source (s_streams); filter (f_err); destination (l_messages); }; log { source (s_streams); filter (f_kern); filter (f_debug); destination (l_messages); }; log { source (s_streams); filter (f_daemon); filter (f_notice); destination (l_messages); }; log { source (s_streams); filter (f_mail); filter (f_crit); destination (l_messages); }; log { source (s_streams); filter (f_mail); filter (f_debug); destination (l_syslog); }; log { source (s_streams); filter (f_auth); filter (f_info); destination (l_authlog); }; # sshd logging However when sending to daemon.err I get duplicate messages... Should I change line 2 to this: log { source (s_streams); filter (f_err); filter (not f_daemon) destination (l_messages); }; Or should I hard-code every facility like so: log { source (s_streams); filter (f_err); filter (f_daemon) destination (l_messages); }; log { source (s_streams); filter (f_err); filter (f_mail) destination (l_messages); }; log { source (s_streams); filter (f_err); filter (f_auth) destination (l_messages); }; ... and so on? Thx,.CC

1 0

patterndb: grep on steroids
by Balazs Scheidler 20 Jul '10

20 Jul '10

Hi, This is a repost of my blog post at http://bazsi.blogs.balabit.com/ regarding patterndb and syslog-ng OSE 3.2. I felt it was important enough to crosspost it here. You may have heard of my last project to collect log samples from various applications, in order to convert log data from free-form human readable strings into structured information. The first round to collect login/logout messages from sshd is now complete. You could ask: ok, but what is the immediate benefit? You supposedly have a lot of unprocessed log files, and syslog-ng's db-parser() has not been used to process them, thus they are stored as good-old plain text files. I spent a couple of hours to add a "grep"-like functionality to pdbtool which makes it easy to process already existing log files, giving you immediate benefit for each and every sample added to patterndb. For example, if you are interested in login failure events, you could say: zcat logfile.gz | pdbtool match -p access/sshd.pdb \ --file - \ --filter 'tags("usracct") and match('REJECT' type(string) value("secevt.verdict"));' \ --template '${usracct.type},${secevt.verdict},${usracct.username}\n' What the command above does is the following: * reads a compressed logfile from logfile.gz * tells pdbtool to use access/sshd.pdb (in the patterndb git repo) as its pattern database file * tells pdbtool to read its stdin as a logfile, and * apply the db-parser() for each log message * apply the syslog-ng filter specified above * and print matching messages using the template also specified above As a combination, it results in a CSV file, containing login failure records found in the logfile. Also please note that as long there's a pattern in the pdb file, it doesn't really matter how that originally looked like, the fact that ssh can use 3-5 different messages for the same meaning is hidden nicely under the hood. And imagine we'd have patterns for all common applications running on our computers: this would mean that the same command above would produce login-failure reports independently from the application/OS combination being used. Try that with grep. :) This pdbtool is in the OSE 3.2 tree, clone the tree from: git://git.balabit.hu/bazsi/syslog-ng-3.2.git If you want to experiment with the OSE 3.2 tree, please read the OSE 3.2 caveats post at: http://bazsi.blogs.balabit.com/2010/07/syslog-ng-ose-32-caveats.html -- Bazsi

1 0

configuring different sources on the syslog-ng server
by Khaleelah Peerbocus 20 Jul '10

20 Jul '10

Hello I would like to get some clarification on the source tag in both syslog client and server. I have different clients hosting apache / weblogic My Syslog-ng Server has ip 192.x.x.x Apache (client) has ip 192.x.x.1 Weblogic (client) has ip 192.x.x.2 In all the clients syslog configuration files, i have added destination apacheaccesslog { tcp("192.x.x.x " port(3331)); }; destination weblogiclog { tcp("192.x.x.x " port(3332)); }; My issue is in the syslog-ng server configuration file, if i add source apache_access {tcp(ip(192.x.x.x) port(3331)); }; Should the ip be the server ip or the client ip (in this example, i have put the syslog-ng server ip). Does the port number differentiate the different sources we are receiving the log files ??? logo-emailsignature.jpgKhaleelah Peerbocus - Systems Analyst Mauritius Cargo Community Services Business Registration No.: C08077158 VAT Registration No.: VAT20427044 1st Floor, Trade and Marketing Centre, Mer Rouge Port-Louis, Mauritius Tel: +230 206 2970 - Cell: +230 498 7897 Fax: +230 216 8858 Email: <mailto:HDagent6@maccs.mu> systems2(a)maccs.mu - Website: <http://www.maccs.mu/> http://www.maccs.mu

3 6

[announcement] syslog-ng contributions redefined
by Balazs Scheidler 14 Jul '10

14 Jul '10

Dear syslog-ng users, This is a repost of my blog post at http://bazsi.blogs.balabit.com/ which I felt was important enough to repeat here. Links may not be properly reproduced, so please follow the link above if you'd like to browse the references as well. syslog-ng has been around for about 12 years now, but I think the biggest change in the project's life is imminent: with the upcoming release of syslog-ng OSE 3.2, syslog-ng will become an independent entity. Until now, syslog-ng was primarily maintained & developed by BalaBit, copyrights needed to be reassigned in order to grant BalaBit special privileges. BalaBit used her privileges to create a dual-licensed fork of syslog-ng, named "syslog-ng Premium Edition". The value we offer over the Open Source Edition of syslog-ng are things that larger enterprises require: * support on a large number of UNIX platforms (27 as of 3.1), * smaller and larger feature differences (like the encrypted/digitally signed logfile feature) * better test coverage and release management * longer term support Although perfectly legal, this business model was not welcome in various Free Software communities, and has caused friction and harm, because BalaBit has enjoyed a privilege that no others could get. We plan to fix this situation and we've worked hard in the past months to make this possible. We're letting syslog-ng go: no signed Contributory License Agreements will be required in order to contribute to syslog-ng in the future. This is not just a matter of policy: while BalaBit wants to be a true citizen of the Free Software world, we also need to ensure the continued revenue stream that the Premium Edition provides. The adjusted business model allows us to deliver what our customers need, and at the same time make it possible for anyone else in the community to have the same privileges as BalaBit has. The syslog-ng changes in 3.2 that I feel are most important are detailed below. Plugins syslog-ng was transformed from being a monolithic executable, to a core and a set of plugins. Plugins range from source and destination drivers, filters, parsers, rewrite objects, anything can be extended with the use of a relatively simple plugin. And this change didn't transform the configuration file format, it remained the same as before: readable and flexible. If you don't want to look under the hood, no functionality has changed, except plugins are loaded at runtime, and as a side effect, the syntax error reports are way better. The core is licensed under the LGPL and plugins are licensed under the GPL. This legal framework allows BalaBit to deliver value to its customers in non-free plugins and syslog-ng related services. syslog-ng Configuration Library syslog-ng is an infrastructure element: very flexible, but sometimes intimidating. For example, having to specify the pad_size() option for an HP-UX /dev/log device explicitly makes syslog-ng flexible, but not very user-friendly. It is common to share working configuration snippets on the mailing list or elsewhere in the syslog-ng community. We try to concentrate this effort with the creation of the syslog-ng Configuration Library (aka SCL). This library is a set of config snippets that can be included into a syslog-ng configuration file, using proper defaults but still allowing customization. For example: we've created a source driver named "system()" which automatically expands to the local log devices as needed by the current Operating System syslog-ng runs on, but it is also possible to create application specific configuration snippets (for example apache source?) and ship it as a ready to use config block. See this post for more information. This makes it possible to create a configuration file that will run on each of your platforms. In fact, we did that too: syslog-ng now comes with a default configuration file. Support for non-syslog messages Since syslog-ng is now plugin based and even the "syslog" message format as such is a plugin, it is now quite easy to add support for non-syslog message sources. As an example, we've added a plugin to parse Linux process accounting records, which makes it trivial to collect this data as well and possibly use it as a source of information when correllating data. Future plugins like creating a MIB aware SNMP listener, or possibly processing netflow data, but I'd like to create a generic SQL source as well. Support for these formats doesn't mean that syslog-ng would transform them into textual messages: proper name-value structure is kept and it is possible to put them into nicely structured SQL tables. Of course if all you want to transform them to text for readability, that's possible too with the use of a template(). patterndb project Like I announced before, we're starting a parallel project to create a set of message patterns directly usable with syslog-ng's db-parser(). Current state The changes are happening in the syslog-ng 3.2 repository at: http://git.balabit.hu/ We have created a source-only 3.2alpha1 release of the current state. It runs our automated test harness on Linux, but of course it is not yet recommended in production environments. The release can be downloaded from www.balabit.com. There were significant changes in how syslog-ng is compiled, thus it is expected that we have build issues on non-Linux systems. We expect to address these issues as they are found and create a 3.2beta1 release, for a longer beta testing. Future Parallel to fixing up the remaining issues on syslog-ng 3.2, we're going to open new ways to improve syslog-ng in the 3.3 branch. The most important item on our product backlog is to address the scalability problems and improve performance of syslog-ng. Summary With the upcoming changes in syslog-ng OSE 3.2 possible contributions will be greatly expanded: * write a pattern for your favourite application and process log data faster + easier * write an SCL configuration snippet, make your application easier to integrate with syslog-ng * write a plugin for your favourite NoSQL database, * write a plugin for a transformation that syslog-ng is not capable of doing right now (what about facility / severity rewrite rules? easy as a piece of cake) * write a plugin for things that you do with an external script * or contribute to the core of syslog-ng. And all this without having to sign paperwork... -- Bazsi

1 0

monitoring SQL-Server-ERRORLOG
by Schöppel Marcus 14 Jul '10

14 Jul '10

Hi all@syslog-ng-mailing-list! I'm trying to monitor the MSSQLServer ERROR-logfile but I get a logentry for every single letter that is added to the logfile: -> May 18 13:57:40 test mssql: L May 18 13:57:40 test mssql: o May 18 13:57:40 test mssql: g May 18 13:57:40 test mssql: i May 18 13:57:40 test mssql: n May 18 13:57:40 test mssql: May 18 13:57:40 test mssql: f May 18 13:57:40 test mssql: a May 18 13:57:40 test mssql: i May 18 13:57:40 test mssql: l May 18 13:57:40 test mssql: e May 18 13:57:40 test mssql: d <- This seems to stem from the encoding of the file (UTF-16LE) because with other logfiles this problem doesn't arise. I suppose it would help if the "encoding"-Parameter of the file-source would work but syslog-ng/cygwin (Version 3.0.1) won't start when I add it no matter which encoding I use. I tried this workaround-cronjob: - copy file somewhere - convert to UTF-8 using iconv - give the same timestamp to the converted logfile so that syslog-ng doesn't transfer the whole file again - monitoring the converted file with syslog-ng This doesn't work completely correct as despite assigning the same timestamp the whole file is sometimes transferred again (and not just the new entries). Is this problem (following this UTF-16LE-Windowsfile (easily)) not resolvable by using the cygwin-version of syslog-ng?? Or am I maybe using a wrong encoding spelling? Does it work using the commercial windows-version of the agent?

2 1

[announce] patterndb project
by Balazs Scheidler 13 Jul '10

13 Jul '10

Hi, By now probably most of you know about patterndb, a powerful framework in syslog-ng that lets you extract structured information from log messages and perform classification at a high speed: http://www.balabit.com/dl/html/syslog-ng-ose-v3.1-guide-admin-en.html/conce… Until now, syslog-ng offered the feature, but no release-quality patterns were produced by the syslog-ng developers. Some samples based on the logcheck database were created, but otherwise every syslog-ng user had to create her samples manually, possibly repeating work performed by others. Since this calls out to be a community project, I'm hereby starting one. Goals ===== Create release-quality pattern databases that can simply be deployed to an existing syslog-ng installation. The goal of the patterns is to extract structured information from the free-form syslog messages, e.g. create name-value pairs based on the syslog message. Since the key factor when doing something like this is the naming of fields, we're going to create our generic naming guidelines that can be applied to any application in the industry. It is not our goal to implement correllation or any other advanced form of analysis, although we feel that with the results of this project, event correllation and analysis can be performed much easier than without it. Related projects ================ I know there are other efforts in the field, why not simply join them? CEF - is the log message format for a proprietary log analysis engine, primarily meant to be used to hold IP security device logs (firewalls, IPSs, virus gateways etc). The patterndb project aims to create patterns for a wider range of device logs and be more generic in the approach. On the other hand we feel that it might be useful to create a solution for converting db-parser output to the CEF format. CEE - Common Event Expression project by Mitre has a focus on creating a nv pair dictionary for all kinds of devices/log messages out there. Although I might be missing something, but I didn't find the concrete results so far, apart from a nicely looking white paper. If the CEE delivers something, then patterndb would probably adapt the naming/taxonomy structure. But I guess not all devices will start logging in the new shiny format, thus the existing devices would need their logs converted, so the patterndb work wouldn't be wasted. Infrastructure ============== Our original patterndb related plans were to create an easy to use web based interface for editing patterns, but since that project is progressing slowly, I'm calling for a minimalist approach: git based version control of simple plain text files. Of course once the nice web based interface is finished, we're going to be ready to use it. First steps =========== I have created a git repository at: http://git.balabit.hu/bazsi/syslog-ng-patterndb.git This contains the initial version of the naming policy document and a simple schema for SIEM-style and a user login-logout naming schema. If you are interested please read the file README.txt in the git archive, or if you prefer a web browser, use this link: http://git.balabit.hu/?p=bazsi/syslog-ng-patterndb.git;a=blob;f=README.txt;… Licensing ========= I do not have a decision yet, but for sure this is going to use one of the open source licenses or Creative Commons. Let me know if you have a preference in this area. Getting involved ================ Join the syslog-ng mailing list, a start discussing! If you have existing patterns, great. If you don't, it is not late to join. http://lists.balabit.hu/mailman/listinfo/syslog-ng -- Bazsi

3 20

Syslog-ng 3.0.7 fails to insert into microsoft sql server
by Khaleelah Peerbocus 13 Jul '10

13 Jul '10

Dear All, Im currently configuring syslog-ng on SUSE Linux 10 SP2 to communicate with microsoft sql server 2005. I have installed freeTds 8.0. Tested TSQL ..works fine ISQL works fine. when testing with syslog-ng Conf file destination d_mssql { sql(type(mssql) host("ip address") port("1433") username("sa") password("password") database("syslogng") table("test1") columns("datetime varchar(16)","host varchar(32)" ,"program varchar(32)","pid varchar(8)","message varchar(4096)") values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY") indexes() ); }; log { source(src); destination(d_mssql); }; I get the following error Note I created the table test1 manually in SQL server Code: Jul 12 09:57:06 syslogng syslog-ng[25034]: Error running SQL query; type='freetds', host='ip address', port='1433', user='sa', database='syslogng', error='', query='INSERT INTO test1 (datetime, host, program, pid, message) VALUES (\'Jul 12 08:59:46\', \'syslogng\', \'agetty\', \'25154\', \'/dev/xvc0: No such file or directory\')' Jul 12 09:57:36 syslogng syslog-ng[25034]: Error opening pipe, underlying file is not a FIFO, it should be used by file(); filename='/dev/tty10' Jul 12 09:57:36 syslogng syslog-ng[25034]: Error opening file for writing; filename='/dev/tty10', error='Illegal seek (29)' Jul 12 09:59:06 syslogng syslog-ng[25034]: Error running SQL query; type='freetds', host='ip address', port='1433', user='sa', database='syslogng', error='', query='INSERT INTO test1 (datetime, host, program, pid, message) VALUES (\'Jul 12 08:59:46\', \'syslogng\', \'agetty\', \'25154\', \'/dev/xvc0: No such file or directory\')' Jul 12 09:59:06 syslogng syslog-ng[25034]: Error inserting message into database. It is tried to insert 3 times, but it's failed. Message is dropped; Grateful if someone can help me figure out the problem Regards KP

2 4

ts_format(iso); frac_digits(6); == 000000+00:00 always ?
by Daniel Maher 13 Jul '10

13 Jul '10

Hello, I wish to enable the ISO timestamp format, and to have a granularity of six digits. I have enabled the following in my global options : frac_digits (6); ts_format(iso); The timestamp format is enabled as expected, but the fractional digits always appear as straight zeros. For example : 2010-07-12T16:49:27.000000+00:00 s_sys@srv-01 postfix/qmgr[2851]: 2EF1B1F9034E: removed Is there a way to enable the desired millisecond timestamps ? Details : $ uname -s -r -v -p -o Linux 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010 x86_64 GNU/Linux $ cat /etc/redhat-release CentOS release 5.5 (Final) $ rpm -qa | grep syslog-ng syslog-ng-2.1.4-1.el5 Thank you, and have an excellent day. -- Daniel Maher <dma PLUS syslogng AT witbe DOT net>

3 3

Syslog messages being received as kern/emerg MARK
by Lee, Steve 13 Jul '10

13 Jul '10

Good day, We are evaluating syslog-ng Premium using the Windows Agent sending syslogs back to a client on Linux. Everything was working fine for about a week. Now anytime a message is sent from the agent, it shows up on the receiving end as "[kern] [emerg] Jul 9 10:32:38 <IP Address> <server name> --- MARK ---", where MARK is actually in the message field. The messages should be coming in as local6/notice instead of kern/emerg. Everything seems to be setup correctly on the Windows agent. We have tried restarting the agent to no avail. The windows server shows nothing in the event log for the time that the MARK message comes across. If anyone has any insight into this problem, please let me know. Thanks. Steve ------------- Steve Lee Technical Operations Center University Technology Services Emory University ------------- ________________________________ This e-mail message (including any attachments) is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message (including any attachments) is strictly prohibited. If you have received this message in error, please contact the sender by reply e-mail message and destroy all copies of the original message (including attachments).

2 1