- syslog-ng - lists.balabit.hu

[announcement] syslog-ng contributions redefined
by Balazs Scheidler 14 Jul '10

14 Jul '10

Dear syslog-ng users, This is a repost of my blog post at http://bazsi.blogs.balabit.com/ which I felt was important enough to repeat here. Links may not be properly reproduced, so please follow the link above if you'd like to browse the references as well. syslog-ng has been around for about 12 years now, but I think the biggest change in the project's life is imminent: with the upcoming release of syslog-ng OSE 3.2, syslog-ng will become an independent entity. Until now, syslog-ng was primarily maintained & developed by BalaBit, copyrights needed to be reassigned in order to grant BalaBit special privileges. BalaBit used her privileges to create a dual-licensed fork of syslog-ng, named "syslog-ng Premium Edition". The value we offer over the Open Source Edition of syslog-ng are things that larger enterprises require: * support on a large number of UNIX platforms (27 as of 3.1), * smaller and larger feature differences (like the encrypted/digitally signed logfile feature) * better test coverage and release management * longer term support Although perfectly legal, this business model was not welcome in various Free Software communities, and has caused friction and harm, because BalaBit has enjoyed a privilege that no others could get. We plan to fix this situation and we've worked hard in the past months to make this possible. We're letting syslog-ng go: no signed Contributory License Agreements will be required in order to contribute to syslog-ng in the future. This is not just a matter of policy: while BalaBit wants to be a true citizen of the Free Software world, we also need to ensure the continued revenue stream that the Premium Edition provides. The adjusted business model allows us to deliver what our customers need, and at the same time make it possible for anyone else in the community to have the same privileges as BalaBit has. The syslog-ng changes in 3.2 that I feel are most important are detailed below. Plugins syslog-ng was transformed from being a monolithic executable, to a core and a set of plugins. Plugins range from source and destination drivers, filters, parsers, rewrite objects, anything can be extended with the use of a relatively simple plugin. And this change didn't transform the configuration file format, it remained the same as before: readable and flexible. If you don't want to look under the hood, no functionality has changed, except plugins are loaded at runtime, and as a side effect, the syntax error reports are way better. The core is licensed under the LGPL and plugins are licensed under the GPL. This legal framework allows BalaBit to deliver value to its customers in non-free plugins and syslog-ng related services. syslog-ng Configuration Library syslog-ng is an infrastructure element: very flexible, but sometimes intimidating. For example, having to specify the pad_size() option for an HP-UX /dev/log device explicitly makes syslog-ng flexible, but not very user-friendly. It is common to share working configuration snippets on the mailing list or elsewhere in the syslog-ng community. We try to concentrate this effort with the creation of the syslog-ng Configuration Library (aka SCL). This library is a set of config snippets that can be included into a syslog-ng configuration file, using proper defaults but still allowing customization. For example: we've created a source driver named "system()" which automatically expands to the local log devices as needed by the current Operating System syslog-ng runs on, but it is also possible to create application specific configuration snippets (for example apache source?) and ship it as a ready to use config block. See this post for more information. This makes it possible to create a configuration file that will run on each of your platforms. In fact, we did that too: syslog-ng now comes with a default configuration file. Support for non-syslog messages Since syslog-ng is now plugin based and even the "syslog" message format as such is a plugin, it is now quite easy to add support for non-syslog message sources. As an example, we've added a plugin to parse Linux process accounting records, which makes it trivial to collect this data as well and possibly use it as a source of information when correllating data. Future plugins like creating a MIB aware SNMP listener, or possibly processing netflow data, but I'd like to create a generic SQL source as well. Support for these formats doesn't mean that syslog-ng would transform them into textual messages: proper name-value structure is kept and it is possible to put them into nicely structured SQL tables. Of course if all you want to transform them to text for readability, that's possible too with the use of a template(). patterndb project Like I announced before, we're starting a parallel project to create a set of message patterns directly usable with syslog-ng's db-parser(). Current state The changes are happening in the syslog-ng 3.2 repository at: http://git.balabit.hu/ We have created a source-only 3.2alpha1 release of the current state. It runs our automated test harness on Linux, but of course it is not yet recommended in production environments. The release can be downloaded from www.balabit.com. There were significant changes in how syslog-ng is compiled, thus it is expected that we have build issues on non-Linux systems. We expect to address these issues as they are found and create a 3.2beta1 release, for a longer beta testing. Future Parallel to fixing up the remaining issues on syslog-ng 3.2, we're going to open new ways to improve syslog-ng in the 3.3 branch. The most important item on our product backlog is to address the scalability problems and improve performance of syslog-ng. Summary With the upcoming changes in syslog-ng OSE 3.2 possible contributions will be greatly expanded: * write a pattern for your favourite application and process log data faster + easier * write an SCL configuration snippet, make your application easier to integrate with syslog-ng * write a plugin for your favourite NoSQL database, * write a plugin for a transformation that syslog-ng is not capable of doing right now (what about facility / severity rewrite rules? easy as a piece of cake) * write a plugin for things that you do with an external script * or contribute to the core of syslog-ng. And all this without having to sign paperwork... -- Bazsi

1 0

monitoring SQL-Server-ERRORLOG
by Schöppel Marcus 14 Jul '10

14 Jul '10

Hi all@syslog-ng-mailing-list! I'm trying to monitor the MSSQLServer ERROR-logfile but I get a logentry for every single letter that is added to the logfile: -> May 18 13:57:40 test mssql: L May 18 13:57:40 test mssql: o May 18 13:57:40 test mssql: g May 18 13:57:40 test mssql: i May 18 13:57:40 test mssql: n May 18 13:57:40 test mssql: May 18 13:57:40 test mssql: f May 18 13:57:40 test mssql: a May 18 13:57:40 test mssql: i May 18 13:57:40 test mssql: l May 18 13:57:40 test mssql: e May 18 13:57:40 test mssql: d <- This seems to stem from the encoding of the file (UTF-16LE) because with other logfiles this problem doesn't arise. I suppose it would help if the "encoding"-Parameter of the file-source would work but syslog-ng/cygwin (Version 3.0.1) won't start when I add it no matter which encoding I use. I tried this workaround-cronjob: - copy file somewhere - convert to UTF-8 using iconv - give the same timestamp to the converted logfile so that syslog-ng doesn't transfer the whole file again - monitoring the converted file with syslog-ng This doesn't work completely correct as despite assigning the same timestamp the whole file is sometimes transferred again (and not just the new entries). Is this problem (following this UTF-16LE-Windowsfile (easily)) not resolvable by using the cygwin-version of syslog-ng?? Or am I maybe using a wrong encoding spelling? Does it work using the commercial windows-version of the agent?

2 1

[announce] patterndb project
by Balazs Scheidler 13 Jul '10

13 Jul '10

Hi, By now probably most of you know about patterndb, a powerful framework in syslog-ng that lets you extract structured information from log messages and perform classification at a high speed: http://www.balabit.com/dl/html/syslog-ng-ose-v3.1-guide-admin-en.html/conce… Until now, syslog-ng offered the feature, but no release-quality patterns were produced by the syslog-ng developers. Some samples based on the logcheck database were created, but otherwise every syslog-ng user had to create her samples manually, possibly repeating work performed by others. Since this calls out to be a community project, I'm hereby starting one. Goals ===== Create release-quality pattern databases that can simply be deployed to an existing syslog-ng installation. The goal of the patterns is to extract structured information from the free-form syslog messages, e.g. create name-value pairs based on the syslog message. Since the key factor when doing something like this is the naming of fields, we're going to create our generic naming guidelines that can be applied to any application in the industry. It is not our goal to implement correllation or any other advanced form of analysis, although we feel that with the results of this project, event correllation and analysis can be performed much easier than without it. Related projects ================ I know there are other efforts in the field, why not simply join them? CEF - is the log message format for a proprietary log analysis engine, primarily meant to be used to hold IP security device logs (firewalls, IPSs, virus gateways etc). The patterndb project aims to create patterns for a wider range of device logs and be more generic in the approach. On the other hand we feel that it might be useful to create a solution for converting db-parser output to the CEF format. CEE - Common Event Expression project by Mitre has a focus on creating a nv pair dictionary for all kinds of devices/log messages out there. Although I might be missing something, but I didn't find the concrete results so far, apart from a nicely looking white paper. If the CEE delivers something, then patterndb would probably adapt the naming/taxonomy structure. But I guess not all devices will start logging in the new shiny format, thus the existing devices would need their logs converted, so the patterndb work wouldn't be wasted. Infrastructure ============== Our original patterndb related plans were to create an easy to use web based interface for editing patterns, but since that project is progressing slowly, I'm calling for a minimalist approach: git based version control of simple plain text files. Of course once the nice web based interface is finished, we're going to be ready to use it. First steps =========== I have created a git repository at: http://git.balabit.hu/bazsi/syslog-ng-patterndb.git This contains the initial version of the naming policy document and a simple schema for SIEM-style and a user login-logout naming schema. If you are interested please read the file README.txt in the git archive, or if you prefer a web browser, use this link: http://git.balabit.hu/?p=bazsi/syslog-ng-patterndb.git;a=blob;f=README.txt;… Licensing ========= I do not have a decision yet, but for sure this is going to use one of the open source licenses or Creative Commons. Let me know if you have a preference in this area. Getting involved ================ Join the syslog-ng mailing list, a start discussing! If you have existing patterns, great. If you don't, it is not late to join. http://lists.balabit.hu/mailman/listinfo/syslog-ng -- Bazsi

3 20

Syslog-ng 3.0.7 fails to insert into microsoft sql server
by Khaleelah Peerbocus 13 Jul '10

13 Jul '10

Dear All, Im currently configuring syslog-ng on SUSE Linux 10 SP2 to communicate with microsoft sql server 2005. I have installed freeTds 8.0. Tested TSQL ..works fine ISQL works fine. when testing with syslog-ng Conf file destination d_mssql { sql(type(mssql) host("ip address") port("1433") username("sa") password("password") database("syslogng") table("test1") columns("datetime varchar(16)","host varchar(32)" ,"program varchar(32)","pid varchar(8)","message varchar(4096)") values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY") indexes() ); }; log { source(src); destination(d_mssql); }; I get the following error Note I created the table test1 manually in SQL server Code: Jul 12 09:57:06 syslogng syslog-ng[25034]: Error running SQL query; type='freetds', host='ip address', port='1433', user='sa', database='syslogng', error='', query='INSERT INTO test1 (datetime, host, program, pid, message) VALUES (\'Jul 12 08:59:46\', \'syslogng\', \'agetty\', \'25154\', \'/dev/xvc0: No such file or directory\')' Jul 12 09:57:36 syslogng syslog-ng[25034]: Error opening pipe, underlying file is not a FIFO, it should be used by file(); filename='/dev/tty10' Jul 12 09:57:36 syslogng syslog-ng[25034]: Error opening file for writing; filename='/dev/tty10', error='Illegal seek (29)' Jul 12 09:59:06 syslogng syslog-ng[25034]: Error running SQL query; type='freetds', host='ip address', port='1433', user='sa', database='syslogng', error='', query='INSERT INTO test1 (datetime, host, program, pid, message) VALUES (\'Jul 12 08:59:46\', \'syslogng\', \'agetty\', \'25154\', \'/dev/xvc0: No such file or directory\')' Jul 12 09:59:06 syslogng syslog-ng[25034]: Error inserting message into database. It is tried to insert 3 times, but it's failed. Message is dropped; Grateful if someone can help me figure out the problem Regards KP

2 4

ts_format(iso); frac_digits(6); == 000000+00:00 always ?
by Daniel Maher 13 Jul '10

13 Jul '10

Hello, I wish to enable the ISO timestamp format, and to have a granularity of six digits. I have enabled the following in my global options : frac_digits (6); ts_format(iso); The timestamp format is enabled as expected, but the fractional digits always appear as straight zeros. For example : 2010-07-12T16:49:27.000000+00:00 s_sys@srv-01 postfix/qmgr[2851]: 2EF1B1F9034E: removed Is there a way to enable the desired millisecond timestamps ? Details : $ uname -s -r -v -p -o Linux 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010 x86_64 GNU/Linux $ cat /etc/redhat-release CentOS release 5.5 (Final) $ rpm -qa | grep syslog-ng syslog-ng-2.1.4-1.el5 Thank you, and have an excellent day. -- Daniel Maher <dma PLUS syslogng AT witbe DOT net>

3 3

Syslog messages being received as kern/emerg MARK
by Lee, Steve 13 Jul '10

13 Jul '10

Good day, We are evaluating syslog-ng Premium using the Windows Agent sending syslogs back to a client on Linux. Everything was working fine for about a week. Now anytime a message is sent from the agent, it shows up on the receiving end as "[kern] [emerg] Jul 9 10:32:38 <IP Address> <server name> --- MARK ---", where MARK is actually in the message field. The messages should be coming in as local6/notice instead of kern/emerg. Everything seems to be setup correctly on the Windows agent. We have tried restarting the agent to no avail. The windows server shows nothing in the event log for the time that the MARK message comes across. If anyone has any insight into this problem, please let me know. Thanks. Steve ------------- Steve Lee Technical Operations Center University Technology Services Emory University ------------- ________________________________ This e-mail message (including any attachments) is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message (including any attachments) is strictly prohibited. If you have received this message in error, please contact the sender by reply e-mail message and destroy all copies of the original message (including attachments).

2 1

cannot find -lglib-2.0 while cross compiling
by Hasan Ümit Ezerçe 10 Jul '10

10 Jul '10

Hi, I need to run syslog-ng on an itanium RHEL 4.7 server but cannot find any RPM's for ia64 arch. So decided to cross compile it. I am trying to build it on a centos 4.7 VM and get the following error: ia64-unknown-linux-gnu-gcc -Os -Wall -o syslog-ng main.o libsyslog-ng.a -lnsl -lrt -Wl,-Bstatic -lglib-2.0 -levtlog -lnet -Wl,-Bdynamic -ldl /opt/crosstool/gcc-3.4.5-glibc-2.3.6/ia64-unknown-linux-gnu/lib/gcc/ia64-unknown-linux-gnu/3.4.5/../../../../ia64-unknown-linux-gnu/bin/ld: cannot find -lglib-2.0 collect2: ld returned 1 exit status However it seems that I already have glib-2.0 libs installed: $ pkg-config glib-2.0 --cflags --libs -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -lglib-2.0 Can you tell what am I missing here? Best regards.

3 2

syslog-ng 3.0.8 has been released
by devel＠balabit.hu 09 Jul '10

09 Jul '10

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.0.8 SUMMARY : new stable release DATE : Jul 9, 2010 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.0.8) has been released. For latest fixes in the 3.0.x branch you are recommended to upgrade to this version. CHANGES: 3.0.8 Fri, 09 Jul 2010 09:27:37 +0200 Bugfixes: * The flush_timeout() fix in 3.0.7 was not working properly for file() destinations. This has been fixed. * Fixed the use of the greedy and drop-invalid flags for csv-parser. * Fixed a possible but rare memory leak in the handling of message suppression. DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system…

1 0

Syslog-NG to parse HTTP query strings?
by Steve Barnes 09 Jul '10

09 Jul '10

Hello Is Syslog-NG capable of parsing HTTP query string key=value pairs? I've read through both the CSV and pattern DB parser documentation and it seems they both presume a consistent ordering in the $MSG field. Neither seem appropriate when if the order of query string pairs changes, such as: http://somehost.com/somefile.cgi?keyA=valueA&keyB=valueB or http://somehost.com/somefile.cgi?keyB=valueB&keyA=valueA Basically, I want to grab keyA's value irrespective of which order it appears and use it as a value in an SQL insert statement. Is this possible with Syslog-NG's native parsers or am I going to have to handle this in a Perl script/program? Cheers Steve

2 3

[Bug 90] New: csv-parser's flags(drop-invalid) doesn' t work with escape-{backslash, double-char}
by bugzilla＠bugzilla.balabit.com 09 Jul '10

09 Jul '10

https://bugzilla.balabit.com/show_bug.cgi?id=90 Summary: csv-parser's flags(drop-invalid) doesn't work with escape-{backslash,double-char} Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: andrewn(a)locus.net Type of the Report: bug Estimated Hours: 0.0 Created an attachment (id=19) --> (https://bugzilla.balabit.com/attachment.cgi?id=19) Possible fix The example parser p_apache in syslog-ng-v3.0-guide-admin-en.pdf on numbered page 63 does not work when drop-invalid is added. The 2 main loops of log_csv_parser_process seem to treat cur_column and src differently upon termination. The patch attached attempts to fix that. It also checks to see that the whole string was parsed. I don't know if this is the correct fix, but so far it seems to Work For Me (tm). -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5