January 2012 - syslog-ng - lists.balabit.hu

[Bug 142] New: --worker-threads=0 support
by bugzilla＠bugzilla.balabit.com 25 Jan '12

25 Jan '12

https://bugzilla.balabit.com/show_bug.cgi?id=142 Summary: --worker-threads=0 support Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: arekm(a)maven.pl Type of the Report: enhancement Estimated Hours: 0.0 Hello, I was hoping that threading support would solve my problem described in https://bugzilla.balabit.com/show_bug.cgi?id=113. Unfortunately it looks like it is impossible to create configuration for which each target will use one thread by default (as only such configuration is safe in case of some targets blocking on I/O). This feature request is about adding some option like: --worker-threads=0 that would limit number of threads to numer of sources + number of destinations, so each possible blocking I/O would be happening in separate thread. Yes, it would be more than cpu cores but this configuration is about reliability and not performance. (right now I have only one, network logging target that needs to always work, so probably disabling threading globally and enabling threading only for this network target would get me "always working" logging via net, even if some infinite I/O blocking happens. What would be best is to have ability to have such config on by default for all targets) -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 12

syslog-ng 3.3.4 has been released
by devel＠balabit.hu 24 Jan '12

24 Jan '12

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.3.4 SUMMARY : new stable release DATE : Jan 19, 2012 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.3.4) has been released. For latest fixes in the 3.3.x feature branch you are recommended to upgrade to this version. CHANGES: 3.3.4 Mon, 16 Jan 2012 23:07:46 +0100 Highlights ========== This release is aimed fixing bugs in the 3.3.x series, with one more important change: there's an incompatible change in the processing of <action> tags in the db-parser() database format (aka patterndb), see below for more details. Bugfixes: ========= * Fixed set() and subst() rewrite operations to work properly on the value() parameter specified in the configuration even if they are referenced at multiple spots in the configuration file. Earlier the 2nd and subsequent invocation of the rewrite rule changed $MESSAGE. * Fixed csv-parser() to work even if it is invoked at multiple spots in the configuration file. Earlier, the 2nd and subsequent references of the parser rule forgot the list of column names and the input template. * Fixed the processing of condition() parameter in rewrite rules, which was broken if it contained a filter() function call. * Fixed program() destination to properly kill the child process on reload and shutdown. * Fixed a potential division by zero error which could happen for large data rates due to a race in an unlocked region. * Fixed an assertion failure in mongodb destination that happened due to a race condition at high data rates. * Fixed an fd leak in the control socket code, that caused the control connection file descriptors to be leaked. Other changes ============= * db-parser() automatically sets a tag named '.classifier.unknown' if the message doesn't match. * The use of actions in db-parser() for messages without a correllation context was inconsistently indexing messages. For actions in rules that had correllation @0 was the new message being generated, and @1 was the message that triggered the rule. Without correllation @0 was used for the triggering message, which is greatly inconsistent and unintuitive. This was fixed by changing the behaviour for rules without correllation, now both correllation and non-correllation rules use @0 for the new message, and @1 for the triggering message. This is an incompatible change in the db-parser() format. * The value of the $TAGS macro is added to pdbtool match output. * unix-dgram() and unix-stream() error logging on systemd failures became more detailed for easier troubleshooting. Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Attila M. Magyar (BalaBit) Balazs Scheidler (BalaBit) Costa Farber Dmitry Gilev Evan Rempel (University of Victoria) Fried Zoltan (BalaBit) Gergely Nagy (BalaBit) Jakub Jankowski (SuperHost.pl) John Morrissey (Horde.net) Michael Hocke (New York University) Michal Schmidt (RedHat) Peter Eisenlohr (Inform Software) Svante Signell (Telia.se) Thomas Wollner DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

5 6

sql scaling commits idea
by Patrick Hemmer 24 Jan '12

24 Jan '12

A technique that we use for an in-house application I wrote -- that writes log entries to a database -- I think would be very useful for syslog-ng in high-performance setups. The idea is basically to commit as frequently as the backend database can handle. The way it works is that you basically commit when your queue is completely empty. So if you have a low volume of logs coming it, it would commit after every log entry. This is quite demanding on the database for high volumes, so as the database loses the ability to keep up, the queue in syslog-ng starts to build and so it starts to commit less. When it starts doing more log entries per commit/transaction, performance on the database increases. So you end up writing out to the database as fast as the database can handle, with as little data loss as possible were syslog-ng to die somehow. You would also need some sort of max-transaction-size so you dont end up with too many uncommited log entries if the database is being really really slow. As mentioned I've been using this technique in our in-house application for a couple years now and it performs beautifully. Very low latency between syslog-ng getting the message, and it being available (for a select query) in the database, without overwhelming the database. In theory you could also use this same principle for file destinations as well, but I dont think it'd be as useful there. But anyway, just an idea :-)

2 4

[Bug 158] New: syslog-ng tries to add already existing column when using sqlite3
by bugzilla＠bugzilla.balabit.com 23 Jan '12

23 Jan '12

https://bugzilla.balabit.com/show_bug.cgi?id=158 Summary: syslog-ng tries to add already existing column when using sqlite3 Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.syslogng(a)feystorm.net Type of the Report: --- Estimated Hours: 0.0 I'm trying to use an sqlite3 database through the sql driver, however syslog-ng seems to be unable to verify the table structure and is trying to add a column which already exists, and so the add fails. 2012-01-21 18:58:10 syslog-ng[14564]: Error running SQL query; type='sqlite3', host='', port='', user='syslog-ng', database='/var/log/logs.sql3', error='1: duplicate column name: time', query='ALTER TABLE logs ADD time text' 2012-01-21 18:58:10 syslog-ng[14564]: Error adding missing column, giving up; table='logs', column='time' 2012-01-21 18:58:10 syslog-ng[14564]: Error checking table, disconnecting from database, trying again shortly; time_reopen='60' Relevant portion of my config destination d_sqlite { sql( type('sqlite3') database("/var/log/logs.sql3") table("logs") columns("time", "time_r", "host", "facility", "priority", "program", "pid", "tag", "message") values("$S_UNIXTIME", "$R_UNIXTIME", "$FULLHOST", "$FACILITY_NUM", "$PRIORITY", "$PROGRAM", "$PID", "$DBTAG", "$MSG") null("") flags(explicit-commits) flush_lines(10) flush_timeout(200) ); }; My database schema CREATE TABLE logs (id integer primary key autoincrement, time unsigned integer not null, time_r unsigned integer not null, host text not null, facility unsigned integer not null, priority unsigned integer not null, program text not null, pid unsigned integer, tag text, message text not null); This is occurring with syslog-ng 3.3.4 and libdbi 0.8.3 -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

Re: [syslog-ng] Forward syslog-ng messages
by Michael Kolowicz 21 Jan '12

21 Jan '12

> Please verify first that syslog-ng actually gets the incoming traffic. The incoming traffic will be regnozie and processed. I have configure to write a database and to a local file - it works fine But not the forwarding - BTW: The address is be changed, why I use another machine to test my application ;-) Here my full config as reference @version: 3.2 #ADM_NTINST @ 11:45 20.01.2012 source s_udp {udp(port(514));}; destination d_SYSLOG { file("/var/log/syslog");}; destination d_udp1514 {udp("192.168.2.108" port (1514));}; destination SUA002254_localhost_SYS_SYSLOG_uaagent {sql( type(mysql) host("localhost") username("USR_SYS_SYSLOG") password("pw_Syslog") database("SYS_SYSLOG") table("uaagent") columns("host", "source_ip", "facility", "priority", "tag", "unixtime", "program", "msg") values("$HOST","$SOURCEIP","$FACILITY","$PRIORITY","$TAG", "$UNIXTIME","$PROGRAM","$MSGONLY") indexes("host", "source_ip", "facility", "priority", "tag", "unixtime", "program", "msg"));}; #log_uaagent log { source(s_udp); destination(d_udp1514); destination(SUA002254_localhost_SYS_SYSLOG_uaagent); destination(d_SYSLOG); };

1 0

Re: [syslog-ng] Forwarding Syslog-ng Messages
by Michael Kolowicz 20 Jan '12

20 Jan '12

> The config has syntax errors, it should read like this: > > destination d_udp1514 {udp("192.168.2.108" port (1514));}; > > keep-hostname() is not a valid destination option, it is related to source > drivers instead OK - Thanks for you hint - I have change it - but no success. The messages come not to my Application LG

2 1

JSON vs PatternDB for parser performance
by Martin Holste 20 Jan '12

20 Jan '12

Does anyone have a good idea as to which parser is fastest? The pattern matching tree of PatternDB makes me think that it will be faster, but I'd like to see what others think. Testing in my environment would be non-trivial, so I am hoping someone already has. Thanks, Martin

2 3

Error on start of Syslog-NG
by Michael Kolowicz 19 Jan '12

19 Jan '12

I using Syslog-NG on 3.2.6 on Ubuntu 10.04 If I start / reload syslog-ng I see following error: Reloading syslog-ng's config file: WARNING: Configuration file format is too old, please update it to use the 3.2 format as some constructs might operate inefficiently; WARNING: the expected message format is being changed for unix-domain transports to improve syslogd compatibity with syslog-ng 3.2. If you are using custom applications which bypass the syslog() API, you might need the 'expect-hostname' flag to get the old behaviour back; * My Configuration is: @version: 3.2 #ADM_NTINST @ 09:41 13.01.2012 source s_local {internal(); unix-stream("/dev/log");}; source s_udp {udp(port(514));}; destination d_SYSLOG { file("/var/log/syslog");}; destination SUA002254_localhost_SYS_SYSLOG_local {program("/usr/bin/mysql --host=localhost --user=USR_SYS_SYSLOG --password=pw_Syslog SYS_SYSLOG" template("INSERT INTO local (host, source_ip,facility, priority, tag, unixtime,program, msg) VALUES ('$HOST','$SOURCEIP','$FACILITY','$PRIORITY','$TAG','$UNIXTIME','$PROGRAM',' $MSGONLY' );\n") template-escape(yes));}; destination d_udp1514 {udp("192.168.2.108" port (1514); keep-hostname(););}; destination SUA002254_localhost_SYS_SYSLOG_uaagent {sql( type(mysql) host("localhost") username("USR_SYS_SYSLOG") password("pw_Syslog") database("SYS_SYSLOG") table("uaagent") columns("host", "source_ip", "facility", "priority", "tag", "unixtime", "program", "msg") values("$HOST","$SOURCEIP","$FACILITY","$PRIORITY","$TAG", "$UNIXTIME","$PROGRAM","$MSGONLY") indexes("host", "source_ip", "facility", "priority", "tag", "unixtime", "program", "msg"));}; #log_local log { source(s_local); destination(d_SYSLOG); destination(SUA002254_localhost_SYS_SYSLOG_local); }; #log_uaagent log { source(s_udp); destination(d_udp1514); destination(SUA002254_localhost_SYS_SYSLOG_uaagent); destination(d_SYSLOG); }; Freundliche Grüße / Best regards

3 2

Forwarding Syslog-ng Messages
by Michael Kolowicz 19 Jan '12

19 Jan '12

Hello, I want to forward incomming messages to a "LiveMonitor"-Application which is running on a remote machine and listening on UDP-Port 1514. I have create a destination "{udp("192.168.2.122" port (1514));}" but my Monitor-Application didn´t recieve messages (I using Syslog-ng 3.2.5 on ubuntu 10.04) If I sending direct to my application then I see the messages - is my destination configuration wrong? Here my Configuration source s_udp {udp(port(514));}; destination d_udp1514 {udp("192.168.2.108" port (1514); keep-hostname(););}; destination d_SYSLOG { file("/var/log/syslog");}; log { source(s_udp); destination(d_udp1514); destination(d_SYSLOG); }; I have run Whireshark on the System with the application and I see no incoming messages The the log messages are being written to /var/log/syslog? Freundliche Grüße / Best regards

2 1

Syntax error on threading(yes)
by John Morrissey 18 Jan '12

18 Jan '12

syslog-ng 3.3 snapshots as recent as 9 January yield: -- Error parsing config, syntax error, unexpected LL_IDENTIFIER, expecting '}' in /etc/syslog-ng/options.conf at line 6, column 2: included from /etc/syslog-ng/syslog-ng.conf line 5, column 23 threading(yes); ^^^^^^^^^ -- even with an options statement as simple as: options { threading(yes); }; syslog-ng is linked against libpthread and libgthread, so that much seems to be in order. Is there something else I'm missing? john -- John Morrissey _o /\ ---- __o jwm(a)horde.net _-< \_ / \ ---- < \, www.horde.net/ __(_)/_(_)________/ \_______(_) /_(_)__

2 2