- syslog-ng - lists.balabit.hu

sql scaling commits idea
by Patrick Hemmer 24 Jan '12

24 Jan '12

A technique that we use for an in-house application I wrote -- that writes log entries to a database -- I think would be very useful for syslog-ng in high-performance setups. The idea is basically to commit as frequently as the backend database can handle. The way it works is that you basically commit when your queue is completely empty. So if you have a low volume of logs coming it, it would commit after every log entry. This is quite demanding on the database for high volumes, so as the database loses the ability to keep up, the queue in syslog-ng starts to build and so it starts to commit less. When it starts doing more log entries per commit/transaction, performance on the database increases. So you end up writing out to the database as fast as the database can handle, with as little data loss as possible were syslog-ng to die somehow. You would also need some sort of max-transaction-size so you dont end up with too many uncommited log entries if the database is being really really slow. As mentioned I've been using this technique in our in-house application for a couple years now and it performs beautifully. Very low latency between syslog-ng getting the message, and it being available (for a select query) in the database, without overwhelming the database. In theory you could also use this same principle for file destinations as well, but I dont think it'd be as useful there. But anyway, just an idea :-)

2 4

[Bug 158] New: syslog-ng tries to add already existing column when using sqlite3
by bugzilla＠bugzilla.balabit.com 23 Jan '12

23 Jan '12

https://bugzilla.balabit.com/show_bug.cgi?id=158 Summary: syslog-ng tries to add already existing column when using sqlite3 Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.syslogng(a)feystorm.net Type of the Report: --- Estimated Hours: 0.0 I'm trying to use an sqlite3 database through the sql driver, however syslog-ng seems to be unable to verify the table structure and is trying to add a column which already exists, and so the add fails. 2012-01-21 18:58:10 syslog-ng[14564]: Error running SQL query; type='sqlite3', host='', port='', user='syslog-ng', database='/var/log/logs.sql3', error='1: duplicate column name: time', query='ALTER TABLE logs ADD time text' 2012-01-21 18:58:10 syslog-ng[14564]: Error adding missing column, giving up; table='logs', column='time' 2012-01-21 18:58:10 syslog-ng[14564]: Error checking table, disconnecting from database, trying again shortly; time_reopen='60' Relevant portion of my config destination d_sqlite { sql( type('sqlite3') database("/var/log/logs.sql3") table("logs") columns("time", "time_r", "host", "facility", "priority", "program", "pid", "tag", "message") values("$S_UNIXTIME", "$R_UNIXTIME", "$FULLHOST", "$FACILITY_NUM", "$PRIORITY", "$PROGRAM", "$PID", "$DBTAG", "$MSG") null("") flags(explicit-commits) flush_lines(10) flush_timeout(200) ); }; My database schema CREATE TABLE logs (id integer primary key autoincrement, time unsigned integer not null, time_r unsigned integer not null, host text not null, facility unsigned integer not null, priority unsigned integer not null, program text not null, pid unsigned integer, tag text, message text not null); This is occurring with syslog-ng 3.3.4 and libdbi 0.8.3 -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

Re: [syslog-ng] Forward syslog-ng messages
by Michael Kolowicz 21 Jan '12

21 Jan '12

> Please verify first that syslog-ng actually gets the incoming traffic. The incoming traffic will be regnozie and processed. I have configure to write a database and to a local file - it works fine But not the forwarding - BTW: The address is be changed, why I use another machine to test my application ;-) Here my full config as reference @version: 3.2 #ADM_NTINST @ 11:45 20.01.2012 source s_udp {udp(port(514));}; destination d_SYSLOG { file("/var/log/syslog");}; destination d_udp1514 {udp("192.168.2.108" port (1514));}; destination SUA002254_localhost_SYS_SYSLOG_uaagent {sql( type(mysql) host("localhost") username("USR_SYS_SYSLOG") password("pw_Syslog") database("SYS_SYSLOG") table("uaagent") columns("host", "source_ip", "facility", "priority", "tag", "unixtime", "program", "msg") values("$HOST","$SOURCEIP","$FACILITY","$PRIORITY","$TAG", "$UNIXTIME","$PROGRAM","$MSGONLY") indexes("host", "source_ip", "facility", "priority", "tag", "unixtime", "program", "msg"));}; #log_uaagent log { source(s_udp); destination(d_udp1514); destination(SUA002254_localhost_SYS_SYSLOG_uaagent); destination(d_SYSLOG); };

1 0

Re: [syslog-ng] Forwarding Syslog-ng Messages
by Michael Kolowicz 20 Jan '12

20 Jan '12

> The config has syntax errors, it should read like this: > > destination d_udp1514 {udp("192.168.2.108" port (1514));}; > > keep-hostname() is not a valid destination option, it is related to source > drivers instead OK - Thanks for you hint - I have change it - but no success. The messages come not to my Application LG

2 1

JSON vs PatternDB for parser performance
by Martin Holste 20 Jan '12

20 Jan '12

Does anyone have a good idea as to which parser is fastest? The pattern matching tree of PatternDB makes me think that it will be faster, but I'd like to see what others think. Testing in my environment would be non-trivial, so I am hoping someone already has. Thanks, Martin

2 3

Error on start of Syslog-NG
by Michael Kolowicz 19 Jan '12

19 Jan '12

I using Syslog-NG on 3.2.6 on Ubuntu 10.04 If I start / reload syslog-ng I see following error: Reloading syslog-ng's config file: WARNING: Configuration file format is too old, please update it to use the 3.2 format as some constructs might operate inefficiently; WARNING: the expected message format is being changed for unix-domain transports to improve syslogd compatibity with syslog-ng 3.2. If you are using custom applications which bypass the syslog() API, you might need the 'expect-hostname' flag to get the old behaviour back; * My Configuration is: @version: 3.2 #ADM_NTINST @ 09:41 13.01.2012 source s_local {internal(); unix-stream("/dev/log");}; source s_udp {udp(port(514));}; destination d_SYSLOG { file("/var/log/syslog");}; destination SUA002254_localhost_SYS_SYSLOG_local {program("/usr/bin/mysql --host=localhost --user=USR_SYS_SYSLOG --password=pw_Syslog SYS_SYSLOG" template("INSERT INTO local (host, source_ip,facility, priority, tag, unixtime,program, msg) VALUES ('$HOST','$SOURCEIP','$FACILITY','$PRIORITY','$TAG','$UNIXTIME','$PROGRAM',' $MSGONLY' );\n") template-escape(yes));}; destination d_udp1514 {udp("192.168.2.108" port (1514); keep-hostname(););}; destination SUA002254_localhost_SYS_SYSLOG_uaagent {sql( type(mysql) host("localhost") username("USR_SYS_SYSLOG") password("pw_Syslog") database("SYS_SYSLOG") table("uaagent") columns("host", "source_ip", "facility", "priority", "tag", "unixtime", "program", "msg") values("$HOST","$SOURCEIP","$FACILITY","$PRIORITY","$TAG", "$UNIXTIME","$PROGRAM","$MSGONLY") indexes("host", "source_ip", "facility", "priority", "tag", "unixtime", "program", "msg"));}; #log_local log { source(s_local); destination(d_SYSLOG); destination(SUA002254_localhost_SYS_SYSLOG_local); }; #log_uaagent log { source(s_udp); destination(d_udp1514); destination(SUA002254_localhost_SYS_SYSLOG_uaagent); destination(d_SYSLOG); }; Freundliche Grüße / Best regards

3 2

Forwarding Syslog-ng Messages
by Michael Kolowicz 19 Jan '12

19 Jan '12

Hello, I want to forward incomming messages to a "LiveMonitor"-Application which is running on a remote machine and listening on UDP-Port 1514. I have create a destination "{udp("192.168.2.122" port (1514));}" but my Monitor-Application didn´t recieve messages (I using Syslog-ng 3.2.5 on ubuntu 10.04) If I sending direct to my application then I see the messages - is my destination configuration wrong? Here my Configuration source s_udp {udp(port(514));}; destination d_udp1514 {udp("192.168.2.108" port (1514); keep-hostname(););}; destination d_SYSLOG { file("/var/log/syslog");}; log { source(s_udp); destination(d_udp1514); destination(d_SYSLOG); }; I have run Whireshark on the System with the application and I see no incoming messages The the log messages are being written to /var/log/syslog? Freundliche Grüße / Best regards

2 1

Syntax error on threading(yes)
by John Morrissey 18 Jan '12

18 Jan '12

syslog-ng 3.3 snapshots as recent as 9 January yield: -- Error parsing config, syntax error, unexpected LL_IDENTIFIER, expecting '}' in /etc/syslog-ng/options.conf at line 6, column 2: included from /etc/syslog-ng/syslog-ng.conf line 5, column 23 threading(yes); ^^^^^^^^^ -- even with an options statement as simple as: options { threading(yes); }; syslog-ng is linked against libpthread and libgthread, so that much seems to be in order. Is there something else I'm missing? john -- John Morrissey _o /\ ---- __o jwm(a)horde.net _-< \_ / \ ---- < \, www.horde.net/ __(_)/_(_)________/ \_______(_) /_(_)__

2 2

Re: [syslog-ng] Error on start of Syslog-NG
by Michael Kolowicz 18 Jan '12

18 Jan '12

> Try changing that unix-stream to unix-dgram, and see if the warning goes > away. No - the "Message" still come up What I not understand: Why Syslog-ng mean, that the format of file is not right. I have a statement with the version-number @version: 3.2

1 0

No logging on Snow Leopard except internal
by Andrew Eberbach 17 Jan '12

17 Jan '12

Hi I'm trying to get syslog-ng working on snow leopard but when it starts all I get is internal messages. Using logger to send a message won't work nor will using another program. Both work with normal syslogd (which I have turned off so as not to conflict). I've also tried loggen -D localhost 514 and loggen -i localhost 514 to no avail. Here's the details: #/usr/local/Cellar/syslog-ng/3.2.4/sbin/syslog-ng -V syslog-ng 3.2.4 Installer-Version: 3.2.4 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#ef7b91e4a1b1f9628c66138b4ae83de7e4c697c6 Compile-Date: Jan 16 2012 05:44:25 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off and here's the config #cat syslog-ng.conf ############################################################################# # Default syslog-ng.conf file which collects all local logs into a # single file called /var/log/messages. # @version: 3.2 @include "scl.conf" source s_local { system(); internal(); }; source s_network { udp(); }; destination d_local { file("/var/log/messages"); }; log { source(s_local); # uncomment this line to open port 514 to receive messages source(s_network); destination(d_local); }; I'm sure there's something stupid that I'm missing but I haven't been able to find it. Any thoughts? Thanks, Andrew

3 5