- syslog-ng - lists.balabit.hu

Cannot start syslog-ng-3.3.1
by Hery Fanomezantsoa 29 Jan '12

29 Jan '12

Hello I installed syslog-ng-3.3.1 from source on debian lenny. And when I start it, it says "error while loading shared librairies: libsyslog-ng-3.3.1.so: cannot open shared object file: No such file or directory" though this file is in /usr/local/syslog-ng/lib/ libsyslog-ng-3.3.1.so Anyone can help?

3 3

[PATCH] [logproto.c] "Invalid Frame Header" in syslog protocol
by Gergely Nagy 29 Jan '12

29 Jan '12

When using the syslog protocol, in case when we received an EAGAIN during an attempt to send the framing, the frame was not resent, resulting in protocol errors and invalid frame header messages on the other end. This patch corrects the problem by not using log_proto_text_client_post to send framed messages, but a wrapper that can resume sending the frame too. Signed-off-by: Fried Zoltan <deirf(a)balabit.hu> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/logproto.c | 64 ++++++++++++++++++++++++++++++++++++------------------- 1 files changed, 42 insertions(+), 22 deletions(-) diff --git a/lib/logproto.c b/lib/logproto.c index 16ca0d7..e2cf944 100644 --- a/lib/logproto.c +++ b/lib/logproto.c @@ -35,6 +35,26 @@ #include <sys/uio.h> #include <limits.h> + +typedef struct _LogProtoTextClient +{ + LogProto super; + guchar *partial; + gsize partial_len, partial_pos; +} LogProtoTextClient; + +#define LPFCS_FRAME_INIT 0 +#define LPFCS_FRAME_SEND 1 +#define LPFCS_MESSAGE_SEND 2 + +typedef struct _LogProtoFramedClient +{ + LogProtoTextClient super; + gint state; + gchar frame_hdr_buf[9]; + gint frame_hdr_len, frame_hdr_pos; +} LogProtoFramedClient; + gboolean log_proto_set_encoding(LogProto *self, const gchar *encoding) { @@ -70,14 +90,6 @@ log_proto_free(LogProto *s) g_free(s); } - -typedef struct _LogProtoTextClient -{ - LogProto super; - guchar *partial; - gsize partial_len, partial_pos; -} LogProtoTextClient; - static gboolean log_proto_text_client_prepare(LogProto *s, gint *fd, GIOCondition *cond) { @@ -144,7 +156,7 @@ log_proto_text_client_flush(LogProto *s) * successfully sent this message, or if it should be resent by the caller. **/ static LogProtoStatus -log_proto_text_client_post(LogProto *s, guchar *msg, gsize msg_len, gboolean *consumed) +log_proto_client_post_writer(LogProto *s, guchar *msg, gsize msg_len, gboolean *consumed, gboolean syslog_proto) { LogProtoTextClient *self = (LogProtoTextClient *) s; gint rc; @@ -173,6 +185,14 @@ log_proto_text_client_post(LogProto *s, guchar *msg, gsize msg_len, gboolean *co */ return rc; } + else if (self->partial_len != 0 && !self->partial && syslog_proto) + { + LogProtoFramedClient *framed_self = (LogProtoFramedClient *) s; + self->partial_pos =0; + self->partial_len =0; + framed_self->state = LPFCS_FRAME_INIT; + return rc; + } /* OK, partial buffer empty, now flush msg that we just got */ @@ -219,6 +239,18 @@ log_proto_text_client_post(LogProto *s, guchar *msg, gsize msg_len, gboolean *co return LPS_SUCCESS; } +static LogProtoStatus +log_proto_text_client_post(LogProto *s, guchar *msg, gsize msg_len, gboolean *consumed) +{ + return log_proto_client_post_writer(s, msg, msg_len, consumed, FALSE); +} + +static LogProtoStatus +log_proto_framed_client_post_writer(LogProto *s, guchar *msg, gsize msg_len, gboolean *consumed) +{ + return log_proto_client_post_writer(s, msg, msg_len, consumed, TRUE); +} + LogProto * log_proto_text_client_new(LogTransport *transport) { @@ -1741,18 +1773,6 @@ log_proto_dgram_server_new(LogTransport *transport, gint max_msg_size, guint fla return &self->super.super; } -#define LPFCS_FRAME_INIT 0 -#define LPFCS_FRAME_SEND 1 -#define LPFCS_MESSAGE_SEND 2 - -typedef struct _LogProtoFramedClient -{ - LogProtoTextClient super; - gint state; - gchar frame_hdr_buf[9]; - gint frame_hdr_len, frame_hdr_pos; -} LogProtoFramedClient; - static LogProtoStatus log_proto_framed_client_post(LogProto *s, guchar *msg, gsize msg_len, gboolean *consumed) { @@ -1800,7 +1820,7 @@ log_proto_framed_client_post(LogProto *s, guchar *msg, gsize msg_len, gboolean * self->state = LPFCS_MESSAGE_SEND; } case LPFCS_MESSAGE_SEND: - rc = log_proto_text_client_post(s, msg, msg_len, consumed); + rc = log_proto_framed_client_post_writer(s, msg, msg_len, consumed); /* NOTE: we don't check *consumed here, as we might have a pending * message in self->partial before we begin, in which case *consumed -- 1.7.8.3

2 1

Syslog-ng and tomcat6 & apache2 servers
by Ralf W. 27 Jan '12

27 Jan '12

Hello group - Using syslog-ng as central syslog server. I was ask to syslog all tomcat6, as well as apache2 servers using syslog-ng. I followed some instructions on using sockets, but it doesn't work. Can somebody please point me in the right direction or has any ideas on how to do this. Thank YOU, Ralf The problem with troubleshooting is that trouble shoots back. - Ralf Wiegand 1999

2 1

AUTO: James B Horwath/TheGuardian is out of the office. (returning 01/30/2012)
by James B Horwath 27 Jan '12

27 Jan '12

I am out of the office until 01/30/2012. I am out of the office until Monday, January 30, 2012 with limited access to e-mail. If there are Nitro issues please contact Dante Rodino at 212-919-2819 or Sunil Gupta at 610-807-6710. Any other issues please contact Christopher Clarke manager of Security Operations Thanks, Jim Note: This is an automated response to your message "syslog-ng Digest, Vol 81, Issue 38" sent on 1/27/2012 6:00:15. This is the only notification you will receive while this person is away. ----------------------------------------- This message, and any attachments to it, may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are notified that any use, dissemination, distribution, copying, or communication of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately by return e-mail and delete the message and any attachments. Thank you.

1 0

[Bug 142] New: --worker-threads=0 support
by bugzilla＠bugzilla.balabit.com 25 Jan '12

25 Jan '12

https://bugzilla.balabit.com/show_bug.cgi?id=142 Summary: --worker-threads=0 support Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: arekm(a)maven.pl Type of the Report: enhancement Estimated Hours: 0.0 Hello, I was hoping that threading support would solve my problem described in https://bugzilla.balabit.com/show_bug.cgi?id=113. Unfortunately it looks like it is impossible to create configuration for which each target will use one thread by default (as only such configuration is safe in case of some targets blocking on I/O). This feature request is about adding some option like: --worker-threads=0 that would limit number of threads to numer of sources + number of destinations, so each possible blocking I/O would be happening in separate thread. Yes, it would be more than cpu cores but this configuration is about reliability and not performance. (right now I have only one, network logging target that needs to always work, so probably disabling threading globally and enabling threading only for this network target would get me "always working" logging via net, even if some infinite I/O blocking happens. What would be best is to have ability to have such config on by default for all targets) -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 12

syslog-ng 3.3.4 has been released
by devel＠balabit.hu 24 Jan '12

24 Jan '12

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.3.4 SUMMARY : new stable release DATE : Jan 19, 2012 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.3.4) has been released. For latest fixes in the 3.3.x feature branch you are recommended to upgrade to this version. CHANGES: 3.3.4 Mon, 16 Jan 2012 23:07:46 +0100 Highlights ========== This release is aimed fixing bugs in the 3.3.x series, with one more important change: there's an incompatible change in the processing of <action> tags in the db-parser() database format (aka patterndb), see below for more details. Bugfixes: ========= * Fixed set() and subst() rewrite operations to work properly on the value() parameter specified in the configuration even if they are referenced at multiple spots in the configuration file. Earlier the 2nd and subsequent invocation of the rewrite rule changed $MESSAGE. * Fixed csv-parser() to work even if it is invoked at multiple spots in the configuration file. Earlier, the 2nd and subsequent references of the parser rule forgot the list of column names and the input template. * Fixed the processing of condition() parameter in rewrite rules, which was broken if it contained a filter() function call. * Fixed program() destination to properly kill the child process on reload and shutdown. * Fixed a potential division by zero error which could happen for large data rates due to a race in an unlocked region. * Fixed an assertion failure in mongodb destination that happened due to a race condition at high data rates. * Fixed an fd leak in the control socket code, that caused the control connection file descriptors to be leaked. Other changes ============= * db-parser() automatically sets a tag named '.classifier.unknown' if the message doesn't match. * The use of actions in db-parser() for messages without a correllation context was inconsistently indexing messages. For actions in rules that had correllation @0 was the new message being generated, and @1 was the message that triggered the rule. Without correllation @0 was used for the triggering message, which is greatly inconsistent and unintuitive. This was fixed by changing the behaviour for rules without correllation, now both correllation and non-correllation rules use @0 for the new message, and @1 for the triggering message. This is an incompatible change in the db-parser() format. * The value of the $TAGS macro is added to pdbtool match output. * unix-dgram() and unix-stream() error logging on systemd failures became more detailed for easier troubleshooting. Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Attila M. Magyar (BalaBit) Balazs Scheidler (BalaBit) Costa Farber Dmitry Gilev Evan Rempel (University of Victoria) Fried Zoltan (BalaBit) Gergely Nagy (BalaBit) Jakub Jankowski (SuperHost.pl) John Morrissey (Horde.net) Michael Hocke (New York University) Michal Schmidt (RedHat) Peter Eisenlohr (Inform Software) Svante Signell (Telia.se) Thomas Wollner DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

5 6

sql scaling commits idea
by Patrick Hemmer 24 Jan '12

24 Jan '12

A technique that we use for an in-house application I wrote -- that writes log entries to a database -- I think would be very useful for syslog-ng in high-performance setups. The idea is basically to commit as frequently as the backend database can handle. The way it works is that you basically commit when your queue is completely empty. So if you have a low volume of logs coming it, it would commit after every log entry. This is quite demanding on the database for high volumes, so as the database loses the ability to keep up, the queue in syslog-ng starts to build and so it starts to commit less. When it starts doing more log entries per commit/transaction, performance on the database increases. So you end up writing out to the database as fast as the database can handle, with as little data loss as possible were syslog-ng to die somehow. You would also need some sort of max-transaction-size so you dont end up with too many uncommited log entries if the database is being really really slow. As mentioned I've been using this technique in our in-house application for a couple years now and it performs beautifully. Very low latency between syslog-ng getting the message, and it being available (for a select query) in the database, without overwhelming the database. In theory you could also use this same principle for file destinations as well, but I dont think it'd be as useful there. But anyway, just an idea :-)

2 4

[Bug 158] New: syslog-ng tries to add already existing column when using sqlite3
by bugzilla＠bugzilla.balabit.com 23 Jan '12

23 Jan '12

https://bugzilla.balabit.com/show_bug.cgi?id=158 Summary: syslog-ng tries to add already existing column when using sqlite3 Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.syslogng(a)feystorm.net Type of the Report: --- Estimated Hours: 0.0 I'm trying to use an sqlite3 database through the sql driver, however syslog-ng seems to be unable to verify the table structure and is trying to add a column which already exists, and so the add fails. 2012-01-21 18:58:10 syslog-ng[14564]: Error running SQL query; type='sqlite3', host='', port='', user='syslog-ng', database='/var/log/logs.sql3', error='1: duplicate column name: time', query='ALTER TABLE logs ADD time text' 2012-01-21 18:58:10 syslog-ng[14564]: Error adding missing column, giving up; table='logs', column='time' 2012-01-21 18:58:10 syslog-ng[14564]: Error checking table, disconnecting from database, trying again shortly; time_reopen='60' Relevant portion of my config destination d_sqlite { sql( type('sqlite3') database("/var/log/logs.sql3") table("logs") columns("time", "time_r", "host", "facility", "priority", "program", "pid", "tag", "message") values("$S_UNIXTIME", "$R_UNIXTIME", "$FULLHOST", "$FACILITY_NUM", "$PRIORITY", "$PROGRAM", "$PID", "$DBTAG", "$MSG") null("") flags(explicit-commits) flush_lines(10) flush_timeout(200) ); }; My database schema CREATE TABLE logs (id integer primary key autoincrement, time unsigned integer not null, time_r unsigned integer not null, host text not null, facility unsigned integer not null, priority unsigned integer not null, program text not null, pid unsigned integer, tag text, message text not null); This is occurring with syslog-ng 3.3.4 and libdbi 0.8.3 -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

Re: [syslog-ng] Forward syslog-ng messages
by Michael Kolowicz 21 Jan '12

21 Jan '12

> Please verify first that syslog-ng actually gets the incoming traffic. The incoming traffic will be regnozie and processed. I have configure to write a database and to a local file - it works fine But not the forwarding - BTW: The address is be changed, why I use another machine to test my application ;-) Here my full config as reference @version: 3.2 #ADM_NTINST @ 11:45 20.01.2012 source s_udp {udp(port(514));}; destination d_SYSLOG { file("/var/log/syslog");}; destination d_udp1514 {udp("192.168.2.108" port (1514));}; destination SUA002254_localhost_SYS_SYSLOG_uaagent {sql( type(mysql) host("localhost") username("USR_SYS_SYSLOG") password("pw_Syslog") database("SYS_SYSLOG") table("uaagent") columns("host", "source_ip", "facility", "priority", "tag", "unixtime", "program", "msg") values("$HOST","$SOURCEIP","$FACILITY","$PRIORITY","$TAG", "$UNIXTIME","$PROGRAM","$MSGONLY") indexes("host", "source_ip", "facility", "priority", "tag", "unixtime", "program", "msg"));}; #log_uaagent log { source(s_udp); destination(d_udp1514); destination(SUA002254_localhost_SYS_SYSLOG_uaagent); destination(d_SYSLOG); };

1 0

Re: [syslog-ng] Forwarding Syslog-ng Messages
by Michael Kolowicz 20 Jan '12

20 Jan '12

> The config has syntax errors, it should read like this: > > destination d_udp1514 {udp("192.168.2.108" port (1514));}; > > keep-hostname() is not a valid destination option, it is related to source > drivers instead OK - Thanks for you hint - I have change it - but no success. The messages come not to my Application LG

2 1

JSON vs PatternDB for parser performance
by Martin Holste 20 Jan '12

20 Jan '12

Does anyone have a good idea as to which parser is fastest? The pattern matching tree of PatternDB makes me think that it will be faster, but I'd like to see what others think. Testing in my environment would be non-trivial, so I am hoping someone already has. Thanks, Martin

2 3

Error on start of Syslog-NG
by Michael Kolowicz 19 Jan '12

19 Jan '12

I using Syslog-NG on 3.2.6 on Ubuntu 10.04 If I start / reload syslog-ng I see following error: Reloading syslog-ng's config file: WARNING: Configuration file format is too old, please update it to use the 3.2 format as some constructs might operate inefficiently; WARNING: the expected message format is being changed for unix-domain transports to improve syslogd compatibity with syslog-ng 3.2. If you are using custom applications which bypass the syslog() API, you might need the 'expect-hostname' flag to get the old behaviour back; * My Configuration is: @version: 3.2 #ADM_NTINST @ 09:41 13.01.2012 source s_local {internal(); unix-stream("/dev/log");}; source s_udp {udp(port(514));}; destination d_SYSLOG { file("/var/log/syslog");}; destination SUA002254_localhost_SYS_SYSLOG_local {program("/usr/bin/mysql --host=localhost --user=USR_SYS_SYSLOG --password=pw_Syslog SYS_SYSLOG" template("INSERT INTO local (host, source_ip,facility, priority, tag, unixtime,program, msg) VALUES ('$HOST','$SOURCEIP','$FACILITY','$PRIORITY','$TAG','$UNIXTIME','$PROGRAM',' $MSGONLY' );\n") template-escape(yes));}; destination d_udp1514 {udp("192.168.2.108" port (1514); keep-hostname(););}; destination SUA002254_localhost_SYS_SYSLOG_uaagent {sql( type(mysql) host("localhost") username("USR_SYS_SYSLOG") password("pw_Syslog") database("SYS_SYSLOG") table("uaagent") columns("host", "source_ip", "facility", "priority", "tag", "unixtime", "program", "msg") values("$HOST","$SOURCEIP","$FACILITY","$PRIORITY","$TAG", "$UNIXTIME","$PROGRAM","$MSGONLY") indexes("host", "source_ip", "facility", "priority", "tag", "unixtime", "program", "msg"));}; #log_local log { source(s_local); destination(d_SYSLOG); destination(SUA002254_localhost_SYS_SYSLOG_local); }; #log_uaagent log { source(s_udp); destination(d_udp1514); destination(SUA002254_localhost_SYS_SYSLOG_uaagent); destination(d_SYSLOG); }; Freundliche Grüße / Best regards

3 2

Forwarding Syslog-ng Messages
by Michael Kolowicz 19 Jan '12

19 Jan '12

Hello, I want to forward incomming messages to a "LiveMonitor"-Application which is running on a remote machine and listening on UDP-Port 1514. I have create a destination "{udp("192.168.2.122" port (1514));}" but my Monitor-Application didn´t recieve messages (I using Syslog-ng 3.2.5 on ubuntu 10.04) If I sending direct to my application then I see the messages - is my destination configuration wrong? Here my Configuration source s_udp {udp(port(514));}; destination d_udp1514 {udp("192.168.2.108" port (1514); keep-hostname(););}; destination d_SYSLOG { file("/var/log/syslog");}; log { source(s_udp); destination(d_udp1514); destination(d_SYSLOG); }; I have run Whireshark on the System with the application and I see no incoming messages The the log messages are being written to /var/log/syslog? Freundliche Grüße / Best regards

2 1

Syntax error on threading(yes)
by John Morrissey 18 Jan '12

18 Jan '12

syslog-ng 3.3 snapshots as recent as 9 January yield: -- Error parsing config, syntax error, unexpected LL_IDENTIFIER, expecting '}' in /etc/syslog-ng/options.conf at line 6, column 2: included from /etc/syslog-ng/syslog-ng.conf line 5, column 23 threading(yes); ^^^^^^^^^ -- even with an options statement as simple as: options { threading(yes); }; syslog-ng is linked against libpthread and libgthread, so that much seems to be in order. Is there something else I'm missing? john -- John Morrissey _o /\ ---- __o jwm(a)horde.net _-< \_ / \ ---- < \, www.horde.net/ __(_)/_(_)________/ \_______(_) /_(_)__

2 2

Re: [syslog-ng] Error on start of Syslog-NG
by Michael Kolowicz 18 Jan '12

18 Jan '12

> Try changing that unix-stream to unix-dgram, and see if the warning goes > away. No - the "Message" still come up What I not understand: Why Syslog-ng mean, that the format of file is not right. I have a statement with the version-number @version: 3.2

1 0

No logging on Snow Leopard except internal
by Andrew Eberbach 17 Jan '12

17 Jan '12

Hi I'm trying to get syslog-ng working on snow leopard but when it starts all I get is internal messages. Using logger to send a message won't work nor will using another program. Both work with normal syslogd (which I have turned off so as not to conflict). I've also tried loggen -D localhost 514 and loggen -i localhost 514 to no avail. Here's the details: #/usr/local/Cellar/syslog-ng/3.2.4/sbin/syslog-ng -V syslog-ng 3.2.4 Installer-Version: 3.2.4 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#ef7b91e4a1b1f9628c66138b4ae83de7e4c697c6 Compile-Date: Jan 16 2012 05:44:25 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off and here's the config #cat syslog-ng.conf ############################################################################# # Default syslog-ng.conf file which collects all local logs into a # single file called /var/log/messages. # @version: 3.2 @include "scl.conf" source s_local { system(); internal(); }; source s_network { udp(); }; destination d_local { file("/var/log/messages"); }; log { source(s_local); # uncomment this line to open port 514 to receive messages source(s_network); destination(d_local); }; I'm sure there's something stupid that I'm missing but I haven't been able to find it. Any thoughts? Thanks, Andrew

3 5

syslog-ng.ctl socket fd leak in 3.3.3
by Jakub Jankowski 17 Jan '12

17 Jan '12

Hi, can anyone confirm my observations? $ lsof -n -p $(cat /var/run/syslog-ng.pid) | grep -c /var/lib/syslog-ng/syslog-ng.ctl 129 $ syslog-ng-ctl stats >/dev/null; echo $? 0 $ lsof -n -p $(cat /var/run/syslog-ng.pid) | grep -c /var/lib/syslog-ng/syslog-ng.ctl 130 $ I'm running a cronjob that dumps stats every minute, so the number of open fds is growing linearly, soon reaching configured open files limits. I'm using self-compiled 3.3.3: # syslog-ng -V syslog-ng 3.3.3 Installer-Version: 3.3.3 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.3#master#d199a1980be6b23fe24189e86a882812288e292c Compile-Date: Nov 18 2011 00:16:55 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat,afsql Available-Modules: afprog,syslogformat,convertfuncs,afuser,afsocket,afsocket-tls,csvparser,confgen,afsql,afmongodb,dummy,dbparser,affile,basicfuncs Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: off Am I the only one seeing this? Regards, -- Jakub Jankowski|shasta@toxcorp.com|http://toxcorp.com/ GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D

3 5

tuning syslog-ng 3.3.3
by Johnson, Chris (HP TippingPoint Roseville) 17 Jan '12

17 Jan '12

I'm trying to tune syslog-ng 3.3.3. My first attempt is to use the log_fetch_limit parameter, but I'm running into the problem of where to define it! The Admin guide says I can put it in the global options{}, but that generates and error saying it has been taken out of the global area and needs to be specified by the source and I can't seem to find the correct source syntax. My config (pared down): @version: 3.3 @include "scl.conf" options { # log_fetch_limit(100); }; source s_local { system(); internal(); }; ############################################################################# # Service all filter f_all_pgm_01{program("*" type("glob"));}; filter f_all_lvl_01.01{level(info..emerg)}; destination d_all_01{file("/var/log/system.log" suppress(30));}; log { source(s_local); filter(f_all_pgm_01); filter(f_all_lvl_01.01); destination(d_all_01); }; I've tried putting it in: source s_local{ system( log_fetch_limit(100); ); ...}; source s_local{ system(); ... log_fetch_limit(100); }; log{ source(s_local log_fetch_limit(100);); ... }; each generate a 'syslog-ng[1567]: Error parsing configuration;' error. Any ideas on *where* I should put it? Thanks, Chris ---------------------------------------- Christopher Johnson chris.johnson3(a)hp.com<mailto:chris.johnson3@hp.com> HP Software - Security Product Group (916) 785-2817 ----------------------------------------

5 6

Centos 6 Syslog-ng
by Pigri 16 Jan '12

16 Jan '12

Hello! I have problem for Centos 6.Rebuild syslog-ng package for epel repo, because not have SQL support. But i have this problem: Error: The number of columns and values do not match; len_columns='10', len_values='11' Error initializing dest driver; dest='d_apache_mysql', id='d_apache_mysql#0' Error initializing message pipeline; Config: parser p_apache { csv-parser( columns( "APACHE.CLIENT_IP", "APACHE.IDENT_NAME", "APACHE.USER_NAME", "APACHE.TIMESTAMP", "APACHE.REQUEST_URL", "APACHE.REQUEST_STATUS", "APACHE.CONTENT_LENGTH", "APACHE.REFERER", "APACHE.USER_AGENT", "APACHE.VIRTUAL_HOST", "APACHE.SERVER_NAME") flags(escape-none,escape-backslash,escape-double-char,escape-double-char,strip-whitespace) delimiters(" ") quote-pairs('""[]') ); }; destination d_apache_mysql { sql(type(mysql) host("localhost") username("root") password("xxxxx") database("data") table("accesslog") columns("1", "2", "3", "4", "5", "6", "7", "8","9","10") values("$APACHE.CLIENT_IP","$APACHE.IDENT_NAME","$APACHE.USER_NAME","$APACHE.TIMESTAMP","$APACHE.REQUEST_URL","$APACHE.REQUEST_STATUS","$APACHE.CONTENT_LENGTH","$APACHE.REFERER","$APACHE.USER_AGENT","$APACHE.VIRTUAL_HOST","$APACHE.SERVER_NAME") indexes("1", "2", "3", "4", "5", "6", "7", "8","9","10")); }; destination d_apache_me { file("/var/log/test.me.log"); }; filter f_apache_edata { facility(local6) and program(test.me); }; log { source(s_sys); parser(p_apache); filter(f_apache_edata); destination(d_apache_mysql); }; Version: LIBDBI: Version : 0.8.3 Syslog-ng: syslog-ng 3.2.5 Installer-Version: 3.2.5 Revision: ssh+git://bazsi@git.balabit //var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#9d4bea28198bd731df1a61e980a2af5b88d81116 Compile-Date: Jan 8 2012 22:19:13 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: on Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off Strace: http://dl.dropbox.com/u/4683046/debug Thank you for all. Best regards, pigri

3 3

trouble with chain_hostnames
by Matt Van Mater 16 Jan '12

16 Jan '12

Hi all, Many thanks to the developers, I've been a user for many years and have run into a deployment that I can't quite get working the way I want to. Please correct me if I am wrong, but the chain_hostnames global option is intended to append the current syslog-ng hostname to every message processed, no matter how many hosts a message chains through? My messages are being relayed through these hosts: 1) webserver: Ubuntu LTS 10.4.3, syslog-ng 2.0.9 (from standard apt-get sources) 2) proxy: OpenBSD 5.0, syslog-ng 3.1.4 (from prebuilt packages) 3) logserver: Ubuntu LTS 10.4.3, syslog-ng 2.0.9 (from standard apt-get sources) The traffic flows through them this way: webserver --> proxy (tcp 514) --> proxy (tcp 1514) --> ssh tunnel --> logserver (tcp 514) I have chain_hostnames(yes) in the options global variable section all three syslog-ng.conf files, and messages relay through all hosts properly EXCEPT that the proxy server does not seem to append its hostname or IP address to the syslog message when forwarding it on to the logserver. I have experimented with keep_hostnames as both yes and no on all hosts and have not seen what i am expecting... (example: timestamp s_all@webserver_ip/proxy_ip/logserver_ip actual_log_message) An example of what i see today is this: On webserver # echo "webserver sending message to localhost tcp 514" | nc localhsot 514 On loghost# tail -n1 /var/log/messages Jan 4 17:22:16 webserver/webserver/loghost_ip webserver sending message to localhost tcp 514 Note the fact that the message gets transported through all 3 hosts but the proxy hostname isn't being inserted into the chain. So my questions are: 1) Is chain_hostnames supposed to append the hostname to the message, no matter how many hosts a message is relayed through? 1a) Suggestion: can you update the FAQ here: http://www.campin.net/syslog-ng/faq.html#hostname to include an example of a log message relayed through multiple hosts and the expected behavior 2) Is there a known bug in syslog-ng v3.1.4 and / or OpenBSD's example syslog-ng.conf file that is the cause of hostname chaining not working? 2a) Suggestion: can you publish the syslog-ng changelogs in a single file appending changes, or a searchable database? As far as I can tell, I currently have to download the changelog for each individual syslog-ng version released in order to search and see if an issue was released and that is a huge PITA. Thanks! Matt

3 2

[PATCH (3.4) 0/4]: json-parser updates
by Gergely Nagy 16 Jan '12

16 Jan '12

Following this mail, a couple of patches will come - they were written in last November, but I don't think I posted them (as I didn't expect to see the json-parser merged so soon). Until now, they were sitting on my feature/3.4/json/parser branch, and that's where they sit now, too. I rebased that branch onto 3.4 master + center.c syntax error fixes (which I posted a week or two ago), and am sending the json parser patches now. They accomplish a few things: it makes the parser handle invalid JSON input properly (by returning an error, instead of crashing); adds support for parsing boolean, array and nested object types. Booleans will get parsed into TRUE or FALSE, nested objects will be turned into dotted notation (which also means that keys with a dot in them won't be handled properly by format-json, once it is updated to support reconstructing nested objects from dotted notation), and arrays will be treated like nested objects, with the array indexes used as a key. As an example, lets look at a JSON input: {"is-example": true, "hats": ["top hat", "baseball cap", "pointy witch-hat"], "auth": { "method": ["looking angry at the security guy", "bribery"], "result": "did not work" } } This would be parsed into the following name-value paris: "is-example"="TRUE" "hats.0"="top hat" "hats.1"="baseball cap" "hats.2"="pointy witch-hat" "auth.method.0"="looking angry at the security guy" "auth.method.1"="bribery" "auth.result"="did not work" The other part of this will be an enhahcement to value-pairs and format-json, that will make syslog-ng able to reconstruct the original JSON (more or less... booleans probably won't be supported, except perhaps optionally with a flag). But that is something I haven't started working on yet.

2 6

Hello, and project for Win32 port
by Ramón García Fernández 16 Jan '12

16 Jan '12

Hello, I am Ramón García Fernández and I work for the IT department of the Comisión Nacional de Mercado de Valores, the Spanish regulator of securities. Would you be interested in an opensource Win32 port of syslog-ng? Best regards, Aviso legal – Comisión Nacional del Mercado de Valores Este mensaje y, en su caso, los ficheros que lleve incorporados, está dirigido exclusivamente a su destinatario y es de carácter confidencial. Si fuere recibido por error o se tuviere conocimiento del mismo sin ser su destinatario, rogamos nos lo comunique por la misma vía o telefónicamente (91 585 15 00) y proceda a su destrucción, debiendo abstenerse de utilizar, transmitir, divulgar o reproducir la información contenida en el mismo. La CNMV se reserva las acciones legales que procedan contra todo tercero que acceda de forma ilegítima al contenido de cualquier mensaje externo procedente de la entidad Para información y consultas visite nuestra web: http://www.cnmv.es Disclaimer - Comisión Nacional del Mercado de Valores This message, its content and any file attached thereto is for the intended recipient only and is confidential. If you have received this e-mail in error or had access to it, you should note that the information in it is private and any use thereof is unauthorised. In such an event please notify us by e-mail or by telephone (+ 34 91 585 15 00). Any reproduction of this e-mail by whatsoever means and any transmission or dissemination thereof to other persons is prohibited. The Comisión Nacional del Mercado de Valores reserves the right to take legal action against any persons unlawfully gaining access to the content of any external message it has emitted For additional information, please visit our website: http://www.cnmv.es

3 4

SQL driver - Table names with dots failing to create
by phil＠mindfury.net 16 Jan '12

16 Jan '12

Hello. I'm running v3.3.3 built from source on a Red Hat EL 5.3 machine. I've been trying to narrow down a problem where hosts sending syslogs to this box aren't showing up in MySQL, so I checked the internal log and found this: Error running SQL query; type='mysql', host='localhost', port='', user='syslog-ng', database='syslog-ng', error='1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near \'.3.4_20120113 (date text, facility text, level text, host text, program text, pi\' at line 1', query='CREATE TABLE messages_172.25.3.4_20120113 (date text, facility text, level text, host text, program text, pid text, message text)' Looks like in MySQL, that table name can be misinterpreted as <database name>.<table name>, the suggestion is to quote the table identifier with backticks. My question is, is there an option for the SQL driver that does this that I'm just not seeing? I know a workaround would be to ensure the IPs for my hosts reverse map to a name, however it may not be feasible in all situations. Any pointers would be appreciated. --Philip

3 2

CSV parser ignoring template()?
by John Morrissey 16 Jan '12

16 Jan '12

I'm upgrading from 3.1.3 to 3.3.3. My configuration has a CSV parser: parser p_find_role { csv-parser( columns("ROLE", "REMAINDER") delimiters("0123456789.") template("${HOST}") ); }; With 3.1.3, this takes the first word of the machine's hostname and stores it in $ROLE, but 3.3.3 seems to ignore template() and operates on the log message instead, storing the first field of the message in $ROLE. $HOST expands correctly in a log template, so that token is still available in 3.3.x. Am I missing something else related to the upgrade? john -- John Morrissey _o /\ ---- __o jwm(a)horde.net _-< \_ / \ ---- < \, www.horde.net/ __(_)/_(_)________/ \_______(_) /_(_)__

3 4

patterndb that kills syslong-ng with a segfault.
by Russell Fulton 16 Jan '12

16 Jan '12

I tried to rewrite the program macro with a <value> clause and this caused syslog-ng to die. syslog-ng-3.2.4 I no longer have the example but can reconstruct it if you have problems reproducing the problem. R

2 1