- syslog-ng - lists.balabit.hu

Cannot start syslog-ng-3.3.1
by Hery Fanomezantsoa 29 Jan '12

29 Jan '12

Hello I installed syslog-ng-3.3.1 from source on debian lenny. And when I start it, it says "error while loading shared librairies: libsyslog-ng-3.3.1.so: cannot open shared object file: No such file or directory" though this file is in /usr/local/syslog-ng/lib/ libsyslog-ng-3.3.1.so Anyone can help?

3 3

[PATCH] [logproto.c] "Invalid Frame Header" in syslog protocol
by Gergely Nagy 29 Jan '12

29 Jan '12

When using the syslog protocol, in case when we received an EAGAIN during an attempt to send the framing, the frame was not resent, resulting in protocol errors and invalid frame header messages on the other end. This patch corrects the problem by not using log_proto_text_client_post to send framed messages, but a wrapper that can resume sending the frame too. Signed-off-by: Fried Zoltan <deirf(a)balabit.hu> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/logproto.c | 64 ++++++++++++++++++++++++++++++++++++------------------- 1 files changed, 42 insertions(+), 22 deletions(-) diff --git a/lib/logproto.c b/lib/logproto.c index 16ca0d7..e2cf944 100644 --- a/lib/logproto.c +++ b/lib/logproto.c @@ -35,6 +35,26 @@ #include <sys/uio.h> #include <limits.h> + +typedef struct _LogProtoTextClient +{ + LogProto super; + guchar *partial; + gsize partial_len, partial_pos; +} LogProtoTextClient; + +#define LPFCS_FRAME_INIT 0 +#define LPFCS_FRAME_SEND 1 +#define LPFCS_MESSAGE_SEND 2 + +typedef struct _LogProtoFramedClient +{ + LogProtoTextClient super; + gint state; + gchar frame_hdr_buf[9]; + gint frame_hdr_len, frame_hdr_pos; +} LogProtoFramedClient; + gboolean log_proto_set_encoding(LogProto *self, const gchar *encoding) { @@ -70,14 +90,6 @@ log_proto_free(LogProto *s) g_free(s); } - -typedef struct _LogProtoTextClient -{ - LogProto super; - guchar *partial; - gsize partial_len, partial_pos; -} LogProtoTextClient; - static gboolean log_proto_text_client_prepare(LogProto *s, gint *fd, GIOCondition *cond) { @@ -144,7 +156,7 @@ log_proto_text_client_flush(LogProto *s) * successfully sent this message, or if it should be resent by the caller. **/ static LogProtoStatus -log_proto_text_client_post(LogProto *s, guchar *msg, gsize msg_len, gboolean *consumed) +log_proto_client_post_writer(LogProto *s, guchar *msg, gsize msg_len, gboolean *consumed, gboolean syslog_proto) { LogProtoTextClient *self = (LogProtoTextClient *) s; gint rc; @@ -173,6 +185,14 @@ log_proto_text_client_post(LogProto *s, guchar *msg, gsize msg_len, gboolean *co */ return rc; } + else if (self->partial_len != 0 && !self->partial && syslog_proto) + { + LogProtoFramedClient *framed_self = (LogProtoFramedClient *) s; + self->partial_pos =0; + self->partial_len =0; + framed_self->state = LPFCS_FRAME_INIT; + return rc; + } /* OK, partial buffer empty, now flush msg that we just got */ @@ -219,6 +239,18 @@ log_proto_text_client_post(LogProto *s, guchar *msg, gsize msg_len, gboolean *co return LPS_SUCCESS; } +static LogProtoStatus +log_proto_text_client_post(LogProto *s, guchar *msg, gsize msg_len, gboolean *consumed) +{ + return log_proto_client_post_writer(s, msg, msg_len, consumed, FALSE); +} + +static LogProtoStatus +log_proto_framed_client_post_writer(LogProto *s, guchar *msg, gsize msg_len, gboolean *consumed) +{ + return log_proto_client_post_writer(s, msg, msg_len, consumed, TRUE); +} + LogProto * log_proto_text_client_new(LogTransport *transport) { @@ -1741,18 +1773,6 @@ log_proto_dgram_server_new(LogTransport *transport, gint max_msg_size, guint fla return &self->super.super; } -#define LPFCS_FRAME_INIT 0 -#define LPFCS_FRAME_SEND 1 -#define LPFCS_MESSAGE_SEND 2 - -typedef struct _LogProtoFramedClient -{ - LogProtoTextClient super; - gint state; - gchar frame_hdr_buf[9]; - gint frame_hdr_len, frame_hdr_pos; -} LogProtoFramedClient; - static LogProtoStatus log_proto_framed_client_post(LogProto *s, guchar *msg, gsize msg_len, gboolean *consumed) { @@ -1800,7 +1820,7 @@ log_proto_framed_client_post(LogProto *s, guchar *msg, gsize msg_len, gboolean * self->state = LPFCS_MESSAGE_SEND; } case LPFCS_MESSAGE_SEND: - rc = log_proto_text_client_post(s, msg, msg_len, consumed); + rc = log_proto_framed_client_post_writer(s, msg, msg_len, consumed); /* NOTE: we don't check *consumed here, as we might have a pending * message in self->partial before we begin, in which case *consumed -- 1.7.8.3

2 1

Syslog-ng and tomcat6 & apache2 servers
by Ralf W. 27 Jan '12

27 Jan '12

Hello group - Using syslog-ng as central syslog server. I was ask to syslog all tomcat6, as well as apache2 servers using syslog-ng. I followed some instructions on using sockets, but it doesn't work. Can somebody please point me in the right direction or has any ideas on how to do this. Thank YOU, Ralf The problem with troubleshooting is that trouble shoots back. - Ralf Wiegand 1999

2 1

AUTO: James B Horwath/TheGuardian is out of the office. (returning 01/30/2012)
by James B Horwath 27 Jan '12

27 Jan '12

I am out of the office until 01/30/2012. I am out of the office until Monday, January 30, 2012 with limited access to e-mail. If there are Nitro issues please contact Dante Rodino at 212-919-2819 or Sunil Gupta at 610-807-6710. Any other issues please contact Christopher Clarke manager of Security Operations Thanks, Jim Note: This is an automated response to your message "syslog-ng Digest, Vol 81, Issue 38" sent on 1/27/2012 6:00:15. This is the only notification you will receive while this person is away. ----------------------------------------- This message, and any attachments to it, may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are notified that any use, dissemination, distribution, copying, or communication of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately by return e-mail and delete the message and any attachments. Thank you.

1 0

[Bug 142] New: --worker-threads=0 support
by bugzilla＠bugzilla.balabit.com 25 Jan '12

25 Jan '12

https://bugzilla.balabit.com/show_bug.cgi?id=142 Summary: --worker-threads=0 support Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: arekm(a)maven.pl Type of the Report: enhancement Estimated Hours: 0.0 Hello, I was hoping that threading support would solve my problem described in https://bugzilla.balabit.com/show_bug.cgi?id=113. Unfortunately it looks like it is impossible to create configuration for which each target will use one thread by default (as only such configuration is safe in case of some targets blocking on I/O). This feature request is about adding some option like: --worker-threads=0 that would limit number of threads to numer of sources + number of destinations, so each possible blocking I/O would be happening in separate thread. Yes, it would be more than cpu cores but this configuration is about reliability and not performance. (right now I have only one, network logging target that needs to always work, so probably disabling threading globally and enabling threading only for this network target would get me "always working" logging via net, even if some infinite I/O blocking happens. What would be best is to have ability to have such config on by default for all targets) -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 12

syslog-ng 3.3.4 has been released
by devel＠balabit.hu 24 Jan '12

24 Jan '12

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.3.4 SUMMARY : new stable release DATE : Jan 19, 2012 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.3.4) has been released. For latest fixes in the 3.3.x feature branch you are recommended to upgrade to this version. CHANGES: 3.3.4 Mon, 16 Jan 2012 23:07:46 +0100 Highlights ========== This release is aimed fixing bugs in the 3.3.x series, with one more important change: there's an incompatible change in the processing of <action> tags in the db-parser() database format (aka patterndb), see below for more details. Bugfixes: ========= * Fixed set() and subst() rewrite operations to work properly on the value() parameter specified in the configuration even if they are referenced at multiple spots in the configuration file. Earlier the 2nd and subsequent invocation of the rewrite rule changed $MESSAGE. * Fixed csv-parser() to work even if it is invoked at multiple spots in the configuration file. Earlier, the 2nd and subsequent references of the parser rule forgot the list of column names and the input template. * Fixed the processing of condition() parameter in rewrite rules, which was broken if it contained a filter() function call. * Fixed program() destination to properly kill the child process on reload and shutdown. * Fixed a potential division by zero error which could happen for large data rates due to a race in an unlocked region. * Fixed an assertion failure in mongodb destination that happened due to a race condition at high data rates. * Fixed an fd leak in the control socket code, that caused the control connection file descriptors to be leaked. Other changes ============= * db-parser() automatically sets a tag named '.classifier.unknown' if the message doesn't match. * The use of actions in db-parser() for messages without a correllation context was inconsistently indexing messages. For actions in rules that had correllation @0 was the new message being generated, and @1 was the message that triggered the rule. Without correllation @0 was used for the triggering message, which is greatly inconsistent and unintuitive. This was fixed by changing the behaviour for rules without correllation, now both correllation and non-correllation rules use @0 for the new message, and @1 for the triggering message. This is an incompatible change in the db-parser() format. * The value of the $TAGS macro is added to pdbtool match output. * unix-dgram() and unix-stream() error logging on systemd failures became more detailed for easier troubleshooting. Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Attila M. Magyar (BalaBit) Balazs Scheidler (BalaBit) Costa Farber Dmitry Gilev Evan Rempel (University of Victoria) Fried Zoltan (BalaBit) Gergely Nagy (BalaBit) Jakub Jankowski (SuperHost.pl) John Morrissey (Horde.net) Michael Hocke (New York University) Michal Schmidt (RedHat) Peter Eisenlohr (Inform Software) Svante Signell (Telia.se) Thomas Wollner DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

5 6

sql scaling commits idea
by Patrick Hemmer 24 Jan '12

24 Jan '12

A technique that we use for an in-house application I wrote -- that writes log entries to a database -- I think would be very useful for syslog-ng in high-performance setups. The idea is basically to commit as frequently as the backend database can handle. The way it works is that you basically commit when your queue is completely empty. So if you have a low volume of logs coming it, it would commit after every log entry. This is quite demanding on the database for high volumes, so as the database loses the ability to keep up, the queue in syslog-ng starts to build and so it starts to commit less. When it starts doing more log entries per commit/transaction, performance on the database increases. So you end up writing out to the database as fast as the database can handle, with as little data loss as possible were syslog-ng to die somehow. You would also need some sort of max-transaction-size so you dont end up with too many uncommited log entries if the database is being really really slow. As mentioned I've been using this technique in our in-house application for a couple years now and it performs beautifully. Very low latency between syslog-ng getting the message, and it being available (for a select query) in the database, without overwhelming the database. In theory you could also use this same principle for file destinations as well, but I dont think it'd be as useful there. But anyway, just an idea :-)

2 4

[Bug 158] New: syslog-ng tries to add already existing column when using sqlite3
by bugzilla＠bugzilla.balabit.com 23 Jan '12

23 Jan '12

https://bugzilla.balabit.com/show_bug.cgi?id=158 Summary: syslog-ng tries to add already existing column when using sqlite3 Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.syslogng(a)feystorm.net Type of the Report: --- Estimated Hours: 0.0 I'm trying to use an sqlite3 database through the sql driver, however syslog-ng seems to be unable to verify the table structure and is trying to add a column which already exists, and so the add fails. 2012-01-21 18:58:10 syslog-ng[14564]: Error running SQL query; type='sqlite3', host='', port='', user='syslog-ng', database='/var/log/logs.sql3', error='1: duplicate column name: time', query='ALTER TABLE logs ADD time text' 2012-01-21 18:58:10 syslog-ng[14564]: Error adding missing column, giving up; table='logs', column='time' 2012-01-21 18:58:10 syslog-ng[14564]: Error checking table, disconnecting from database, trying again shortly; time_reopen='60' Relevant portion of my config destination d_sqlite { sql( type('sqlite3') database("/var/log/logs.sql3") table("logs") columns("time", "time_r", "host", "facility", "priority", "program", "pid", "tag", "message") values("$S_UNIXTIME", "$R_UNIXTIME", "$FULLHOST", "$FACILITY_NUM", "$PRIORITY", "$PROGRAM", "$PID", "$DBTAG", "$MSG") null("") flags(explicit-commits) flush_lines(10) flush_timeout(200) ); }; My database schema CREATE TABLE logs (id integer primary key autoincrement, time unsigned integer not null, time_r unsigned integer not null, host text not null, facility unsigned integer not null, priority unsigned integer not null, program text not null, pid unsigned integer, tag text, message text not null); This is occurring with syslog-ng 3.3.4 and libdbi 0.8.3 -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

Re: [syslog-ng] Forward syslog-ng messages
by Michael Kolowicz 21 Jan '12

21 Jan '12

> Please verify first that syslog-ng actually gets the incoming traffic. The incoming traffic will be regnozie and processed. I have configure to write a database and to a local file - it works fine But not the forwarding - BTW: The address is be changed, why I use another machine to test my application ;-) Here my full config as reference @version: 3.2 #ADM_NTINST @ 11:45 20.01.2012 source s_udp {udp(port(514));}; destination d_SYSLOG { file("/var/log/syslog");}; destination d_udp1514 {udp("192.168.2.108" port (1514));}; destination SUA002254_localhost_SYS_SYSLOG_uaagent {sql( type(mysql) host("localhost") username("USR_SYS_SYSLOG") password("pw_Syslog") database("SYS_SYSLOG") table("uaagent") columns("host", "source_ip", "facility", "priority", "tag", "unixtime", "program", "msg") values("$HOST","$SOURCEIP","$FACILITY","$PRIORITY","$TAG", "$UNIXTIME","$PROGRAM","$MSGONLY") indexes("host", "source_ip", "facility", "priority", "tag", "unixtime", "program", "msg"));}; #log_uaagent log { source(s_udp); destination(d_udp1514); destination(SUA002254_localhost_SYS_SYSLOG_uaagent); destination(d_SYSLOG); };

1 0

Re: [syslog-ng] Forwarding Syslog-ng Messages
by Michael Kolowicz 20 Jan '12

20 Jan '12

> The config has syntax errors, it should read like this: > > destination d_udp1514 {udp("192.168.2.108" port (1514));}; > > keep-hostname() is not a valid destination option, it is related to source > drivers instead OK - Thanks for you hint - I have change it - but no success. The messages come not to my Application LG

2 1