- syslog-ng - lists.balabit.hu

Re: [syslog-ng] Filtering duplicate messages
by chris packham 30 May '08

30 May '08

Here's the updated patch. I've ditched the "suppress" name in favour of the more appropriate "summary" option and "last_msg" function naming. References are used instead of making a copy of the last message. Timer based flushing is now implemented. Implement "Last message repeated N times" functionality. This behaviour can be enabled by adding the new "summary(<num>)" option to an output configuration e.g. destination tologfile { file("/var/log/messages" template(t) summary(30)); }; As a log message is added to the queue it is remembered as the last message seen. When a new message is added to the queue it is checked against the last message. If its contents are the same the message is dropped and a counter incremented. A message summary indicating the value of the last message counter and a snippet of the message will be inserted into the log queue if a new message that differs is seen or if the configurable timeout period expires. --- src/cfg-grammar.y | 3 +- src/cfg-lex.l | 3 +- src/logmsg.c | 19 ++++++++++ src/logmsg.h | 2 +- src/logwriter.c | 100 ++++++++++++++++++++++++++++++++++++++++++++++++++++- src/logwriter.h | 5 +++ src/messages.c | 4 +-- 7 files changed, 129 insertions(+), 7 deletions(-) diff --git a/src/cfg-grammar.y b/src/cfg-grammar.y index 39ff172..99b033f 100644 --- a/src/cfg-grammar.y +++ b/src/cfg-grammar.y @@ -62,7 +62,7 @@ gint last_addr_family = AF_INET; %token KW_USER KW_DOOR KW_SUN_STREAMS KW_PROGRAM /* option items */ -%token KW_FSYNC KW_MARK_FREQ KW_STATS_FREQ KW_FLUSH_LINES KW_FLUSH_TIMEOUT KW_LOG_MSG_SIZE KW_FILE_TEMPLATE KW_PROTO_TEMPLATE +%token KW_FSYNC KW_MARK_FREQ KW_STATS_FREQ KW_FLUSH_LINES KW_SUMMARY KW_FLUSH_TIMEOUT KW_LOG_MSG_SIZE KW_FILE_TEMPLATE KW_PROTO_TEMPLATE %token KW_CHAIN_HOSTNAMES KW_NORMALIZE_HOSTNAMES KW_KEEP_HOSTNAME KW_CHECK_HOSTNAME KW_BAD_HOSTNAME %token KW_KEEP_TIMESTAMP @@ -671,6 +671,7 @@ dest_writer_option : KW_FLAGS '(' dest_writer_options_flags ')' { last_writer_options->options = $3; } | KW_LOG_FIFO_SIZE '(' NUMBER ')' { last_writer_options->fifo_size = $3; } | KW_FLUSH_LINES '(' NUMBER ')' { last_writer_options->flush_lines = $3; } + | KW_SUMMARY '(' NUMBER ')' { last_writer_options->summary= $3; } | KW_FLUSH_TIMEOUT '(' NUMBER ')' { last_writer_options->flush_timeout = $3; } | KW_TEMPLATE '(' string ')' { last_writer_options->template = cfg_lookup_template(configuration, $3); diff --git a/src/cfg-lex.l b/src/cfg-lex.l index 5c5f54f..d8418b6 100644 --- a/src/cfg-lex.l +++ b/src/cfg-lex.l @@ -6,7 +6,7 @@ /*************************************************************************** * - * Copyright (c) 1999 Balzs Scheidler + * Copyright (c) 1999 Bal�zs Scheidler * Copyright (c) 1999-2007 BalaBit IT Ltd. * * This program is free software; you can redistribute it and/or modify @@ -86,6 +86,7 @@ static struct keyword keywords[] = { { "flush_timeout", KW_FLUSH_TIMEOUT }, { "sync_freq", KW_FLUSH_LINES }, /* obsolete */ { "sync", KW_FLUSH_LINES }, /* obsolete */ + { "summary", KW_SUMMARY }, { "fsync", KW_FSYNC }, { "long_hostnames", KW_CHAIN_HOSTNAMES }, { "chain_hostnames", KW_CHAIN_HOSTNAMES }, diff --git a/src/logmsg.c b/src/logmsg.c index 05354b5..9219a01 100644 --- a/src/logmsg.c +++ b/src/logmsg.c @@ -653,6 +653,25 @@ log_msg_new(gchar *msg, gint length, GSockAddr *saddr, guint flags, regex_t *bad } /** + * log_msg_new_internal: + * @prio: message priority (LOG_*) + * @msg: message text + * @flags: parse flags (LP_*) + * + * This function creates a new log message for messages originating + * internally to syslog-ng + **/ +LogMessage * +log_msg_new_internal(gint prio, gchar *msg, guint flags) +{ + gchar *buf = g_strdup_printf("<%d> syslog-ng[%d]: %s\n", prio, getpid(), msg); + LogMessage *self = log_msg_new(buf, s trlen(buf), NULL, flags, NULL); + g_free(buf); + + return self; +} + +/** * log_msg_new_mark: * * This function returns a new MARK message. MARK messages have the LF_MARK diff --git a/src/logmsg.h b/src/logmsg.h index 420a711..d780282 100644 --- a/src/logmsg.h +++ b/src/logmsg.h @@ -100,7 +100,7 @@ void log_msg_unref(LogMessage *m); LogMessage *log_msg_new(gchar *msg, gint length, GSockAddr *saddr, guint flags, regex_t *bad_hostname); LogMessage *log_msg_new_mark(void); - +LogMessage *log_msg_new_internal(gint prio, gchar *msg, guint flags); void log_msg_ack_block_inc(LogMessage *m); void log_msg_ack_block_start(LogMessage *m, LMAckFunc func, gpointer user_data); void log_msg_ack_block_end(LogMessage *m); diff --git a/src/logwriter.c b/src/logwriter.c index bb82b43..864b193 100644 --- a/src/logwriter.c +++ b/src/logwriter.c @@ -179,10 +179,104 @@ log_writer_watch_new(LogWriter *writer, FDWrite *fd) } static void +log_writer_last_msg_release(LogWriter *self) +{ + if(self->last_msg_timerid) + g_source_remove(self->last_msg_timerid); + + if(self->last_msg) + log_msg_unref(self->last_msg); + + self->last_msg = NULL; + self->last_msg_count = 0; + self->last_msg_timerid = 0; +} + +static void +log_writer_last_msg_flush(LogWriter *self) +{ + gchar *msg; + LogMessage *m; + + msg = g_strdup_printf("Last message '%.20s' repeated %d times\n", + self->last_msg->msg.str, + self->last_msg_count); + + m = log_msg_new_internal(self->last_msg->pri, msg, 0); + + g_string_assign(&m->host,self->last_msg->host.str ); + g_string_assign(&m->host_from,self->last_msg->host_from.str ); + g_queue_push_tail(self->queue, m); + g_queue_push_tail(self->queue, GUINT_TO_POINTER(0x80000000)); + g_free(msg); + + log_writer_last_msg_release(self); +} + +static gboolean +last_msg_timer(gpointer pt) +{ + LogWriter *self = (LogWriter *)pt; + + if(self->last_msg_count) + log_writer_last_msg_flush(self); + else + log_writer_last_msg_release(self); + + return FALSE; +} + +static void +log_writer_last_msg_record(LogWriter *self, LogMessage *lm) +{ + if(self->last_msg) + log_msg_unref(self->last_msg); + + log_msg_ref(lm); + self->last_msg = lm; + self->last_msg_count = 0; + self->last_msg_timerid = g_timeout_add(self->options->summary * 1000, last_msg_timer, self); +} + +static gboolean +log_writer_last_msg_check(LogWriter *self, LogMessage *lm, gint path_flags) +{ + if (self->options->summary <= 0) + return FALSE; + + if(self->last_msg) + { + if(strcmp(self->last_msg->msg.str, lm->msg.str) == 0 && + strcmp(self->last_msg->host.str, lm->host.str) == 0) + { + if (self->dropped_messages) + (*self->dropped_messages)++; + msg_debug("Dropping duplicate message", + NULL); + self->last_msg_count++; + log_msg_drop(lm, path_flags); + return TRUE; + } + + if (self->last_msg_count) + log_writer_last_msg_flush(self); + else + log_writer_last_msg_release(self); + } + + log_writer_last_msg_record(self, lm); + + return FALSE; +} + +static void log_writer_queue(LogPipe *s, LogMessage *lm, gint path_flags) { LogWriter *self = (LogWriter *) s; - + + if(log_writer_last_msg_check(self, lm, path_flags)) + return; + if ((self->queue->length / 2) == self->options->fifo_size) { /* drop incoming message, we must ack here, otherwise the sender might @@ -462,6 +556,9 @@ log_writer_new(guint32 flags, LogPipe *control, LogWriterOptions *options) self->queue = g_queue_new(); self->flags = flags; self->control = control; + self->last_msg = NULL; + self->last_msg_count = 0; + self->last_msg_timerid = 0; return &self->super; } @@ -476,6 +573,+ options->summary = 0; } void diff --git a/src/logwriter.h b/src/logwriter.h index 284899b..452693c 100644 --- a/ src/logwriter.h +++ b/src/logwriter.h @@ -66,6 +66,7 @@ typedef struct _LogWriterOptions gshort ts_format; glong zone_offset; gshort frac_digits; + gint summary; } LogWriterOptions; typedef struct _LogWriter @@ -79,6 +80,10 @@ typedef struct _LogWriter gint partial_pos; LogPipe *control; LogWriterOptions *options; + LogMessage *last_msg; + guint32 last_msg_count; + guint last_msg_timerid; + } LogWriter; void log_writer_set_options(LogWriter *self, LogPipe *control, LogWriterOptions *options); diff --git a/src/messages.c b/src/messages.c index 6c69f48..206206a 100644 --- a/src/messages.c +++ b/src/messages.c @@ -56,10 +56,8 @@ msg_send_internal_message(int prio, const char *msg) if (G_LIKELY(internal_msg_queue)) { - buf = g_strdup_printf("<%d> syslog-ng[%d]: %s\n", prio, getpid(), msg); - m = log_msg_new(buf, strlen(buf), NULL, LP_INTERNAL | LP_LOCAL, NULL); + m = log_msg_new_internal(prio, msg, LP_INTERNAL | LP_LOCAL); g_queue_push_tail(internal_msg_queue, m); - g_free(buf); } } } -- 1.5.4.5

2 1

Re: [syslog-ng] Filtering duplicate messages
by chris packham 29 May '08

29 May '08

Here's a rough cut of my patch. It still needs a bit of cleanup and I have yet to implement the time-based behaviour. I just wanted to get some feedback. One particular thing I wanted to clarify is where in the log message pipeline would best to attach this behaviour to? I've currently attached it to the LogWriter. The disadvantage of that is that the new message I generate does not get passed through the filters or templates (and we've just done a bunch of wasted processing on a message that gets dropped) The behaviour implemented is as follows. As a log message is output to a destination it is remembered as the last message seen. When a new message is to be output it is checked against the last message seen. If its contents (message text, including hostname and pid if present) are the same the message is dropped and a couter incremented. If the message is different to the last message seen a new log message indicating the value of the last message counter will be inserted into the queue and the new message will be processed. TODO: Implement a time based flush of last seen messages --- src/cfg-grammar.y | 3 +- src/cfg-lex.l | 3 +- src/logmsg.c | 26 +++++++++++++++++++++++ src/logmsg.h | 2 + src/logwriter.c | 59 ++++++++++++++++++++++++++++++++++++++++++++++++++++- src/logwriter.h | 9 ++++++++ 6 files changed, 99 insertions(+), 3 deletions(-) diff --git a/src/cfg-grammar.y b/src/cfg-grammar.y index 39ff172..386c231 100644 --- a/src/cfg-grammar.y +++ b/src/cfg-grammar.y @@ -62,7 +62,7 @@ gint last_addr_family = AF_INET; %token KW_USER KW_DOOR KW_SUN_STREAMS KW_PROGRAM /* option items */ -%token KW_FSYNC KW_MARK_FREQ KW_STATS_FREQ KW_FLUSH_LINES KW_FLUSH_TIMEOUT KW_LOG_MSG_SIZE KW_FILE_TEMPLATE KW_PROTO_TEMPLATE +%token KW_FSYNC KW_MARK_FREQ KW_STATS_FREQ KW_FLUSH_LINES KW_SUPPRESS KW_FLUSH_TIMEOUT KW_LOG_MSG_SIZE KW_FILE_TEMPLATE KW_PROTO_TEMPLATE %token KW_CHAIN_HOSTNAMES KW_NORMALIZE_HOSTNAMES KW_KEEP_HOSTNAME KW_CHECK_HOSTNAME KW_BAD_HOSTNAME %token KW_KEEP_TIMESTAMP @@ -671,6 +671,7 @@ dest_writer_option : KW_FLAGS '(' dest_writer_options_flags ')' { last_writer_options->options = $3; } | KW_LOG_FIFO_SIZE '(' NUMBER ')' { last_writer_options->fifo_size = $3; } | KW_FLUSH_LINES '(' NUMBER ')' { last_writer_options->flush_lines = $3; } + | KW_SUPPRESS '(' yesno ')' { last_writer_options->suppress= $3; } | KW_FLUSH_TIMEOUT '(' NUMBER ')' { last_writer_options->flush_timeout = $3; } | KW_TEMPLATE '(' string ')' { last_writer_options->template = cfg_lookup_template(configuration, $3); diff --git a/src/cfg-lex.l b/src/cfg-lex.l index 5c5f54f..4eb0c63 100644 --- a/src/cfg-lex.l +++ b/src/cfg-lex.l @@ -6,7 +6,7 @@ /*************************************************************************** * - * Copyright (c) 1999 Balzs Scheidler + * Copyright (c) 1999 Bal�zs Scheidler * Copyright (c) 1999-2007 BalaBit IT Ltd. * * This program is free software; you can redistribute it and/or modify @@ -86,6 +86,7 @@ static struct keyword keywords[] = { { "flush_timeout", KW_FLUSH_TIMEOUT }, { "sync_freq", KW_FLUSH_LINES }, /* obsolete */ { "sync", KW_FLUSH_LINES }, /* obsolete */ + { "suppress", KW_SUPPRESS }, { "fsync", KW_FSYNC }, { "long_hostnames", KW_CHAIN_HOSTNAMES }, { "chain_hostnames", KW_CHAIN_HOSTNAMES }, diff --git a/src/logmsg.c b/src/logmsg.c index 05354b5..4da3af5 100644 --- a/src/logmsg.c +++ b/src/logmsg.c @@ -766,3 +766,29 @@ log_msg_drop(LogMessage *m, guint path_flags) log_msg_ack(m); log_msg_unref(m); } + +/** + * log_msg_dup: + * @m: LogMessage instance + * + * This function duplicates an existing log message. The new message is not parsed. + */ +LogMessage * +log_msg_dup(LogMessage *m) +{ + LogMessage *nm = g_new0(LogMessage, 1); + + log_msg_init(nm, m->saddr); + nm->flags = m->flags; + nm->pri = m->pri; + g_string_assign_len(&nm->date, m->date.str, m->date.len); + g_string_assign_len(&nm->host, m->host.str, m->host.len); + g_string_assign_len(&nm->host_from, m->host_from.str, m->host_from.len); + g_string_assign_len(&nm->program, m->program.str, m->program.len); + g_string_assign_len(&nm->msg, m->msg.str, m->msg.len); + memcpy(&nm->stamp, &m->stamp, sizeof(LogStamp)); + memcpy(&nm->recvd, &m->recvd, sizeof(LogStamp)); + + return nm; +} + diff --git a/src/logmsg.h b/src/logmsg.h index 420a711..9dbb886 100644 --- a/src/logmsg.h +++ b/src/logmsg.h @@ -107,6 +107,8 @@ void log_msg_ack_block_end(LogMessage *m); void log_msg_ack(LogMessage *msg); void log_msg_drop(LogMessage *msg, guint path_flags); void log_msg_clear_matches(LogMessage *self); +LogMessage *log_msg_dup(LogMessage *m); + gchar *log_msg_find_cr_or_lf(gchar *s, gsize n); diff --git a/src/logwriter.c b/src/logwriter.c index bb82b43..4843100 100644 --- a/src/logwriter.c +++ b/src/logwriter.c @@ -178,11 +178,63 @@ log_writer_watch_new(LogWriter *writer, FDWrite *fd) return &self->super; } +static gboolean +log_writer_supress_duplicate(LogWriter *self, LogMessage *lm, gint path_flags) +{ + if (!self->options->suppress) + return FALSE; + + if(self->suppress.hash) + { + guint hash = g_string_hash(&lm->msg); + if(self->suppress.hash == hash) + { + msg_debug("Dropping duplicate message", + NULL); + self->suppress.count ++; + log_msg_drop(lm, path_flags); + return TRUE; + } + + if (self->suppress.count) + { + gchar *buf; + LogMessage *m; + GString *line = g_string_sized_new(128); + + buf = g_strdup_printf("<%d> syslog-ng[%d]: Last message '%.20s' repeated %d times\n", + self->suppress.msg->pri, + getpid(), + self->suppress.msg->msg.str, + self->suppress.count); + m = log_msg_new(buf, strlen(buf), self->suppress.msg->saddr, 0, NULL); + g_queue_push_tail(self->queue, m); + g_queue_push_tail(self->queue, GUINT_TO_POINTER(0x80000000)); + g_free(buf); + } + log_msg_drop(self->suppress.msg, 0); + self->suppress.msg = log_msg_dup(lm); + self->suppress.hash = g_string_hash(&lm->msg); + self->suppress.count = 0; + } + else + { + self->suppress.msg = log_msg_dup(lm); + self->suppress.hash = g_string_hash(&lm->msg); + self->suppress.count = 0; + } + + return FALSE; +} + static void log_writer_queue(LogPipe *s, LogMessage *lm, gint path_flags) { LogWriter *self = (LogWriter *) s; - + + if(log_writer_supress_duplicate(self, lm, path_flags)) + return; + if ((self->queue->length / 2) == self->options->fifo_size) { /* drop incoming message, we must ack here, otherwise the sender might @@ -462,6 +514,9 @@ log_writer_new(guint32 flags, LogPipe *control, LogWriterOptions *options) self->queue = g_queue_new(); self->flags = flags; self->control = control; + self->suppress.msg = NULL; + self->suppress.hash = 0; + self->suppress.count = 0; return &self->super; } @@ -476,6 +531,8 @@ log_writer_options_defaults(LogWriterOptions *options) options->ts_format = -1; options->zone_offset = -1; options->frac_digits = -1; + options->suppress = FALSE; + } void diff --git a/src/logwriter.h b/src/logwriter.h index 284899b..d3a136f 100644 --- a/src/logwriter.h +++ b/src/logwriter.h @@ -41,6 +41,13 @@ /* several writers use the same counter */ #define LWOF_SHARE_STATS 0x0002 +typedef struct _LogSuppress +{ + LogMessage *msg; + guint hash; + guint32 count; +} LogSuppress; + glong zone_offset; gshort frac_digits; + gboolean suppress; } LogWriterOptions; typedef struct _LogWriter @@ -79,6 +87,7 @@ typedef struct _LogWriter gint partial_pos; LogPipe *control; LogWriterOptions *options; + LogSuppress suppress; } LogWriter; void log_writer_set_options(LogWriter *self, LogPipe *control, LogWriterOptions *options); -- 1.5.4.5

2 1

Re: [syslog-ng] Filtering duplicate messages
by chris packham 29 May '08

29 May '08

Actually the new message does hit the templates, just not the filters. On Thu, 2008-05-29 at 12:57 +1200, chris packham wrote: > Here's a rough cut of my patch. It still needs a bit of cleanup and I > have yet to implement the time-based behaviour. I just wanted to get > some > feedback. > > One particular thing I wanted to clarify is where in the log message > pipeline would best to attach this behaviour to? I've currently > attached it to the LogWriter. The disadvantage of that is that the new > message I generate does not get passed through the filters or templates > (and we've just done a bunch of wasted processing on a message that > gets dropped) > > The behaviour implemented is as follows. > > As a log message is output to a destination it is remembered > as the last message seen. > > When a new message is to be output it is checked against the > last message seen. If its contents (message text, including > hostname and pid if present) are the same the message is dropped > and a couter incremented. If the message is different to the > last message seen a new log message indicating the value of the > last message counter will be inserted into the queue and the new > message will be processed. > > TODO: Implement a time based flush of last seen messages > > --- > src/cfg-grammar.y | 3 +- > src/cfg-lex.l | 3 +- > src/logmsg.c | 26 +++++++++++++++++++++++ > src/logmsg.h | 2 + > src/logwriter.c | 59 > ++++++++++++++++++++++++++++++++++++++++++++++++++++- > src/logwriter.h | 9 ++++++++ > 6 files changed, 99 insertions(+), 3 deletions(-) > > diff --git a/src/cfg-grammar.y b/src/cfg-grammar.y > index 39ff172..386c231 100644 > --- a/src/cfg-grammar.y > +++ b/src/cfg-grammar.y > @@ -62,7 +62,7 @@ gint last_addr_family = AF_INET; > %token KW_USER KW_DOOR KW_SUN_STREAMS KW_PROGRAM > > /* option items */ > -%token KW_FSYNC KW_MARK_FREQ KW_STATS_FREQ KW_FLUSH_LINES > KW_FLUSH_TIMEOUT KW_LOG_MSG_SIZE KW_FILE_TEMPLATE KW_PROTO_TEMPLATE > +%token KW_FSYNC KW_MARK_FREQ KW_STATS_FREQ KW_FLUSH_LINES KW_SUPPRESS > KW_FLUSH_TIMEOUT KW_LOG_MSG_SIZE KW_FILE_TEMPLATE KW_PROTO_TEMPLATE > > %token KW_CHAIN_HOSTNAMES KW_NORMALIZE_HOSTNAMES KW_KEEP_HOSTNAME > KW_CHECK_HOSTNAME KW_BAD_HOSTNAME > %token KW_KEEP_TIMESTAMP > @@ -671,6 +671,7 @@ dest_writer_option > : KW_FLAGS '(' dest_writer_options_flags ')' { > last_writer_options->options = $3; } > | KW_LOG_FIFO_SIZE '(' NUMBER ')' { > last_writer_options->fifo_size = $3; } > | KW_FLUSH_LINES '(' NUMBER ')' { > last_writer_options->flush_lines = $3; } > + | KW_SUPPRESS '(' yesno ')' { > last_writer_options->suppress= $3; } > | KW_FLUSH_TIMEOUT '(' NUMBER ')' { > last_writer_options->flush_timeout = $3; } > | KW_TEMPLATE '(' string ')' { > > last_writer_options->template = cfg_lookup_template(configuration, $3); > diff --git a/src/cfg-lex.l b/src/cfg-lex.l > index 5c5f54f..4eb0c63 100644 > --- a/src/cfg-lex.l > +++ b/src/cfg-lex.l > @@ -6,7 +6,7 @@ > > > /*************************************************************************** > * > - * Copyright (c) 1999 Balzs Scheidler > + * Copyright (c) 1999 Bal�zs Scheidler > * Copyright (c) 1999-2007 BalaBit IT Ltd. > * > * This program is free software; you can redistribute it and/or modify > @@ -86,6 +86,7 @@ static struct keyword keywords[] = { > { "flush_timeout", KW_FLUSH_TIMEOUT }, > { "sync_freq", KW_FLUSH_LINES }, /* obsolete */ > { "sync", KW_FLUSH_LINES }, /* obsolete */ > + { "suppress", KW_SUPPRESS }, > { "fsync", KW_FSYNC }, > { "long_hostnames", KW_CHAIN_HOSTNAMES }, > { "chain_hostnames", KW_CHAIN_HOSTNAMES }, > diff --git a/src/logmsg.c b/src/logmsg.c > index 05354b5..4da3af5 100644 > --- a/src/logmsg.c > +++ b/src/logmsg.c > @@ -766,3 +766,29 @@ log_msg_drop(LogMessage *m, guint path_flags) > log_msg_ack (m); > log_msg_unref(m); > } > + > +/** > + * log_msg_dup: > + * @m: LogMessage instance > + * > + * This function duplicates an existing log message. The new message is > not parsed. > + */ > +LogMessage * > +log_msg_dup(LogMessage *m) > +{ > + LogMessage *nm = g_new0(LogMessage, 1); > + > + log_msg_init(nm, m->saddr); > + nm->flags > = m->flags; > + nm->pri = m->pri; > + g_string_assign_len(&nm->date, m->date.str, m->date.len); > + g_string_assign_len(&nm->host, m->host.str, m->host.len); > + g_string_assign_len(&nm->host_from, m->host_from.str, > m->host_from.len); > + g_string_assign_len(&nm->program, m->program.str, m->program.len); > + g_string_assign_len(&nm->msg, m->msg.str, m->msg.len); > + memcpy(&nm->stamp, &m->stamp, sizeof(LogStamp)); > + memcpy(&nm->recvd, &m->recvd, sizeof(LogStamp)); > + > + return nm; > +} > + > diff --git a/src/logmsg.h b/src/logmsg.h > index 420a711..9dbb886 100644 > --- a/src/logmsg.h > +++ b/src/logmsg.h > @@ -107,6 +107,8 @@ void log_msg_ack_block_end(LogMessage *m); > void log_msg_ack(LogMessage *msg); > void log_msg_drop(LogMessage *msg, guint path_flags); > void log_msg_clear_matches(LogMessage *self); > +LogMessage *log_msg_dup(LogMessage *m); > + > > gchar *log_msg_find_cr_or_lf(gchar *s, gsize n); > > diff --git a/src/logwriter.c b/src/logwriter.c > index bb82b43..4843100 100644 > --- a/src/logwriter.c > +++ b/src/logwriter.c > @@ -178,11 +178,63 @@ log_writer_watch_new(LogWriter *writer, FDWrite > *fd) > return &self->super; > } > > +static gboolean > +log_writer_supress_duplicate(LogWriter *self, LogMessage *lm, gint > path_flags) > +{ > + if (!self->options->suppress) > + return FALSE; > + > + if(self->suppress.hash) > + { > + guint hash = g_string_hash(&lm->msg); > + if(self->suppress.hash == hash) > + { > + msg_debug("Dropping duplicate message", > + NULL); > + self->suppress.count ++; > + log_msg_drop(lm, path_flags); > + return TRUE; > + } > + > + if (self->suppress.count) > + { > + gchar *buf; > + LogMessage *m; > + GString *line = g_string_sized_new(128); > + > + buf = g_strdup_printf("<%d> syslog-ng[%d]: Last message > '%.20s' repeated %d times\n", > + self->suppress.msg->pri, > + getpid(), > + self->suppress.msg->msg.str, > + self->suppress.count); > + m = log_msg_new(buf, strlen(buf), self->suppress.msg->saddr, > 0, NULL); > + g_queue_push_tail(self->queue, m); > + g_queue_push_tail(self->queue, GUINT_TO_POINTER(0x80000000)); > + g_free(buf); > + } > + log_msg_drop(self->suppress.msg, 0); > + self->suppress.msg = log_msg_dup(lm); > + self->suppress.hash = g_string_hash(&lm->msg); > + self->suppress.count = 0; > + } > + else > + { > + self->suppress.msg = log_msg_dup(lm); > + self->suppress.hash = g_string_hash(&lm->msg); > + self->suppress.count = 0; > + } > + > + return FALSE; > +} > + > static void > log_writer_queue(LogPipe *s, LogMessage *lm, gint path_flags) > { > LogWriter *self = (LogWriter *) s; > - > + > + if(log_writer_supress_duplicate(self, lm, path_flags)) > + return; > + > if ((self->queue->length / 2) == self->options->fifo_size) > { > /* drop incoming message, we must ack here, otherwise the sender > might > @@ -462,6 +514,9 @@ log_writer_new(guint32 flags, LogPipe *control, > LogWriterOptions *options) > self->queue = g_queue_new(); > self->flags = flags; > self->control = control; > + self->suppress.msg = NULL; > + self->suppress.hash = 0; > + self->suppress.count = 0; > return &self->super; > > options->zone_offset = -1; > options->frac_digits = -1; > + options->suppress = FALSE; > + > } > > void > diff --git a/src/logwriter.h b/src/logwriter.h > index 284899b..d3a136f 100644 > --- a/src/logwriter.h > +++ b/src/logwriter.h > @@ -41,6 +41,13 @@ > /* several writers use the same counter */ > #define LWOF_SHARE_STATS 0x0002 > > +typedef struct _LogSuppress > +{ > + LogMessage *msg; > + guint hash; > + guint32 count; > +} LogSuppress; > + > glong zone_offset; > gshort frac_digits; > + gboolean suppress; > } LogWriterOptions; > > typedef struct _LogWriter > @@ > -79,6 +87,7 @@ typedef struct _LogWriter > gint partial_pos; > LogPipe *control; > LogWriterOptions *options; > + LogSuppress suppress; > } LogWriter; > > void log_writer_set_options(LogWriter *self, LogPipe *control, > LogWriterOptions *options); > -- > 1.5.4.5 > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.campin.net/syslog-ng/faq.html >

1 0

Re: [syslog-ng] Syslog-ng not restarting
by Francisco Javier Lizarán Vilches 28 May '08

28 May '08

I have installed syslog-ng-premium-edition-2.1.10a-1.i386.rpm for RHEL-5. These are some commands and its outputs: -Stop ok. # /etc/init.d/syslog-ng stop Shutting down syslog-ng: [ OK ] -Start doesn't work. # /etc/init.d/syslog-ng start Starting syslog-ng: /bin/bash: line 1: 20231 Abortado /sbin/syslog-ng [FALLÃ] -Status ok. # /etc/init.d/syslog-ng status syslog-ng estÃ¡ parado After that (and after the system booting) I have to execute syslog-ng manually: # syslog-ng -f /etc/syslog-ng/syslog-ng.conf & [1] 20350 # /etc/init.d/syslog-ng status Se estÃ¡ ejecutando syslog-ng (pid 20351)... [1]+ Done syslog-ng -f /etc/syslog-ng/syslog-ng.conf -Restart doesn't work. # /etc/init.d/syslog-ng restart Restarting syslog-ng: /etc/init.d/syslog-ng: line 41: 20493 Abortado $binary --syntax-only Apart from that, syslog-ng operation is normal. FJ -----Mensaje original----- De: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] En nombre de Balazs Scheidler Enviado el: miércoles, 28 de mayo de 2008 16:31 Para: Syslog-ng users' and developers' mailing list Asunto: Re: [syslog-ng] Syslog-ng not restarting On Wed, 2008-05-28 at 15:58 +0200, Francisco Javier Lizarán Vilches wrote: > > > Hi all: > > > > Just installed and configured syslog-ng-pe rpm with mysql > support on a Centos 5.1 system. When I try to restart syslog-ng > issuing: > > #/etc/init.d/syslog-ng restart > > I get: > > Restarting syslog-ng: /etc/init.d/syslog-ng: line 41: 18237 > $binary --syntax-only > > > > Moreover, when I reboot the system the service fails too. > > Hmm what version are you running? Do you have a longer output? if an executed program gets a signal, then bash reports the received signal in the next line. In the meanwhile I have this tested on CentOS5. -- Bazsi ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html ___________________________________________________________________________ Este mensaje se dirige exclusivamente a su destinatario y puede contener información privilegiada o confidencial. Si no es vd. el destinatario indicado, queda notificado de que la utilización, divulgación y/o copia sin autorización está prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción. This message is intended exclusively for its addressee and may contain information that is CONFIDENTIAL and protected by professional privilege. If you are not the intended recipient you are hereby notified that any dissemination, copy or disclosure of this communication is strictly prohibited by law. If this message has been received in error, please immediately notify us via e-mail and delete it. ___________________________________________________________________________

2 1

Syslog-ng not restarting
by Francisco Javier Lizarán Vilches 28 May '08

28 May '08

Hi all: Just installed and configured syslog-ng-pe rpm with mysql support on a Centos 5.1 system. When I try to restart syslog-ng issuing: #/etc/init.d/syslog-ng restart I get: Restarting syslog-ng: /etc/init.d/syslog-ng: line 41: 18237 $binary --syntax-only Moreover, when I reboot the system the service fails too. Thanks: FJ ___________________________________________________________________________ Este mensaje se dirige exclusivamente a su destinatario y puede contener información privilegiada o confidencial. Si no es vd. el destinatario indicado, queda notificado de que la utilización, divulgación y/o copia sin autorización está prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción. This message is intended exclusively for its addressee and may contain information that is CONFIDENTIAL and protected by professional privilege. If you are not the intended recipient you are hereby notified that any dissemination, copy or disclosure of this communication is strictly prohibited by law. If this message has been received in error, please immediately notify us via e-mail and delete it. ___________________________________________________________________________

2 1

Re: [syslog-ng] Filtering duplicate messages
by chris packham 27 May '08

27 May '08

Yes that's the first problem I ran into with my patch. It worked well with a single host but when I tried it with multiple hosts things got a bit messy. The point about timing information is definitely worth noting. This is often crucial in debugging issues. I wouldn't suggest making this default behaviour, admins would have to turn it on with the knowledge that they may lose some timing info (although the first message will have the correct time). On Tue, 2008-05-27 at 18:25 +0200, Balazs Scheidler wrote: > On Mon, 2008-05-26 at 20:05 -0700, stucky wrote: > > I've been after this since I switched to ng and realized this feature > > was not migrated over (for reasons beyond me since this was the only > > good feature of syslog !) > > I'd be very interested in having this ported to the 2.x branch but I'm > > not a programmer so I need to rely on you code gurus for that. > > Nice to see that someone is on it and I'd be happy to help test it ! > > the problem with suppressing duplicate messages is that it loses too > much information, and once you collect messages from several devices > into the same file, the message "Last message repeated N times" does not > really have too much information. You lose: > * host information > * timing > * the message itself > > So analyzing this is almost impossible. I might integrate a patch that > implements this, though. >

1 0

Filtering duplicate messages
by chris packham 27 May '08

27 May '08

Hi, I've done a bit of searching on this mailing list and found some messages requesting a feature where duplicate messages are filtered/suppressed/summarised (I'm after the same thing). There appears to be a patch for syslog-ng 1.6.12 in openSUSE that addresses this for the older version. Is anyone actually working on this functionality for a 2.x release? I've got a proof of concept working on top of the latest 2.0.x git branch but before I get too carried away I wanted to check if anyone else is working on it (or if someone has already implemented it). Thanks, Chris

3 3

rsyslog to syslog-ng
by Robert Shaw 22 May '08

22 May '08

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, I'm having a problem with logs from rsyslog to syslog-ng can't detect facility so I can't separate applications, this is important, mostly because we have 2 instances of postgres that need to be in separate logfiles. So instead of having pgsql0 and pgsql1 files created its all dumped into syslog, so we can't differentiate between the two postgres instances. The syslog-ng instance is our central logging host, and the reason we are using rsyslog is because one of our specific applications needs it. I hope these config files help, I really need to get this sorted, if anyone could help it would be great rsyslog.conf # PostgreSQL logging stuff local0.* @loghost;RSYSLOG_TraditionalFileFormat local1.* @loghost;RSYSLOG_TraditionalFileFormat syslog-ng.conf options { chain_hostnames(no); create_dirs (yes); dir_perm(0755); keep_hostname(yes); log_fifo_size(1024); perm(0644); sync(0); time_reopen (10); use_dns(yes); use_fqdn(yes); sync(0); }; # Local logging and getting logs from remote via UDP source net { udp(); tcp(port(5140) keep-alive(yes)); }; destination net_log { file("/var/log/hosts/${HOST}/${YEAR}/${MONTH}/${DAY}/syslog"); }; destination pgnet_log0 { file("/var/log/hosts/${HOST}/${YEAR}/${MONTH}/${DAY}/pgsql0"); }; destination pgnet_log1 { file("/var/log/hosts/${HOST}/${YEAR}/${MONTH}/${DAY}/pgsql1"); }; filter f_pgsql0 { facility(local0); }; filter f_pgsql1 { facility(local1); }; log { source(net); filter(f_pgsql0); destination(pgnet_log0); }; log { source(net); filter(f_pgsql1); destination(pgnet_log1); }; log { source(net); destination(net_log); }; Cheers, Rob Shaw -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQEVAwUBSDUKu3LIVYILPVW0AQL9JAf9E28U7AQK25o9FwAd4xVb11GObxuZkusu q/IwL/+92LDB97PQ1fB9fdLUmP7MABYk+N6+kohrCJtWM+TpY5OoUBt0kE6PMBI7 Nj5GVi/IlkRMCTHrVlsNdfcciQeY2+WPlJAtFJJbwIA+kzj0O/keG6lSpFcfZ0BN Eg536qlaOLVHQIUq7N8NzjZTkLr2FkAxEMVfOKpOryDjuTcnyrdmZ5iqx7n+nY4y VPtQW1mNwTs9BRf80g/Vx+DzNdoK0MbXKTogkMSNBoNSia8tvatA7CW+45TO3ucG gwo4KtRlS5BgRS++cMgmFwkDSPiI4cNYDGrAMWwch8o2yqSyIXf8Cw== =Tfas -----END PGP SIGNATURE-----

3 2

syslog-ng with piped fifo
by Steve Bielen 21 May '08

21 May '08

greetings I have setup a syslog-ng server, which I intent to use to filter the messages from squid. Currently I filter all the messages from squid and send them to a pipe. on the other end of this pipe I have my perl script which formats the data and stores it into my postgres database. the problem is that when I try to run /tmp/pgsql.pipe > script.perl I get the error permission denied. I run that command as root, the pipe is owned by root, and the owner. and the same goes for script.perl. second question: In most tutorials I see for database usage with syslog-ng everyone used a template insert command with variables. does anyone know if such variables exist for the squid access logs ? best wishes -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.

2 1

Syslog-ng DNS Problem
by zarahel＠iol.pt 20 May '08

20 May '08

Good morning to you all I need your help I´ve installed syslog-ng and it´s cacti plugin and it´s working fine, except for it´s name resolution. For some reason it resolves some names, and doesn´t resolve others. I´m able to ping my hosts by their names without any problem, and I´ve set use_dns(yes) in syslog-ng.conf I´ve done nslookup, and everything´s as it should. Does anybody have any idea why syslog-ng is resolving only some of my hosts names? Any help is very apreciated. Thkx I apreciate any help Thkx ________________________________________________________________________________ Cartão Citi: receba GRÁTIS uma máquina de café Peça aqui o seu http://www.iol.pt/correio/rodape.php?dst=0805131

2 2