July 2011 - syslog-ng - lists.balabit.hu

[Bug 107] New: when trying to log to postgres database if port is not specified it gets defaulted to 0
by bugzilla＠bugzilla.balabit.com 23 Oct '11

23 Oct '11

https://bugzilla.balabit.com/show_bug.cgi?id=107 Summary: when trying to log to postgres database if port is not specified it gets defaulted to 0 Product: syslog-ng Version: 3.2.x Platform: PC OS/Version: Linux Status: NEW Severity: minor Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: jhnsmth64(a)gmail.com Type of the Report: bug Estimated Hours: 0.0 syslog-ng.conf ... destination d_pgsql { sql(type(pgsql) host("localhost") username("user") password("pass") ... }; ... If destination stays defined like above, without port specified explicitly, syslog sends log messages to port 0 instead defaulting port to the Postgres port (5432). Destination settings below works: ... destination d_pgsql { sql(type(pgsql) host("localhost") port(5432) username("user") password("pass") ... }; ... -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

2 4

[Bug 106] New: Syslog-ng memory strdup and g_malloc leaks
by bugzilla＠bugzilla.balabit.com 23 Oct '11

23 Oct '11

https://bugzilla.balabit.com/show_bug.cgi?id=106 Summary: Syslog-ng memory strdup and g_malloc leaks Product: syslog-ng Version: 3.2.x Platform: Other OS/Version: Solaris Status: NEW Severity: minor Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: daniel_novak(a)kb.cz Type of the Report: refactorization Estimated Hours: 0.0 We tried to compile a few last git version of syslog-3.2.1(syslog-ng-3.2.git-5f25ce47ab57774f8fe8df17bc96a006e535cb53) as well as 3.2.1 stable release. Everything is running well, but there are several minor memory leaks.We run on Solaris 10/08 on sparc platform.With libumem.so in memory debug mode MDB output is following : > ::findleaks CACHE LEAKED BUFCTL CALLER 00052288 2 00110d98 libc.so.1`strdup+0xc 00052788 1 0007de00 libglib-2.0.so.0.2513.0`g_malloc0+0x64 ---------------------------------------------------------------------- Total 3 buffers, 64 bytes =========================================================================================== > ::findleaks -dvf findleaks: maximum buffers => 4121 findleaks: actual buffers => 3967 mdb: [fef80000, fef84000): couldn't read 16384 bytes at fef80000: no mapping for address findleaks: findleaks: potential pointers => 341574 findleaks: dismissals => 196071 (57.4%) findleaks: misses => 130325 (38.1%) findleaks: dups => 11214 ( 3.2%) findleaks: follows => 3964 ( 1.1%) findleaks: findleaks: elapsed wall time => 0 seconds findleaks: CACHE LEAKED BUFCTL CALLER 00052288 2 00110d98 libc.so.1`strdup+0xc 00052788 1 0007de00 libglib-2.0.so.0.2513.0`g_malloc0+0x64 ---------------------------------------------------------------------- Total 3 buffers, 64 bytes umem_alloc_16 leak: 2 buffers, 16 bytes each, 32 bytes total ADDR BUFADDR TIMESTAMP THREAD CACHE LASTLOG CONTENTS 110d98 1097e0 1b08cf3fb0c96c 1 52288 43af4 0 libumem.so.1`umem_cache_alloc+0x210 libumem.so.1`umem_alloc+0x60 libumem.so.1`malloc+0x28 libc.so.1`strdup+0xc libsyslog-ng.so.0.0.0`cfg_lexer_lookup_keyword+0xf0 libsyslog-ng.so.0.0.0`_cfg_lexer_lex+0x18a4 libsyslog-ng.so.0.0.0`cfg_lexer_lex+0x284 libsyslog-ng.so.0.0.0`main_lex+0x1c libsyslog-ng.so.0.0.0`main_parse+0x7a4 libsyslog-ng.so.0.0.0`cfg_parser_parse+0xcc libsyslog-ng.so.0.0.0`cfg_run_parser+0x1ac libsyslog-ng.so.0.0.0`cfg_read_config+0x98 initial_init+0x5c main+0x244 _start+0x5c umem_alloc_32 leak: 1 buffer, 32 bytes ADDR BUFADDR TIMESTAMP THREAD CACHE LASTLOG CONTENTS 7de00 77b98 1b08cf406e21b0 1 52788 373ec 0 libumem.so.1`umem_cache_alloc+0x210 libumem.so.1`umem_alloc+0x60 libumem.so.1`malloc+0x28 libumem.so.1`calloc+0x58 libglib-2.0.so.0.2513.0`g_malloc0+0x64 libsyslog-ng.so.0.0.0`serialize_file_archive_new+0x1c libsyslog-ng.so.0.0.0`persist_state_load+0x54 libsyslog-ng.so.0.0.0`persist_state_start+0x34 libsyslog-ng.so.0.0.0`cfg_initial_init+0x28 initial_init+0xbc main+0x244 _start+0x5c ========================================================================= findleaks: using cached results (-f will force a full run) CACHE LEAKED BUFCTL CALLER 00052288 2 00110d98 libc.so.1`strdup+0xc 00052788 1 0007de00 libglib-2.0.so.0.2513.0`g_malloc0+0x64 ---------------------------------------------------------------------- Total 3 buffers, 64 bytes > 00110d98::umem_verify mdb: cache 110d98 () does not have redzone checking enabled > 00052288::umem_verify Summary for cache 'umem_alloc_16' clean > 00052788::umem_verify Summary for cache 'umem_alloc_32' clean ============================================================================= -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 4

[Bug 104] New: Add libtool option -module
by bugzilla＠bugzilla.balabit.com 23 Oct '11

23 Oct '11

https://bugzilla.balabit.com/show_bug.cgi?id=104 Summary: Add libtool option -module Product: syslog-ng Version: 3.2.x Platform: All OS/Version: All Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: vinschen(a)redhat.com Type of the Report: bug Estimated Hours: 0.0 Created an attachment (id=30) --> (https://bugzilla.balabit.com/attachment.cgi?id=30) Patch to add libtool option -module Libtool 2.2.6 and later provides an option -module which should be used when creating shared libraries which are supposed to be used as modules which get loaded via dlopen. See the "Building modules to dlopen" chapter in `info libtool'. For Cygwin the additional benefit is that the modules are not copied into a parallel "bin" directory at installation time, which is the default otherwise to accommodate the DLL search algorithm on Windows. The attached patch just adds the -module option to each of the Makefile.am files in the modules subdir. Corinna -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

2 4

[Bug 120] New: afsql. c afsql_dd_format_* routines should include table in persist_name string
by bugzilla＠bugzilla.balabit.com 23 Oct '11

23 Oct '11

https://bugzilla.balabit.com/show_bug.cgi?id=120 Summary: afsql.c afsql_dd_format_* routines should include table in persist_name string Product: syslog-ng Version: 3.2.x Platform: Other URL: https://lists.balabit.hu/pipermail/syslog-ng/2011- February/015913.html OS/Version: Other Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: jtk(a)cymru.com Type of the Report: bug Estimated Hours: 0.0 Created an attachment (id=36) --> (https://bugzilla.balabit.com/attachment.cgi?id=36) include table name in persist_name As originally discovered in the syslog-ng list post entitled "[syslog-ng] duplicate config elements in sql destinations" by John Kristoff on 2011-02-11, syslog-ng will complain, when multiple destinations all point to the same database, but different tables with the message: Internal error, duplicate configuration elements refer to the same persistent config; name='afsql_dd(pgsql,127.0.0.1,,bar)' Balazs responded describing the potential problems with counters being swapped and a fix that appears to work in testing. A Patch for syslog-ng 3.2.4 is attached, but similar patch should work for earlier versions as indicated by Balazs' posting. Thank you, John -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

[PATCH] afmongodb: Support system-installed libmongo-client at configure time.
by Gergely Nagy 30 Sep '11

30 Sep '11

Modify configure.in to recognise a suitable version (>= 0.1.0) of the libmongo-client library, and use it instead of the internal version if requested so. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- configure.in | 10 ++++++---- 1 files changed, 6 insertions(+), 4 deletions(-) diff --git a/configure.in b/configure.in index e5bf174..e7038fb 100644 --- a/configure.in +++ b/configure.in @@ -28,6 +28,7 @@ IVYKIS_MIN_VERSION="0.18" JSON_C_MIN_VERSION="0.7" JSON_GLIB_MIN_VERSION="0.12" PCRE_MIN_VERSION="6.1" +LMC_MIN_VERSION="0.1.0" dnl *************************************************************************** dnl Initial setup @@ -743,12 +744,13 @@ if test "x$with_libmongo_client" = "xinternal"; then AC_MSG_WARN([Internal libmongo-client sources not found in modules/afmongodb/libmongo-client]) with_libmongo_client="no" fi -else - AC_MSG_WARN([Don't know how to locate system installed libmongo-client.]) - with_libmongo_client="no" +elif test "x$with_libmongo_client" = "xsystem"; then + PKG_CHECK_MODULES(LIBMONGO, libmongo-client >= $LMC_MIN_VERSION,,with_libmongo_client="no") fi - +if test "x$with_libmongo_client" = "xno"; then + enable_mongodb="no" +fi dnl *************************************************************************** dnl misc features to be enabled -- 1.7.2.5

4 14

Reading from standard input
by Remy Zandwijk 28 Sep '11

28 Sep '11

Hi list, I use syslog-ng to create log files from within a script. The script sends the message through a pipe to syslog-ng, which I use as command line tool. In the syslog-ng.conf I created a source which reads the standard input. very simplified: Config: source src { pipe("/dev/fd/0"); }; destination dst { file("/tmp/test.log"); }; log { source(src); destination(dst); }; Called as: echo "test" | /usr/sbin/syslog-ng -f /tmp/syslog-ng.conf This works perfectly in 2.0.9. Now I am in the process of upgrading the system and syslog-ng 3.2.4. With this version of syslog-ng the config file above results in an error: Error opening file for reading; filename='/dev/fd/0', error='Text file busy (26)' Error initializing source driver; source='src', id='src#0' Error initializing message pipeline; I tried several things to get rid of this error, like replacing 'pipe' in 'file', but nothing seems to work. Any ideas? Regards, Remy

2 3

Re: [syslog-ng] log file size limit
by Sergei Zhirikov 28 Sep '11

28 Sep '11

On 2011-07-25 17:50, William Tambellini wrote: > Dear Syslog-ng team, > > We are currently evaluating syslog-ng in order to satisfy some requests from our customers regarding logs from our software. > One of their requests is to limit log file size using some kind of rotation algo : if size > 6Mo then rotate to another file. > We have found some options in file destination like fifo buffer control but nothing about files rotation. > Is there anything for that kind of feature ? > I have a patch that does something of that kind. When the log file grows above certain size syslog-ng switches to a new one and renames the old one. Further processing (such as compression) should be done externally. I posted that patch in this list quite some time ago, but as far as I can tell, it went unnoticed... -- Sergei.

4 7

[Bug 14] New: Maintain symlinks to current and previous log output files...
by bugzilla＠bugzilla.balabit.com 28 Sep '11

28 Sep '11

https://bugzilla.balabit.com/show_bug.cgi?id=14 Summary: Maintain symlinks to current and previous log output files... Product: syslog-ng Version: 2.0.x Platform: PC URL: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=218688 OS/Version: Linux Status: NEW Severity: minor Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: sasa(a)debian.org Type of the Report: enhancement Estimated Hours: 0.0 >From bugs.debian.org: I'm hoping that upstream might consider adding functionality to syslog-ng to allow it to track current and previous log files and maintain symlinks to them, as a user option. This replicates one of the options that I (and others) find useful in cronolog. This has been previously discussed on the syslog-ng mailing list, though apparently nothing fully came of it. The archive of that discussion is at: https://lists.balabit.hu/pipermail/syslog-ng/2002-June/003452.html Obviously, it's probably somewhat of a low priority, but it would be nice. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[PATCH] afsql: Fix sql stuck problema
by Juhasz Viktor 11 Aug '11

11 Aug '11

[sql] send a signal to the database thread only in case when the queue is syncronised, because in other cases the database thread wake up unnecessary many times in case of threaded mode And the sql stucked if the explicit commit is turned on and flush_lines and log_iw_size and fetch_limit and log_fifo_size have the same value. The reason of the stuck was that the queue become full but the database thread can't pop any messages form the queue. So it can be occurred that the reader finished the io_job and suspended and the database thread wait for the signal with 0 read message. Signed-off-by: Viktor Juhasz <jviktor(a)balabit.hu> diff --git a/lib/logqueue.c b/lib/logqueue.c index 29c39e4..43ed69e 100644 --- a/lib/logqueue.c +++ b/lib/logqueue.c @@ -55,6 +55,17 @@ log_queue_reset_parallel_push(LogQueue *self) g_static_mutex_unlock(&self->lock); } +void +log_queue_set_parallel_push(LogQueue *self, gint notify_limit, LogQueuePushNotifyFunc parallel_push_notify, gpointer user_data, GDestroyNotify user_data_destroy) +{ + g_static_mutex_lock(&self->lock); + self->parallel_push_notify = parallel_push_notify; + self->parallel_push_data = user_data; + self->parallel_push_notify_limit = notify_limit; + self->parallel_push_data_destroy = user_data_destroy; + g_static_mutex_unlock(&self->lock); +} + /* * * @batch_items: the number of items processed in a batch (e.g. the number of items the consumer is preferred to process at a single invocation) diff --git a/lib/logqueue.h b/lib/logqueue.h index 1d86f3f..ef04c99 100644 --- a/lib/logqueue.h +++ b/lib/logqueue.h @@ -139,6 +139,7 @@ log_queue_set_throttle(LogQueue *self, gint throttle) void log_queue_push_notify(LogQueue *self); void log_queue_reset_parallel_push(LogQueue *self); +void log_queue_set_parallel_push(LogQueue *self, gint notify_limit, LogQueuePushNotifyFunc parallel_push_notify, gpointer user_data, GDestroyNotify user_data_destroy); gboolean log_queue_check_items(LogQueue *self, gint batch_items, gboolean *partial_batch, gint *timeout, LogQueuePushNotifyFunc parallel_push_notify, gpointer user_data, GDestroyNotify user_data_destroy); void log_queue_set_counters(LogQueue *self, StatsCounterItem *stored_messages, StatsCounterItem *dropped_messages); void log_queue_init_instance(LogQueue *self, const gchar *persist_name); diff --git a/modules/afsql/afsql.c b/modules/afsql/afsql.c index 039270d..bbdd1d2 100644 --- a/modules/afsql/afsql.c +++ b/modules/afsql/afsql.c @@ -116,6 +116,7 @@ typedef struct _AFSqlDestDriver dbi_conn dbi_ctx; GHashTable *validated_tables; guint32 failed_message_counter; + } AFSqlDestDriver; static gboolean dbi_initialized = FALSE; @@ -685,11 +686,12 @@ afsql_dd_insert_db(AFSqlDestDriver *self) success = log_queue_pop_head(self->queue, &msg, &path_options, (self->flags & AFSQL_DDF_EXPLICIT_COMMITS), FALSE); g_mutex_unlock(self->db_thread_mutex); if (!success) - return TRUE; + { + return TRUE; + } } msg_set_context(msg); - table = g_string_sized_new(32); value = g_string_sized_new(256); query_string = g_string_sized_new(512); @@ -767,7 +769,6 @@ afsql_dd_insert_db(AFSqlDestDriver *self) if (success && self->flush_lines_queued != -1) { self->flush_lines_queued++; - if (self->flush_lines && self->flush_lines_queued == self->flush_lines && !afsql_dd_commit_txn(self, TRUE)) return FALSE; } @@ -1155,6 +1156,16 @@ afsql_dd_deinit(LogPipe *s) } static void +afsql_dd_queue_notify(gpointer user_data) +{ + AFSqlDestDriver *self = (AFSqlDestDriver *) user_data; + g_mutex_lock(self->db_thread_mutex); + g_cond_signal(self->db_thread_wakeup_cond); + log_queue_reset_parallel_push(self->queue); + g_mutex_unlock(self->db_thread_mutex); +} + +static void afsql_dd_queue(LogPipe *s, LogMessage *msg, const LogPathOptions *path_options, gpointer user_data) { AFSqlDestDriver *self = (AFSqlDestDriver *) s; @@ -1165,7 +1176,9 @@ afsql_dd_queue(LogPipe *s, LogMessage *msg, const LogPathOptions *path_options, log_queue_push_tail(self->queue, msg, path_options); if (queue_was_empty && !self->db_thread_suspended) - g_cond_signal(self->db_thread_wakeup_cond); + { + log_queue_set_parallel_push(self->queue, 1, afsql_dd_queue_notify, self, NULL); + } g_mutex_unlock(self->db_thread_mutex); }

2 2

Parsing Question
by Brandon Phelps 05 Aug '11

05 Aug '11

Hello All, Could anyone explain how I would parse a message that looks like this: Jul 29 08:58:38 192.168.1.1 id=firewall sn=0017C5158708 time="2011-07-29 08:58:38" fw=100.1.1.1 pri=6 c=262144 m=98 msg="Connection Opened" n=0 src=192.168.2.100:123:X0 dst=74.1.2.3:X1 proto=udp/ntp I am logging to mysql and would like to extract the 'src' and 'dst' fields from the above message so that I can insert them into indexed fields in my database. Currently my destination looks like this: destination d_mysql { sql( type(mysql) host("localhost") username("myusername") password("mypassword") database("syslog") table("logs") columns("host", "facility", "priority", "level", "tag", "datetime", "program", "msg") values("$HOST_FROM", "$FACILITY", "$PRIORITY", "$LEVEL", "$TAG", "$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC", "$PROGRAM", "$MSG") indexes("host", "facility", "priority", "datetime", "program") ); }; This table (logs) also has source_ip and destination_ip fields which are currently unused since I don't know how to extract that from the message. For the above example, I would want those fields to contact '192.168.2.100' and '74.1.2.3' respectively. Is my only option in this case to write a perl script or something that watches a named pipe and have syslog-ng log to the named pipe instead, while my perl script does the actual parsing? Or can I do what I want with syslog-ng alone? Any help would be greatly appreciated. Thanks, Brandon

4 5