- syslog-ng - lists.balabit.hu

Struggling with regexp, store-matches and csv-parser macros
by Remy Zandwijk 14 Aug '11

14 Aug '11

Hi list, so, I am new to syslog-ng and I am struggling with the features regexp/store-matches and csv-parser macros. I searched the web and mailinglist archive for these two subjects, but that didn't answer my questions. The syslog-ng 3.1.3 I'm using is running on Debian Squeeze. filter f_mytest { facility(local0) and level(debug) and match("^www" value("MSGHDR") flags("store-matches")); }; template t_mytest { template("${MSG}--${1}\n"); } destination d_mytest { file("/var/tmp/mytest.log"); }; log { source(s_src); filter(f_mytest); template(t_mytest); destination(f_mytest); }; When I send a message to syslog-ng with: logger -p local0.debug -t www testmessage nothing appears in the logfile. However, when flags("store-matches") is omitted, the message appears like expected. Why does this filter not work? I am playing around with the csv-parser function as well. The (user-defined) macros can be used in file() functions, but I found out that they don't work in owner() and group() functions. Is that expected? Thanks, Remy

2 2

How to manage the Syslog-NG messages coming from different sites? Each sites has their Syslog-NG clients?
by Marcos Tang 12 Aug '11

12 Aug '11

Hi, I have a problem to manage the Syslog-NG messages sending from different Syslog-NG clients at different remote sites. For example, I have 10 sites (site1, site2, site3 to site 10) running Syslog-NG clients. Each site has one Syslog-NG server and all the Syslog messages will be forwarded to the centralized Syslog-NG server and insert into the MySQL database at the headquarter finally. Now, I can see all the Syslog messages at the headquarter MySQL database; but it is hard for me to manage them. For example, if I know the hostname of a particular host, I can query the MySQL database to search for the Syslog related to that host. However, if I want to know the Syslog messages coming from a particular site such as "site1", how can I do that? There is no "site1" information inside the Syslog messages. I am thinking if I can modify the Syslog-NG configuration file such that I can add some information such as "site1" and make it as part of the Syslog messages, I can query the MySQL database for "site1" pattern. But I am not sure if I can do that or not. If you have any suggestions/ideas, please let me know that. Regards, Marcos

6 6

[PATCH] afsql: Fix sql stuck problema
by Juhasz Viktor 11 Aug '11

11 Aug '11

[sql] send a signal to the database thread only in case when the queue is syncronised, because in other cases the database thread wake up unnecessary many times in case of threaded mode And the sql stucked if the explicit commit is turned on and flush_lines and log_iw_size and fetch_limit and log_fifo_size have the same value. The reason of the stuck was that the queue become full but the database thread can't pop any messages form the queue. So it can be occurred that the reader finished the io_job and suspended and the database thread wait for the signal with 0 read message. Signed-off-by: Viktor Juhasz <jviktor(a)balabit.hu> diff --git a/lib/logqueue.c b/lib/logqueue.c index 29c39e4..43ed69e 100644 --- a/lib/logqueue.c +++ b/lib/logqueue.c @@ -55,6 +55,17 @@ log_queue_reset_parallel_push(LogQueue *self) g_static_mutex_unlock(&self->lock); } +void +log_queue_set_parallel_push(LogQueue *self, gint notify_limit, LogQueuePushNotifyFunc parallel_push_notify, gpointer user_data, GDestroyNotify user_data_destroy) +{ + g_static_mutex_lock(&self->lock); + self->parallel_push_notify = parallel_push_notify; + self->parallel_push_data = user_data; + self->parallel_push_notify_limit = notify_limit; + self->parallel_push_data_destroy = user_data_destroy; + g_static_mutex_unlock(&self->lock); +} + /* * * @batch_items: the number of items processed in a batch (e.g. the number of items the consumer is preferred to process at a single invocation) diff --git a/lib/logqueue.h b/lib/logqueue.h index 1d86f3f..ef04c99 100644 --- a/lib/logqueue.h +++ b/lib/logqueue.h @@ -139,6 +139,7 @@ log_queue_set_throttle(LogQueue *self, gint throttle) void log_queue_push_notify(LogQueue *self); void log_queue_reset_parallel_push(LogQueue *self); +void log_queue_set_parallel_push(LogQueue *self, gint notify_limit, LogQueuePushNotifyFunc parallel_push_notify, gpointer user_data, GDestroyNotify user_data_destroy); gboolean log_queue_check_items(LogQueue *self, gint batch_items, gboolean *partial_batch, gint *timeout, LogQueuePushNotifyFunc parallel_push_notify, gpointer user_data, GDestroyNotify user_data_destroy); void log_queue_set_counters(LogQueue *self, StatsCounterItem *stored_messages, StatsCounterItem *dropped_messages); void log_queue_init_instance(LogQueue *self, const gchar *persist_name); diff --git a/modules/afsql/afsql.c b/modules/afsql/afsql.c index 039270d..bbdd1d2 100644 --- a/modules/afsql/afsql.c +++ b/modules/afsql/afsql.c @@ -116,6 +116,7 @@ typedef struct _AFSqlDestDriver dbi_conn dbi_ctx; GHashTable *validated_tables; guint32 failed_message_counter; + } AFSqlDestDriver; static gboolean dbi_initialized = FALSE; @@ -685,11 +686,12 @@ afsql_dd_insert_db(AFSqlDestDriver *self) success = log_queue_pop_head(self->queue, &msg, &path_options, (self->flags & AFSQL_DDF_EXPLICIT_COMMITS), FALSE); g_mutex_unlock(self->db_thread_mutex); if (!success) - return TRUE; + { + return TRUE; + } } msg_set_context(msg); - table = g_string_sized_new(32); value = g_string_sized_new(256); query_string = g_string_sized_new(512); @@ -767,7 +769,6 @@ afsql_dd_insert_db(AFSqlDestDriver *self) if (success && self->flush_lines_queued != -1) { self->flush_lines_queued++; - if (self->flush_lines && self->flush_lines_queued == self->flush_lines && !afsql_dd_commit_txn(self, TRUE)) return FALSE; } @@ -1155,6 +1156,16 @@ afsql_dd_deinit(LogPipe *s) } static void +afsql_dd_queue_notify(gpointer user_data) +{ + AFSqlDestDriver *self = (AFSqlDestDriver *) user_data; + g_mutex_lock(self->db_thread_mutex); + g_cond_signal(self->db_thread_wakeup_cond); + log_queue_reset_parallel_push(self->queue); + g_mutex_unlock(self->db_thread_mutex); +} + +static void afsql_dd_queue(LogPipe *s, LogMessage *msg, const LogPathOptions *path_options, gpointer user_data) { AFSqlDestDriver *self = (AFSqlDestDriver *) s; @@ -1165,7 +1176,9 @@ afsql_dd_queue(LogPipe *s, LogMessage *msg, const LogPathOptions *path_options, log_queue_push_tail(self->queue, msg, path_options); if (queue_was_empty && !self->db_thread_suspended) - g_cond_signal(self->db_thread_wakeup_cond); + { + log_queue_set_parallel_push(self->queue, 1, afsql_dd_queue_notify, self, NULL); + } g_mutex_unlock(self->db_thread_mutex); }

2 2

syslog-ng 3.3 git snapshot
by Peter Czanik 10 Aug '11

10 Aug '11

Hello, There were many fixes added since the last 3.3 beta1 release. These are currently available only from git. I just learned, that accessing git is troublesome for those behind strict firewalls. So I prepared a tgz from the latest git snapshot with 'make dist'. It is available at http://people.balabit.hu/czanik/syslog-ng-3.3.0beta1.tar.gz (don't be confused by the 'beta1' suffix, it's from this mornings git checkout) Known problem: mongodb support is broken, when using the output of 'make dist'. Please use the --disable-mongodb configure option to disable mongodb for now. I'll try to periodically update this snapshot until an 'official' release is available. Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

3 2

syslog-ng event capability
by curious curious 08 Aug '11

08 Aug '11

Hi, What are the capabilities , limits of syslog-ng daemon? When does it fail to send the logs to the destination? How many events/logs per second can it handle without dropping anything? Regards

2 1

syslog-ng OSE 3.3 Administrator Guide released
by Fekete Róbert 05 Aug '11

05 Aug '11

Hi, Even though OSE 3.3 is still in a beta state, the documentation has been finally updated with the new features of syslog-ng OSE 3.3, so it contains a basic description for the mongo database support and for running syslog-ng in multihreaded mode, and a bunch of error corrections and clarifications. For a list of changes, see http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guid… The guide is available at the BalaBit Documentation Page at http://www.balabit.com/support/documentation?&product_line=46&product_type=… , in PDF, HTML, and single-page HTML format. As always, comments, criticism, error reports, and feedback is welcome here on the list or at the documentation(a)balabit.com e-mail address. Kind Regards, Robert Fekete

1 0

Parsing Question
by Brandon Phelps 05 Aug '11

05 Aug '11

Hello All, Could anyone explain how I would parse a message that looks like this: Jul 29 08:58:38 192.168.1.1 id=firewall sn=0017C5158708 time="2011-07-29 08:58:38" fw=100.1.1.1 pri=6 c=262144 m=98 msg="Connection Opened" n=0 src=192.168.2.100:123:X0 dst=74.1.2.3:X1 proto=udp/ntp I am logging to mysql and would like to extract the 'src' and 'dst' fields from the above message so that I can insert them into indexed fields in my database. Currently my destination looks like this: destination d_mysql { sql( type(mysql) host("localhost") username("myusername") password("mypassword") database("syslog") table("logs") columns("host", "facility", "priority", "level", "tag", "datetime", "program", "msg") values("$HOST_FROM", "$FACILITY", "$PRIORITY", "$LEVEL", "$TAG", "$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC", "$PROGRAM", "$MSG") indexes("host", "facility", "priority", "datetime", "program") ); }; This table (logs) also has source_ip and destination_ip fields which are currently unused since I don't know how to extract that from the message. For the above example, I would want those fields to contact '192.168.2.100' and '74.1.2.3' respectively. Is my only option in this case to write a perl script or something that watches a named pipe and have syslog-ng log to the named pipe instead, while my perl script does the actual parsing? Or can I do what I want with syslog-ng alone? Any help would be greatly appreciated. Thanks, Brandon

4 5

MongoDB and syslog-ng 3.3 on Centos 5.5
by Viacheslav Biriukov 03 Aug '11

03 Aug '11

Hello. Can you help me with syslog-ng 3.3 on the CentoOS 5.5 with MongoDB 1.6 I install syslog-ng version 3.3 where mongodb support exists. But when I add to the config mongodb destination I get an error: Starting syslog-ng: Error parsing destination, destination plugin mongodb > not found in /etc/syslog-ng/syslog-ng.conf at line 70, column 5: > mongodb( > ^^^^^^^ syslog-ng.conf @version: 3.3 > # > # configuration file for syslog-ng, customized for remote logging > # > source s_internal { internal(); }; > destination d_syslognglog { file("/var/log/syslog-ng.log"); }; > log { source(s_internal); destination(d_syslognglog); }; > # Local sources, filters and destinations are commented out > # If you want to replace sysklogd simply uncomment the following > # parts and disable sysklogd > # > # Local sources > # > source s_local { > unix-dgram("/dev/log"); > file("/proc/kmsg" program_override("kernel")); > }; > # > # Local filters > # > filter f_messages { level(info..emerg); }; > filter f_secure { facility(authpriv); }; > filter f_mail { facility(mail); }; > filter f_cron { facility(cron); }; > filter f_emerg { level(emerg); }; > filter f_spooler { level(crit..emerg) and facility(uucp, news); }; > filter f_local7 { facility(local7); }; > # > # Local destinations > # > destination d_messages { file("/var/log/messages"); }; > destination d_secure { file("/var/log/secure"); }; > destination d_maillog { file("/var/log/maillog"); }; > destination d_cron { file("/var/log/cron"); }; > destination d_console { usertty("root"); }; > destination d_spooler { file("/var/log/spooler"); }; > destination d_bootlog { file("/var/log/boot.log"); }; > # > # Local logs - order DOES matter ! > # > log { source(s_local); filter(f_emerg); destination(d_console); }; > log { source(s_local); filter(f_secure); destination(d_secure); > flags(final); }; > log { source(s_local); filter(f_mail); destination(d_maillog); > flags(final); }; > log { source(s_local); filter(f_cron); destination(d_cron); flags(final); > }; > log { source(s_local); filter(f_spooler); destination(d_spooler); }; > log { source(s_local); filter(f_local7); destination(d_bootlog); }; > log { source(s_local); filter(f_messages); destination(d_messages); }; > > # Remote logging > source s_remote { > tcp(ip(0.0.0.0) port(514)); > udp(ip(0.0.0.0) port(514)); > }; > destination d_separatedbyhosts { > file("/var/log/syslog-ng/$HOST/messages" owner("root") group("root") > perm(0640) dir_perm(0750) create_dirs(yes)); > }; > log { source(s_remote); destination(d_separatedbyhosts); }; > > > #MongoDB > destination d_mongodb { > mongodb( > host("127.0.0.1") > port(27017) > database("syslog-ng") > collection("logs") > keys("date", "facility", "level", "host", "program", "pid", > "message") > values("${R_YEAR}-${R_MONTH}-${R_DAY} > ${R_HOUR}:${R_MIN}:${R_SEC}", "$FACILITY", "$LEVEL", "$HOST","$PROGRAM", > "$PID", "$MSGONLY") > ); > }; > log { source(s_local); destination(d_mongodb); }; > In the modules.conf I try to add @afmongodb but this isn't change the situation. What's wrong? -- Viacheslav Biriukov BR http://biriukov.com

2 1

[Bug 128] New: template() broken in latest 3.3 git
by bugzilla＠bugzilla.balabit.com 01 Aug '11

01 Aug '11

https://bugzilla.balabit.com/show_bug.cgi?id=128 Summary: template() broken in latest 3.3 git Product: syslog-ng Version: unspecified Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: algernon(a)balabit.hu Type of the Report: --- Estimated Hours: 0.0 Hi! While updating my 3.3 installation from an ancient version to latest git head, I noticed that template functions stopped working. If I have any kind of template in the config, I get a segmentation fault: ,---- | Program received signal SIGSEGV, Segmentation fault. | log_template_reset_compiled (self=0x0) at templates.c:661 | 661 while (self->compiled_template) | (gdb) bt | #0 log_template_reset_compiled (self=0x0) at templates.c:661 | #1 0x00007ffff7b9f302 in log_template_compile (self=0x0, template=0x63d7e0 "foo: bar", error=0x7fffffffb5d8) at templates.c:788 | #2 0x00007ffff5e96761 in affile_parse (lexer=<value optimized out>, instance=<value optimized out>, arg=<value optimized out>) at affile-grammar.y:592 | #3 0x00007ffff7b9bd72 in cfg_parser_parse (self=0x7ffff609ce30, cfg=<value optimized out>, yylloc=<value optimized out>, arg=<value optimized out>) at cfg-parser.h:83 | #4 plugin_parse_config (self=0x7ffff609ce30, cfg=<value optimized out>, yylloc=<value optimized out>, arg=<value optimized out>) at plugin.c:124 | #5 0x00007ffff7ba89f7 in main_parse (lexer=<value optimized out>, dummy=<value optimized out>, arg=<value optimized out>) at cfg-grammar.y:592 | #6 0x00007ffff7b7ee93 in cfg_parser_parse (self=0x611720, lexer=0x6188a0, parser=0x7ffff7dd92a0, result=<value optimized out>, arg=<value optimized out>) at cfg-parser.h:83 | #7 cfg_run_parser (self=0x611720, lexer=0x6188a0, parser=0x7ffff7dd92a0, result=<value optimized out>, arg=<value optimized out>) at cfg.c:370 | #8 0x00007ffff7b7efa6 in cfg_read_config (self=0x611720, fname=0x7ffff7bb7068 "/home/algernon/tmp/sng/install/3.3/etc/syslog-ng.conf", syntax_only=<value optimized out>, preprocess_into=0x0) at cfg.c:392 | #9 0x00007ffff7b970c1 in main_loop_init () at mainloop.c:611 | #10 0x0000000000401865 in main (argc=1, argv=0x7fffffffdc78) at main.c:242 | (gdb) `---- The config that triggers this is as follows: ,---- | @version: 3.3 | @include "scl.conf" | | source s_local { internal(); }; | destination d_local { file("/tmp/template-test" template("foo: bar")); }; | log { source(s_local); destination(d_local); }; `---- After bisecting the problem, I ended up with the following commit: ,---- | 2443b07b90c822414a628aae6a9ca513357b52fc is the first bad commit | commit 2443b07b90c822414a628aae6a9ca513357b52fc | Author: Balazs Scheidler <bazsi(a)balabit.hu> | Date: Sat Jun 11 14:09:01 2011 +0200 | | template: make template compilation thread safe | | Instead of doing template compilation on-demand, require the application | code to do it explicitly, at initialization, in order to avoid having to | acquire a lock during runtime. | | This patch contains an incompatible API change, and requires updating | all call-sites. | | The change makes it also possible to change template string of a constructed | LogTemplate class multiple times with a call to log_template_compile(). | | Signed-off-by: Balazs Scheidler <bazsi(a)balabit.hu> | | :040000 040000 a40fc910d215dac2184aa02ae1498c6748e3e64a dea1d2a7fc02d1d0ec8e095f2f62499ce42b895a M lib | :040000 040000 1641cf3152897406874716d4f090b5cc795292ab f66d2e5c482c3ff91b9d4219812766d66067c7a1 M modules | :040000 040000 2c2a37fd3a4f9c5bddc81813605c20ece033ce2f dff570d331b8e7ada6c1202761066878b9bb3910 M tests `---- I'm not quite sure why this happens yet, I'll keep digging. I'll post a patch if and when I figure out what exactly broke. Meanwhile, this bug report will serve as a reminder for me to keep looking. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

4 11

TCP / max-connections & flow-control
by Paul Muther 01 Aug '11

01 Aug '11

Greetings all, I have not been able to find confirmation on this but I wanted to find details on TCP connections in 3.2 open-source. When a client establishes a TCP connection to a host log server, does it reuse that connection for all subsequent log transmissions? Does it have a threshold for destroying the connection and creating a new one if it does? Thanks much, PM

3 2