- syslog-ng - lists.balabit.hu

Make Errors on Solaris 10
by Tony MacDoodle 31 Mar '11

31 Mar '11

Hello, I am trying to compile syslog-ng 3.2.2 for Solaris 10 Sparc and getting the following errors when I run a make: ./configure --enable-tcp-wrapper syslog-ng Open Source Edition 3.2.2 configured Compiler options: compiler : gcc -std=gnu99 compiler options : -g -O2 -Wall -D_REENTRANT -D_PTHREADS -I/usr/local/include/glib-2.0 -I/usr/local/lib/glib-2.0/include -I/usr/local/include/eventlog -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 linker flags : -lpthread prefix : /usr/local linking mode : dynamic Features: Sun STREAMS support (module): yes Debug symbols : no GCC profiling : no Memtrace : no IPV6 support : yes spoof-source support : no tcp-wrapper support : yes SSL support (module) : no SQL support (module) : no Linux capability support : no PCRE support : no Env wrapper support : no PACCT module (EXPERIMENTAL) : no root:~/syslog/syslog-ng-3.2.2#* make* make all-recursive Making all in lib make: Fatal error in reader: Makefile, line 637: Unexpected end of line seen Current working directory /root/syslog/syslog-ng-3.2.2/lib *** Error code 1 The following command caused the error: fail= failcom='exit 1'; \ for f in x $MAKEFLAGS; do \ case $f in \ *=* | --[!k]*);; \ *k*) failcom='fail=yes';; \ esac; \ done; \ dot_seen=no; \ target=`echo all-recursive | sed s/-recursive//`; \ list='lib modules syslog-ng scripts tests doc contrib scl debian tgz2build'; for subdir in $list; do \ echo "Making $target in $subdir"; \ if test "$subdir" = "."; then \ dot_seen=yes; \ local_target="$target-am"; \ else \ local_target="$target"; \ fi; \ (CDPATH="${ZSH_VERSION+.}:" && cd $subdir && make $local_target) \ || eval $failcom; \ done; \ if test "$dot_seen" = "no"; then \ make "$target-am" || exit 1; \ fi; test -z "$fail" make: Fatal error: Command failed for target `all-recursive' Current working directory /root/syslog/syslog-ng-3.2.2 *** Error code 1 make: Fatal error: Command failed for target `all' Thanks

2 1

Embedded log statements
by Hendrik Visage 29 Mar '11

29 Mar '11

Hi there, The example in the Administrator's Guide, looks (simplified) like this: log{ Source(source); filter(f1); destination(d1) log { filter(f1.1); destination(d1.1) log {filter(f1.1.1); destination(d1.1.1)}; }; }; ie. the equivalence of: log{ Source(source); filter(f1); destination(d1)} log{ Source(source); filter(f1); filter(f1.1); destination(d1.1)} log{ Source(source); filter(f1); filter(f1.1); filter(f1.1.1); destination(d1.1.1)}; I need to know if the following also allowed/expected behavior of embedded logs: log{ Source(source); filter(f1); destination(d1) log { filter(f1.1); destination(d1.1) log {filter(f1.1.1); destination(d1.1.1)}; }; log {filter(f1.2); destination(d1.2)}; }; ie. equivalent to: log{ Source(source); filter(f1); destination(d1)} log{ Source(source); filter(f1); filter(f1.1); destination(d1.1)} log{ Source(source); filter(f1); filter(f1.1); filter(f1.1.1); destination(d1.1.1)}; log{ Source(source); filter(f1); filter(f1.2); destination(d1.2)} And example to this nature in the documentation would be appreciated ;) Thank you Hendrik

2 2

Re: [syslog-ng] Regexp values in comparison
by Roland Kamke 29 Mar '11

29 Mar '11

Thank you, Robert. I guess since I only have syslog-ng OSE 3.0.10-1.rhel5 this regexp comparison stuff won't work anyway - that's a pity. I have to use CentOS 5 - do you know some yum repository providing newer OSE versions then 3.0.10-1.rhel5, e. g. OSE 3.3 or 3.2? Regards, Roland -----Ursprüngliche Nachricht----- Von: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] Im Auftrag von Fekete Robert Gesendet: Dienstag, 29. März 2011 11:57 An: Syslog-ng users' and developers' mailing list Betreff: Re: [syslog-ng] Regexp values in comparison Hi, yes, you can do such things in syslog-ng OSE 3.2 and 3.3, but not in earlier versions. Check out https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-gu… for details. Regards, Robert On 03/29/2011 10:27 AM, Roland Kamke wrote: > A newbie question: Is it possible to use values matching a regular expression in > a comparison? > > Imagine I receive messges like this: "Disk usage is 60%", "Disk usage is 80%", > etc. All match the regular expression "Disk usage is [0-9]+%". I do not want to > log all these messages, but only those where disk usage is at least 75 per cent. > Can this be achieved by proper configuration? > Where to look in the documentation? > > Currently I am stuck with this approch: > parser p_disk { > csv-parser(columns("fix.disk","fix.usage", "fix.is", "diskusage") > delimiters(" %") > ); > }; > filter f_disk_syntax { > message("Disk usage is [0-9]+%"); > }; > filter f_disk_usage { "$diskusage" > "75" }; > log l_mylog { > source(...); > filter(f_disk_syntax); > parser(p_disk); > filter(f_disk_usage); > destination(...); > }; > > Is this a useful approach? Can it be accomplished with fewer definitions? > The definition of f_disk_usage results in a syntax error. What would be the > right syntax? > filter f_disk_usage { "$diskusage" > "75" }; > filter f_disk_usage { "$diskusage" ge "75" }; > filter f_disk_usage { value("diskusage") > "75" }; > filter f_disk_usage { "$diskusage" > 75}; > filter f_disk_usage { "${diskusage}" > "75" }; > filter f_disk_usage { ${diskusage} > "75" }; > > Will I need to set up a pattern database to get rid of all unwanted messages? > > Thank you > Roland > > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.campin.net/syslog-ng/faq.html > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

2 1

Regexp values in comparison
by Roland Kamke 29 Mar '11

29 Mar '11

A newbie question: Is it possible to use values matching a regular expression in a comparison? Imagine I receive messges like this: "Disk usage is 60%", "Disk usage is 80%", etc. All match the regular expression "Disk usage is [0-9]+%". I do not want to log all these messages, but only those where disk usage is at least 75 per cent. Can this be achieved by proper configuration? Where to look in the documentation? Currently I am stuck with this approch: parser p_disk { csv-parser(columns("fix.disk","fix.usage", "fix.is", "diskusage") delimiters(" %") ); }; filter f_disk_syntax { message("Disk usage is [0-9]+%"); }; filter f_disk_usage { "$diskusage" > "75" }; log l_mylog { source(...); filter(f_disk_syntax); parser(p_disk); filter(f_disk_usage); destination(...); }; Is this a useful approach? Can it be accomplished with fewer definitions? The definition of f_disk_usage results in a syntax error. What would be the right syntax? filter f_disk_usage { "$diskusage" > "75" }; filter f_disk_usage { "$diskusage" ge "75" }; filter f_disk_usage { value("diskusage") > "75" }; filter f_disk_usage { "$diskusage" > 75}; filter f_disk_usage { "${diskusage}" > "75" }; filter f_disk_usage { ${diskusage} > "75" }; Will I need to set up a pattern database to get rid of all unwanted messages? Thank you Roland

2 1

[PATCH] syslog-ng development module
by Juhasz Viktor 28 Mar '11

28 Mar '11

Hi! This patch creates a new package: syslog-ng-dev. Using this you can create your own syslog-ng modules BR, Viktor Juhasz

2 1

Configuring syslog to log on both central server as well as local host( 3.0.7 version)
by Rakesh Rajasekharan 28 Mar '11

28 Mar '11

Hi, I am using syslog-ng 3.0.7 version. I have a requirement to log the messages on both the central server as well as my local machine. The central server is a HPUX machine and my cleint is a SUSE machine. From my SUSE machine I am able to get he messages logged on to the HPUX machine by this entry in my conf file destination loghost { tcp("20.5.68.85" port(514)); }; 20.5.68.85 being the ip for the central server. This works perfectly fine. But is this possible to ge the logs generated on my SUSE machine as well simulatneously. Please let me know if there seems any way around to do this. Thanks, Rakesh

2 1

Re: [syslog-ng] Rewrite facility names of remote logs
by Steve Smith 25 Mar '11

25 Mar '11

I'm actually writing logs out to syslog as local6 facility using log4j. Then using rsyslog, I am forwarding those logs to the syslog-ng server using 'local6.* @centrallogserverhost'. So when syslog-ng receives those logs, it writes them out to a file named local5.$DAY according to my destination configuration. So, I don't know that $PROGRAM would accurately record the name. Will give it a shot and see. -----Original Message----- From: Martin Holste [mcholste(a)gmail.com] Received: Friday, 25 Mar 2011, 12:21pm To: Syslog-ng users' and developers' mailing list [syslog-ng(a)lists.balabit.hu] CC: Steve Smith [ssmith(a)xpressdocs.com] Subject: Re: [syslog-ng] Rewrite facility names of remote logs Is "tomcat" the $PROGRAM or the $FACILITY name? I would think it would be $PROGRAM like "ssh" or "apache." On Fri, Mar 25, 2011 at 9:44 AM, Steve Smith <ssmith(a)xpressdocs.com> wrote: > I’ve setup Syslog-NG to receive logs from other servers which have been > configured as follows - > Tomcat servers are forwarding logs as facility6 to rsyslog, which then > forwards to central log server. > Apache servers are forwarding logs as faility5 to rsyslog which then > forwards to central log server. > > When I receive these logs on the central log server, they are written to > files as their facility name, i.e. local6.$DAY. > Is there a way to change or re-write the facility name on the fly so that > instead of local6.$DAY I can get the file written as tomcat.$DAY? > > Here is the configuration I’m using to store the logs - > > destination d_net { > file("/var/log/hosts/$YEAR/$MONTH/$HOST/$FACILITY.$DAY" > owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes) > ); > }; > > log { > source(s_net); > destination(d_net); > }; > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.campin.net/syslog-ng/faq.html > > >

1 0

[RFC/RFR] value-pairs() code preview
by Gergely Nagy 25 Mar '11

25 Mar '11

Hi! It's been a while I came to the list with value-pairs(), and the last time was a syntax RFC. This time, after almost two months of barely doing a thing with it, I'm able to present some code! The downside is, that the code is horribly inefficient at this time, but it's good enough to introduce the API (as can be used by destination drivers), as that part of it should be fairly complete by now. A lengthy description will follow - I hope I won't bore anyone out of their skull. If I do, apologies, I'll try to be more entertaining next time! To refresh our memories (mine included), the syntax we're trying to implement is something like the following: ,---- | destination d_mongo { | mongodb( | value-pairs( | scope(selected_macros nv_pairs) | exclude("R_*") | exclude("S_*") | exclude("HOST_FROM") | exclude("MSG") | pair("test" "test: ${loggen.runid}") | ) | database("test") | collection("syslog") | ); | }; `---- We'd like this to include every possible name-value pair, except the ones starting with R_ or S_, MSG and HOST_FROM. As a bonus, we'll add a custom key: "test" (with a custom template; more about that later). The purpose of this, is to be able to store a whole lot of (usually structured) data that are either dynamically generated (by patterndb) or cannot be easily predicted at configuration time (like SDATA), without the need to explicitly list all keys in a template. The code is available from the 'work/value-pairs/base' branch of my git tree at git://git.balabit.hu/algernon/syslog-ng-3.3.git (or browseable online at: http://git.balabit.hu/?p=algernon/syslog-ng-3.3.git;a=shortlog;h=work/value…) Adding value-pairs() support to drivers ======================================= >From the driver's point of view, it needs to do three things to support value-pairs(): * It needs to be able to understand value-pairs(). This is fairly easy: the core grammar rules provide 'value_pair_stmt', which can be used to parse these things. To use it, one should do something along these lines (in the driver's grammar file): ,---- | driver_option | : ... | | value_pair_stmt { driver_set_value_pairs(driver_instance, last_value_pairs); } `---- Where the _set_value_pairs() function can be as simple as this: ,---- | void | driver_set_value_pairs (LogDriver *d, ValuePairs *vp) | { | MyDriver *self = (MyDriver *)d; | | value_pairs_free (self->vp); | self->vp = vp; | } `---- With this, value-pairs() are recognised. * The driver needs to set up defaults. This can be done with calling the various functions in value-pairs.h. I'm not going to describe them, they're pretty darn self explanatory, especially after one had a look at the core grammar file. The idea is, that if no value-pairs() are defined, we'll use a sane default. If the user specifies value-pairs(), then the _set_value_pairs() function will free up the defaults, and replace them with the user supplied configuration. * The driver needs to make use of value-pairs(). To actually _use_ value pairs, the driver needs to iterate over these pairs. The way to do that, is the value_pairs_foreach() function. This takes a couple of parameters: + The ValuePairs object. + A driver-supplied foreach callback (more about this later) + An NVTable (the message payload in most cases) + An NVRegistry (logmsg_registry in most cases) + A LogMessage object + An extra user_data argument, which will be passed down to the callback. The interesting things here are the callback and the user_data. The user_data pointer can be used to pass custom data to the callback, so that it can actually do something with the data it receives. The callback itself will be called for each and every name-value pair that matches the user-configured criteria, with the following arguments: + The key name + The key value + The user_data argument passed to value_pairs_foreach() Using these, the callback can do whatever it wants with the data. Do note, that both the key name and key value will likely be freed after the foreach loop completed, so if their value will be used later by the driver, it needs to copy them. An example ========== I have updated my mongodb destination driver to use value-pairs(), the code is available from the work/afmongodb-vp branch of my git tree, or browsable online at http://git.balabit.hu/?p=algernon/syslog-ng-3.3.git;a=shortlog;h=work/afmon… Using a configuration like the following: ,---- | @version: 3.3 | @include "scl.conf" | | @module afmongodb | | source s_network { | tcp(port (10514) tags("tcp-tag")); | syslog(port (10515) tags("syslog-tag")); | }; | | destination d_mongo { | mongodb( | value-pairs( | scope(selected_macros nv_pairs) | exclude("R_*") | exclude("S_*") | exclude("HOST_FROM") | exclude("MSG") | pair("test" "test: ${loggen.runid}") | ) | database("test") | collection("syslog") | ); | }; | | parser p_loggen { | db_parser( | file("etc/loggen.pdb") | ); | }; | | log { | source(s_network); | parser(p_loggen); | destination(d_mongo); | }; `---- (for etc/loggen.pdb, see the attachment of this message) When poking syslog-ng with a couple of standard loggen messages, mongodb would contain documents like the following: ,---- | { | "DATE" : "Mar 25 16:06:37", | "FACILITY" : "auth", | "HOST" : "localhost", | "MESSAGE" : "seq: 0000000018, thread: 0000, runid: 1301065597, stamp: 2011-03-25T16:06:37 PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPAD", | "PID" : "1234", | "PRIORITY" : "info", | "PROGRAM" : "prg00000", | "SDATA" : "", | "SEQNUM" : "", | "SOURCE" : "s_network", | "SOURCEIP" : "127.0.0.1", | "TAGS" : ".classifier.system,syslog-tag,.source.s_network,loggen", | "_id" : ObjectId("4d8caf7d8ccffe2227000013"), | "classifier" : { | "class" : "debug", | "rule_id" : "d7d3ada4-6907-4dad-924f-d254e8f29f92" | }, | "loggen" : { | "runid" : "1301065597", | "seq" : "0000000018", | "stamp" : "2011-03-25T16:06:37", | "thread" : "0000" | }, | "test" : "test: 1301065597" | } `---- Even though SDATA and SEQNUM are empty (there's no SDATA in the message, and SEQNUM is not supported by value-pairs() at the moment), the document in mongodb is neatly structured. Implementation details ====================== The way value-pairs() work right now, is very simple: during setup, we just store a couple of things (the scopes, as an ORed together value; the exclude patterns and the explicitly added keys). The bulk of the work is done by value_pairs_foreach(), which will construct a base set of name-value pairs, based on the scope, then iterate over them, filter out anything that is excluded, and pass the rest to the callback. When done, it goes over the explicitly added keys, and runs those through the callback too. The implementation is horribly inefficient in many ways, but for a preview, it's good enough. I'll tune it for efficiency in the near future. However, the public API should not change anymore. And that's about it! TODO ==== * Add helpers that can be used by template functions that want to support value-pairs() (for example tfjson). * Support SEQNUM * Possibly filter out empty, zero-length values, since they're kinda useless. * Performance tuning: + Pre-allocate LogTemplate structures, if possible + Try to reduce the number of memory allocations + Possibly move away from GHashTable (as used for the temporary base-set and for the explicit keys) to something lighter. * Clean up the grammar Right now, the grammar is messy, there's a lot of global symbols we use, which really should be local to value-pairs(). I still need to get familiar with bison/flex again to understand how these things work. Once these are sorted out, I'll flatten the patch set and submit a merge request. In the meantime, if anyone feels up to it, I'd appreciate any code reviews and comments. -- |8]

1 1

Rewrite facility names of remote logs
by Steve Smith 25 Mar '11

25 Mar '11

I've setup Syslog-NG to receive logs from other servers which have been configured as follows - Tomcat servers are forwarding logs as facility6 to rsyslog, which then forwards to central log server. Apache servers are forwarding logs as faility5 to rsyslog which then forwards to central log server. When I receive these logs on the central log server, they are written to files as their facility name, i.e. local6.$DAY. Is there a way to change or re-write the facility name on the fly so that instead of local6.$DAY I can get the file written as tomcat.$DAY? Here is the configuration I'm using to store the logs - destination d_net { file("/var/log/hosts/$YEAR/$MONTH/$HOST/$FACILITY.$DAY" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes) ); }; log { source(s_net); destination(d_net); };

2 1

AUTO: James B Horwath/TheGuardian is out of the office. (returning 03/28/2011)
by James B Horwath 25 Mar '11

25 Mar '11

I am out of the office until 03/28/2011. I am out of the office until Monday March 28, 2011 and will have no access to e-mail. If there are urgent logging issues please contact Sunil Gupta at 610-807-6717. Any other issues please contact Christopher Clarke. Thanks, Jim Note: This is an automated response to your message "syslog-ng Digest, Vol 71, Issue 39" sent on 3/25/11 7:00:02. This is the only notification you will receive while this person is away. ----------------------------------------- This message, and any attachments to it, may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are notified that any use, dissemination, distribution, copying, or communication of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately by return e-mail and delete the message and any attachments. Thank you.

1 0

[PATCH] configure.in: Bump minimum eventlog version to 0.2.10.
by Gergely Nagy 23 Mar '11

23 Mar '11

As of a few patches ago, eventlog >= 0.2.10 is required due to evt_tag_long() use. As such, bump the requirement in configure.in to match reality. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- configure.in | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/configure.in b/configure.in index 7e42b3e..94fda95 100644 --- a/configure.in +++ b/configure.in @@ -21,7 +21,7 @@ dnl *************************************************************************** dnl dependencies GLIB_MIN_VERSION="2.10.1" -EVTLOG_MIN_VERSION="0.2" +EVTLOG_MIN_VERSION="0.2.10" OPENSSL_MIN_VERSION="0.9.8" LIBDBI_MIN_VERSION="0.8.0" IVYKIS_MIN_VERSION="0.18" -- 1.7.0.4

2 1

[PATCH] Fix --enable-mixed-linking mode.
by Gergely Nagy 23 Mar '11

23 Mar '11

When mixed linking is enabled, the tools (pdbtool, in particular) weren't linked against ivykis, which resulted in a link-time error, due to libsyslog-ng using ivykis, but not being linked to it. A similar problem was present in the linking of syslog-ng: -livykis DID appear in @SYSLOGNG_DEPS_LIBS@, but the neccessary library paths (when using the internal ivykis) did not. This patch fixes both problems. There is probably a more correct solution, but this will do for the time being. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- configure.in | 2 +- syslog-ng/Makefile.am | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/configure.in b/configure.in index 7e42b3e..4e0160c 100644 --- a/configure.in +++ b/configure.in @@ -801,7 +801,7 @@ else # tools & unit tests that link against libsyslog-ng.so should be linked with this set of libraries. # other tools that do not use libsyslog-ng.so can use the _LIBS variables directly. - TOOL_DEPS_LIBS="$LIBS $BASE_LIBS $GLIB_LIBS $EVTLOG_LIBS $RESOLV_LIBS $LIBCAP_LIBS $PCRE_LIBS $REGEX_LIBS $DL_LIBS" + TOOL_DEPS_LIBS="$LIBS $BASE_LIBS $GLIB_LIBS $EVTLOG_LIBS $RESOLV_LIBS $LIBCAP_LIBS $PCRE_LIBS $REGEX_LIBS $IVYKIS_LIBS $DL_LIBS" # libsyslog-ng.so is linked with this set of libraries CORE_DEPS_LIBS="" diff --git a/syslog-ng/Makefile.am b/syslog-ng/Makefile.am index 33378e0..889483d 100644 --- a/syslog-ng/Makefile.am +++ b/syslog-ng/Makefile.am @@ -14,7 +14,7 @@ syslog_ng_SOURCES = main.c # libtool, because in mixed mode libtool is not used for the final linking # phase. See the comment in the configure script for details. -syslog_ng_LDADD = -L../lib/.libs -lsyslog-ng @SYSLOGNG_DEPS_LIBS@ +syslog_ng_LDADD = -L../lib/.libs -L../lib/ivykis/lib/.libs -L../lib/ivykis/modules/.libs -lsyslog-ng @SYSLOGNG_DEPS_LIBS@ syslog_ng_LINK = @SYSLOGNG_LINK@ syslog_ng_wrapper_SOURCES = wrapper.c -- 1.7.2.3

3 2

escapes in SQL statements (was Re: dbtype in SQL logging)
by Hendrik Visage 21 Mar '11

21 Mar '11

Hi there, Okay, looks like I was looking/thinking about something else, so lets explain my problem. I want the pid inseted to be a "proper" number/integer, not a string/character, and the fun starts with several of my sources not having a pid entry, thus that field I would like to be filled with NULL values. However, it still gets escaped, and when it gets escaped, the NULL is not null anymore, but the string 'NULL' :( I was hoping that in some why I could coerce syslog-ng to understand that that column is a integer column, thus should not be escaped as it is currently done ;( I have the following destination: destination d_pgsql { sql(type(pgsql) host("127.0.0.1") username("logwriter") password("logwriter") port("5432") database("syslog") table("logs_${HOST}_${R_YEAR}${R_MONTH}${R_DAY}") #or whatever you want, example ${HOST}" for hosts, ${LEVEL}" for levels.. etc columns("datetime varchar(16)", "host varchar(32)", "program varchar(16)", "pid varchar(8) default null", "message varchar") values("$R_DATE", "$HOST", "$PROGRAM", "${PID:-NULL}", "$MSG") indexes("datetime", "host", "program", "pid", "message")); }; and the following log lines: Incoming log entry; line='<166>Feb 22 08:38:59 Vpxa: [2011-02-22 08:38:59.862 1B7A7B90 verbose \'App\'] [VpxaHalServices] HostChanged Event Fired, properties changed []' Running SQL query; query='INSERT INTO logs_somedevice_20110222 (datetime, host, program, pid, message) VALUES (\'Feb 22 11:39:27\', \'somedevice\', \'Vpxa\', \'NULL\', \'[2011-02-22 08:38:59.862 1B7A7B90 verbose \'\'App\'\'] [VpxaHalServices] HostChanged Event Fired, properties changed []\')'

5 8

syslog-ng stats
by Ioan Indreias 21 Mar '11

21 Mar '11

Hello, Running with syslog-ng OSE 3.2.1 and wondering what exactly mean some of the parameters by STATS. Please find below an extract of the output we get (with the default stats_level) , hoping to receive some hints from you. global > payload_reallocs = ? global > msg_clones = ? global > sdata_updates = ? why center > received is 0 ? From OSE 3.2 manual, Chapter 14, it should be the "total number of messages received from every configured sources." BR, Ioan ### [root@myhost TOOLS]# /opt/syslog-ng/sbin/syslog-ng-ctl stats SourceName;SourceId;SourceInstance;State;Type;Number global;payload_reallocs;;a;processed;1643977 source;s_net;;a;processed;615301996 global;msg_clones;;a;processed;1230603992 destination;d_mysql1;;a;processed;247656 destination;d_mysql2;;a;processed;1354498 global;sdata_updates;;a;processed;0 center;;received;a;processed;0 center;;queued;a;processed;0 ###

2 2

Syslog-ng returning a TCP Reset
by Andy Ruch 20 Mar '11

20 Mar '11

Hello, I was evaluating syslog-ng and have come across a strange problem. When connecting to syslog-ng using the 'syslog' source, syslog-ng will return a TCP Reset at random intervals. The interesting thing about this error is it only seems to appear when the message length varies between logs. After some experimenting, I was able to reproduce the error using loggen to send logs from a file. However, when loggen generates its own fixed length messages, syslog-ng will receive everything properly. I'm concerned about the reliability of syslog-ng. I don’t want it to close the connection if it gets overloaded. I know that I can add some options to improve performance, but that seems like a patch and not a fix. I would think syslog-ng would utilize TCP flow control to prevent the sender from sending too fast. Am I doing something wrong? Does this look like a bug or have I just reached the max performance of syslog-ng? Any help is appreciated. Below I’ve included some information about the commands that I was running as well as my system setup. Thanks, Andy ***** Details ***** - Two machine setup (one sender and one receiver) - Receiver is using an evaluation copy of Syslog-PE-4.0.1a (but I first saw this issue using open source version 3.2.2 ) - CentOS 5.5 ***** Send Command ***** This is the console output that I receive from loggen. As you can see, the failure did not happen every time. In this particular case, it was the second execution that failed. I normally just run it back to back until it fails. $ ./loggen -P -r 40000 -R /tmp/syslog_10-40 10.64.27.38 6514 average rate = 16503.22 msg/sec, count=50000, time=3.297, (last) msg size=112, bandwidth=1627.75 kB/sec $ ./loggen -P -r 40000 -R /tmp/syslog_10-40 10.64.27.38 6514 Send error Connection reset by peer5 msg/sec average rate = 15483.69 msg/sec, count=15435, time=0.9968, (last) msg size=114, bandwidth=1527.27 kB/sec ***** Send File ***** The following pattern was put into a file and repeated for a total of 50,000 lines. I tried to attach the complete file but it was too large for this mailing list. Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Mar 16 10:35:28 guinness syslog-tester[proc-1234]: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ***** Syslog-ng.conf ***** @version: 4.0 options { }; ###### # sources source s_local { # message generated by Syslog-NG internal(); # standard Linux log source (this is the default place for the syslog() # function to send logs to) unix-stream("/dev/log"); # messages from the kernel file("/proc/kmsg" program_override("kernel")); }; source s_net { syslog( ip(0.0.0.0) transport("tcp") port(6514) ); }; ###### # destinations destination d_messages { file("/var/log/messages"); }; destination d_tmp_file { file("/tmp/log_messages"); }; ###### # paths log { source(s_local); destination(d_messages); }; log { source(s_net); destination(d_tmp_file); flags(flow-control); };

3 2

Local7 issue
by Cardia E. - GDI/BH/O&I 20 Mar '11

20 Mar '11

Gentlemen, I've got an issue that the bootlog info doesn't appear in the bootlog file as I would like ( running Red Hat 5.6 EL ). Here are the syslog-ng.conf file parameters : destination bootlog { file("/var/remote2_log/$HOST/$YEAR$MONTH$DAY/boot.log"); }; filter f_local7 { facility(local7); }; log { source(system); filter(f_local7); destination(bootlog); }; Someone an idea ?? Greetz, Efisio --------------------------------------------------------------------------------- Dit bericht kan informatie bevatten die niet voor u is bestemd. Indien u niet de geadresseerde bent of dit bericht abusievelijk aan u is toegezonden, wordt u verzocht dat aan de afzender te melden en het bericht te verwijderen. De Staat aanvaardt geen aansprakelijkheid voor schade, van welke aard ook, die verband houdt met risico's verbonden aan het elektronisch verzenden van berichten. Ministerie van Veiligheid en Justitie This message may contain information that is not intended for you. If you are not the addressee or if this message was sent to you by mistake, you are requested to inform the sender and delete the message. The State accepts no liability for damage of any kind resulting from the risks inherent in the electronic transmission of messages. Ministry of Security and Justice ---------------------------------------------------------------------------------

2 1

TLS - Authentication problem
by Honza Mach 18 Mar '11

18 Mar '11

Hi everyone, I am currently implementing central syslog server using syslog-ng and I have encontered some problems when configuring mutual TLS authentication. On client machine I keep getting the following errors: Mar 18 12:54:22 machine syslog-ng[5432]: Syslog connection established; fd='9', server='AF_INET(xxx.113.161.23:514)', local='AF_INET(0.0.0.0:0)' Mar 18 12:54:22 machine syslog-ng[5432]: Certificate validation failed; subject='CN=AddTrust External CA Root, OU=AddTrust External TTP Network, O=AddTrust AB, C=SE', issuer='CN=AddTrust External CA Root, OU=AddTrust External TTP Network, O=AddTrust AB, C=SE', error='self signed certificate in certificate chain', depth='3' Mar 18 12:54:22 machine syslog-ng[5432]: SSL error while writing stream; tls_error='SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed' Mar 18 12:54:22 machine syslog-ng[5432]: I/O error occurred while writing; fd='9', error='Broken pipe (32)' Mar 18 12:54:22 machine syslog-ng[5432]: Syslog connection broken; fd='9', server='AF_INET(xxx.113.161.23:514)', time_reopen='60' This is the exact same problem as in the following thread: https://lists.balabit.hu/pipermail/syslog-ng/2011-January/015613.html I have verified the functionality without TLS settings and everything works perfectly, so there is no problem on the way (firewall, etc.). I studied carefully the manual and everything is as it is supposed to be: CLIENT MACHINE -------------- OS - Debian 6.0 Squeeze SYSLOG-NG VERSION: client:# syslog-ng --version syslog-ng 3.1.3 Installer-Version: 3.1.3 Revision: ssh +git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.1#master#def34661b08109f8148904b860457d5747c425b3 Compile-Date: Feb 1 2011 19:36:27 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-Sun-Door: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: on Enable-Linux-Caps: on Enable-Pcre: on DESTINATION CONFIGURATION FROM SYSLOG-NG.CONF: destination d_net { tcp("xxx.113.161.23" port(514) tls( key_file("/etc/ssl/private/client.company.cz-nopass.key") cert_file("/etc/ssl/certs/client.company.cz.pem") ca_dir("/etc/ssl/trusted_ca")) ); }; CA HASH: client:# openssl x509 -in /etc/ssl/trusted_ca/tcs.pem -noout -hash 9df51c42 CA CERTIFICATE DIRECTORY: client:# ll /etc/ssl/trusted_ca total 12 lrwxrwxrwx 1 root root 27 Mar 18 12:30 9df51c42.0 -> /etc/ssl/trusted_ca/tcs.pem -rw-r--r-- 1 root root 1655 Mar 18 12:29 tcs.pem VALIDATE CERTIFICATE AGAINST CA: client:# openssl verify -CAfile /etc/ssl/trusted_ca/9df51c42.0 -verbose /etc/ssl/certs/client.company.cz.pem /etc/ssl/certs/client.company.cz.pem: OK SERVER MACHINE -------------- OS - Debian 6.0 Squeeze SYSLOG-NG VERSION: server:# syslog-ng --version syslog-ng 3.1.3 Installer-Version: 3.1.3 Revision: ssh +git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.1#master#def34661b08109f8148904b860457d5747c425b3 Compile-Date: Feb 1 2011 19:36:27 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-Sun-Door: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: on Enable-Linux-Caps: on Enable-Pcre: on SOURCE CONFIGURATION FROM SYSLOG-NG.CONF: source s_net { tcp( ip(0.0.0.0) port(514) tls( key_file("/etc/ssl/private/server.company.cz-nopass.key") cert_file("/etc/ssl/certs/server.company.cz.pem") ca_dir("/etc/ssl/trusted_ca")) ); }; CA HASH: server:# openssl x509 -in /etc/ssl/trusted_ca/tcs.pem -noout -hash 9df51c42 CA CERTIFICATE DIRECTORY: server:# ll /etc/ssl/trusted_ca total 12 lrwxrwxrwx 1 root root 27 Mar 18 12:30 9df51c42.0 -> /etc/ssl/trusted_ca/tcs.pem -rw-r--r-- 1 root root 1655 Mar 18 12:29 tcs.pem VALIDATE CERTIFICATE AGAINST CA: server:# openssl verify -CAfile /etc/ssl/trusted_ca/9df51c42.0 -verbose /etc/ssl/certs/server.company.cz.pem /etc/ssl/certs/server.company.cz.pem: OK Certificates for the CA are the same on both machines. I tried to put either the whole chain up to the root certificate, or the last certificate, but the error keeps to appear. It the thread mentioned above the community member Baszi suggested following: ------ It's the whole point in certificate validation to use self-signed CA certificates. (after all, a centrally trusted CA is self-signed too). Try validating that your directory is correct usign "openssl verify", once that validates the cert, syslog-ng should too. -- Bazsi ------- What is meant by validate the directory, I do not understand that. As I said, I succefully validated both client and server certificate against the CA certificate using openssl command, but syslog-ng is still complaining. Am I missing something? I will gladly provide more information, if needed. Thank you for your time Regards Honza Mach

2 1

SEC-2.6.0 released
by Risto Vaarandi 18 Mar '11

18 Mar '11

hi all, since a number of sites are running syslog-ng with SEC event log correlator, I thought I'll let you know about the new major release of SEC: http://sourceforge.net/projects/simple-evcorr/files/sec/2.6.0/ kind regards, risto

1 0

Syslog-ng Cisco CSS switch
by add gy 17 Mar '11

17 Mar '11

Hi, I am not ablle to recive syslog messsages from Cisco content switches I am able to recive messages from rest of the devices ex: cisco routers , switches ,firewalls , loadbalancers etc .. But not able to recive messages from cisco CSS Syslog version 3.0.9 stable Operating system Cento OS 4.7 Is there any configuration changes to be done on syslog file? Thanks Add

1 0

logging to the cloud
by Peter Czanik 16 Mar '11

16 Mar '11

Hello, Two weeks ago I wrote about Loganalyzer, which was one of my most read blog post ever. This time I wrote about a different approach: logging to the cloud: http://czanik.blogs.balabit.com/2011/03/logging-to-the-cloud/ which also provides a nice, web based gui for syslog-ng logs, without the need to set anything up locally. There are already a few more syslog web gui apps on my radar, as I wrote at the end Loganalzer blog. Please let me know, if you have any other syslog related web applications you find interesting! Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

FreeBSD install with SSL
by John Kristoff 16 Mar '11

16 Mar '11

I have a FreeBSD system that has the SSL libraries not built from ports. Apparently the syslog-ng port expects ssl to be installed from ports. I'd prefer not to do that at this time. However, when building syslog-ng from source, it expects pkg-config (or gnome-config as this FreeBSD still refers to it) for SSL, which also is not going to work. I've not tried the balabit supplied FreeBSD package. Any guidance on the best approach to get syslog-ng with SSL support installed onto this FreeBSD box? Keep in mind I'd like to avoid changing the installed openssl libaries at this time. Will the balabit package do the trick? Perhaps it is better to modify the build process from source? John

5 4

afsql module error
by Eshan Bhide 15 Mar '11

15 Mar '11

Hi guys, I'm trying to setup a master syslog-ng server which listens on UDP syslog port to incoming squid access log and Cisco syslog messages. It uses a db_parser to parse and store these access logs in different variables. I now need to store these values in a simple mysql database. I have created the database schema and thought of using a pipe and a wrapper script to write the query. But read that there are problems sometimes with this approach. I then thought about doing it using libddb and downloaded the package syslog-ng_3.2.2 from the website. So far so good, but when I try to run syslog-ng it gives me an error: syslog-ng Error opening plugin module; module='afsql', error='/usr/local/lib/syslog-ng/libafsql.so: undefined symbol: dbi_result_free' Error opening plugin module; module='afsql', error='/usr/local/lib/syslog-ng/libafsql.so: undefined symbol: dbi_result_free' Error parsing destination, destination plugin sql not found in /usr/local/etc/syslog-ng.conf at line 61, column 3: sql(type(mysql) ^^^ Any pointers on what I'm missing here ? I have commented out the '@module afsql' line in modules.conf, have made sure I've got the latest libdb and libdb-devel packages. I'm running CentOS 5.x and just don't know how to proceed :( Appreciate all and any help guys thank you. -- Eshan Bhide Engineer, Core Infrastructure and Systems Development Global Gossip 14 Wentworth Ave Surry Hills, NSW 2010 Australia ------------------------------------------------- IMPORTANT NOTICE : The information in this email is confidential and may also be privileged. If you are not the intended recipient, any use or dissemination of the information and any disclosure or copying of this email is unauthorised and strictly prohibited. If you have received this email in error, please promptly inform us by reply email or telephone. You should also delete this email and destroy any hard copies produced.

5 14

[PATCH] Fix autogen.sh to autogen submodules by default.
by Gergely Nagy 15 Mar '11

15 Mar '11

Call autogen_submodules even if $skip_modules is unset (which it would be on most systems that just cloned the repository). Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- autogen.sh | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/autogen.sh b/autogen.sh index 3569af3..569f134 100755 --- a/autogen.sh +++ b/autogen.sh @@ -31,7 +31,7 @@ autogen_submodules() done } -if [ "$skip_submodules" = 0 ]; then +if [ -z "$skip_submodules" ] || [ "$skip_modules" = 0 ]; then autogen_submodules fi -- 1.7.2.3

2 1

logging stopped
by Len Conrad 15 Mar '11

15 Mar '11

uname -a FreeBSD 7.0-RELEASE syslog-ng --version syslog-ng 2.0.10 change date on syslog-ng.conf is "Apr 20 2009" been running untouched for at least that long. about 00:20 today Friday, all syslogging to syslog-ng stopped. chkrootkit shows nothing wrong stop syslog-ng then pkg_delete, and then cd /usr/ports/sysutils/syslog-ng2 make && make install start it, no change I rebooted the syslog server. no change trafshow -i bce0 -n then filter 514 shows 100KBs arriving from our syslog clients. df shows plenty of disk space for /var suggestions? Len

2 1

Re: [syslog-ng] afsql module error
by Eshan Bhide 15 Mar '11

15 Mar '11

Updated to the latest 3.3.0alpha2 build, libdbi-drivers and latest ivykis from git, and after much messing with the source I got it to compile and install. Now I get the error: "The database access library (DBI) reports no usable SQL drivers, perhaps DBI drivers are not installed properly; Error initializing dest driver; dest='d_sql', id='d_sql#0' Error initializing message pipeline;" I'm pulling my hair out at this point - could I get some help here please :) ? On 15 March 2011 14:13, Eshan Bhide <ebhide(a)globalgossip.net> wrote: > Here's another try: > > http://pastebin.com/7TV0nHzD > > On 14 March 2011 21:54, Hendrik Visage <hvjunk(a)gmail.com> wrote: >> >> >> On Mon, Mar 14, 2011 at 6:37 AM, Eshan Bhide <ebhide(a)globalgossip.net> >> wrote: >>> >>> Hi Hendrik, >>> Still having this problem of "No such DBI driver; type='mysql'" >>> >>> Strace is attached at: >>> http://pastebin.com/qaJZ2ZbE >> >> I see libdbi getting loaded, but what I am missing in that strace, is the >> libdbi searching for it's drivers.... >> >> >> > ------------------------------------------------- IMPORTANT NOTICE : The information in this email is confidential and may also be privileged. If you are not the intended recipient, any use or dissemination of the information and any disclosure or copying of this email is unauthorised and strictly prohibited. If you have received this email in error, please promptly inform us by reply email or telephone. You should also delete this email and destroy any hard copies produced.

2 1