- syslog-ng - lists.balabit.hu

hostname issue with aggregation from aggregator of end-system
by Ian Veach 16 Mar '12

16 Mar '12

We're currently running loghosts/aggregators for each data center and two OS types. This works fine. We are now trying to forward all syslogs from our aggregators to another single aggregator (a security loghost, sec-loghost, which is proprietary). We are having a problem retaining hostname when forwarding from a loghost aggregator to ANOTHER loghost - but only from one OS type / syslog-ng install. We've been running syslog-ng for some time on RHEL (and previously SLES) and AIX, in a loghost configuration, with no issues. Our current architecture (example): data center 1: rhelhostX -> rhel-loghost1 aixhostX -> aix-loghost1 data center 2: rhelhostX -> rhel-loghost2 aixhostX -> aix-loghost2 aix-loghost's are running syslog-ng 3.0.5, RHEL loghost's are running syslog-ng 3.1.4. Both are similarly configured (see below), and are working fine as advertised by themselves - e.g. each loghost gets $HOST_FROM and properly stores the logs for that host in our architecture (/logs/$YEAR/$MONTH/$DAY/$HOST_FROM/). However, given data center 1, if syslogs follow this path: aixhostX -> aix-loghost1 -> sec-loghost rhelhostX -> rhel-loghost1 -> sec-loghost rhelhostX -> rhel-loghost1 -> rhel-loghost2 sec-loghost sees syslogs from all aixhostXs (even though they passed through aix-loghost1) correctly. However, sec-loghost (and we also tested going to rhel-loghost2) sees ALL syslog entries as coming from rhel-loghost, even though it's simply aggregating from rhelhostX's. loghost1's all correctly get $HOST_FROM and see syslog record hosts correctly. This issue only occurs when forwarding again to another loghost. Our nutshell configuration on BOTH loghosts (related to this) is: options { chain_hostnames(no); keep_hostname(yes); use_fqdn(no); use_dns(yes); }; Any ideas on how to fix this? The only hint I've gotten is from our networks group, who uses kiwi syslog and had to make sure a checkbox for RFC 3164 headers compliance was turned on. I don't see an equivalent of that not already set with the above config. Thanks, Ian

2 1

Profiling syslog-ng memory usage and performance
by William Sweat 16 Mar '12

16 Mar '12

Hello, I'm having issues with syslog-ng hitting swap on a set of servers that handle a lot of web traffic. Are there good techniques to figure out an optimal configuration or if there's something with syslog-ng that may prove problematic with large amounts of requests? I'm using the syslog-ng premium client (LTS v4.0.3b), on Linux systems with 256GB of Memory and RAID10. I've enabled more informational logging, but because each system handles multiple gigabytes of traffic per day, debugging is problematic as this is a problem that manifests after a day (or three). Also I am using nightly logrotate, so syslog-ng does get restarted everyday to write new log files. These are only high traffic servers that are experiencing swap issues, the syslog-ng server doesn't have any problems. Thanks, Will Sweat Demand Media ________________________________ Please NOTE: This electronic message, including any attachments, may include privileged, confidential and/or inside information owned by Demand Media, Inc. Any distribution or use of this communication by anyone other than the intended recipient(s) is strictly prohibited and may be unlawful. If you are not the intended recipient, please notify the sender by replying to this message and then delete it from your system. Thank you.

2 1

Compilation errors syslog-ng 3.2.5 on solaris 10
by Arya, Manish Kumar 15 Mar '12

15 Mar '12

Hi Guys, I have compiled eventlog-0.2.12 and glib-2.12.12 in /opt/syslog/eventlog and /opt/syslog/glib directories. now when I try to compile syslog-ng with following options in configure ./configure --prefix=/opt/syslog/syslog-ng --enable-ipv6 EVTLOG_LIBS=/opt/syslog/eventlog/lib EVTLOG_CFLAGS=/opt/syslog/eventlog/include GLIB_CFLAGS=/opt/syslog/glib/include GLIB_LIBS=/opt/syslog/glib/lib PKG_CONFIG_PATH=/opt/syslog/glib/lib/pkgconfig I see this error checking for GLIB... yes checking sanity checking Glib headers... no configure: error: Glib headers inconsistent with current compiler setting. You might be using 32 bit Glib with a 64 bit compiler, check PKG_CONFIG_PATH here is the errors o/p from config.log bash-3.00# grep -i error config.log /usr/include/stdbool.h:42:2: #error "Use of <stdbool.h> is valid only in a c99 compilation environment." conftest.c:59: error: syntax error before "text" conftest.c:64: error: 'for' loop initial declaration used outside C99 mode conftest.c:64: error: `text' undeclared (first use in this function) conftest.c:64: error: (Each undeclared identifier is reported only once conftest.c:64: error: for each function it appears in.) conftest.c:108: error: `false' undeclared (first use in this function) conftest.c:112: error: `true' undeclared (first use in this function) conftest.c:113: error: syntax error before "newvar" conftest.c:123: error: 'for' loop initial declaration used outside C99 mode conftest.c:139: error: `newvar' undeclared (first use in this function) ld: fatal: Symbol referencing errors. No output written to conftest | static void yy_fatal_error YY_PROTO(( yyconst char msg[] )); | if ( c == EOF && ferror( yyin ) ) \ | YY_FATAL_ERROR( "input in flex scanner failed" ); \ | && ferror( yyin ) ) \ | YY_FATAL_ERROR( "input in flex scanner failed" ); | /* Report a fatal error. */ | #ifndef YY_FATAL_ERROR | #define YY_FATAL_ERROR(msg) yy_fatal_error( msg ) | YY_FATAL_ERROR( | "fatal flex scanner internal error--no action found" ); | YY_FATAL_ERROR( | "fatal flex scanner internal error--end of buffer missed" ); | YY_FATAL_ERROR( | YY_FATAL_ERROR( | "fatal error - scanner input buffer overflow" ); | YY_FATAL_ERROR( "flex scanner push-back overflow" ); | YY_FATAL_ERROR( "out of dynamic memory in yy_create_buffer()" ); | YY_FATAL_ERROR( "out of dynamic memory in yy_create_buffer()" ); | YY_FATAL_ERROR( "out of dynamic memory in yy_scan_buffer()" ); | YY_FATAL_ERROR( "out of dynamic memory in yy_scan_bytes()" ); | YY_FATAL_ERROR( "bad buffer in yy_scan_bytes()" ); | YY_FATAL_ERROR( | YY_FATAL_ERROR( "start-condition stack underflow" ); | static void yy_fatal_error( yyconst char msg[] ) | static void yy_fatal_error( msg ) ld: fatal: Symbol referencing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. ld: fatal: File processing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. configure:12119: gcc -std=gnu99 -c -g -O2 -Wall -pedantic -Werror -Wno-pointer-sign conftest.c >&5 cc1: error: unrecognized command line option "-Wno-pointer-sign" cc1: error: unrecognized command line option "-Wno-pointer-sign" conftest.c:30: error: size of array `off_t_is_large' is negative conftest.c:41: error: structure has no member named `tm_gmtoff' conftest.c:41: error: structure has no member named `tm_gmtoff' ld: fatal: File processing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. ld: fatal: File processing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. ld: fatal: Symbol referencing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. ld: fatal: Symbol referencing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. ld: fatal: Symbol referencing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. ld: fatal: Symbol referencing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. ld: fatal: Symbol referencing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. configure:13336: $PKG_CONFIG --exists --print-errors "dbi >= $LIBDBI_MIN_VERSION" configure:13352: $PKG_CONFIG --exists --print-errors "dbi >= $LIBDBI_MIN_VERSION" ld: fatal: File processing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. conftest.c:46: error: `GLIB_SIZEOF_LONG' undeclared (first use in this function) conftest.c:46: error: (Each undeclared identifier is reported only once conftest.c:46: error: for each function it appears in.) configure:13605: error: Glib headers inconsistent with current compiler setting. You might be using 32 bit Glib with a 64 bit compiler, check PKG_CONFIG_PATH bash-3.00# help me !! Regards, -Manish

1 2

[PATCH] value-pairs: Fix a double free on parse errors
by Gergely Nagy 14 Mar '12

14 Mar '12

In the command-line parser code, when we encountered an error, we explicitly free'd the value-pairs structure, but did not return. A few lines later, we called vp_cmdline_parse_rekey_finish(), which also tried to free the same structure. Instead, we should only call _rekey_finish() if we didn't NULL out the value pairs structure earlier. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/value-pairs.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/lib/value-pairs.c b/lib/value-pairs.c index 12cf831..912164f 100644 --- a/lib/value-pairs.c +++ b/lib/value-pairs.c @@ -678,7 +678,8 @@ value_pairs_new_from_cmdline (GlobalConfig *cfg, vp = NULL; } g_option_context_free (ctx); - vp_cmdline_parse_rekey_finish (user_data_args); + if (vp) + vp_cmdline_parse_rekey_finish (user_data_args); return vp; } -- 1.7.9.1

2 2

segfault upon starting
by Jenei Gábor 14 Mar '12

14 Mar '12

Hello, On friday I said about this problem already, but now I could captured some messages by rsyslog(it has other logfiles set manually than syslog-ng). So, syslog-ng sends the following messages upon its init: Mar 4 11:32:18 gabor-netbook supervise/syslog-ng[18834]: Daemon exited due to a deadlock/signal/failure, restarting; exitcode='11' Mar 4 11:32:19 gabor-netbook kernel: [ 3039.971733] syslog-ng[21594]: segfault at 8f95000 ip b74ce758 sp bfd86e88 error 4 in libc-2.13.so[b73be000+153000] Do you know what the problem is? Maybe I should disable linux capabilities?(actually I am using linux mint, so it should work though) I'm using version 3.4 complied on my own, so there maybe some configuration issues. Thanks for your help, Gabor

3 2

Excluding the local host
by Clayton Dukes 14 Mar '12

14 Mar '12

Hey all, Seems like it would be simple to exclude the local host, but I can't find anything in the docs. What is the best way to do this? ______________________________________________________________ Clayton Dukes ______________________________________________________________

3 2

patterndb can't match empty log strings
by Kurt Yoder 13 Mar '12

13 Mar '12

All, Sometimes an empty log message is emitted, but I can't match it using pdbtool. I've tried patterns such as: <pattern>@ANYSTRING@</pattern> <pattern></pattern> <pattern> </pattern> Nothing works. Other log messages from the same application are definitely being matched by the @ANYSTRING@, so I know my rule is working. So is there a way to specify an empty log string? Do I have to use a regex instead? Thanks, -Kurt

1 0

unable to use multiple application patterns per ruleset
by Kurt Yoder 13 Mar '12

13 Mar '12

Hi list, Using patterndb, has anyone successfully used multiple application patterns per ruleset? I have the following: <?xml version='1.0' encoding='UTF-8'?> <patterndb version='4' pub_date='2012-03-08'> <ruleset name='rdc-pam' id='rdc-pam'> <patterns> <pattern>blahblah</pattern> <pattern>sshd</pattern> </patterns> <rules> <rule class='system' id='pam_session_opened_for_user_by_uid' provider='rdc'> <description></description> <patterns> <pattern>pam_unix(sshd:session): session opened for user kurt by (uid=0)</pattern> </patterns> <values> </values> <tags> <tag>rdc.alert.ok</tag> </tags> </rule> </rules> </ruleset> </patterndb> As soon as I remove "<pattern>blahblah</pattern>", or move it below "<pattern>sshd</pattern>", this rule starts matching events. When I add it back, it stops matching events. The documentation at http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-gui… says (incidentally, the closing "<patterns>" in this example should instead read "</patterns>"): " Specifying multiple patterns is useful if two or more applications have different names (that is, different $PROGRAM fields), but otherwise send identical log messages. <patterns> <pattern>firstapplication</pattern> <pattern>otherapplication</pattern> <patterns> This is exactly what I am trying to do, but it does not appear to work as documented. I manually backported my syslog-ng from Ubuntu Oneiric: $ syslog-ng --version syslog-ng 3.2.4 Installer-Version: 3.2.4 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#ef7b91e4a1b1f9628c66138b4ae83de7e4c697c6 Compile-Date: Jan 19 2012 02:57:58 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: on Enable-Linux-Caps: on Enable-Pcre: on Enable-Pacct: off Can anyone offer a suggestion toward making this work? Thanks

1 1

[Bug 160] New: memory leak in message buffering
by bugzilla＠bugzilla.balabit.com 12 Mar '12

12 Mar '12

https://bugzilla.balabit.com/show_bug.cgi?id=160 Summary: memory leak in message buffering Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 Created an attachment (id=47) --> (https://bugzilla.balabit.com/attachment.cgi?id=47) tunable read rate program destination When writing to a destination that requires syslog-ng to buffer messages, the memory footprint of syslog-ng grows endlessly, even when there is sufficient time to flush the buffers. I am including a read rate tunable perl program destination that I use to slow the destination down. I am including a syslog-ng.conf file I am including the output of valgrind. I run syslog-ng, cat a /var/messages file (100,000 lines) to the input pipe. valgrind now shows definitely lost: 2,620,649 bytes in 4,439 blocks -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 8

[Bug 162] New: Invalid frame header; header=''
by bugzilla＠bugzilla.balabit.com 12 Mar '12

12 Mar '12

https://bugzilla.balabit.com/show_bug.cgi?id=162 Summary: Invalid frame header; header='' Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: yuri(a)fused.com Type of the Report: bug Estimated Hours: 0.0 Got always reproducible bug when sending huge logs between machines. Here are the steps: * setup default OSE installs of client & server on rhel5 * configure client to read file("/tmp/input" flags(no-parse)) and send it to syslog(transport("tcp")) * configure server to listen on syslog(transport(tcp)) and store messages in file() * Now let client send big chunk of data: ruby -e 'loop { puts (97+rand(26)).chr*250 }' | head -1000 >> /tmp/input * See on server the following message: syslog-ng[xxx]: Invalid frame header; header='' server's RSTing the connection afterwards It doesn't happen when you start with a couple of lines and then raise the "chunk" to 100-1000 messages per insertion. If you need more info, ask. Thanks. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 6