- syslog-ng - lists.balabit.hu

Losing to much remote sent logs
by Daniel Neubacher 18 Mar '12

18 Mar '12

Hello there, I've started playing around with syslog-ng 3.3.4 ose a few days ago but I'm still experiencing some trouble. First of all we want to use syslog-ng to send all of our logs via udp to a central syslog server. This includes of course syslogs, apache logs and custom generated applogs. These logs are generated from 400 clients and produces a minimum of 300 mio. log lines a day. The problem is really simple: I'm losing log lines :P Most of the time everything goes well but when the logs are peaking high 1-5% logs are getting lost. Last night the stats of the server and a client said 0 drops but when I counted the lines I found lost lines. The server has 24g ram & 8 cores and I can rule out a network problem for sure. So now to my questions, has anyone else an idea where I can tweak my cfg or where I have to look to find more clues? Is tcp the only way to get around it? I've attached my syslog server cfg. The so_rcvbuf buffer is the same size as the os net.core.rmem settings. And as described in the various balabit blog posts I played around with log_fetch_limit and flush_lines already. syslog-ng.conf: @version: 3.3 options { threaded(yes); owner("root"); group("root"); perm(0660); dir_owner("root"); dir_group("root"); dir_perm(0770); create_dirs(yes); stats_freq(600); stats_level(2); chain_hostnames(yes); normalize_hostnames(yes); check_hostname(yes); dns_cache(yes); dns_cache_size(16384); dns_cache_expire(3600); dns_cache_expire_failed(60); log_msg_size(16384); log_fifo_size(100000); use_fqdn(yes); #disabled 4 debugging # flush_lines(200); }; source s_src { unix-dgram("/dev/log"); internal(); file("/proc/kmsg" program_override("kernel")); }; source s_net { udp( log_fetch_limit(400) so_rcvbuf(51200000) keep_hostname(yes) keep_timestamp(no) ip("10.8.4.10") port(514) ); tcp( so_rcvbuf(51200000) so_keepalive(yes) keep_hostname(no) keep_timestamp(no) ip("10.8.4.10") port(514) ); syslog(); }; filter f_syslog { not program(access.log) and not program(error.log) and not program(beetle.log) and not program(edge.log); }; filter f_apache { program(access.log) or program(error.log); }; filter f_applogs { program(beetle.log) or program(edge.log); }; template t_plain { template("$MSG\n"); template_escape(no); }; destination d_messages { file("/var/log/messages"); }; destination d_remote { file("/log/syslog/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST"); }; destination d_apache { file("/log/apache/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST/$PROGRAM" template(t_plain)); }; destination d_applogs { file("/log/applogs/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST/$PROGRAM" template(t_plain)); }; log { source(s_src); destination(d_messages); }; log { source(s_net); filter(f_syslog); destination(d_remote); }; log { source(s_net); filter(f_apache); destination(d_apache); }; log { source(s_net); filter(f_applogs); destination(d_applogs); }; Thanks Daniel Neubacher

5 11

using syslog-ng and parsing data from both Windows XP and Windows 7 machines
by Mary A Waddick 18 Mar '12

18 Mar '12

Hi, I am using nxlog to send data from both Windows XP and Windows 7 machines to a Unix machine using syslog-ng . My nxlog.conf files are configured to send im_mseventlog data for the Windows XP boxes and im_msvistalog data for the Windows 7 boxes. (See attached file: new nxlog.conf) Therefore I get slightly different data for each machine. The examples on your website don't show me how to parse out all of the im_mseventlog or im_msvistalog data from the different columns in msg. Can you help me with getting the data? My syslog-ng.conf looks like this. I was told that the eventlog data would have the columns that I included in my table, but I am unable to figure out how to pull that data out of the msg column using the provided macros. I have searched and searched, but have not found any examples. (See attached file: syslog-ng.conf) Thank you for your help. Mary Anne Waddick Raytheon Technical Services Senior Software Engineer II (317) 306-2691 (desk)

2 1

syslog-ng blocking application
by Ulrich David 16 Mar '12

16 Mar '12

Hi, I have an ISC-DHCP who is logging on a local facility on a chrooted system : #options { # chain_hostnames(no); # use_dns(no); # flush_lines(1); # stats_freq(3600); #}; #source dhcp_src { # unix-stream("/chroot/dhcp/dev/log" max-connections(256)); #}; My syslog-ng take logs from this source, filter on program and then put the logs on two destinations : #filter f_chroot_l2 { # program(dhcpd); #}; # #destination dhcp_log { file("/var/log/dhcpd.log" perm(0644)); }; #destination dhcp_sql { program("/usr/local/sbin/dhcp_log_to_mysql"); }; # #log { # source(dhcp_src); # filter(f_chroot_l2); # destination(dhcp_log); # destination(dhcp_sql); # flags(final ); #}; Everything goes well… until I block my program "dhcp_log_to_mysql" (I do it with a lock on the mysql table it uses or with a sleep in the script - for test). When "dhcp_log_to_mysql" is blocked, logs continues to be written on dhcp_log for some minutes/seconds then it stops. My chrooted dhcpd hangs on that too. I don't know why syslog-ng is not just dropping logs (continuing eating them on the source). I don't have flow-control enabled… and I find nothing which can explain the problem. I try to resize the buffer output with very big number (log_fifo_size(45000000)) but it changes nothing. Logs and dhcpd hangs at the same moment. What can I do? Regards David

3 4

delay in forwarding logs
by Ivanov, Konstantin 16 Mar '12

16 Mar '12

I have two servers running syslog-ng 2.0.9 on Dell R710's. I did some benchmarking with loggen [1] and was able to generate 41000 msg/sec locally. When I try to forward log messages from one server to the other I only get about 60 msg/sec. tcpdump shows that only that many are sent over the wire as well. I test this by writing to the log file from a script. Any idea what can be the problem? Here are both configs from the sending and receiving servers: ----------- Sending syslog-ng server -------------- options { log_fifo_size(120000); time_reopen (1); time_reap(1); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); group(adm); perm(0640); dir_perm(0755); keep_hostname (yes); log_fetch_limit(60000); }; source s_zxtm { file("/var/log/zxtm/access_log",follow_freq(1), flags(no-parse)); }; destination d_logrelay { tcp(somehost.com port(514)); }; log { source(s_zxtm); destination(d_logrelay); flags(flow-control); }; --------------- Receiving syslog-ng server ------------------ options { log_fifo_size(3000000); time_reopen (1); time_reap(1); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); group(adm); perm(0640); dir_perm(0755); keep_hostname (yes); stats_freq(600); sync(0); }; source s_zxtm { tcp(max-connections(500), port(514)); }; destination d_zxtm { file("/var/log/zxtm/incoming/$YEAR$MONTH$DAY$HOUR-access_log.aggregated" owner("glassfish") perm(0644) template("$MESSAGE\n") template_escape(no)); }; log { source(s_zxtm); destination(d_zxtm); flags(flow-control); }; -------------- [1]. loggen -iS -r 10000000 -s 200 -I 60 localhost 514 average rate = 41053.50 msg/sec, count=410535

2 1

hostname issue with aggregation from aggregator of end-system
by Ian Veach 16 Mar '12

16 Mar '12

We're currently running loghosts/aggregators for each data center and two OS types. This works fine. We are now trying to forward all syslogs from our aggregators to another single aggregator (a security loghost, sec-loghost, which is proprietary). We are having a problem retaining hostname when forwarding from a loghost aggregator to ANOTHER loghost - but only from one OS type / syslog-ng install. We've been running syslog-ng for some time on RHEL (and previously SLES) and AIX, in a loghost configuration, with no issues. Our current architecture (example): data center 1: rhelhostX -> rhel-loghost1 aixhostX -> aix-loghost1 data center 2: rhelhostX -> rhel-loghost2 aixhostX -> aix-loghost2 aix-loghost's are running syslog-ng 3.0.5, RHEL loghost's are running syslog-ng 3.1.4. Both are similarly configured (see below), and are working fine as advertised by themselves - e.g. each loghost gets $HOST_FROM and properly stores the logs for that host in our architecture (/logs/$YEAR/$MONTH/$DAY/$HOST_FROM/). However, given data center 1, if syslogs follow this path: aixhostX -> aix-loghost1 -> sec-loghost rhelhostX -> rhel-loghost1 -> sec-loghost rhelhostX -> rhel-loghost1 -> rhel-loghost2 sec-loghost sees syslogs from all aixhostXs (even though they passed through aix-loghost1) correctly. However, sec-loghost (and we also tested going to rhel-loghost2) sees ALL syslog entries as coming from rhel-loghost, even though it's simply aggregating from rhelhostX's. loghost1's all correctly get $HOST_FROM and see syslog record hosts correctly. This issue only occurs when forwarding again to another loghost. Our nutshell configuration on BOTH loghosts (related to this) is: options { chain_hostnames(no); keep_hostname(yes); use_fqdn(no); use_dns(yes); }; Any ideas on how to fix this? The only hint I've gotten is from our networks group, who uses kiwi syslog and had to make sure a checkbox for RFC 3164 headers compliance was turned on. I don't see an equivalent of that not already set with the above config. Thanks, Ian

2 1

Profiling syslog-ng memory usage and performance
by William Sweat 16 Mar '12

16 Mar '12

Hello, I'm having issues with syslog-ng hitting swap on a set of servers that handle a lot of web traffic. Are there good techniques to figure out an optimal configuration or if there's something with syslog-ng that may prove problematic with large amounts of requests? I'm using the syslog-ng premium client (LTS v4.0.3b), on Linux systems with 256GB of Memory and RAID10. I've enabled more informational logging, but because each system handles multiple gigabytes of traffic per day, debugging is problematic as this is a problem that manifests after a day (or three). Also I am using nightly logrotate, so syslog-ng does get restarted everyday to write new log files. These are only high traffic servers that are experiencing swap issues, the syslog-ng server doesn't have any problems. Thanks, Will Sweat Demand Media ________________________________ Please NOTE: This electronic message, including any attachments, may include privileged, confidential and/or inside information owned by Demand Media, Inc. Any distribution or use of this communication by anyone other than the intended recipient(s) is strictly prohibited and may be unlawful. If you are not the intended recipient, please notify the sender by replying to this message and then delete it from your system. Thank you.

2 1

Compilation errors syslog-ng 3.2.5 on solaris 10
by Arya, Manish Kumar 15 Mar '12

15 Mar '12

Hi Guys, I have compiled eventlog-0.2.12 and glib-2.12.12 in /opt/syslog/eventlog and /opt/syslog/glib directories. now when I try to compile syslog-ng with following options in configure ./configure --prefix=/opt/syslog/syslog-ng --enable-ipv6 EVTLOG_LIBS=/opt/syslog/eventlog/lib EVTLOG_CFLAGS=/opt/syslog/eventlog/include GLIB_CFLAGS=/opt/syslog/glib/include GLIB_LIBS=/opt/syslog/glib/lib PKG_CONFIG_PATH=/opt/syslog/glib/lib/pkgconfig I see this error checking for GLIB... yes checking sanity checking Glib headers... no configure: error: Glib headers inconsistent with current compiler setting. You might be using 32 bit Glib with a 64 bit compiler, check PKG_CONFIG_PATH here is the errors o/p from config.log bash-3.00# grep -i error config.log /usr/include/stdbool.h:42:2: #error "Use of <stdbool.h> is valid only in a c99 compilation environment." conftest.c:59: error: syntax error before "text" conftest.c:64: error: 'for' loop initial declaration used outside C99 mode conftest.c:64: error: `text' undeclared (first use in this function) conftest.c:64: error: (Each undeclared identifier is reported only once conftest.c:64: error: for each function it appears in.) conftest.c:108: error: `false' undeclared (first use in this function) conftest.c:112: error: `true' undeclared (first use in this function) conftest.c:113: error: syntax error before "newvar" conftest.c:123: error: 'for' loop initial declaration used outside C99 mode conftest.c:139: error: `newvar' undeclared (first use in this function) ld: fatal: Symbol referencing errors. No output written to conftest | static void yy_fatal_error YY_PROTO(( yyconst char msg[] )); | if ( c == EOF && ferror( yyin ) ) \ | YY_FATAL_ERROR( "input in flex scanner failed" ); \ | && ferror( yyin ) ) \ | YY_FATAL_ERROR( "input in flex scanner failed" ); | /* Report a fatal error. */ | #ifndef YY_FATAL_ERROR | #define YY_FATAL_ERROR(msg) yy_fatal_error( msg ) | YY_FATAL_ERROR( | "fatal flex scanner internal error--no action found" ); | YY_FATAL_ERROR( | "fatal flex scanner internal error--end of buffer missed" ); | YY_FATAL_ERROR( | YY_FATAL_ERROR( | "fatal error - scanner input buffer overflow" ); | YY_FATAL_ERROR( "flex scanner push-back overflow" ); | YY_FATAL_ERROR( "out of dynamic memory in yy_create_buffer()" ); | YY_FATAL_ERROR( "out of dynamic memory in yy_create_buffer()" ); | YY_FATAL_ERROR( "out of dynamic memory in yy_scan_buffer()" ); | YY_FATAL_ERROR( "out of dynamic memory in yy_scan_bytes()" ); | YY_FATAL_ERROR( "bad buffer in yy_scan_bytes()" ); | YY_FATAL_ERROR( | YY_FATAL_ERROR( "start-condition stack underflow" ); | static void yy_fatal_error( yyconst char msg[] ) | static void yy_fatal_error( msg ) ld: fatal: Symbol referencing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. ld: fatal: File processing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. configure:12119: gcc -std=gnu99 -c -g -O2 -Wall -pedantic -Werror -Wno-pointer-sign conftest.c >&5 cc1: error: unrecognized command line option "-Wno-pointer-sign" cc1: error: unrecognized command line option "-Wno-pointer-sign" conftest.c:30: error: size of array `off_t_is_large' is negative conftest.c:41: error: structure has no member named `tm_gmtoff' conftest.c:41: error: structure has no member named `tm_gmtoff' ld: fatal: File processing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. ld: fatal: File processing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. ld: fatal: Symbol referencing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. ld: fatal: Symbol referencing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. ld: fatal: Symbol referencing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. ld: fatal: Symbol referencing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. ld: fatal: Symbol referencing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. configure:13336: $PKG_CONFIG --exists --print-errors "dbi >= $LIBDBI_MIN_VERSION" configure:13352: $PKG_CONFIG --exists --print-errors "dbi >= $LIBDBI_MIN_VERSION" ld: fatal: File processing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. conftest.c:46: error: `GLIB_SIZEOF_LONG' undeclared (first use in this function) conftest.c:46: error: (Each undeclared identifier is reported only once conftest.c:46: error: for each function it appears in.) configure:13605: error: Glib headers inconsistent with current compiler setting. You might be using 32 bit Glib with a 64 bit compiler, check PKG_CONFIG_PATH bash-3.00# help me !! Regards, -Manish

1 2

[PATCH] value-pairs: Fix a double free on parse errors
by Gergely Nagy 15 Mar '12

15 Mar '12

In the command-line parser code, when we encountered an error, we explicitly free'd the value-pairs structure, but did not return. A few lines later, we called vp_cmdline_parse_rekey_finish(), which also tried to free the same structure. Instead, we should only call _rekey_finish() if we didn't NULL out the value pairs structure earlier. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/value-pairs.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/lib/value-pairs.c b/lib/value-pairs.c index 12cf831..912164f 100644 --- a/lib/value-pairs.c +++ b/lib/value-pairs.c @@ -678,7 +678,8 @@ value_pairs_new_from_cmdline (GlobalConfig *cfg, vp = NULL; } g_option_context_free (ctx); - vp_cmdline_parse_rekey_finish (user_data_args); + if (vp) + vp_cmdline_parse_rekey_finish (user_data_args); return vp; } -- 1.7.9.1

2 2

segfault upon starting
by Jenei Gábor 14 Mar '12

14 Mar '12

Hello, On friday I said about this problem already, but now I could captured some messages by rsyslog(it has other logfiles set manually than syslog-ng). So, syslog-ng sends the following messages upon its init: Mar 4 11:32:18 gabor-netbook supervise/syslog-ng[18834]: Daemon exited due to a deadlock/signal/failure, restarting; exitcode='11' Mar 4 11:32:19 gabor-netbook kernel: [ 3039.971733] syslog-ng[21594]: segfault at 8f95000 ip b74ce758 sp bfd86e88 error 4 in libc-2.13.so[b73be000+153000] Do you know what the problem is? Maybe I should disable linux capabilities?(actually I am using linux mint, so it should work though) I'm using version 3.4 complied on my own, so there maybe some configuration issues. Thanks for your help, Gabor

3 2

Excluding the local host
by Clayton Dukes 14 Mar '12

14 Mar '12

Hey all, Seems like it would be simple to exclude the local host, but I can't find anything in the docs. What is the best way to do this? ______________________________________________________________ Clayton Dukes ______________________________________________________________

3 2