February 2012 - syslog-ng - lists.balabit.hu

[Bug 156] New: rewrite rule not working on Linux
by bugzilla＠bugzilla.balabit.com 20 Apr '12

20 Apr '12

https://bugzilla.balabit.com/show_bug.cgi?id=156 Summary: rewrite rule not working on Linux Product: syslog-ng Version: 3.3.x Platform: Other OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: ne24(a)georgetown.edu Type of the Report: bug Estimated Hours: 0.0 Hi, I have syslog-ng 3.3.3 running Red Hat Enterprise Linux Server release 6.2 (Santiago). I compiled with the following flags. ./configure --prefix=/usr/local --enable-spoof-source --enable-pcre --enable-ssl \ --enable-debug I have the following rewrite rule: rewrite n_dst_router { subst("^(%(?:ASA|PIX|FWSM)\-\d\-\d{6}):", "", value("MESSAGE"), type("pcre"), flags("global")); }; But it is not working. The line below minus the IP addresses shows that it is not replacing the %FWSM-6-305011 with a blank. Jan 7 15:59:06 HOSTNAME-REMOVED %FWSM-6-305011: Built dynamic tcp translation from GRT:IP-ADDRESS-REMOVED/53328 to BACKBONE-VRF:IP-ADDRESS-REMOVED/29361 I will also attach my whole syslog-ng.conf file so that you would see the whole setup. Here is the output of the debug output and as you can see, it looks like it is doing what it is supposed to, but it does not write it to the file with the change. Incoming log entry; line='<163>%FWSM-3-710003: udp access denied by ACL from IP-ADDRESS-REMOVED/35390 to BACKBONE-VRF:IP-ADDRESS-REMOVED/46897\x0a' Rewrite expression evaluation result; value='MESSAGE', new_value='udp access denied by ACL from IP-ADDRESS-REMOVED/35390 to BACKBONE-VRF:IP-ADDRESS-REMOVED/46897' I searched the bug reports but could not find anything related to this issue. I would appreciate it your help on this. Thank you Nadim El-Khoury -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 163] New: afmongo does not send log
by bugzilla＠bugzilla.balabit.com 20 Apr '12

20 Apr '12

https://bugzilla.balabit.com/show_bug.cgi?id=163 Summary: afmongo does not send log Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: critical Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: whille(a)163.com Type of the Report: bug Estimated Hours: 0.0 i'm using syslog-ng to store mongo, about 2k logs/sec. About half an hour, no message sent out. trying to locate it, I set log_fifo_size(16), it's easy to reproduce the bug. i found codes related: afmongodb_dd_queue(LogPipe *s, LogMessage *msg, const LogPathOptions *path_options, gpointer user_data) { ... g_mutex_lock(self->queue_mutex); self->last_msg_stamp = cached_g_current_time_sec (); queue_was_empty = log_queue_get_length(self->queue) == 0; g_mutex_unlock(self->queue_mutex); log_queue_push_tail(self->queue, msg, path_options); g_mutex_lock(self->suspend_mutex); if (queue_was_empty && !self->writer_thread_suspended) { g_mutex_lock(self->queue_mutex); log_queue_set_parallel_push(self->queue, 1, afmongodb_dd_queue_notify, self, NULL); g_mutex_unlock(self->queue_mutex); } g_mutex_unlock(self->suspend_mutex); } since in afmongodb_worker_thread(){ ... g_mutex_unlock(self->suspend_mutex); g_mutex_lock(self->queue_mutex); if (log_queue_get_length(self->queue) == 0) { g_cond_wait(self->writer_thread_wakeup_cond, self->queue_mutex); } g_mutex_unlock(self->queue_mutex); ... } the whole rough process i think is, worker wait fo afmongodb_dd_queue_notify(), if 0 queue. But in multi-thread race condition, queue_was_empty offend simplicity. Here's a situation: afmongodb_dd_queue() afmongodb_dd_queue() afmongodb_dd_queue() { // worker thread afmongodb_worker_thread{ whille(){... // run in a while circle afmongodb_worker_thread{ whille(){... afmongodb_worker_thread{ whille(){... all queue are empty now, g_cond_wait(self->writer_thread_wakeup_cond, self->queue_mutex); log_queue_push_tail queue_was_empty is FALSE, so afmongodb_dd_queue_notify is not called. afmongodb_dd_queue() // a queue has been put to tail last time, so queue_was_empty will be still FALSE. i'm trying to get queue length later and judge it as <=1, instead of ==0, for a queue just put to tail. wonder if it work, though a bit ugly. afmongodb_dd_queue(){ ... if(!self->writer_thread_suspended){ g_mutex_lock(self->queue_mutex); if (log_queue_get_length(self->queue)<=1){ log_queue_set_parallel_push(self->queue, 1, afmongodb_dd_queue_notify, self, NULL); } g_mutex_unlock(self->queue_mutex); } -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 10

syslog-ng large NVTable patch
by Peter Eisenlohr 31 Mar '12

31 Mar '12

Hi, as requested by bazsi in the forums http://www.syslog.org/forum/index.php?topic=1218.0 here's a patch against syslog-ng enabling it to handle log messages larger than the current 256k limit. This is done by extending NVTable's data structures from 16 to 32 bit. The patch also contains simple code for automatic resizing of the receive buffer. 'log_msg_size' is still honoured, though. I have been using this to regularly receive multi-MB XML messages and process them into log files. I developed the patch against 3.3.1, but also tested it with 3.3.3 and on x64 as well. 'make check' runs fine, too -- but still, use at your own risk. Peter

3 6

Installing syslog-ng 3.3.4 OSS on multiple Centos servers
by Srinivasan Subramanian 26 Mar '12

26 Mar '12

Hi I have a need to install syslog-ng 3.3.4 on multiple Centos servers. Unfortunately i am unable to find RPMs for the newer versions and only find the source downloads. 1. Any links to contributed repositories for Centos 5.x?2. Alternatively can anyone point me to Source RPMs for Centos 5.x so that I can at least make the RPMs myself? Even EPEL 6 I find only 3.2.x source RPMs RegardsSrini

6 5

[Bug 154] New: Processes forked via program() do not properly terminate
by bugzilla＠bugzilla.balabit.com 23 Mar '12

23 Mar '12

https://bugzilla.balabit.com/show_bug.cgi?id=154 Summary: Processes forked via program() do not properly terminate Product: syslog-ng Version: 3.2.x Platform: Other OS/Version: Solaris Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: hendrik.volker(a)de.verizonbusiness.com Type of the Report: --- Estimated Hours: 0.0 I am using a SyslogNG 3.2.4 build for our company internal version of Solaris 10 on some SUN V245 SPARC servers. In a project I am using the following config: ---8<--- source s_whsvoipgwv2_ngrep_bge1 { program ( "/prod/tools/ngrep -d bge1 -t -q -W single SIP port 5060" flags (no-parse) tags ("IF1") ); }; parser p_whsvoipgwv2_ngrep { csv-parser ( columns ("NGREP.PROTO" "NGREP.DATE", "NGREP.TIME", "NGREP.SRCADDR", "NGREP.DUMMY", "NGREP.DSTADDR", "NGREP.MSG" ) flags (drop-invalid, escape-none, greedy, strip-whitespace) delimiters (" ") ); }; template t_whsvoipgwv2_ngrep { template ( "$MONTH_ABBREV $DAY ${NGREP.TIME} $HOST ${NGREP.PROTO} ${NGREP.SRCADDR} ${NGREP.DSTADDR} ${ NGREP.MSG}\n" ); }; destination d_whsvoipgwv2_sip_voipslp { file ("/var/home/voipslp/sip/sip.$YEAR$MONTH$DAY$HOUR.log" owner("voipslp") group("voipsl") perm(0644) template(t_whsvoipgwv2_ngrep) ); }; log { source(s_whsvoipgwv2_ngrep_bge1); parser(p_whsvoipgwv2_ngrep); destination(d_whsvoipgwv2_sip_voipslp); flags (final); }; ---8<--- With the log file rotation every hour it seems that SyslogNG is starting new instances of ngrep, while not terminating the old instances. This results in lots! of running but unused ngreps. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

2 11

[Bug 150] New: Possible memory leak for TLS connections
by bugzilla＠bugzilla.balabit.com 23 Mar '12

23 Mar '12

https://bugzilla.balabit.com/show_bug.cgi?id=150 Summary: Possible memory leak for TLS connections Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: eric(a)sevenscale.com Type of the Report: --- Estimated Hours: 0.0 I've noticed that memory usage seems to grow as clients using TLS connect and disconnect. I ran syslog-ng under valgrind while I caused a number of clients to connect and disconnect a few times and saved the leak summary from valgrind: https://gist.github.com/893a9687bf98dc53f0e7 -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 6

syslog-ng blocking application
by Ulrich David 16 Mar '12

16 Mar '12

Hi, I have an ISC-DHCP who is logging on a local facility on a chrooted system : #options { # chain_hostnames(no); # use_dns(no); # flush_lines(1); # stats_freq(3600); #}; #source dhcp_src { # unix-stream("/chroot/dhcp/dev/log" max-connections(256)); #}; My syslog-ng take logs from this source, filter on program and then put the logs on two destinations : #filter f_chroot_l2 { # program(dhcpd); #}; # #destination dhcp_log { file("/var/log/dhcpd.log" perm(0644)); }; #destination dhcp_sql { program("/usr/local/sbin/dhcp_log_to_mysql"); }; # #log { # source(dhcp_src); # filter(f_chroot_l2); # destination(dhcp_log); # destination(dhcp_sql); # flags(final ); #}; Everything goes well… until I block my program "dhcp_log_to_mysql" (I do it with a lock on the mysql table it uses or with a sleep in the script - for test). When "dhcp_log_to_mysql" is blocked, logs continues to be written on dhcp_log for some minutes/seconds then it stops. My chrooted dhcpd hangs on that too. I don't know why syslog-ng is not just dropping logs (continuing eating them on the source). I don't have flow-control enabled… and I find nothing which can explain the problem. I try to resize the buffer output with very big number (log_fifo_size(45000000)) but it changes nothing. Logs and dhcpd hangs at the same moment. What can I do? Regards David

3 4

delay in forwarding logs
by Ivanov, Konstantin 16 Mar '12

16 Mar '12

I have two servers running syslog-ng 2.0.9 on Dell R710's. I did some benchmarking with loggen [1] and was able to generate 41000 msg/sec locally. When I try to forward log messages from one server to the other I only get about 60 msg/sec. tcpdump shows that only that many are sent over the wire as well. I test this by writing to the log file from a script. Any idea what can be the problem? Here are both configs from the sending and receiving servers: ----------- Sending syslog-ng server -------------- options { log_fifo_size(120000); time_reopen (1); time_reap(1); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); group(adm); perm(0640); dir_perm(0755); keep_hostname (yes); log_fetch_limit(60000); }; source s_zxtm { file("/var/log/zxtm/access_log",follow_freq(1), flags(no-parse)); }; destination d_logrelay { tcp(somehost.com port(514)); }; log { source(s_zxtm); destination(d_logrelay); flags(flow-control); }; --------------- Receiving syslog-ng server ------------------ options { log_fifo_size(3000000); time_reopen (1); time_reap(1); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); group(adm); perm(0640); dir_perm(0755); keep_hostname (yes); stats_freq(600); sync(0); }; source s_zxtm { tcp(max-connections(500), port(514)); }; destination d_zxtm { file("/var/log/zxtm/incoming/$YEAR$MONTH$DAY$HOUR-access_log.aggregated" owner("glassfish") perm(0644) template("$MESSAGE\n") template_escape(no)); }; log { source(s_zxtm); destination(d_zxtm); flags(flow-control); }; -------------- [1]. loggen -iS -r 10000000 -s 200 -I 60 localhost 514 average rate = 41053.50 msg/sec, count=410535

2 1

hostname issue with aggregation from aggregator of end-system
by Ian Veach 16 Mar '12

16 Mar '12

We're currently running loghosts/aggregators for each data center and two OS types. This works fine. We are now trying to forward all syslogs from our aggregators to another single aggregator (a security loghost, sec-loghost, which is proprietary). We are having a problem retaining hostname when forwarding from a loghost aggregator to ANOTHER loghost - but only from one OS type / syslog-ng install. We've been running syslog-ng for some time on RHEL (and previously SLES) and AIX, in a loghost configuration, with no issues. Our current architecture (example): data center 1: rhelhostX -> rhel-loghost1 aixhostX -> aix-loghost1 data center 2: rhelhostX -> rhel-loghost2 aixhostX -> aix-loghost2 aix-loghost's are running syslog-ng 3.0.5, RHEL loghost's are running syslog-ng 3.1.4. Both are similarly configured (see below), and are working fine as advertised by themselves - e.g. each loghost gets $HOST_FROM and properly stores the logs for that host in our architecture (/logs/$YEAR/$MONTH/$DAY/$HOST_FROM/). However, given data center 1, if syslogs follow this path: aixhostX -> aix-loghost1 -> sec-loghost rhelhostX -> rhel-loghost1 -> sec-loghost rhelhostX -> rhel-loghost1 -> rhel-loghost2 sec-loghost sees syslogs from all aixhostXs (even though they passed through aix-loghost1) correctly. However, sec-loghost (and we also tested going to rhel-loghost2) sees ALL syslog entries as coming from rhel-loghost, even though it's simply aggregating from rhelhostX's. loghost1's all correctly get $HOST_FROM and see syslog record hosts correctly. This issue only occurs when forwarding again to another loghost. Our nutshell configuration on BOTH loghosts (related to this) is: options { chain_hostnames(no); keep_hostname(yes); use_fqdn(no); use_dns(yes); }; Any ideas on how to fix this? The only hint I've gotten is from our networks group, who uses kiwi syslog and had to make sure a checkbox for RFC 3164 headers compliance was turned on. I don't see an equivalent of that not already set with the above config. Thanks, Ian

2 1

Profiling syslog-ng memory usage and performance
by William Sweat 16 Mar '12

16 Mar '12

Hello, I'm having issues with syslog-ng hitting swap on a set of servers that handle a lot of web traffic. Are there good techniques to figure out an optimal configuration or if there's something with syslog-ng that may prove problematic with large amounts of requests? I'm using the syslog-ng premium client (LTS v4.0.3b), on Linux systems with 256GB of Memory and RAID10. I've enabled more informational logging, but because each system handles multiple gigabytes of traffic per day, debugging is problematic as this is a problem that manifests after a day (or three). Also I am using nightly logrotate, so syslog-ng does get restarted everyday to write new log files. These are only high traffic servers that are experiencing swap issues, the syslog-ng server doesn't have any problems. Thanks, Will Sweat Demand Media ________________________________ Please NOTE: This electronic message, including any attachments, may include privileged, confidential and/or inside information owned by Demand Media, Inc. Any distribution or use of this communication by anyone other than the intended recipient(s) is strictly prohibited and may be unlawful. If you are not the intended recipient, please notify the sender by replying to this message and then delete it from your system. Thank you.

2 1