- syslog-ng - lists.balabit.hu

Syslog-ng unable to configure the client for sending messages to central server
by Rakesh Rajasekharan 26 Jun '13

26 Jun '13

I am getting some errors when i try to set up my client machine and try to send the messages to a remote central server. This is the error that I get, syslog-ng service starting. Plugin module not found in 'module-path'; module-path='/opt/soe/local/syslog-ng-3.3.7/lib/syslog-ng', module='afsocket' Plugin module not found in 'module-path'; module-path='/opt/soe/local/syslog-ng-3.3.7/lib/syslog-ng', module='afsocket' Error parsing destination, destination plugin tcp not found in /opt/soe/local/syslog_ng/etc/syslog-ng.conf at line 24, column 9: tcp("10.5.68.164" port(514)); ^^^ syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng. Any help on this please...

3 2

patterndb and log analysis
by Matt Zagrabelny 26 Jun '13

26 Jun '13

Greetings syslog-ng users, I've never used patterndb, but have been aware of it for a while. I just cloned the git://git.balabit.hu/bazsi/syslog-ng-patterndb.git and it looks like the project has not seen much activity since 2010. Are people still using patterndb? Do the patterns not change much and that is the reason that the git database has not changed much? What are free software enthusiasts using for log analysis? Thanks for the answers! -mz

3 3

problems with syslog-ng host filter
by Orangepeel Beef 26 Jun '13

26 Jun '13

I've got some strange problems with trying to use the syslog-ng host filter. It seems that every logline matches the filter f_comware but none of them match f_netscaler. It makes very little sense to me. Reverse dns is working as the logs that end up in /var/log/remote have the correct hostname being logged from the $HOST template. I have a second issue that SEC does not die when syslog-ng is restarted. (I have used this setup in the past and have had no problems, but i suppose things may have changed in both syslog-ng and sec since the last time) This is driving me crazy.. please help ;) #config @version: 3.1 # # Syslog-ng configuration file, compatible with default Debian syslogd # First, set some global options. options { long_hostnames(on); flush_lines(0); use_dns(yes); use_fqdn(yes); owner("syslog"); group("adm"); perm(0640); dns_cache_size(2000); dns_cache_expire(21600); dir_perm(0755); dir_group("adm"); stats_freq(0); log_fifo_size(200000); create_dirs(yes); bad_hostname("^gconfd$"); chain_hostnames(no); keep_hostname(no); }; source s_remote { udp();tcp(); }; destination d_remote { file("/var/log/remote/$HOST/$YEAR-$MONTH-$DAY.log" template("$R_DATE $HOST $MSG\n") template_escape(no)); }; destination d_netscaler { program("/usr/local/sbin/sec_netscaler" template("$R_DATE $HOST $MSG\n") template_escape(no)); }; destination d_comware { program("/usr/local/sbin/sec_comware" template("$R_DATE $HOST $MSG\n") template_escape(no)); }; filter f_netscaler { host("lb*ae1.mydomain.com"); }; filter f_comware { host("(as|cs|r)*ae1.mydomain.com"); }; log { source(s_remote); destination(d_remote); }; log { source(s_remote); filter(f_netscaler); destination(d_netscaler); }; log { source(s_remote); filter(f_comware); destination(d_comware); }; #debug logs [13:38:54] Filter rule evaluation result; filter_result='match', filter_rule='f_comware' [13:38:56] Incoming log entry; line='<190>Jun 21 20:37:54 2013 R0507S3Z3AE1 %%10MSTP/6/MSTP_FORWARDING(l): -DevIP=x.x.x.x; Instance 0\'s Ten-GigabitEthernet1/0/2 has been set to forwarding state.' [13:38:58] Filter rule evaluation begins; filter_rule='f_netscaler' [13:39:00] Filter node evaluation result; filter_result='not-match' [13:39:02] Filter rule evaluation result; filter_result='not-match', filter_rule='f_netscaler' [13:39:04] Filter rule evaluation begins; filter_rule='f_comware' [13:39:06] Filter node evaluation result; filter_result='match' [13:39:08] Filter rule evaluation result; filter_result='match', filter_rule='f_comware' [13:39:10] Incoming log entry; line='<190>Jun 21 20:37:54 2013 R0507S3Z3AE1 %%10MSTP/6/MSTP_FORWARDING(l): -DevIP=x.x.x.x; Instance 1\'s Ten-GigabitEthernet1/0/2 has been set to forwarding state.' [13:39:12] Filter rule evaluation begins; filter_rule='f_netscaler' [13:39:14] Filter node evaluation result; filter_result='not-match' [13:39:16] Filter rule evaluation result; filter_result='not-match', filter_rule='f_netscaler' [13:39:18] Filter rule evaluation begins; filter_rule='f_comware' [13:39:20] Filter node evaluation result; filter_result='match' [13:39:22] Filter rule evaluation result; filter_result='match', filter_rule='f_comware' [13:39:24] Incoming log entry; line='<134> 06/21/2013:20:37:54 GMT lb1o1ae1 0-PPE-2 : UI CMD_EXECUTED 232044114 0 : User nsroot - Remote_ip x.x.x.x - Command "login nsroot "********"" - Status "Success"' [13:39:26] Initializing destination file writer; template='/var/log/remote/$HOST/$YEAR-$MONTH-$DAY.log', filename='/var/log/remote/lb1o1ae1.mydomain.com/2013-06-21.log' [13:39:28] Filter rule evaluation begins; filter_rule='f_netscaler' [13:39:31] Filter node evaluation result; filter_result='not-match' [13:39:33] Filter rule evaluation result; filter_result='not-match', filter_rule='f_netscaler' [13:39:35] Filter rule evaluation begins; filter_rule='f_comware' [13:39:37] Filter node evaluation result; filter_result='match' [13:39:39] Filter rule evaluation result; filter_result='match', filter_rule='f_comware' [13:39:41] ^CTermination requested via signal, terminating; [13:39:43] syslog-ng shutting down; version='3.1.3' #logs in /var/log/remote/lb2z2ae1.mydomain.com Jun 21 20:23:34 lb2z2ae1.mydomain.com 20:23:34 GMT lb2z2ae1 0-PPE-3 : UI CMD_EXECUTED 28261 0 : User nsroot - Remote_ip x.x.x.x - Command "show service GL-AE1-2AZ1-DB0001_9191" - Status "Success"

2 4

gsoc 2013 - syslog-ng: redis destination
by Tihamér Petrovics 24 Jun '13

24 Jun '13

Hi, I'm Tihamér Petrovics and I will spend this summer in a project of gsoc 2013 to develop redis destination for syslog-ng. If you are curious, you can find my blog here, I will update it approximately weekly: http://tichygsoc.blogspot.com/ All the best, Tihi

1 0

I/O errors
by Russell Fulton 17 Jun '13

17 Jun '13

getting these errors at a rate of several per second but I can't figure out what is causing them: Jun 16 03:45:01 s_self(a)itslogprd01.its.auckland.ac.nz syslog-ng[31159]: I/O error occurred while reading; fd='758', error='Connection reset by peer (104)' Jun 16 03:45:01 s_self(a)itslogprd01.its.auckland.ac.nz syslog-ng[31159]: I/O error occurred while reading; fd='808', error='Connection reset by peer (104)' Jun 16 03:45:01 s_self(a)itslogprd01.its.auckland.ac.nz syslog-ng[31159]: I/O error occurred while reading; fd='906', we have one network destination which is a tcp connection and which appears to be fine. No errors on the other end and stats show now dropped packets. Russell

2 3

Syslog-ng under Windows (Cygwin - copying from another pc)
by Tom Doberstein 17 Jun '13

17 Jun '13

Hello, I am using syslog-ng in cygwin under WinXP. On my developer pc I have no problems, everything works very fine. But when I copy the cygwin directory (from c:/Programs to c:/Programs) on another pc, I got the following errors: ----------------- Error opening plugin module; module='afsocket', error='Exec format error' Error opening plugin module; module='afsocket', error='Exec format error' Error parsing source, source plugin unix-dgram not found in source confgen syste m at line 1, column 1: included from /etc/syslog-ng/syslo g-ng.conf line 10, column 9 unix-dgram ^^^^^^^^^^ syslog-ng documentation: http://www.balabit.com/support/documentation/?product=s yslog-ng mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng ----------------- I already started the syslog-ng-config and the service was installed successfully, but it will not start. cygrunsrv.exe -S syslog-ng will produce this error: "cygrunsrv: Error starting a service: QueryServiceStatus: Win32 error 1062: (...)". Is there now any right problem that the process cannot read the module correctly or something else? What I noticed is, when I copy the cygwin directory to the server and back to the same pc (in the same directory) the error occurs too. I hope that somebody can help me with that problem. Regards, Tom

3 4

AUTO: James B Horwath is out of the office (returning 06/19/2013)
by James B Horwath 17 Jun '13

17 Jun '13

I am out of the office until 06/19/2013. I will be out of the office until 06/19//2013 with limited access to e-mail. If you have an urgent issue please contact Sunil Gupta at (610)-807-6710 or Dylan Yoder at (610)-807-7230. Regards Jim Horwath IT Security Services & Governance, Corporate BTS and IT Security/Risk Tel 610-807-8795 Mobile 610-533-6972 3900 Burgess Place, Bethlehem, Pa. 18017 Jim_Horwath(a)glic.com Note: This is an automated response to your message "syslog-ng Digest, Vol 98, Issue 25" sent on 6/17/2013 6:00:03. This is the only notification you will receive while this person is away. ----------------------------------------- This message, and any attachments to it, may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are notified that any use, dissemination, distribution, copying, or communication of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately by return e-mail and delete the message and any attachments. Thank you.

1 0

[PATCH (3.5) 1/2] tests: Add a testcase for template_escape()
by Gergely Nagy 15 Jun '13

15 Jun '13

In order not to break template_escape() functionality, add a test case for basic functionality testing. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- libtest/template_lib.c | 17 +++++++++++++---- libtest/template_lib.h | 5 +++-- tests/unit/test_template.c | 10 +++++++++- 3 files changed, 25 insertions(+), 7 deletions(-) diff --git a/libtest/template_lib.c b/libtest/template_lib.c index 38dfeff..7992608 100644 --- a/libtest/template_lib.c +++ b/libtest/template_lib.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2012 BalaBit IT Ltd, Budapest, Hungary + * Copyright (c) 2012-2013 BalaBit IT Ltd, Budapest, Hungary * Copyright (c) 2012 Balázs Scheidler * * This library is free software; you can redistribute it and/or @@ -53,6 +53,7 @@ create_sample_message(void) log_msg_set_value(msg, log_msg_get_value_handle("APP.STRIP3"), " value ", -1); log_msg_set_value(msg, log_msg_get_value_handle("APP.STRIP4"), "value", -1); log_msg_set_value(msg, log_msg_get_value_handle("APP.STRIP5"), "", -1); + log_msg_set_value(msg, log_msg_get_value_handle("APP.QVALUE"), "\"value\"", -1); log_msg_set_match(msg, 0, "whole-match", -1); log_msg_set_match(msg, 1, "first-match", -1); log_msg_set_tag_by_name(msg, "alma"); @@ -70,12 +71,13 @@ create_sample_message(void) } LogTemplate * -compile_template(const gchar *template) +compile_template(const gchar *template, gboolean escaping) { LogTemplate *templ = log_template_new(configuration, NULL); gboolean success; GError *error = NULL; + log_template_set_escape(templ, escaping); success = log_template_compile(templ, template, &error); assert_true(success, "template expected to compile cleanly, but it didn't, template=%s, error=%s", template, error ? error->message : "(none)"); g_clear_error(&error); @@ -86,6 +88,13 @@ compile_template(const gchar *template) void assert_template_format(const gchar *template, const gchar *expected) { + assert_template_format_with_escaping(template, FALSE, expected); +} + +void +assert_template_format_with_escaping(const gchar *template, gboolean escaping, + const gchar *expected) +{ LogMessage *msg; LogTemplate *templ; GString *res = g_string_sized_new(128); @@ -93,7 +102,7 @@ assert_template_format(const gchar *template, const gchar *expected) msg = create_sample_message(); - templ = compile_template(template); + templ = compile_template(template, escaping); log_template_format(templ, msg, NULL, LTZ_LOCAL, 999, context_id, res); assert_string(res->str, expected, "template test failed, template=%s", template); log_template_unref(templ); @@ -113,7 +122,7 @@ assert_template_format_with_context(const gchar *template, const gchar *expected msg = create_sample_message(); context[0] = context[1] = msg; - templ = compile_template(template); + templ = compile_template(template, FALSE); log_template_format_with_context(templ, context, 2, NULL, LTZ_LOCAL, 999, context_id, res); assert_string(res->str, expected, "context template test failed, template=%s", template); diff --git a/libtest/template_lib.h b/libtest/template_lib.h index 760858c..f875e4f 100644 --- a/libtest/template_lib.h +++ b/libtest/template_lib.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2012 BalaBit IT Ltd, Budapest, Hungary + * Copyright (c) 2012-2013 BalaBit IT Ltd, Budapest, Hungary * Copyright (c) 2012 Balázs Scheidler * * This library is free software; you can redistribute it and/or @@ -29,11 +29,12 @@ #include "templates.h" void assert_template_format(const gchar *template, const gchar *expected); +void assert_template_format_with_escaping(const gchar *template, gboolean escaping, const gchar *expected); void assert_template_format_with_context(const gchar *template, const gchar *expected); void assert_template_failure(const gchar *template, const gchar *expected_failure); LogMessage *create_sample_message(void); -LogTemplate *compile_template(const gchar *template); +LogTemplate *compile_template(const gchar *template, gboolean escaping); void init_template_tests(void); void deinit_template_tests(void); diff --git a/tests/unit/test_template.c b/tests/unit/test_template.c index e5ff1a0..8546dd4 100644 --- a/tests/unit/test_template.c +++ b/tests/unit/test_template.c @@ -53,7 +53,7 @@ assert_template_format_multi_thread(const gchar *template, const gchar *expected gint i; msg = create_sample_message(); - templ = compile_template(template); + templ = compile_template(template, FALSE); args[0] = msg; args[1] = templ; args[2] = (gpointer) expected; @@ -248,6 +248,13 @@ test_multi_thread(void) assert_template_format_multi_thread("dani $(echo $HOST $DATE $(echo huha)) balint", "dani bzorp Feb 11 10:34:56.000 huha balint"); } +static void +test_escaping(void) +{ + assert_template_format_with_escaping("${APP.QVALUE}", FALSE, "\"value\""); + assert_template_format_with_escaping("${APP.QVALUE}", TRUE, "\\\"value\\\""); +} + int main(int argc G_GNUC_UNUSED, char *argv[] G_GNUC_UNUSED) { @@ -269,6 +276,7 @@ main(int argc G_GNUC_UNUSED, char *argv[] G_GNUC_UNUSED) test_syntax_errors(); test_compat(); test_multi_thread(); + test_escaping(); /* multi-threaded expansion */ -- 1.7.10.4

2 2

Re: [syslog-ng] TCP packet collapse error
by Balazs Scheidler 15 Jun '13

15 Jun '13

Sorry to tune in so late in the xonversation. Without responding to all poonts here are some related facts. 1. Incoming messages in the same connection are processed serially, eg it will not scale to multiple CPUs for the same connection. 2. Output is separate from input and runs in a separate thread. 3. The more the input/output the more syslog-ng scales CPU wise 4. Input performance depends a lot on the number of messages processed in a single iteration. That is affected by log-fetch-limit and window size. Increasing these needs a bump in log-fifo-size too. 5. Output performance depends on flush-lines a lot as syslog-ng batches writes in those chunks. This was added in 3.3. 6. It might make sense to tune VM related parameters in the kernel to start write out earlier, this way smoothing the output side, as that increases latency on the input. Hope this helps Cheers, On Jun 4, 2013 10:47 AM, "Xuri Nagarin" <secsubs(a)gmail.com> wrote: > Yes, multi-threading is enabled. To eliminate disk IO as an issue, I > created a tmpfs filesystem to write logs to. > > I am watching TCP collapsed errors with this simple script in one window: > -----xxxxxxxxx--------- > #!/bin/bash > > while true > do > r1=`/bin/netstat -s | /bin/grep collapsed | /bin/awk '{print $1}'` > sleep 2 > r2=`/bin/netstat -s | /bin/grep collapsed | /bin/awk '{print $1}'` > let loss=$r2-$r1 > echo $loss > done > ------xxxxxxxxxx-------- > > In another window, I watch syslog-ng stats with the script posted earlier > in this thread. > > In a third window, I watch tcp receive queue for only one source that I > enabled - sudo netstat --tcp -p -o -e -e -v | grep "$ip" > > The receive queue builds to hundreds of thousands of packets to millions > depending on the syslog-ng config but rarely drops to zero. As packets in > the receive queue stack up, the collapsed packet count rises from a lower > thousands to tens of thousands. Raising the receive buffers at the OS level > makes the problem worse as does increasing the so_rcvbuf in syslog-ng. In a > fourth window I watch CPU utilization with mpstat. A single core > utilization stays high. > > I tried a vanilla syslog-ng config with no tweaking to varying values of > log_fifo_size, log_fetch_limit, log-iw-size, and so_rcvbuf. Packet collapse > errors stubbornly stay at low thousands to hundreds of thousands. > > Looking at iptraf stats, inbound pps is 15-20k. > > Looking at tcpdump window sizing, the syslog-ng host slows down the sender > by sending a windows size of as small as 1. > > Without looking at the syslog-ng code, it looks like a single source is > getting tied to a single thread/core and if traffic is high from that > source, it ends up saturating the core causing packet loss. > > I will continue to investigate more but need more ideas :) > > > > > On Tue, Jun 4, 2013 at 12:21 AM, Daniel Neubacher < > daniel.neubacher(a)xing.com> wrote: > >> If I remember this correctly you have to adjust the linux networking >> options too. Like here: >> http://wwwx.cs.unc.edu/~sparkst/howto/network_tuning.php**** >> >> Maybe you missed an option.**** >> >> And to activate multithreading you need threaded(yes); But it’s odd that >> the syslog problem is gone now.**** >> >> ** ** >> >> ** ** >> >> ** ** >> >> *Von:* syslog-ng-bounces(a)lists.balabit.hu [mailto: >> syslog-ng-bounces(a)lists.balabit.hu] *Im Auftrag von *Xuri Nagarin >> *Gesendet:* Montag, 3. Juni 2013 23:53 >> >> *An:* Syslog-ng users' and developers' mailing list >> *Betreff:* Re: [syslog-ng] TCP packet collapse errors**** >> >> ** ** >> >> I switched to 3.4.1 (multi-threaded) but still no respite from the TCP >> collapse issue.**** >> >> ** ** >> >> I tried to set so_rcvbuf to '51200000' and got this error:**** >> >> ** ** >> >> "Starting syslog-ng: The kernel refused to set the receive buffer >> (SO_RCVBUF) to the requested size, you probably need to adjust buffer >> related kernel parameters; so_rcvbuf='51200000', so_rcvbuf_set='20971520'" >> **** >> >> ** ** >> >> In terms of lps processed, may be it's just a case of Monday but I am >> seeing numbers as high as 40k consistently now with no drops to zero. Disk >> IO is low but more consistent between 4-8 Mbytes/sec.**** >> >> ** ** >> >> In terms of multi-threading, not sure how well the code does it because I >> am seeing individual cores spike to 100% even when several others are at >> idle.**** >> >> ** ** >> >> ** ** >> >> ** ** >> >> On Fri, May 31, 2013 at 12:41 PM, Xuri Nagarin <secsubs(a)gmail.com> wrote: >> **** >> >> Thanks Daniel. I think I owe you a six-pack of your favourite beer just >> for the script :) **** >> >> ** ** >> >> Very interesting stats. For both hosts, lps ranges from anywhere between >> few hundred to 30K per second but falls to zero for several seconds. All >> the packet loss happens when the lps is peaking. What has me concerned is >> that lps falls to zero for 15-20 seconds at a time. During this period disk >> IO falls to near zero too and packet hemorrhaging pauses. **** >> >> ** ** >> >> All my sources are tcp and there are only four of them but those are >> proprietary products so they may be sending data in batches/bursts. I will >> tweak the tcp source options and see what more I can squeeze out of this >> build of syslog-ng and the disks. But you are right - I do need to upgrade >> to the multi-threaded version asap. At some point, a single core process is >> going to stall on writing to disk and not be able to reap tcp buffers >> during that time.**** >> >> ** ** >> >> I think the disk scheduler probably did not have any impact but one >> system has the disk scheduler tweak and other doesn't but I don't see any >> significant differences with packet loss on either. **** >> >> ** ** >> >> Right now, I think the issue simply is that when a burst of data comes >> in, the single core syslog-ng blocks of IO and cannot switch back to >> reading TCP buffers fast enough to clear up everything in the buffer. High >> network IO, high Disk IO, high lps and packet loss - all match up.**** >> >> ** ** >> >> A larger receive buffer for the tcp source in syslog config will be a >> band-aid, I think until I can build a multi-threaded syslog-ng. Eventually, >> given that these are 7200RPM disks in RAID-1, the disk sub-system will >> start being a bottleneck but by then, hopefully, I will find a host with >> better disk througput :-)**** >> >> ** ** >> >> ** ** >> >> ** ** >> >> ** ** >> >> On Fri, May 31, 2013 at 2:41 AM, Daniel Neubacher < >> daniel.neubacher(a)xing.com> wrote:**** >> >> I've forgot to ask for your syslog source settings. **** >> >> Here is my cfg. Maybe it helps. **** >> >> ** ** >> >> tcp(**** >> >> log_fetch_limit(1000)**** >> >> max-connections(5000)**** >> >> so_rcvbuf(51200000)**** >> >> keep_timestamp(yes)**** >> >> port(514)**** >> >> log-iw-size(500000)**** >> >> );**** >> >> ** ** >> >> I've got it from a great artice >> http://codeascraft.com/2012/08/13/performance-tuning-syslog-ng/**** >> >> ** ** >> >> ** ** >> >> ** ** >> ------------------------------ >> >> *Von:* syslog-ng-bounces(a)lists.balabit.hu [ >> syslog-ng-bounces(a)lists.balabit.hu]" im Auftrag von "Xuri Nagarin [ >> secsubs(a)gmail.com]**** >> >> *Gesendet:* Freitag, 31. Mai 2013 10:12**** >> >> >> *An:* Syslog-ng users' and developers' mailing list**** >> >> *Betreff:* Re: [syslog-ng] TCP packet collapse errors**** >> >> Thanks for the quick response, Daniel.**** >> >> ** ** >> >> I look at statistics for an hour before tweaking flush_lines to zero and >> setting log_fifo_size to 10000. In that period, syslog-ng reported >> processing 7,898,310,589 messages across all destinations and >> dropped 4,200,260. **** >> >> ** ** >> >> After making the change (flush_lines set to 0 and log_fifo_size to >> 10000), I looked at three sets (half hour) of stats (default, every 10 >> minutes). The dropped messages are now zero across all destinations.**** >> >> ** ** >> >> But the collapsed TCP packets count keeps incrementing. I ran 'iostat -xm >> 5' and "watch -d 'netstat -s | grep collpased' " in two windows >> side-by-side. Each time that disk IO spikes up, the TCP collapsed counter >> starts incrementing. Disk IO remains almost zero for about half a minute >> and then spikes up to ~4-25 Mbytes/sec for half a minute.**** >> >> ** ** >> >> Does this mean I need to bump up log_fifo_size even higher? I think >> ideally we want the disk to be consistently written to instead of bursts of >> write activity. Right?**** >> >> ** ** >> >> ** ** >> >> ** ** >> >> ** ** >> >> ** ** >> >> ** ** >> >> ** ** >> >> ** ** >> >> On Thu, May 30, 2013 at 10:56 PM, Daniel Neubacher < >> daniel.neubacher(a)xing.com> wrote:**** >> >> I don't know how much logs you are getting but should tweak >> "log_fifo_size (1000);" to a higher number. Your flush_lines is really high >> too.. I tested around with flush lines but I ended setting it to 0 with 50k >> log per second. And they greatest of all tweaks would be a newer syslog >> version because of the threading.**** >> ------------------------------ >> >> *Von:* syslog-ng-bounces(a)lists.balabit.hu [ >> syslog-ng-bounces(a)lists.balabit.hu]" im Auftrag von "Xuri Nagarin [ >> secsubs(a)gmail.com] >> *Gesendet:* Freitag, 31. Mai 2013 07:46 >> *An:* Syslog-ng users' and developers' mailing list >> *Betreff:* [syslog-ng] TCP packet collapse errors**** >> >> I have a pair of Syslog-NG servers running 3.2.5-3. The hardware specs >> are - Quad Xeon E5-2680 (32 cores), 32GB RAM, and two 1TB SAS 7200 RPM >> disks in RAID-1. **** >> >> **** >> >> OS is RHEL6.2 - Kernel 2.6.32-279.5.2. Filesystem is ext3.**** >> >> ** ** >> >> Global options are set as:**** >> >> options {**** >> >> flush_lines (1000);**** >> >> time_reopen (10);**** >> >> log_fifo_size (1000);**** >> >> long_hostnames (off);**** >> >> use_dns (no);**** >> >> use_fqdn (no);**** >> >> create_dirs (yes);**** >> >> keep_hostname (yes);**** >> >> keep_timestamp(yes);**** >> >> dir_group("syslog");**** >> >> perm(0640);**** >> >> dir_perm(0750);**** >> >> group("syslog");**** >> >> };**** >> >> ** ** >> >> I have already set TCP kernel buffers to 128MB max and set disk scheduler >> to "deadline".**** >> >> ** ** >> >> But even under light disk IO load, from ~8-25MB, I see "1320811067 >> packets collapsed in receive queue due to low socket buffer". I had some >> other processes on the host writing to disk. Stopping them reduced the >> packet errors but this number still keeps incrementing.**** >> >> ** ** >> >> To rule out other issues, I temporarily pointed my disk-based >> destinations to /dev/null and then packet losses/errors stopped. So either >> Syslog-NG isn't able to write to disk fast enough or there is an underlying >> OS/hardware issue.**** >> >> ** ** >> >> Both hosts have the same issue. Any pointers in troubleshooting it will >> be appreciated.**** >> >> ** ** >> >> TIA.**** >> >> ** ** >> >> ** ** >> >> >> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> **** >> >> ** ** >> >> >> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> **** >> >> ** ** >> >> ** ** >> >> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> >> > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > >

1 0

Google Summer of Code 2013 - syslog-ng: MySQL destination driver
by Gyula Petrovics 15 Jun '13

15 Jun '13

Hi everyone, I am Gyula Petrovics, one of the students who got the opportunity to help in the development of the syslog-ng. My goal is to redesign the current libdbi based solution to store logs in MySQL database. I made a blog here <http://petrovicsgyula.blogspot.com>, please visit this blog if you are interested in the current progress and the details of the project. Thanks for reading my e-mail. Best regards, Gyula

4 6

[libsyslog_ng_la-afinter.lo] Error 1, syslog-ng-3.4.1 on x86_64
by Flur Blubr 15 Jun '13

15 Jun '13

Dear all, looking for help, for 4 days trying to compile syslog-ng-3.4.1 on some kind of custom RED HAT derivate and still failing. RPMs and YUM are forbidden and system has libraries spread around all directories / using exports export EVTLOG_CFLAGS, export EVTLOG_LIBS, export GLIB_CFLAGS, export GLIB_LIBS to overwrite PKG CONFIG . uname -a Linux 2.6.32-279.1.1.el6.x86_64 #1 SMP Wed Jun 20 11:41:22 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.3 (Santiago) Configuration: ./configure --prefix=/my-prefix/syslog-ng --enable-sql --enable-ipv6 --enable-dynamic-linking --build=x86_64-pc-linux-gnu --host=i686-pc-linux-gnu --enable-shared --disable-static CFLAGS=-O2 Getting this: syslog-ng Open Source Edition 3.4.1 configured Compiler options: compiler : gcc -std=gnu99 compiler options : -O2 -Wall -pthread -I/my-prefix/sources/glib-2.15.4/ -I/my-prefix/eventlog/include/eventlog/ -I$(top_srcdir)/lib/ivykis/src/include -I$(top_builddir)/lib/ivykis/src/include -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 linker flags : prefix : /my-prefix/syslog-ng linking mode : dynamic embedded crypto : no __thread keyword : yes Submodules: ivykis : internal libmongo-client : internal librabbitmq : internal Features: Debug symbols : no GCC profiling : no Memtrace : no IPV6 support : yes spoof-source support : no tcp-wrapper support : no Linux capability support : yes PCRE support : yes Env wrapper support : no systemd support : no (unit dir: none) Modules: Module search path : /my-prefix/syslog-ng/lib/syslog-ng Sun STREAMS support (module): no SSL support (module) : yes SQL support (module) : yes PACCT module (EXPERIMENTAL) : no MongoDB destination (module): yes JSON support (module) : no SMTP support (module) : no AMQP destination (module) : yes GEOIP support (module) : no But make is failing no matter what I do: afinter.c: At top level: afinter.c:385: error: expected ')' before 'mark_freq' afinter.c: In function 'afinter_message_posted': afinter.c:404: error: 'internal_msg_lock' undeclared (first use in this function) afinter.c:405: error: 'internal_msg_queue' undeclared (first use in this function) afinter.c:407: warning: implicit declaration of function 'g_queue_new' afinter.c:409: warning: implicit declaration of function 'g_queue_push_tail' afinter.c:411: error: 'AFInterSource' has no member named 'post' afinter.c: At top level: afinter.c:416: error: expected ')' before 'hook_type' afinter.c: In function 'afinter_global_init': afinter.c:424: warning: implicit declaration of function 'register_application_hook' afinter.c:424: error: 'afinter_register_posted_hook' undeclared (first use in this function) make[4]: *** [libsyslog_ng_la-afinter.lo] Error 1 make[4]: Leaving directory `/my-prefix/syslog-ng-3.4.1/lib' make[3]: *** [all-recursive] Error 1 make[3]: Leaving directory `/my-prefix/syslog-ng-3.4.1/lib' make[2]: *** [all] Error 2 make[2]: Leaving directory `/my-prefix/syslog-ng-3.4.1/lib' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/my-prefix/syslog-ng-3.4.1' make: *** [all] Error 2 Can somebody advice? Thx for your time. Regards Michal

4 9

syslog-ng 3.3.11 has been released
by devel＠balabit.hu 14 Jun '13

14 Jun '13

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.3.11 SUMMARY : new stable release DATE : Jun 14, 2013 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.3.11) has been released. For latest fixes in the 3.3.x feature branch you are recommended to upgrade to this version. CHANGES: 3.3.11 Fri, 14 Jun 2013 16:00:00 +0100 Highlights ========== This release is a bug-fix release, correcting a handful of issues discovered since the previous one. Bugfixes ======== * Most of the changes in 3.3.10 have been reverted, as they resulted in instability and dead locks. * The configure files were updated to work with the newest automake, and not use long obsolete macros. * Zero-length templates (or template functions with zero-length output) do not result in an infinite loop anymore. * The $(if) and $(grep) template functions now respect the template_escape() setting. Credits ======= syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Balazs Scheidler <bazsi(a)balabit.hu> Evan Rempel <erempel(a)uvic.ca> Gergely Nagy <algernon(a)balabit.hu> Imre Lazar <imre(a)balabit.hu> Laszlo Meszaros <lmesz(a)balabit.hu> Michael Sterrett <mr_bones_(a)gentoo.org> DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

[Bug 230] New: Passing args to blocks is broken in syslog-ng v.3.4
by bugzilla＠bugzilla.balabit.com 13 Jun '13

13 Jun '13

https://bugzilla.balabit.com/show_bug.cgi?id=230 Summary: Passing args to blocks is broken in syslog-ng v.3.4 Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: koldaevav(a)gmail.com Type of the Report: bug Estimated Hours: 0.0 After an upgrade from 3.3.7 to 3.4.1: ... block root mylogs(log()) { }; mylogs(log(123)) ... Gives: Unknown argument; context='root block mylogs', arg='log', value='123' Error parsing config, syntax error, unexpected LL_ERROR, expecting $end in /etc/syslog-ng/syslog-ng.conf at line 11, column 1: mylogs(log(123)) ^^^^^^ -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 6

[Bug 237] New: syslog-ng 3.4.2 (and 3.3.10): test failures in Fedora 18
by bugzilla＠bugzilla.balabit.com 13 Jun '13

13 Jun '13

https://bugzilla.balabit.com/show_bug.cgi?id=237 Summary: syslog-ng 3.4.2 (and 3.3.10): test failures in Fedora 18 Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: jpo(a)di.uminho.pt Type of the Report: --- Estimated Hours: 0.0 The syslog-ng 3.4.2 (and 3.3.10) test suite fails in Fedora 18: ---------- ... $DATE ${HOST:--} ${PROGRAM:--} ${PID:--} ${MSGID:--} ${SDATA:--} $MSG speed: 1056859.015 msg/sec *** glibc detected *** /home/fedora/rpms/BUILD/syslog-ng-3.4.2/tests/unit/.libs/lt-test_template_speed: free(): invalid pointer: 0x0000003a831b1c68 *** ======= Backtrace: ========= /lib64/libc.so.6[0x3a82e7ca8e] /lib64/libglib-2.0.so.0(g_mutex_free+0x9)[0x3a84a1c719] /lib64/libglib-2.0.so.0(g_static_mutex_free+0x16)[0x3a84a1ca66] /home/fedora/rpms/BUILDROOT/syslog-ng-3.4.2-0.1.fc18.x86_64//usr/lib64/libsyslog-ng-3.4.2.so(crypto_deinit+0x4e)[0x7f43c2fef0be] /lib64/ld-linux-x86-64.so.2[0x3a82a0f5f7] /lib64/libc.so.6[0x3a82e38df1] /lib64/libc.so.6[0x3a82e38e75] /lib64/libc.so.6(__libc_start_main+0xfc)[0x3a82e21a0c] /home/fedora/rpms/BUILD/syslog-ng-3.4.2/tests/unit/.libs/lt-test_template_speed[0x401681] ======= Memory map: ======== 00400000-00403000 r-xp 00000000 08:02 1863099 /home/fedora/rpms/BUILD/syslog-ng-3.4.2/tests/unit/.libs/lt-test_template_speed 00602000-00603000 r--p 00002000 08:02 1863099 /home/fedora/rpms/BUILD/syslog-ng-3.4.2/tests/unit/.libs/lt-test_template_speed 00603000-00604000 rw-p 00003000 08:02 1863099 /home/fedora/rpms/BUILD/syslog-ng-3.4.2/tests/unit/.libs/lt-test_template_speed 01f72000-01fb4000 rw-p 00000000 00:00 0 [heap] 3a82a00000-3a82a20000 r-xp 00000000 08:02 828574 /usr/lib64/ld-2.16.so ... 7f43c342c000-7f43c342d000 r--p 0000a000 08:02 1862848 /home/fedora/rpms/BUILDROOT/syslog-ng-3.4.2-0.1.fc18.x86_64/usr/lib64/syslog-ng/libsyslogformat.so 7f43c342d000-7f43c342e000 rw-p 0000b000 08:02 1862848 /home/fedora/rpms/BUILDROOT/syslog-ng-3.4.2-0.1.fc18.x86_64/usr/lib64/syslog-ng/libsyslogformat.so 7f43c342e000-7f43c342f000 rw-p 00000000 00:00 0 7fff57cb7000-7fff57cd8000 rw-p 00000000 00:00 0 [stack] 7fff57cef000-7fff57cf1000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] /bin/sh: line 5: 6107 Aborted (core dumped) ${dir}$tst FAIL: test_template_speed ... ---------- -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

4 6

insider 2013-06: syslog-ng 3.4.2 released; PatternDB update; GSoC; RSS destination
by Peter Czanik 13 Jun '13

13 Jun '13

Dear syslog-ng users, This is the 25th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. FEATURED NEWS syslog-ng 3.4.2 is released --------------------------- Last week syslog-ng 3.4.2 was released, the first bugfix release in the 3.4 series. The most important fix is for a hang when suppress() was used. A full list of changes is available in git at: https://github.com/balabit/syslog-ng-3.4/commits/master Sources are available from git or as tgz from http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… For third-party binary packages for various Linux and UNIX platforms, visithttp://www.balabit.com/network-security/syslog-ng/opensource-logging-s… for 3^rd party binary packages for various distributions and UNIX variants. PatternDB git moved and updated ------------------------------- The BalaBit patterndb git moved recently to github. It’s available now at https://github.com/balabit/syslog-ng-patterndb instead of our server. That way the patterns are available next to the syslog-ng sources, and are also faster to download. We have also updated, reorganized and extended the available patterns. For details, read https://czanik.blogs.balabit.com/2013/05/patterndb-git-moved-and-updated/ syslog-ng is participating GSoC again ------------------------------------- Just as last year, syslog-ng is participating in the Google Summer of Code under the umbrella of the openSUSE project. Two of our candidates have the opportunity to code all summer long as GsoC students. Gyula Petrovicswill work on a faster MySQL destination, which works without libdbi. The other student, Tihamér Petrovics, will add a redis destination, which can not only store logs but also provide counters and help with statistics. Related blogs: http://petrovicsgyula.blogspot.com/ http://tichygsoc.blogspot.hu/ RSS destination --------------- RSS destination is a neat little feature of syslog-ng. It can store up to 100 pieces log lines, and can serve them as an Atom feed. This destination works as a FIFO, so if it is full, the last incoming log line kicks out the first one. Now you are able to read the alerts from your machines in your favorite RSS reader (well, unfortunately not in Google Reader). Available at https://github.com/talien/syslog-ng in the feature/rss-destination branch. SHORT NEWS: * Using a low power gadget as syslog-ng server: https://czanik.blogs.balabit.com/2013/05/using-a-low-power- <https://czanik.blogs.balabit.com/2013/05/using-a-low-power-gadget-as-syslog…>gadget-as-syslog-ng-server/ <https://czanik.blogs.balabit.com/2013/05/using-a-low-power-gadget-as-syslog…> Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

How to log message without a timestamp?
by Anton Koldaev 12 Jun '13

12 Jun '13

Is there any way to send a message from syslog-ng via destination(tcp) with no timestamp at all? -- Best regards, Koldaev Anton

2 1

[PATCH (3.5) 0/3]: Catching template syntax errors at config time
by Gergely Nagy 10 Jun '13

10 Jun '13

Following up on the previous patch that added a check to the file() and pipe() destinations, allowing them to catch and report template syntax errors at config time, following this mail are three others, that do similar things to the set() and subst() rewrite functions, to the pair() statement of value-pairs() and to templatable settings in afamqp. These catch these issues: rewrite s_broken { set("$(+ 0", value(MESSAGE)); subst("original", "$(+ 0", value(MESSAGE)); }; destination d_broken { afamqp(body("$(+ 0") routing-key("$(+ 0") value-pairs(pair("INVALID", "$(+ 0"))); }; There are a couple of more cases in the codebase where the same issue is present, namely afsmtp() (the subject, body and header templates), afsql (fields and table name), and perhaps in patterndb too, I have not throughly inspected that one yet. -- |8]

1 3

Building 3.4.1 RPM
by Xuri Nagarin 07 Jun '13

07 Jun '13

I am trying to build a 3.4.1 RPM for CentOS 6. I already built the eventlog RPM and upgraded the one provided by EPEL repo. If I compile straight after untarring the source by running "./configure" (with options same as listed in the SPEC file below) and "make check", the tests runs ok and I get the syslog-ng binaries. But if I build a RPM from the SPEC file below, build fails with the following error. ------- error --------------- Input is not valid utf8, glob match requires utf8 input, thus it never matches in this case; value='\xe1rv\xedzt\xfbr\xf5t\xfck\xf6rf\xfar\xf3g\xe9p' PASS: test_matcher PASS: test_clone_logmsg PASS: test_serialize PASS: test_msgparse PASS: test_template /bin/sh: line 5: 31254 Segmentation fault (core dumped) ${dir}$tst FAIL: test_template_speed One "invalid regular expression" message is to be expected Error compiling regular expression; re='((', error='Unmatched ( or \(' One "invalid regular expressions" message is to be expected Error compiling regular expression; re='((', error='Unmatched ( or \(' WARNING: Your configuration file uses an obsoleted regexp option, please update your configuration; option='(?i)', change='use ignore-case flag instead of (?i)' One "invalid regular expression" message is to be expected Error compiling regular expression; re='((', error='Unmatched ( or \(' One "invalid regular expression" message is to be expected Error while compiling regular expression; regular_expression='((', error_at='', error_offset='2', error_message='missing )', error_code='14' One "invalid regular expressions" message is to be expected Error while compiling regular expression; regular_expression='((', error_at='', error_offset='2', error_message='missing )', error_code='14' One "invalid regular expressions" message is to be expected Invalid regexp flags; re='(?iana' One "invalid regular expressions" message is to be expected Error while compiling regular expression; regular_expression='(?iana', error_at='ana', error_offset='3', error_message='unrecognized character after (? or (?-', error_code='12' One "invalid regular expression" message is to be expected Error while compiling regular expression; regular_expression='((', error_at='', error_offset='2', error_message='missing )', error_code='14' PASS: test_filters DNS cache speed: 6578947.368 iters/sec inet_ntop speed: 1589572.405 iters/sec PASS: test_dnscache PASS: test_findeom PASS: test_findcrlf PASS: test_tags PASS: test_logwriter PASS: test_logproto PASS: test_serialize Failed to open the time zone file; filename='/usr/share/zoneinfo/Factory', message='Failed to open file \'/usr/share/zoneinfo/Factory\': open() failed: No such file or directory' Bogus timezone spec, must be in the format [+-]HH:MM, offset must be less than 24:00; value='Factory' SKIP: Factory PASS: test_zone PASS: test_persist_state PASS: test_value_pairs ==================== 1 of 21 tests failed ==================== make[3]: *** [check-TESTS] Error 1 make[3]: Leaving directory `/home/makerpm/rpmbuild/BUILD/syslog-ng-3.4.1/tests/unit' make[2]: *** [check-am] Error 2 make[2]: Leaving directory `/home/makerpm/rpmbuild/BUILD/syslog-ng-3.4.1/tests/unit' make[1]: *** [check-recursive] Error 1 make[1]: Leaving directory `/home/makerpm/rpmbuild/BUILD/syslog-ng-3.4.1/tests' make: *** [check-recursive] Error 1 error: Bad exit status from /var/tmp/rpm-tmp.izm78V (%check) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.izm78V (%check) ----------------------------xxxxxxxxxxxxx------------------------------------------------------------------------- ---------------syslog-ng.SPEC------------------------------------ %global evtlog_ver 0.2.12 %global _sbindir /sbin %global _libdir /%{_lib} Name: syslog-ng Version: 3.4.1 Release: 1%{?dist} Summary: Next-generation syslog server Group: System Environment/Daemons License: GPLv2+ Url: http://www.balabit.com/network-security/syslog-ng Source0: http://www.balabit.com/downloads/files?path=/syslog-ng/sources/%{version}/s… Source1: syslog-ng.conf Source2: syslog-ng.init.d Source3: syslog-ng.sysconfig Source4: syslog-ng.logrotate #Patch0: syslog-ng-3.2.4-disable-ssl-tests.patch #Patch1: syslog-ng-3.2.5-tests-functional-control.py.patch #Patch2: syslog-ng-3.2.5-tests-functional-sql-test.patch BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n) BuildRequires: pkgconfig BuildRequires: libtool BuildRequires: eventlog-devel >= %{evtlog_ver} BuildRequires: glib2-devel >= 2.10.1 BuildRequires: libdbi-devel BuildRequires: libnet-devel >= 1.1.4-3 BuildRequires: openssl-devel BuildRequires: pcre-devel >= 6.1 BuildRequires: tcp_wrappers-devel # For the test suite BuildRequires: python # For the SQL tests BuildRequires: libdbi-dbd-sqlite Requires: logrotate Requires(post): /sbin/chkconfig Requires(preun): /sbin/chkconfig Requires(preun): /sbin/service Requires(postun): /sbin/service Provides: syslog # merge separate syslog-vim package into one Provides: syslog-ng-vim = %{version}-%{release} Obsoletes: syslog-ng-vim < 2.0.8-1 %description syslog-ng, as the name shows, is a syslogd replacement, but with new functionality for the new generation. The original syslogd allows messages only to be sorted based on priority/facility pairs; syslog-ng adds the possibility to filter based on message contents using regular expressions. The new configuration scheme is intuitive and powerful. Forwarding logs over TCP and remembering all forwarding hops makes it ideal for firewalled environments. %package libdbi Summary: libdbi support for %{name} Group: Development/Libraries Requires: %{name} = %{version}-%{release} %description libdbi This module supports a large number of database systems via libdbi. %package devel Summary: Development files for %{name} Group: Development/Libraries Requires: %{name} = %{version}-%{release} %description devel The %{name}-devel package contains libraries and header files for developing applications that use %{name}. %prep %setup -q #%patch0 -p1 #%patch1 -p1 #%patch2 -p1 # fix perl path %{__sed} -i 's|^#!/usr/local/bin/perl|#!%{__perl}|' contrib/relogger.pl # fix executable perms on contrib files %{__chmod} -c a-x contrib/syslog2ng # fix authors file /usr/bin/iconv -f iso8859-1 -t utf-8 AUTHORS > AUTHORS.conv && \ %{__mv} -f AUTHORS.conv AUTHORS %build %configure \ --prefix=/ \ --libdir=/%{_lib} \ --includedir=%{_includedir} \ --sysconfdir=%{_sysconfdir}/%{name} \ --localstatedir=%{_sharedstatedir}/%{name} \ --datadir=%{_datadir}/%{name} \ --with-module-dir=/%{_lib}/%{name} \ --enable-ipv6 \ --enable-tcp-wrapper \ --enable-pcre \ --enable-ssl \ --enable-shared \ --disable-static \ --enable-dynamic-linking \ --enable-threads=posix \ --enable-linux-caps # remove rpath sed -i 's|^hardcode_libdir_flag_spec=.*|hardcode_libdir_flag_spec=""|g' libtool sed -i 's|^runpath_var=LD_RUN_PATH|runpath_var=DIE_RPATH_DIE|g' libtool make %{_smp_mflags} %install %{__rm} -rf %{buildroot} make DESTDIR=%{buildroot} install #%{__install} -d -m 755 %{buildroot}%{_sysconfdir}/%{name} %{__install} -p -m 644 %{SOURCE1} %{buildroot}%{_sysconfdir}/%{name}/syslog-ng.conf %{__install} -d -m 755 %{buildroot}%{_sysconfdir}/init.d %{__install} -p -m 755 %{SOURCE2} %{buildroot}%{_sysconfdir}/init.d/%{name} %{__install} -d -m 755 %{buildroot}%{_sysconfdir}/sysconfig %{__install} -p -m 644 %{SOURCE3} %{buildroot}%{_sysconfdir}/sysconfig/%{name} %{__install} -d -m 755 %{buildroot}%{_sysconfdir}/logrotate.d %{__install} -p -m 644 %{SOURCE4} %{buildroot}%{_sysconfdir}/logrotate.d/syslog # create the local state dir %{__install} -d -m 755 %{buildroot}/%{_sharedstatedir}/%{name} # install the main library header files %{__install} -d -m 755 %{buildroot}%{_includedir}/%{name} %{__install} -p -m 644 config.h %{buildroot}%{_includedir}/%{name} %{__install} -p -m 644 lib/*.h %{buildroot}%{_includedir}/%{name} # install vim files %{__install} -d -m 755 %{buildroot}%{_datadir}/%{name} %{__install} -p -m 644 contrib/syslog-ng.vim %{buildroot}%{_datadir}/%{name} for vimver in 70 71 72 73 ; do %{__install} -d -m 755 %{buildroot}%{_datadir}/vim/vim$vimver/syntax cd %{buildroot}%{_datadir}/vim/vim$vimver/syntax ln -s ../../../%{name}/syslog-ng.vim . cd - done find %{buildroot} -name "*.la" -exec rm -f {} \; %check LD_LIBRARY_PATH=%{buildroot}/%{_lib}:%{buildroot}/%{_lib}/%{name} make check %clean rm -rf %{buildroot} %post /sbin/ldconfig /sbin/chkconfig --add %{name} %preun if [ $1 -eq 0 ] ; then # Package removal, not upgrade /sbin/service %{name} stop > /dev/null 2>&1 /sbin/chkconfig --del %{name} fi %postun /sbin/ldconfig if [ $1 -ge 1 ] ; then # Package upgrade, not uninstall /sbin/service %{name} condrestart >/dev/null 2>&1 fi %triggerin -- vim-common VIMVERNEW=`rpm -q --qf='%%{epoch}:%%{version}\n' vim-common | sort | tail -n 1 | sed -e 's/[0-9]*://' | sed -e 's/\.[0-9]*$//' | sed -e 's/\.//'` [ -d %{_datadir}/vim/vim${VIMVERNEW}/syntax ] && \ cd %{_datadir}/vim/vim${VIMVERNEW}/syntax && \ ln -sf ../../../%{name}/syslog-ng.vim . || : %triggerun -- vim-common VIMVEROLD=`rpm -q --qf='%%{epoch}:%%{version}\n' vim-common | sort | head -n 1 | sed -e 's/[0-9]*://' | sed -e 's/\.[0-9]*$//' | sed -e 's/\.//'` [ $2 = 0 ] && rm -f %{_datadir}/vim/vim${VIMVEROLD}/syntax/syslog-ng.vim || : %triggerpostun -- vim-common VIMVEROLD=`rpm -q --qf='%%{epoch}:%%{version}\n' vim-common | sort | head -n 1 | sed -e 's/[0-9]*://' | sed -e 's/\.[0-9]*$//' | sed -e 's/\.//'` VIMVERNEW=`rpm -q --qf='%%{epoch}:%%{version}\n' vim-common | sort | tail -n 1 | sed -e 's/[0-9]*://' | sed -e 's/\.[0-9]*$//' | sed -e 's/\.//'` if [ $1 = 1 ]; then rm -f %{_datadir}/vim/vim${VIMVEROLD}/syntax/syslog-ng.vim || : [ -d %{_datadir}/vim/vim${VIMVERNEW}/syntax ] && \ cd %{_datadir}/vim/vim${VIMVERNEW}/syntax && \ ln -sf ../../../%{name}/syslog-ng.vim . || : fi %files %defattr(-,root,root,-) %doc AUTHORS COPYING README ChangeLog NEWS %doc doc/security/*.txt %doc contrib/{relogger.pl,syslog2ng,syslog-ng.conf.doc} %dir %{_sysconfdir}/%{name} %dir %{_sysconfdir}/%{name}/patterndb.d %config(noreplace) %{_sysconfdir}/%{name}/%{name}.conf %config(noreplace) %{_sysconfdir}/%{name}/modules.conf %config(noreplace) %{_sysconfdir}/%{name}/scl.conf %config(noreplace) %{_sysconfdir}/sysconfig/%{name} %config(noreplace) %{_sysconfdir}/logrotate.d/syslog %{_sysconfdir}/init.d/%{name} %dir %{_sharedstatedir}/%{name} %{_sbindir}/%{name} %{_sbindir}/syslog-ng-ctl %{_bindir}/loggen %{_bindir}/pdbtool %{_bindir}/update-patterndb %{_libdir}/lib%{name}.so.* %{_libdir}/%{name}/*.so %exclude %{_libdir}/%{name}/libafsql.so %dir %{_datadir}/%{name} %{_datadir}/%{name}/syslog-ng.vim %ghost %{_datadir}/vim/ # scl files %{_datadir}/%{name}/include/ # uhm, some better places for those? %{_datadir}/%{name}/xsd/ %{_mandir}/man1/loggen.1* %{_mandir}/man1/pdbtool.1* %{_mandir}/man1/syslog-ng-ctl.1* %{_mandir}/man5/syslog-ng.conf.5* %{_mandir}/man8/syslog-ng.8* %files libdbi %defattr(-,root,root,-) %{_libdir}/%{name}/libafsql.so %files devel %defattr(-,root,root,-) %{_libdir}/libsyslog-ng.so %{_includedir}/%{name}/ %changelog ------------------------------------------------------------------------------------------------------------------------ ----------/var/tmp/rpm-tmp.izm78V-------------------- #!/bin/sh RPM_SOURCE_DIR="/home/makerpm/rpmbuild/SOURCES" RPM_BUILD_DIR="/home/makerpm/rpmbuild/BUILD" RPM_OPT_FLAGS="-O2 -g" RPM_ARCH="x86_64" RPM_OS="linux" export RPM_SOURCE_DIR RPM_BUILD_DIR RPM_OPT_FLAGS RPM_ARCH RPM_OS RPM_DOC_DIR="/usr/share/doc" export RPM_DOC_DIR RPM_PACKAGE_NAME="syslog-ng" RPM_PACKAGE_VERSION="3.4.1" RPM_PACKAGE_RELEASE="1.el6" export RPM_PACKAGE_NAME RPM_PACKAGE_VERSION RPM_PACKAGE_RELEASE LANG=C export LANG unset CDPATH DISPLAY ||: RPM_BUILD_ROOT="/home/makerpm/rpmbuild/BUILDROOT/syslog-ng-3.4.1-1.el6.x86_64" export RPM_BUILD_ROOT PKG_CONFIG_PATH="/lib64/pkgconfig:/usr/share/pkgconfig" export PKG_CONFIG_PATH set -x umask 022 cd "/home/makerpm/rpmbuild/BUILD" cd 'syslog-ng-3.4.1' LD_LIBRARY_PATH=/home/makerpm/rpmbuild/BUILDROOT/syslog-ng-3.4.1-3.el6.x86_64/lib64:/makerpm/rpmbuild/BUILDROOT/syslog-ng-3.4.1-3.el6.x86_64/lib64/syslog-ng make check exit 0 -------------------------------------------------------------------------------------

2 1

Reaping unused destination files
by Jordi Prats 06 Jun '13

06 Jun '13

Hi, After updating to syslog-ng 3.2, I'm no longer getting any logs. It reports some destinations as unused: May 23 17:09:16 borg syslog-ng[27957]: Reaping unused destination files; template='/var/log/firewalls/$YEAR/$MONTH/$DAY/$HOST.log' May 23 17:09:16 borg syslog-ng[27957]: Reaping unused destination files; template='/var/log/troncal/$YEAR/$MONTH/$DAY/$SOURCEIP.log' May 23 17:09:16 borg syslog-ng[27957]: Reaping unused destination files; template='/var/log/remote/$HOST/$YEAR/$MONTH/$DAY/$PROGRAM/$FACILITY' But I have no clue why, as this configuration was working on another system (I'm pretty sure it was an older version, but I can't tell which exactly) My config is as follows: options { time_reopen (10); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (yes); sync (5000); time_reap(5); log_fifo_size (536870912); stats_freq(60); flush_lines(500); flush_timeout(10000); }; source remote_tcp { tcp(port(514) keep-alive(no)); }; source remote_default { udp(ip(192.168.1.1) port(514)); }; source remote_servers { udp(ip(192.168.1.2) port(514)); }; source firewalls { udp(port(1515)); }; source troncal { udp(ip(192.168.1.3) port(514)); }; destination hosts { file("/var/log/remote/$HOST/$YEAR/$MONTH/$DAY/$PROGRAM/$FACILITY" owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes)); }; destination firewalls { file("/var/log/firewalls/$YEAR/$MONTH/$DAY/$HOST.log" owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes)); }; destination troncal { file("/var/log/troncal/$YEAR/$MONTH/$DAY/$SOURCEIP.log" owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes)); }; log { source(remote_default); source(remote_servers); destination(hosts); }; log { source(firewalls); destination(firewalls); }; log { source(troncal); destination(troncal); }; Anyone can help me out? Thank you! -- Jordi

2 1

syslog-ng 3.4.2 has been released
by devel＠balabit.hu 06 Jun '13

06 Jun '13

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.4.2 SUMMARY : new stable release DATE : Jun 3, 2013 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.4.2) has been released. For latest fixes in the 3.4.x feature branch you are recommended to upgrade to this version. CHANGES: 3.4.2 Mon, 3 Jun 2013 16:00:00 +0100 This is the first bug-fix release for the 3.4.x series, collecting significant fixes over the 3.4.1 release. Upgrading to this release is highly recommended. Bugfixes ======== * loggen will bail out earily if using Unix sockets without a path specified. * Using the same filter for multiple destinations does not trigger a crash anymore. [#219] * An issue with log branch propagation has been fixed, restoring compatibility with the 3.3 branch (this mostly affected flags(final) use cases). [#220] * The systemd unit file now specified Restart=on-failure. [#222] * Support for empty blocks in the configuration were restored, syslog-ng does not crash anymore when encountering one. * An assertion was fixed in patterndb, that prevented one from overriding built-in values, such as HOST. [#225] * The persist state file (syslog-ng.persist) is now marked close-on-exec, so it does not leak through to forked subprocesses. * The system() source now treats /proc/kmsg as optional: if it is not readable to the user running syslog-ng, it will not be included. * Proper escape handling in backtick insertion has been implemented, so string arguments to block parameters are - again - treated correctly. [#235] * A rare race condition in the SQL, MongoDB, SMTP and AMQP destinations have been fixed. * A possible deadlock was fixed within the suppress() handling code. Credits ======= Attila M. Magyar <athos(a)balabit.hu> Balazs Scheidler <bazsi(a)balabit.hu> Evan Rempel <erempel(a)uvic.ca> Gergely Nagy <algernon(a)balabit.hu> Paul Dann <pdgiddie+balabit(a)gmail.com> Peter Czanik <czanik(a)balabit.hu> Tamas Pal <folti(a)balabit.hu> DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

5 6

syslog-ng consumes a lot of cpu
by C. L. Martinez 04 Jun '13

04 Jun '13

HI all, I have installed syslog-ng 3.3.8 under an OpenBSD with a simple configuration: @version: 3.3 options { log_fifo_size(30000); use_dns (no); use_fqdn (no); keep_hostname (yes); }; # Define what files to be monitored source s_network { tcp(port(10514) flags(no-multi-line)); }; # Define filters # Define default destinations destination d_ossec_fifo { pipe("/var/log/syslog.fifo"); }; # # Default log actions # log { source(s_network); log { destination(d_ossec_fifo); flags(flow-control,final); }; }; but it is consuming a lot of CPU: load averages: 1.13, 1.09, 1.08 obsdhost 11:17:49 25 processes: 1 running, 23 idle, 1 on processor CPU states: 0.8% user, 0.0% nice, 3.5% system, 0.0% interrupt, 95.7% idle Memory: Real: 29M/356M act/tot Free: 2655M Cache: 297M Swap: 0K/3067M PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND 30813 root 64 0 1260K 4204K run - 60:49 99.02% syslog-ng Any idea why??

2 3

TCP packet collapse errors
by Xuri Nagarin 04 Jun '13

04 Jun '13

I have a pair of Syslog-NG servers running 3.2.5-3. The hardware specs are - Quad Xeon E5-2680 (32 cores), 32GB RAM, and two 1TB SAS 7200 RPM disks in RAID-1. OS is RHEL6.2 - Kernel 2.6.32-279.5.2. Filesystem is ext3. Global options are set as: options { flush_lines (1000); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (yes); keep_timestamp(yes); dir_group("syslog"); perm(0640); dir_perm(0750); group("syslog"); }; I have already set TCP kernel buffers to 128MB max and set disk scheduler to "deadline". But even under light disk IO load, from ~8-25MB, I see "1320811067 packets collapsed in receive queue due to low socket buffer". I had some other processes on the host writing to disk. Stopping them reduced the packet errors but this number still keeps incrementing. To rule out other issues, I temporarily pointed my disk-based destinations to /dev/null and then packet losses/errors stopped. So either Syslog-NG isn't able to write to disk fast enough or there is an underlying OS/hardware issue. Both hosts have the same issue. Any pointers in troubleshooting it will be appreciated. TIA.

2 8

syslog-ng 3.3.10 has been released
by devel＠balabit.hu 03 Jun '13

03 Jun '13

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.3.10 SUMMARY : new stable release DATE : Jun 3, 2013 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.3.10) has been released. For latest fixes in the 3.3.x feature branch you are recommended to upgrade to this version. CHANGES: 3.3.10 Mon, 3 Jun 2013 16:00:00 +0100 Highlights ========== This release is a bug-fix release, correcting a handful of issues discovered since the previous one. Bugfixes ======== * The persist state file (syslog-ng.persist) is now marked close-on-exec, so it does not leak through to forked subprocesses. * A rare race condition in the SQL and MongoDB destinations have been fixed. Credits ======= syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Attila M. Magyar <athos(a)balabit.hu> Balazs Scheidler <bazsi(a)balabit.hu> Evan Rempel <erempel(a)uvic.ca> Gergely Nagy <algernon(a)balabit.hu> DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

[Bug 236] New: syslog-ng loses part of kernel log and replaces it with \0 characters
by bugzilla＠bugzilla.balabit.com 03 Jun '13

03 Jun '13

https://bugzilla.balabit.com/show_bug.cgi?id=236 Summary: syslog-ng loses part of kernel log and replaces it with \0 characters Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: kroz.nn(a)gmail.com Type of the Report: bug Estimated Hours: 0.0 1. syslog-ng with default configuration inserts characters with code 0 into /var/log/messages (2nd line, beginning of kernel log / dmesg). Comparison with output of dmesg has shown, that no log records lost, just characters inserted. 2. syslog-ng with more complicated configuration inserts two batches of \0 characters (2nd line and 12th line). Second batch of characters replaces 118 lines of kernel log (comparing with log of default configuration and dmesg) OS: Gentoo Linux Attachments: - For each of scenario 1 and scenario 2: configuration file, relevant parts of /var/log/messages - Gentoo compilation environment (output of `emerge --info` ) - Beginning of syslog-ng compilation log (configure stage) Syslog-ng version: syslog-ng 3.4.1 Installer-Version: 3.4.1 Revision: ssh+git://algernon@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.4#master-3.4#dea0b2a11bffe1877ec10d29279fbb0f635d70e7 Compile-Date: Jun 2 2013 12:23:49 Available-Modules: syslogformat,dbparser,afsocket-notls,afuser,afamqp,basicfuncs,cryptofuncs,afsocket-tls,afprog,afsocket,system-source,confgen,affile,csvparser Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 9

problems compiling OSE 3.4.1 on redhat enterprise 6.4
by Russell Fulton 01 Jun '13

01 Jun '13

make[4]: Entering directory `/home/rful011/syslog-ng-3.4.1/modules/afsocket' /bin/sh ../../libtool --tag=CC --mode=compile gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../.. -I../../lib -I../../lib -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/local/include/eventlog -D_BSD_SOURCE -D__BSD_SOURCE -D__FAVOR_BSD -DHAVE_NET_ETHERNET_H -I../../lib/ivykis/src/include -I../../lib/ivykis/src/include -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -g -O2 -Wall -pthread -MT libafsocket_notls_la-afinet-dest.lo -MD -MP -MF .deps/libafsocket_notls_la-afinet-dest.Tpo -c -o libafsocket_notls_la-afinet-dest.lo `test -f 'afinet-dest.c' || echo './'`afinet-dest.c libtool: compile: gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../.. -I../../lib -I../../lib -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/local/include/eventlog -D_BSD_SOURCE -D__BSD_SOURCE -D__FAVOR_BSD -DHAVE_NET_ETHERNET_H -I../../lib/ivykis/src/include -I../../lib/ivykis/src/include -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -g -O2 -Wall -pthread -MT libafsocket_notls_la-afinet-dest.lo -MD -MP -MF .deps/libafsocket_notls_la-afinet-dest.Tpo -c afinet-dest.c -fPIC -DPIC -o .libs/libafsocket_notls_la-afinet-dest.o In file included from /usr/include/net/ethernet.h:27, from /usr/include/libnet.h:98, from afinet-dest.c:43: when I first encountered this error I went and did some googling and followed some vague leads to install a bunch of -devel packages (most of which were already installed ) but it did not make any difference. Googling the message brings up a bunch of hits for both redhat and debian families and a range of packages. There did not seem to be any common cause. Russell

4 7