- syslog-ng - lists.balabit.hu

patterndb and log analysis
by Matt Zagrabelny 26 Jun '13

26 Jun '13

Greetings syslog-ng users, I've never used patterndb, but have been aware of it for a while. I just cloned the git://git.balabit.hu/bazsi/syslog-ng-patterndb.git and it looks like the project has not seen much activity since 2010. Are people still using patterndb? Do the patterns not change much and that is the reason that the git database has not changed much? What are free software enthusiasts using for log analysis? Thanks for the answers! -mz

3 3

problems with syslog-ng host filter
by Orangepeel Beef 26 Jun '13

26 Jun '13

I've got some strange problems with trying to use the syslog-ng host filter. It seems that every logline matches the filter f_comware but none of them match f_netscaler. It makes very little sense to me. Reverse dns is working as the logs that end up in /var/log/remote have the correct hostname being logged from the $HOST template. I have a second issue that SEC does not die when syslog-ng is restarted. (I have used this setup in the past and have had no problems, but i suppose things may have changed in both syslog-ng and sec since the last time) This is driving me crazy.. please help ;) #config @version: 3.1 # # Syslog-ng configuration file, compatible with default Debian syslogd # First, set some global options. options { long_hostnames(on); flush_lines(0); use_dns(yes); use_fqdn(yes); owner("syslog"); group("adm"); perm(0640); dns_cache_size(2000); dns_cache_expire(21600); dir_perm(0755); dir_group("adm"); stats_freq(0); log_fifo_size(200000); create_dirs(yes); bad_hostname("^gconfd$"); chain_hostnames(no); keep_hostname(no); }; source s_remote { udp();tcp(); }; destination d_remote { file("/var/log/remote/$HOST/$YEAR-$MONTH-$DAY.log" template("$R_DATE $HOST $MSG\n") template_escape(no)); }; destination d_netscaler { program("/usr/local/sbin/sec_netscaler" template("$R_DATE $HOST $MSG\n") template_escape(no)); }; destination d_comware { program("/usr/local/sbin/sec_comware" template("$R_DATE $HOST $MSG\n") template_escape(no)); }; filter f_netscaler { host("lb*ae1.mydomain.com"); }; filter f_comware { host("(as|cs|r)*ae1.mydomain.com"); }; log { source(s_remote); destination(d_remote); }; log { source(s_remote); filter(f_netscaler); destination(d_netscaler); }; log { source(s_remote); filter(f_comware); destination(d_comware); }; #debug logs [13:38:54] Filter rule evaluation result; filter_result='match', filter_rule='f_comware' [13:38:56] Incoming log entry; line='<190>Jun 21 20:37:54 2013 R0507S3Z3AE1 %%10MSTP/6/MSTP_FORWARDING(l): -DevIP=x.x.x.x; Instance 0\'s Ten-GigabitEthernet1/0/2 has been set to forwarding state.' [13:38:58] Filter rule evaluation begins; filter_rule='f_netscaler' [13:39:00] Filter node evaluation result; filter_result='not-match' [13:39:02] Filter rule evaluation result; filter_result='not-match', filter_rule='f_netscaler' [13:39:04] Filter rule evaluation begins; filter_rule='f_comware' [13:39:06] Filter node evaluation result; filter_result='match' [13:39:08] Filter rule evaluation result; filter_result='match', filter_rule='f_comware' [13:39:10] Incoming log entry; line='<190>Jun 21 20:37:54 2013 R0507S3Z3AE1 %%10MSTP/6/MSTP_FORWARDING(l): -DevIP=x.x.x.x; Instance 1\'s Ten-GigabitEthernet1/0/2 has been set to forwarding state.' [13:39:12] Filter rule evaluation begins; filter_rule='f_netscaler' [13:39:14] Filter node evaluation result; filter_result='not-match' [13:39:16] Filter rule evaluation result; filter_result='not-match', filter_rule='f_netscaler' [13:39:18] Filter rule evaluation begins; filter_rule='f_comware' [13:39:20] Filter node evaluation result; filter_result='match' [13:39:22] Filter rule evaluation result; filter_result='match', filter_rule='f_comware' [13:39:24] Incoming log entry; line='<134> 06/21/2013:20:37:54 GMT lb1o1ae1 0-PPE-2 : UI CMD_EXECUTED 232044114 0 : User nsroot - Remote_ip x.x.x.x - Command "login nsroot "********"" - Status "Success"' [13:39:26] Initializing destination file writer; template='/var/log/remote/$HOST/$YEAR-$MONTH-$DAY.log', filename='/var/log/remote/lb1o1ae1.mydomain.com/2013-06-21.log' [13:39:28] Filter rule evaluation begins; filter_rule='f_netscaler' [13:39:31] Filter node evaluation result; filter_result='not-match' [13:39:33] Filter rule evaluation result; filter_result='not-match', filter_rule='f_netscaler' [13:39:35] Filter rule evaluation begins; filter_rule='f_comware' [13:39:37] Filter node evaluation result; filter_result='match' [13:39:39] Filter rule evaluation result; filter_result='match', filter_rule='f_comware' [13:39:41] ^CTermination requested via signal, terminating; [13:39:43] syslog-ng shutting down; version='3.1.3' #logs in /var/log/remote/lb2z2ae1.mydomain.com Jun 21 20:23:34 lb2z2ae1.mydomain.com 20:23:34 GMT lb2z2ae1 0-PPE-3 : UI CMD_EXECUTED 28261 0 : User nsroot - Remote_ip x.x.x.x - Command "show service GL-AE1-2AZ1-DB0001_9191" - Status "Success"

2 4

gsoc 2013 - syslog-ng: redis destination
by Tihamér Petrovics 24 Jun '13

24 Jun '13

Hi, I'm Tihamér Petrovics and I will spend this summer in a project of gsoc 2013 to develop redis destination for syslog-ng. If you are curious, you can find my blog here, I will update it approximately weekly: http://tichygsoc.blogspot.com/ All the best, Tihi

1 0

I/O errors
by Russell Fulton 17 Jun '13

17 Jun '13

getting these errors at a rate of several per second but I can't figure out what is causing them: Jun 16 03:45:01 s_self(a)itslogprd01.its.auckland.ac.nz syslog-ng[31159]: I/O error occurred while reading; fd='758', error='Connection reset by peer (104)' Jun 16 03:45:01 s_self(a)itslogprd01.its.auckland.ac.nz syslog-ng[31159]: I/O error occurred while reading; fd='808', error='Connection reset by peer (104)' Jun 16 03:45:01 s_self(a)itslogprd01.its.auckland.ac.nz syslog-ng[31159]: I/O error occurred while reading; fd='906', we have one network destination which is a tcp connection and which appears to be fine. No errors on the other end and stats show now dropped packets. Russell

2 3

Syslog-ng under Windows (Cygwin - copying from another pc)
by Tom Doberstein 17 Jun '13

17 Jun '13

Hello, I am using syslog-ng in cygwin under WinXP. On my developer pc I have no problems, everything works very fine. But when I copy the cygwin directory (from c:/Programs to c:/Programs) on another pc, I got the following errors: ----------------- Error opening plugin module; module='afsocket', error='Exec format error' Error opening plugin module; module='afsocket', error='Exec format error' Error parsing source, source plugin unix-dgram not found in source confgen syste m at line 1, column 1: included from /etc/syslog-ng/syslo g-ng.conf line 10, column 9 unix-dgram ^^^^^^^^^^ syslog-ng documentation: http://www.balabit.com/support/documentation/?product=s yslog-ng mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng ----------------- I already started the syslog-ng-config and the service was installed successfully, but it will not start. cygrunsrv.exe -S syslog-ng will produce this error: "cygrunsrv: Error starting a service: QueryServiceStatus: Win32 error 1062: (...)". Is there now any right problem that the process cannot read the module correctly or something else? What I noticed is, when I copy the cygwin directory to the server and back to the same pc (in the same directory) the error occurs too. I hope that somebody can help me with that problem. Regards, Tom

3 4

AUTO: James B Horwath is out of the office (returning 06/19/2013)
by James B Horwath 17 Jun '13

17 Jun '13

I am out of the office until 06/19/2013. I will be out of the office until 06/19//2013 with limited access to e-mail. If you have an urgent issue please contact Sunil Gupta at (610)-807-6710 or Dylan Yoder at (610)-807-7230. Regards Jim Horwath IT Security Services & Governance, Corporate BTS and IT Security/Risk Tel 610-807-8795 Mobile 610-533-6972 3900 Burgess Place, Bethlehem, Pa. 18017 Jim_Horwath(a)glic.com Note: This is an automated response to your message "syslog-ng Digest, Vol 98, Issue 25" sent on 6/17/2013 6:00:03. This is the only notification you will receive while this person is away. ----------------------------------------- This message, and any attachments to it, may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are notified that any use, dissemination, distribution, copying, or communication of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately by return e-mail and delete the message and any attachments. Thank you.

1 0

[PATCH (3.5) 1/2] tests: Add a testcase for template_escape()
by Gergely Nagy 15 Jun '13

15 Jun '13

In order not to break template_escape() functionality, add a test case for basic functionality testing. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- libtest/template_lib.c | 17 +++++++++++++---- libtest/template_lib.h | 5 +++-- tests/unit/test_template.c | 10 +++++++++- 3 files changed, 25 insertions(+), 7 deletions(-) diff --git a/libtest/template_lib.c b/libtest/template_lib.c index 38dfeff..7992608 100644 --- a/libtest/template_lib.c +++ b/libtest/template_lib.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2012 BalaBit IT Ltd, Budapest, Hungary + * Copyright (c) 2012-2013 BalaBit IT Ltd, Budapest, Hungary * Copyright (c) 2012 Balázs Scheidler * * This library is free software; you can redistribute it and/or @@ -53,6 +53,7 @@ create_sample_message(void) log_msg_set_value(msg, log_msg_get_value_handle("APP.STRIP3"), " value ", -1); log_msg_set_value(msg, log_msg_get_value_handle("APP.STRIP4"), "value", -1); log_msg_set_value(msg, log_msg_get_value_handle("APP.STRIP5"), "", -1); + log_msg_set_value(msg, log_msg_get_value_handle("APP.QVALUE"), "\"value\"", -1); log_msg_set_match(msg, 0, "whole-match", -1); log_msg_set_match(msg, 1, "first-match", -1); log_msg_set_tag_by_name(msg, "alma"); @@ -70,12 +71,13 @@ create_sample_message(void) } LogTemplate * -compile_template(const gchar *template) +compile_template(const gchar *template, gboolean escaping) { LogTemplate *templ = log_template_new(configuration, NULL); gboolean success; GError *error = NULL; + log_template_set_escape(templ, escaping); success = log_template_compile(templ, template, &error); assert_true(success, "template expected to compile cleanly, but it didn't, template=%s, error=%s", template, error ? error->message : "(none)"); g_clear_error(&error); @@ -86,6 +88,13 @@ compile_template(const gchar *template) void assert_template_format(const gchar *template, const gchar *expected) { + assert_template_format_with_escaping(template, FALSE, expected); +} + +void +assert_template_format_with_escaping(const gchar *template, gboolean escaping, + const gchar *expected) +{ LogMessage *msg; LogTemplate *templ; GString *res = g_string_sized_new(128); @@ -93,7 +102,7 @@ assert_template_format(const gchar *template, const gchar *expected) msg = create_sample_message(); - templ = compile_template(template); + templ = compile_template(template, escaping); log_template_format(templ, msg, NULL, LTZ_LOCAL, 999, context_id, res); assert_string(res->str, expected, "template test failed, template=%s", template); log_template_unref(templ); @@ -113,7 +122,7 @@ assert_template_format_with_context(const gchar *template, const gchar *expected msg = create_sample_message(); context[0] = context[1] = msg; - templ = compile_template(template); + templ = compile_template(template, FALSE); log_template_format_with_context(templ, context, 2, NULL, LTZ_LOCAL, 999, context_id, res); assert_string(res->str, expected, "context template test failed, template=%s", template); diff --git a/libtest/template_lib.h b/libtest/template_lib.h index 760858c..f875e4f 100644 --- a/libtest/template_lib.h +++ b/libtest/template_lib.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2012 BalaBit IT Ltd, Budapest, Hungary + * Copyright (c) 2012-2013 BalaBit IT Ltd, Budapest, Hungary * Copyright (c) 2012 Balázs Scheidler * * This library is free software; you can redistribute it and/or @@ -29,11 +29,12 @@ #include "templates.h" void assert_template_format(const gchar *template, const gchar *expected); +void assert_template_format_with_escaping(const gchar *template, gboolean escaping, const gchar *expected); void assert_template_format_with_context(const gchar *template, const gchar *expected); void assert_template_failure(const gchar *template, const gchar *expected_failure); LogMessage *create_sample_message(void); -LogTemplate *compile_template(const gchar *template); +LogTemplate *compile_template(const gchar *template, gboolean escaping); void init_template_tests(void); void deinit_template_tests(void); diff --git a/tests/unit/test_template.c b/tests/unit/test_template.c index e5ff1a0..8546dd4 100644 --- a/tests/unit/test_template.c +++ b/tests/unit/test_template.c @@ -53,7 +53,7 @@ assert_template_format_multi_thread(const gchar *template, const gchar *expected gint i; msg = create_sample_message(); - templ = compile_template(template); + templ = compile_template(template, FALSE); args[0] = msg; args[1] = templ; args[2] = (gpointer) expected; @@ -248,6 +248,13 @@ test_multi_thread(void) assert_template_format_multi_thread("dani $(echo $HOST $DATE $(echo huha)) balint", "dani bzorp Feb 11 10:34:56.000 huha balint"); } +static void +test_escaping(void) +{ + assert_template_format_with_escaping("${APP.QVALUE}", FALSE, "\"value\""); + assert_template_format_with_escaping("${APP.QVALUE}", TRUE, "\\\"value\\\""); +} + int main(int argc G_GNUC_UNUSED, char *argv[] G_GNUC_UNUSED) { @@ -269,6 +276,7 @@ main(int argc G_GNUC_UNUSED, char *argv[] G_GNUC_UNUSED) test_syntax_errors(); test_compat(); test_multi_thread(); + test_escaping(); /* multi-threaded expansion */ -- 1.7.10.4

2 2

Re: [syslog-ng] TCP packet collapse error
by Balazs Scheidler 15 Jun '13

15 Jun '13

Sorry to tune in so late in the xonversation. Without responding to all poonts here are some related facts. 1. Incoming messages in the same connection are processed serially, eg it will not scale to multiple CPUs for the same connection. 2. Output is separate from input and runs in a separate thread. 3. The more the input/output the more syslog-ng scales CPU wise 4. Input performance depends a lot on the number of messages processed in a single iteration. That is affected by log-fetch-limit and window size. Increasing these needs a bump in log-fifo-size too. 5. Output performance depends on flush-lines a lot as syslog-ng batches writes in those chunks. This was added in 3.3. 6. It might make sense to tune VM related parameters in the kernel to start write out earlier, this way smoothing the output side, as that increases latency on the input. Hope this helps Cheers, On Jun 4, 2013 10:47 AM, "Xuri Nagarin" <secsubs(a)gmail.com> wrote: > Yes, multi-threading is enabled. To eliminate disk IO as an issue, I > created a tmpfs filesystem to write logs to. > > I am watching TCP collapsed errors with this simple script in one window: > -----xxxxxxxxx--------- > #!/bin/bash > > while true > do > r1=`/bin/netstat -s | /bin/grep collapsed | /bin/awk '{print $1}'` > sleep 2 > r2=`/bin/netstat -s | /bin/grep collapsed | /bin/awk '{print $1}'` > let loss=$r2-$r1 > echo $loss > done > ------xxxxxxxxxx-------- > > In another window, I watch syslog-ng stats with the script posted earlier > in this thread. > > In a third window, I watch tcp receive queue for only one source that I > enabled - sudo netstat --tcp -p -o -e -e -v | grep "$ip" > > The receive queue builds to hundreds of thousands of packets to millions > depending on the syslog-ng config but rarely drops to zero. As packets in > the receive queue stack up, the collapsed packet count rises from a lower > thousands to tens of thousands. Raising the receive buffers at the OS level > makes the problem worse as does increasing the so_rcvbuf in syslog-ng. In a > fourth window I watch CPU utilization with mpstat. A single core > utilization stays high. > > I tried a vanilla syslog-ng config with no tweaking to varying values of > log_fifo_size, log_fetch_limit, log-iw-size, and so_rcvbuf. Packet collapse > errors stubbornly stay at low thousands to hundreds of thousands. > > Looking at iptraf stats, inbound pps is 15-20k. > > Looking at tcpdump window sizing, the syslog-ng host slows down the sender > by sending a windows size of as small as 1. > > Without looking at the syslog-ng code, it looks like a single source is > getting tied to a single thread/core and if traffic is high from that > source, it ends up saturating the core causing packet loss. > > I will continue to investigate more but need more ideas :) > > > > > On Tue, Jun 4, 2013 at 12:21 AM, Daniel Neubacher < > daniel.neubacher(a)xing.com> wrote: > >> If I remember this correctly you have to adjust the linux networking >> options too. Like here: >> http://wwwx.cs.unc.edu/~sparkst/howto/network_tuning.php**** >> >> Maybe you missed an option.**** >> >> And to activate multithreading you need threaded(yes); But it’s odd that >> the syslog problem is gone now.**** >> >> ** ** >> >> ** ** >> >> ** ** >> >> *Von:* syslog-ng-bounces(a)lists.balabit.hu [mailto: >> syslog-ng-bounces(a)lists.balabit.hu] *Im Auftrag von *Xuri Nagarin >> *Gesendet:* Montag, 3. Juni 2013 23:53 >> >> *An:* Syslog-ng users' and developers' mailing list >> *Betreff:* Re: [syslog-ng] TCP packet collapse errors**** >> >> ** ** >> >> I switched to 3.4.1 (multi-threaded) but still no respite from the TCP >> collapse issue.**** >> >> ** ** >> >> I tried to set so_rcvbuf to '51200000' and got this error:**** >> >> ** ** >> >> "Starting syslog-ng: The kernel refused to set the receive buffer >> (SO_RCVBUF) to the requested size, you probably need to adjust buffer >> related kernel parameters; so_rcvbuf='51200000', so_rcvbuf_set='20971520'" >> **** >> >> ** ** >> >> In terms of lps processed, may be it's just a case of Monday but I am >> seeing numbers as high as 40k consistently now with no drops to zero. Disk >> IO is low but more consistent between 4-8 Mbytes/sec.**** >> >> ** ** >> >> In terms of multi-threading, not sure how well the code does it because I >> am seeing individual cores spike to 100% even when several others are at >> idle.**** >> >> ** ** >> >> ** ** >> >> ** ** >> >> On Fri, May 31, 2013 at 12:41 PM, Xuri Nagarin <secsubs(a)gmail.com> wrote: >> **** >> >> Thanks Daniel. I think I owe you a six-pack of your favourite beer just >> for the script :) **** >> >> ** ** >> >> Very interesting stats. For both hosts, lps ranges from anywhere between >> few hundred to 30K per second but falls to zero for several seconds. All >> the packet loss happens when the lps is peaking. What has me concerned is >> that lps falls to zero for 15-20 seconds at a time. During this period disk >> IO falls to near zero too and packet hemorrhaging pauses. **** >> >> ** ** >> >> All my sources are tcp and there are only four of them but those are >> proprietary products so they may be sending data in batches/bursts. I will >> tweak the tcp source options and see what more I can squeeze out of this >> build of syslog-ng and the disks. But you are right - I do need to upgrade >> to the multi-threaded version asap. At some point, a single core process is >> going to stall on writing to disk and not be able to reap tcp buffers >> during that time.**** >> >> ** ** >> >> I think the disk scheduler probably did not have any impact but one >> system has the disk scheduler tweak and other doesn't but I don't see any >> significant differences with packet loss on either. **** >> >> ** ** >> >> Right now, I think the issue simply is that when a burst of data comes >> in, the single core syslog-ng blocks of IO and cannot switch back to >> reading TCP buffers fast enough to clear up everything in the buffer. High >> network IO, high Disk IO, high lps and packet loss - all match up.**** >> >> ** ** >> >> A larger receive buffer for the tcp source in syslog config will be a >> band-aid, I think until I can build a multi-threaded syslog-ng. Eventually, >> given that these are 7200RPM disks in RAID-1, the disk sub-system will >> start being a bottleneck but by then, hopefully, I will find a host with >> better disk througput :-)**** >> >> ** ** >> >> ** ** >> >> ** ** >> >> ** ** >> >> On Fri, May 31, 2013 at 2:41 AM, Daniel Neubacher < >> daniel.neubacher(a)xing.com> wrote:**** >> >> I've forgot to ask for your syslog source settings. **** >> >> Here is my cfg. Maybe it helps. **** >> >> ** ** >> >> tcp(**** >> >> log_fetch_limit(1000)**** >> >> max-connections(5000)**** >> >> so_rcvbuf(51200000)**** >> >> keep_timestamp(yes)**** >> >> port(514)**** >> >> log-iw-size(500000)**** >> >> );**** >> >> ** ** >> >> I've got it from a great artice >> http://codeascraft.com/2012/08/13/performance-tuning-syslog-ng/**** >> >> ** ** >> >> ** ** >> >> ** ** >> ------------------------------ >> >> *Von:* syslog-ng-bounces(a)lists.balabit.hu [ >> syslog-ng-bounces(a)lists.balabit.hu]" im Auftrag von "Xuri Nagarin [ >> secsubs(a)gmail.com]**** >> >> *Gesendet:* Freitag, 31. Mai 2013 10:12**** >> >> >> *An:* Syslog-ng users' and developers' mailing list**** >> >> *Betreff:* Re: [syslog-ng] TCP packet collapse errors**** >> >> Thanks for the quick response, Daniel.**** >> >> ** ** >> >> I look at statistics for an hour before tweaking flush_lines to zero and >> setting log_fifo_size to 10000. In that period, syslog-ng reported >> processing 7,898,310,589 messages across all destinations and >> dropped 4,200,260. **** >> >> ** ** >> >> After making the change (flush_lines set to 0 and log_fifo_size to >> 10000), I looked at three sets (half hour) of stats (default, every 10 >> minutes). The dropped messages are now zero across all destinations.**** >> >> ** ** >> >> But the collapsed TCP packets count keeps incrementing. I ran 'iostat -xm >> 5' and "watch -d 'netstat -s | grep collpased' " in two windows >> side-by-side. Each time that disk IO spikes up, the TCP collapsed counter >> starts incrementing. Disk IO remains almost zero for about half a minute >> and then spikes up to ~4-25 Mbytes/sec for half a minute.**** >> >> ** ** >> >> Does this mean I need to bump up log_fifo_size even higher? I think >> ideally we want the disk to be consistently written to instead of bursts of >> write activity. Right?**** >> >> ** ** >> >> ** ** >> >> ** ** >> >> ** ** >> >> ** ** >> >> ** ** >> >> ** ** >> >> ** ** >> >> On Thu, May 30, 2013 at 10:56 PM, Daniel Neubacher < >> daniel.neubacher(a)xing.com> wrote:**** >> >> I don't know how much logs you are getting but should tweak >> "log_fifo_size (1000);" to a higher number. Your flush_lines is really high >> too.. I tested around with flush lines but I ended setting it to 0 with 50k >> log per second. And they greatest of all tweaks would be a newer syslog >> version because of the threading.**** >> ------------------------------ >> >> *Von:* syslog-ng-bounces(a)lists.balabit.hu [ >> syslog-ng-bounces(a)lists.balabit.hu]" im Auftrag von "Xuri Nagarin [ >> secsubs(a)gmail.com] >> *Gesendet:* Freitag, 31. Mai 2013 07:46 >> *An:* Syslog-ng users' and developers' mailing list >> *Betreff:* [syslog-ng] TCP packet collapse errors**** >> >> I have a pair of Syslog-NG servers running 3.2.5-3. The hardware specs >> are - Quad Xeon E5-2680 (32 cores), 32GB RAM, and two 1TB SAS 7200 RPM >> disks in RAID-1. **** >> >> **** >> >> OS is RHEL6.2 - Kernel 2.6.32-279.5.2. Filesystem is ext3.**** >> >> ** ** >> >> Global options are set as:**** >> >> options {**** >> >> flush_lines (1000);**** >> >> time_reopen (10);**** >> >> log_fifo_size (1000);**** >> >> long_hostnames (off);**** >> >> use_dns (no);**** >> >> use_fqdn (no);**** >> >> create_dirs (yes);**** >> >> keep_hostname (yes);**** >> >> keep_timestamp(yes);**** >> >> dir_group("syslog");**** >> >> perm(0640);**** >> >> dir_perm(0750);**** >> >> group("syslog");**** >> >> };**** >> >> ** ** >> >> I have already set TCP kernel buffers to 128MB max and set disk scheduler >> to "deadline".**** >> >> ** ** >> >> But even under light disk IO load, from ~8-25MB, I see "1320811067 >> packets collapsed in receive queue due to low socket buffer". I had some >> other processes on the host writing to disk. Stopping them reduced the >> packet errors but this number still keeps incrementing.**** >> >> ** ** >> >> To rule out other issues, I temporarily pointed my disk-based >> destinations to /dev/null and then packet losses/errors stopped. So either >> Syslog-NG isn't able to write to disk fast enough or there is an underlying >> OS/hardware issue.**** >> >> ** ** >> >> Both hosts have the same issue. Any pointers in troubleshooting it will >> be appreciated.**** >> >> ** ** >> >> TIA.**** >> >> ** ** >> >> ** ** >> >> >> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> **** >> >> ** ** >> >> >> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> **** >> >> ** ** >> >> ** ** >> >> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> >> > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > >

1 0

Google Summer of Code 2013 - syslog-ng: MySQL destination driver
by Gyula Petrovics 15 Jun '13

15 Jun '13

Hi everyone, I am Gyula Petrovics, one of the students who got the opportunity to help in the development of the syslog-ng. My goal is to redesign the current libdbi based solution to store logs in MySQL database. I made a blog here <http://petrovicsgyula.blogspot.com>, please visit this blog if you are interested in the current progress and the details of the project. Thanks for reading my e-mail. Best regards, Gyula

4 6

[libsyslog_ng_la-afinter.lo] Error 1, syslog-ng-3.4.1 on x86_64
by Flur Blubr 15 Jun '13

15 Jun '13

Dear all, looking for help, for 4 days trying to compile syslog-ng-3.4.1 on some kind of custom RED HAT derivate and still failing. RPMs and YUM are forbidden and system has libraries spread around all directories / using exports export EVTLOG_CFLAGS, export EVTLOG_LIBS, export GLIB_CFLAGS, export GLIB_LIBS to overwrite PKG CONFIG . uname -a Linux 2.6.32-279.1.1.el6.x86_64 #1 SMP Wed Jun 20 11:41:22 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.3 (Santiago) Configuration: ./configure --prefix=/my-prefix/syslog-ng --enable-sql --enable-ipv6 --enable-dynamic-linking --build=x86_64-pc-linux-gnu --host=i686-pc-linux-gnu --enable-shared --disable-static CFLAGS=-O2 Getting this: syslog-ng Open Source Edition 3.4.1 configured Compiler options: compiler : gcc -std=gnu99 compiler options : -O2 -Wall -pthread -I/my-prefix/sources/glib-2.15.4/ -I/my-prefix/eventlog/include/eventlog/ -I$(top_srcdir)/lib/ivykis/src/include -I$(top_builddir)/lib/ivykis/src/include -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 linker flags : prefix : /my-prefix/syslog-ng linking mode : dynamic embedded crypto : no __thread keyword : yes Submodules: ivykis : internal libmongo-client : internal librabbitmq : internal Features: Debug symbols : no GCC profiling : no Memtrace : no IPV6 support : yes spoof-source support : no tcp-wrapper support : no Linux capability support : yes PCRE support : yes Env wrapper support : no systemd support : no (unit dir: none) Modules: Module search path : /my-prefix/syslog-ng/lib/syslog-ng Sun STREAMS support (module): no SSL support (module) : yes SQL support (module) : yes PACCT module (EXPERIMENTAL) : no MongoDB destination (module): yes JSON support (module) : no SMTP support (module) : no AMQP destination (module) : yes GEOIP support (module) : no But make is failing no matter what I do: afinter.c: At top level: afinter.c:385: error: expected ')' before 'mark_freq' afinter.c: In function 'afinter_message_posted': afinter.c:404: error: 'internal_msg_lock' undeclared (first use in this function) afinter.c:405: error: 'internal_msg_queue' undeclared (first use in this function) afinter.c:407: warning: implicit declaration of function 'g_queue_new' afinter.c:409: warning: implicit declaration of function 'g_queue_push_tail' afinter.c:411: error: 'AFInterSource' has no member named 'post' afinter.c: At top level: afinter.c:416: error: expected ')' before 'hook_type' afinter.c: In function 'afinter_global_init': afinter.c:424: warning: implicit declaration of function 'register_application_hook' afinter.c:424: error: 'afinter_register_posted_hook' undeclared (first use in this function) make[4]: *** [libsyslog_ng_la-afinter.lo] Error 1 make[4]: Leaving directory `/my-prefix/syslog-ng-3.4.1/lib' make[3]: *** [all-recursive] Error 1 make[3]: Leaving directory `/my-prefix/syslog-ng-3.4.1/lib' make[2]: *** [all] Error 2 make[2]: Leaving directory `/my-prefix/syslog-ng-3.4.1/lib' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/my-prefix/syslog-ng-3.4.1' make: *** [all] Error 2 Can somebody advice? Thx for your time. Regards Michal

4 9