November 2013 - syslog-ng - lists.balabit.hu

[Bug 253] New: Segfault when reloading configuration
by bugzilla＠bugzilla.balabit.com 21 Nov '13

21 Nov '13

https://bugzilla.balabit.com/show_bug.cgi?id=253 Summary: Segfault when reloading configuration Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: gonzalo.paniagua+slng1(a)acquia.com Type of the Report: bug Estimated Hours: 0.0 syslog-ng version: 3.4.3 built from sources syslog-ng -V: syslog-ng 3.4.3 Installer-Version: 3.4.3 Revision: ssh+git://algernon@git.balabit/var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.4#no_branch#64d670f3cbfb90769f3c7f0fdd9c70bb9136ec5b Compile-Date: Sep 11 2013 05:33:29 Available-Modules: confgen,afsocket-tls,csvparser,affile,system-source,syslogformat,afuser,dbparser,afsocket-notls,basicfuncs,cryptofuncs,afsocket,afprog Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: off Enable-Pcre: on On rare occasions (I can't reproduce it reliably) syslog-ng segfaults when reloading (syslog-ng-ctl reload). I have a core dump file that gave me this stack trace: #0 0x00007fbd30102131 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fbd30101d76 in strdup () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007fbd2f868b19 in evt_tag_str (tag=0x7fbd309a42f0 "encoding", value=0xffffffffffffffff <Address 0xffffffffffffffff out of bounds>) at evttags.c:76 #3 0x00007fbd3097918c in log_proto_server_options_validate (options=<optimized out>) at logproto-server.c:138 #4 0x00007fbd3097d191 in log_proto_server_validate_options (self=<optimized out>) at logproto-server.h:82 #5 log_reader_init (s=0x14a6a30) at logreader.c:638 #6 0x00007fbd2e9d5b07 in log_pipe_init (cfg=0x0, s=0x14a6a30) at ../../lib/logpipe.h:253 #7 log_pipe_init (cfg=0x0, s=0x14a6a30) at afsocket-source.c:763 #8 afsocket_sc_init (s=0x14a68f0) at afsocket-source.c:149 #9 0x00007fbd2e9d67fd in log_pipe_init (cfg=0x0, s=0x14a68f0) at ../../lib/logpipe.h:253 #10 log_pipe_init (cfg=0x0, s=0x14a68f0) at afsocket-source.c:763 #11 afsocket_sd_init (s=0x1489630) at afsocket-source.c:548 #12 0x00007fbd2e9d9719 in afunix_sd_init (s=<optimized out>) at afunix-source.c:194 #13 0x00007fbd3096d27a in log_pipe_init (cfg=<optimized out>, s=0x1489630) at logpipe.h:253 #14 cfg_tree_start (self=0x1c15c48) at cfg-tree.c:1064 #15 0x00007fbd309685de in cfg_init (cfg=0x1c15ae0) at cfg.c:220 #16 0x00007fbd30984faf in main_loop_reload_config_apply () at mainloop.c:501 #17 0x00007fbd3096d699 in control_connection_reload (self=0x1d09e90, command=<optimized out>) at control.c:150 #18 0x00007fbd3096dae8 in control_connection_io_input (s=0x1d09e90) at control.c:242 #19 control_connection_io_input (s=0x1d09e90) at control.c:172 #20 0x00007fbd309a106c in iv_fd_poll_and_run (st=0x1482580, to=<optimized out>) at iv_fd.c:163 #21 0x00007fbd309a1aac in iv_main () at iv_main_posix.c:117 #22 0x00007fbd309859fb in main_loop_run () at mainloop.c:736 #23 0x000000000040142b in main (argc=1, argv=0x7fffd8212a58) at main.c:267 When at frame #5, I got this: (gdb) print *self->proto->options $19 = {destroy = 0, initialized = -1, encoding = 0xffffffffffffffff <Address 0xffffffffffffffff out of bounds>, convert = 0xffffffffffffffff, max_msg_size = -1, max_buffer_size = 0, init_buffer_size = 0} I could not find anywhere in the code where initialized is set to -1, so I assume this is some kind of memory corruption. Let me know if you need anything else from me. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

2 16

[Bug 226] New: 3.3.8: iv_tls_user_register: called after iv_init
by bugzilla＠bugzilla.balabit.com 21 Nov '13

21 Nov '13

https://bugzilla.balabit.com/show_bug.cgi?id=226 Summary: 3.3.8: iv_tls_user_register: called after iv_init Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: arekm(a)maven.pl Type of the Report: bug Estimated Hours: 0.0 When building with libivykis library (internal or external) starting syslog-ng 3.3.8 ends up with: # /sbin/syslog-ng -f /etc/syslog-ng/syslog-ng.conf iv_tls_user_register: called after iv_init System is Linux 3.7.10, glibc 2.17. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 10

redis destination bug
by Otto Berger 21 Nov '13

21 Nov '13

Hi, found a small bug in the redis destination module. I actually dont know where to file a report for it, maybe someone here knows... In modules/redis/redis-parser.c the host and port assignments are missing: { "host", KW_HOST }, { "port", KW_PORT }, without these it is not possible to connect to redis servers other than the default (localhost). The redis destination works very well btw (tested at a quite large deployment). cheers Otto -- www.bergerdata.de

2 2

Re: [syslog-ng] [review request] 3.6queue/f/stats-reset
by Jim Hendrick 20 Nov '13

20 Nov '13

If desired, could a "zero-counters" be added to syslog-ng-ctl ? Not critical but maybe useful for troubleshooting. Jim Sent from my Verizon Wireless 4G LTE Smartphone -------- Original message -------- From: Gergely Nagy <algernon(a)balabit.hu> Date: 11/20/2013 8:47 AM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Cc: syslog-ng-dev-l(a)balabit.hu Subject: Re: [syslog-ng] [review request] 3.6queue/f/stats-reset Balazs Scheidler <bazsi77(a)gmail.com> writes: > On Mon, 2013-11-18 at 18:22 +0100, Gergely Nagy wrote: >> Evan Rempel <erempel(a)uvic.ca> writes: >> >> > Perhaps the same logic can be used to identify those counters that have expired meaning that the stats-lifetime() has been exceeded,but >> > instead of dropping the stat counter at that time, the counter is just flagged for dropping. >> > >> > Then it becomes the job of the statistics logging, based on the stats-freq() value, to log AND drop those counters that >> > have been flagged as having exceeded the stats-lifetime(). >> > >> > This way a counter is never dropped until it has exceeded the >> > stats-lifetime() AND has been logged. >> >> That makes a lot of sense, but syslog-ng-ctl stats should not count (or >> perhaps only with a flag), precisely to avoid the scenario you outlined >> too. >> > > This really makes sense, however my issue is that stats-freq() is often > set to 0 which disables the Log statistics message currently. Relying > the cleanup mechanism on that would disable cleanup altogether in a lot > of setups. > > hmm... maybe we should use stats-freq() if that's nonzero and > stats-lifetime() when it is zero. This way the cleanup mechanism would > kick in, without the risk of losing messages. > > syslog-ng-ctl stats wouldn't prune counters. > > This way, stats-freq() would be the one which actually clears counters, > we get rid of an additional iteration over the set of counters. > > This paves the way for an improved log statistics message that could be > sent into graphite/logstash/whatever graphing application via standard > syslog-ng destination drivers, and all this without the risk of losing > counters. > > What do you think? Sounds very good to me. This is a big step towards what I need, and is simple enough to understand too. -- |8] ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

2 1

Re: [syslog-ng] More on simple filtering problem
by Jim Hendrick 15 Nov '13

15 Nov '13

I can't comment specifically but check for whitespace and map the non space against the various syslog RFCs. Also using tcpdump to inspect the inbound and outbound traffic may help. Be very detailed. I have seen a single space or lack thereof make all the difference. Jim Sent from my Verizon Wireless 4G LTE Smartphone -------- Original message -------- From: Paul Hutton <paul_hutton(a)bigpond.com> Date: 11/15/2013 1:38 AM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] More on simple filtering problem Hi all, I pinged this list some time ago, confessing that I couldn't get simple filtering to work. I still haven't had a satisfactory result to the problem that I have been experiencing, so can I rephrase the situation? Does anyone have syslog-ng configured to read source syslog traffic generated by Cisco switches and firewalls, and to then forward them through to Ciscoworks LMS and successfully displayed on the LMS Admin console? I believe the problem is that syslog-ng's handling of the message changes its contents sufficiently for it not to be recognised as displayable on the LMS console. We do see message traffic arriving at the LMS which gets logged to a file, but it doesn't appear on the GUI console. I have tried the source options no-parse and store-legacy-msghdr with no good effect. Thanks for any responses Paul

1 0

More on simple filtering problem
by Paul Hutton 15 Nov '13

15 Nov '13

Hi all, I pinged this list some time ago, confessing that I couldn't get simple filtering to work. I still haven't had a satisfactory result to the problem that I have been experiencing, so can I rephrase the situation? Does anyone have syslog-ng configured to read source syslog traffic generated by Cisco switches and firewalls, and to then forward them through to Ciscoworks LMS and successfully displayed on the LMS Admin console? I believe the problem is that syslog-ng's handling of the message changes its contents sufficiently for it not to be recognised as displayable on the LMS console. We do see message traffic arriving at the LMS which gets logged to a file, but it doesn't appear on the GUI console. I have tried the source options no-parse and store-legacy-msghdr with no good effect. Thanks for any responses Paul

1 0

insider 2013-11: syslog-ng 3.5 released; Logging to Hadoop; EoL;
by Peter Czanik 14 Nov '13

14 Nov '13

Dear syslog-ng users, This is the 29th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. FEATURED NEWS syslog-ng 3.5 is released ------------------------- Beginning last week syslog-ng 3.5 was finally declared stable, so 3.5.1 was released. Source is available from http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… if you want to compile syslog-ng yourself. Packages and ports for different Linux distributions and FreeBSD are already available from our 3^rd party page: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… New features of 3.5 were detailed here many times, so here are just the headlines: * Redis and Stomp destinations * multiline messages * type hinting For details, check the ChangeLog at http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.5.1/… or algernon's blog at http://asylum.madhouse-project.org/blog/2013/11/06/syslog-ng-3.5.1/ Logging to Hadoop ----------------- Big data is gaining a momentum also in the logging world, so we made some experiments how syslog-ng can log to Hadoop. Even without a dedicated Hadoop destination driver one can already send logs to Hadoop and analyze them using Hadoop tools. For details, check https://tiborbenke.blogs.balabit.com/2013/11/the-syslog-ng-in-the-hadoop-er… syslog-ng EoL policy -------------------- The syslog-ng End of Life policy was updated recently, based on the experiences of syslog-ng stable maintainer, Gergely Nagy. Practically there are four different branches of syslog-ng maintained at any time, representing different level of maturity and support. More details are available at http://asylum.madhouse-project.org/blog/2013/10/22/syslog-ng-eol-dates/ And following the URLs below you can read about how this affects different Linux distributions and FreeBSD: Debian / Ubuntu: https://lists.balabit.hu/pipermail/syslog-ng/2013-November/020884.html openSUSE / SLES: https://lists.balabit.hu/pipermail/syslog-ng/2013- <https://lists.balabit.hu/pipermail/syslog-ng/2013-November/020880.html>November/020880.html <https://lists.balabit.hu/pipermail/syslog-ng/2013-November/020880.html> FreeBSD ports: https://lists.balabit.hu/pipermail/syslog-ng/2013-November/020878.html SHORT NEWS: * Testing syslog-ng PE was never so easy: syslog-ng PE Quick Start Guide: http://czanik.blogs.balabit.com/2013/10/syslog-ng-pe-quick-start-guide/ * syslog-ng git reorganized: https://lists.balabit.hu/pipermail/syslog-ng/2013-November/020875.html * Logging to Loggly using syslog-ng: https://czanik.blogs.balabit.com/2013/11/loggly-and-syslog-ng/ NEW RELEASES * syslog-ng 3.5.1: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-November/000174.… * syslog-ng 3.4.5: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-November/000173.… Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

[Bug 224] New: Segmentation fault in 3.3.7 and 3.4.1
by bugzilla＠bugzilla.balabit.com 13 Nov '13

13 Nov '13

https://bugzilla.balabit.com/show_bug.cgi?id=224 Summary: Segmentation fault in 3.3.7 and 3.4.1 Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 Core dump back trace is #0 0x00007f4020000528 in ?? () #1 0x00007f402a10b1a5 in log_msg_refcache_stop () at logmsg.c:1495 #2 0x00007f402a11760c in log_writer_flush (self=0xa0d460, flush_mode=LW_FLUSH_NORMAL) at logwriter.c:1043 #3 0x00007f402a1177ed in log_writer_work_perform (s=0xa0d460) at logwriter.c:129 #4 0x00007f402a117edb in main_loop_io_worker_job_start (self=0xa0d650) at mainloop.c:371 #5 0x00007f402a1341ca in iv_work_thread_do_work (_thr=0xabdd20) at iv_work.c:118 #6 0x00007f402a13352a in iv_run_tasks (st=0x7f400c40a130) at iv_task.c:48 #7 0x00007f402a13574c in iv_main () at iv_main_posix.c:106 #8 0x00007f402a133fe1 in iv_work_thread (_thr=0xabdd20) at iv_work.c:200 #9 0x00007f402a1361b8 in iv_thread_handler (_thr=0xabddf0) at iv_thread_posix.c:142 #10 0x00007f4028d82851 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f4028ad011d in clone () from /lib64/libc.so.6 some times #2 0x00007f402a11760c in log_writer_flush (self=0xa0d460, flush_mode=LW_FLUSH_NORMAL) at logwriter.c:1043 is #2 0x00007f402a11760c in log_writer_flush (self=0x9b3970, flush_mode=LW_FLUSH_BUFFER) at logwriter.c:1043 of course when this happens all buffers are lost. So much for reliable log tansport :-( -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 8

[Bug 244] New: Pattern beginning with a parser never matches
by bugzilla＠bugzilla.balabit.com 13 Nov '13

13 Nov '13

https://bugzilla.balabit.com/show_bug.cgi?id=244 Summary: Pattern beginning with a parser never matches Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.balabit(a)faxm0dem.org Type of the Report: bug Estimated Hours: 0.0 It would seem the patterns beginning with parsers never match. Example: Consider the following MESSAGE: -- Puppet::Type::Package::ProviderUp2date: file /usr/sbin/up2date-nox does not exist -- The following pattern matches OK: -- <pattern>P@ESTRING:puppet_module: @file @ESTRING:puppet_file: @does not exist</pattern> -- The following pattern does not match: -- <pattern>@ESTRING:puppet_module::: @file @ESTRING:puppet_file: @does not exist</pattern> -- -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

Re: [syslog-ng] syslog-ng Message Deliver Acknowledgment and Actio
by Balazs Scheidler 08 Nov '13

08 Nov '13

On Nov 7, 2013 4:09 PM, "Tamayo, Andres" <Andres.Tamayo(a)viasat.com> wrote: > > Thank you for your feedback. > > > > We are trying to design a log system that protects against log loss, so if any of the failure scenarios that you mentioned occurred, we need to make sure we do not lose any logs. That is the reason why we want to use an audit log file stored in non-volatile memory to store logs that have not been sent to the server yet. However, we are having trouble defining how to know when to delete logs from the file when they are delivered to the server. > > > > I have read a little about RLTP and my understanding is that this protocol is only available for syslog-ng PE edition. We are using syslog-ng OSE because we need to make some minor modifications to the source code. If you could elaborate about the changes, the pe team might be able to work on that to integrate to the pe edition. No promises though, but it'd be useful to know a little more about them. For that reason, we cannot use the RLTP protocol. > > > > Let me ask a few specific questions: > > > > 1) Is there any way to ask the syslog-ng client daemon to send information back to the application that sends messages to it when a message has been successfully delivered to the syslog-ng server daemon (message deliver acknowledgment)? The most reliable would probably be to hand the audit records to syslog-ng as a file, and let syslog-ng track where it's at. syslog-ng keeps counters how many messages are processed by a source, once enough messages are received the files can be removed. The current file position in the current ose release gets updated as soon as the record is queued in the destination's memory based queue. This solution would lose messages when syslogng crashes lacking a disk based queue. Even we had that, it'd have to be resistant to crashes. I'm afraid only the PE edition would solve the problem completely. I am not here to propagate that, but your requirements are way over traditional syslog. The PE team did a lot of work to improve reliability in various failure scenarios effectively converting syslog-ng to a reliable queueing implementation. Bits and pieces are coming to the ose, but it's complex and tricky code, that takes time for me (and the ose team) to take and integrate, and some pieces are proprietary. > > > > 2) I have been looking for a C/C++ API to generate IETF syslog-ng messages but I cannot not find it. I believe that the #include <syslog.h> header is specific for BSD syslog messages. If there is one, where can I find it? I don't know any such implementation for c++, log4j has one. But its really not difficult to write, format the message, and post it using a Unix domain socket, syslog-ng supports that on unix-dgram, so its a single send operation in your c++ app once the message is formatted. > > > > 3) I did a little reading on ampq and zeromp, do you know if I can save messages to non-volatile memory while they are waiting in the queue? I need to make sure that messages are not lost if the system is turned-off or rebooted. These do have disk buffers, I'm not sure what happens if the client crashes. > > > > Thank you for all your help. > > Hope this helps at least somewhat. > > > > From: syslog-ng-bounces(a)lists.balabit.hu [mailto: syslog-ng-bounces(a)lists.balabit.hu] On Behalf Of Balazs Scheidler > Sent: Thursday, November 07, 2013 2:55 AM > To: Syslog-ng users' and developers' mailing list > Subject: Re: [syslog-ng] syslog-ng Message Deliver Acknowledgment and Action > > > > It depends what failure modes you want to protect against. > > There can be > * network connection breaks > * Syslog-ng daemon crashes (client, server) > * audit daemon crashes > > Syslog-ng has most of the infrastructure to carry out (flow control), but some pieces are missing. The PE team did work in this area, they created RLTP for app. Level network acknowledgements that would protect against network and syslogng crashes. Even in that case though the ack information can only be propagated back to your audit app by speaking RLTP. > > Maybe you would need a queueing stack like amqp or 0mq, those tend to provide explicit acks, and there are brokerless solutions as well. > > On Nov 6, 2013 6:15 PM, "Tamayo, Andres" <Andres.Tamayo(a)viasat.com> wrote: > > Hello, > > > > I am new at using syslog-ng and I was hoping someone in the developers mailing list would be able to help me. > > > > I am trying to use a syslog-ng client daemon and a syslog-ng server daemon to implement an audit message system but I cannot find information that would help me configure the daemons for my particular scenario. > > > > Here is my scenario: > > > > I have a separate audit daemon that generates log messages that are written to an audit log file in syslog format. I need the syslog-ng client to read the logs in the audit log file and send them to the syslog-ng server. When the server has received the messages, I need some acknowledgment from the syslog-ng client, so my other audit daemon can remove the submitted log messages from the audit log file (preventing it from reaching maximum capacity). > > > > My scenario does not have to be setup exactly this way if there are better ways to achieve the same result. Basically, I need to keep all logs that have not yet being sent to the syslog-ng server in an audit file. When the messages are delivered to the syslog-ng server, I need to delete them from the audit file. > > > > Can someone tell me if I can achieve this results using syslog-ng daemons? Is there a better way to implement my scenario. > > > > Thank you for your help in advance. > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > >

2 4