- syslog-ng - lists.balabit.hu

syslog-ng 3.6 rebase
by Balazs Scheidler 25 Nov '13

25 Nov '13

Hi, After integrating everything I could, I would like to rebase the current master to merge fixups into the base patches. This is the last such rebase in 3.6. You can see the rebased series in the "next" branch, I intend to reset the master branch to that history tomorrow evening. the HEAD of both branches are identical, only the history differs. In the meantime, you can submit patches either the current "master" or "next" branches, I'll integrate stuff to "next" myself. Thanks. -- Bazsi

1 1

[Bug 253] New: Segfault when reloading configuration
by bugzilla＠bugzilla.balabit.com 21 Nov '13

21 Nov '13

https://bugzilla.balabit.com/show_bug.cgi?id=253 Summary: Segfault when reloading configuration Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: gonzalo.paniagua+slng1(a)acquia.com Type of the Report: bug Estimated Hours: 0.0 syslog-ng version: 3.4.3 built from sources syslog-ng -V: syslog-ng 3.4.3 Installer-Version: 3.4.3 Revision: ssh+git://algernon@git.balabit/var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.4#no_branch#64d670f3cbfb90769f3c7f0fdd9c70bb9136ec5b Compile-Date: Sep 11 2013 05:33:29 Available-Modules: confgen,afsocket-tls,csvparser,affile,system-source,syslogformat,afuser,dbparser,afsocket-notls,basicfuncs,cryptofuncs,afsocket,afprog Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: off Enable-Pcre: on On rare occasions (I can't reproduce it reliably) syslog-ng segfaults when reloading (syslog-ng-ctl reload). I have a core dump file that gave me this stack trace: #0 0x00007fbd30102131 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fbd30101d76 in strdup () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007fbd2f868b19 in evt_tag_str (tag=0x7fbd309a42f0 "encoding", value=0xffffffffffffffff <Address 0xffffffffffffffff out of bounds>) at evttags.c:76 #3 0x00007fbd3097918c in log_proto_server_options_validate (options=<optimized out>) at logproto-server.c:138 #4 0x00007fbd3097d191 in log_proto_server_validate_options (self=<optimized out>) at logproto-server.h:82 #5 log_reader_init (s=0x14a6a30) at logreader.c:638 #6 0x00007fbd2e9d5b07 in log_pipe_init (cfg=0x0, s=0x14a6a30) at ../../lib/logpipe.h:253 #7 log_pipe_init (cfg=0x0, s=0x14a6a30) at afsocket-source.c:763 #8 afsocket_sc_init (s=0x14a68f0) at afsocket-source.c:149 #9 0x00007fbd2e9d67fd in log_pipe_init (cfg=0x0, s=0x14a68f0) at ../../lib/logpipe.h:253 #10 log_pipe_init (cfg=0x0, s=0x14a68f0) at afsocket-source.c:763 #11 afsocket_sd_init (s=0x1489630) at afsocket-source.c:548 #12 0x00007fbd2e9d9719 in afunix_sd_init (s=<optimized out>) at afunix-source.c:194 #13 0x00007fbd3096d27a in log_pipe_init (cfg=<optimized out>, s=0x1489630) at logpipe.h:253 #14 cfg_tree_start (self=0x1c15c48) at cfg-tree.c:1064 #15 0x00007fbd309685de in cfg_init (cfg=0x1c15ae0) at cfg.c:220 #16 0x00007fbd30984faf in main_loop_reload_config_apply () at mainloop.c:501 #17 0x00007fbd3096d699 in control_connection_reload (self=0x1d09e90, command=<optimized out>) at control.c:150 #18 0x00007fbd3096dae8 in control_connection_io_input (s=0x1d09e90) at control.c:242 #19 control_connection_io_input (s=0x1d09e90) at control.c:172 #20 0x00007fbd309a106c in iv_fd_poll_and_run (st=0x1482580, to=<optimized out>) at iv_fd.c:163 #21 0x00007fbd309a1aac in iv_main () at iv_main_posix.c:117 #22 0x00007fbd309859fb in main_loop_run () at mainloop.c:736 #23 0x000000000040142b in main (argc=1, argv=0x7fffd8212a58) at main.c:267 When at frame #5, I got this: (gdb) print *self->proto->options $19 = {destroy = 0, initialized = -1, encoding = 0xffffffffffffffff <Address 0xffffffffffffffff out of bounds>, convert = 0xffffffffffffffff, max_msg_size = -1, max_buffer_size = 0, init_buffer_size = 0} I could not find anywhere in the code where initialized is set to -1, so I assume this is some kind of memory corruption. Let me know if you need anything else from me. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

2 16

[Bug 226] New: 3.3.8: iv_tls_user_register: called after iv_init
by bugzilla＠bugzilla.balabit.com 21 Nov '13

21 Nov '13

https://bugzilla.balabit.com/show_bug.cgi?id=226 Summary: 3.3.8: iv_tls_user_register: called after iv_init Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: arekm(a)maven.pl Type of the Report: bug Estimated Hours: 0.0 When building with libivykis library (internal or external) starting syslog-ng 3.3.8 ends up with: # /sbin/syslog-ng -f /etc/syslog-ng/syslog-ng.conf iv_tls_user_register: called after iv_init System is Linux 3.7.10, glibc 2.17. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 10

redis destination bug
by Otto Berger 21 Nov '13

21 Nov '13

Hi, found a small bug in the redis destination module. I actually dont know where to file a report for it, maybe someone here knows... In modules/redis/redis-parser.c the host and port assignments are missing: { "host", KW_HOST }, { "port", KW_PORT }, without these it is not possible to connect to redis servers other than the default (localhost). The redis destination works very well btw (tested at a quite large deployment). cheers Otto -- www.bergerdata.de

2 2

Re: [syslog-ng] [review request] 3.6queue/f/stats-reset
by Jim Hendrick 20 Nov '13

20 Nov '13

If desired, could a "zero-counters" be added to syslog-ng-ctl ? Not critical but maybe useful for troubleshooting. Jim Sent from my Verizon Wireless 4G LTE Smartphone -------- Original message -------- From: Gergely Nagy <algernon(a)balabit.hu> Date: 11/20/2013 8:47 AM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Cc: syslog-ng-dev-l(a)balabit.hu Subject: Re: [syslog-ng] [review request] 3.6queue/f/stats-reset Balazs Scheidler <bazsi77(a)gmail.com> writes: > On Mon, 2013-11-18 at 18:22 +0100, Gergely Nagy wrote: >> Evan Rempel <erempel(a)uvic.ca> writes: >> >> > Perhaps the same logic can be used to identify those counters that have expired meaning that the stats-lifetime() has been exceeded,but >> > instead of dropping the stat counter at that time, the counter is just flagged for dropping. >> > >> > Then it becomes the job of the statistics logging, based on the stats-freq() value, to log AND drop those counters that >> > have been flagged as having exceeded the stats-lifetime(). >> > >> > This way a counter is never dropped until it has exceeded the >> > stats-lifetime() AND has been logged. >> >> That makes a lot of sense, but syslog-ng-ctl stats should not count (or >> perhaps only with a flag), precisely to avoid the scenario you outlined >> too. >> > > This really makes sense, however my issue is that stats-freq() is often > set to 0 which disables the Log statistics message currently. Relying > the cleanup mechanism on that would disable cleanup altogether in a lot > of setups. > > hmm... maybe we should use stats-freq() if that's nonzero and > stats-lifetime() when it is zero. This way the cleanup mechanism would > kick in, without the risk of losing messages. > > syslog-ng-ctl stats wouldn't prune counters. > > This way, stats-freq() would be the one which actually clears counters, > we get rid of an additional iteration over the set of counters. > > This paves the way for an improved log statistics message that could be > sent into graphite/logstash/whatever graphing application via standard > syslog-ng destination drivers, and all this without the risk of losing > counters. > > What do you think? Sounds very good to me. This is a big step towards what I need, and is simple enough to understand too. -- |8] ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

2 1

Re: [syslog-ng] More on simple filtering problem
by Jim Hendrick 15 Nov '13

15 Nov '13

I can't comment specifically but check for whitespace and map the non space against the various syslog RFCs. Also using tcpdump to inspect the inbound and outbound traffic may help. Be very detailed. I have seen a single space or lack thereof make all the difference. Jim Sent from my Verizon Wireless 4G LTE Smartphone -------- Original message -------- From: Paul Hutton <paul_hutton(a)bigpond.com> Date: 11/15/2013 1:38 AM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] More on simple filtering problem Hi all, I pinged this list some time ago, confessing that I couldn't get simple filtering to work. I still haven't had a satisfactory result to the problem that I have been experiencing, so can I rephrase the situation? Does anyone have syslog-ng configured to read source syslog traffic generated by Cisco switches and firewalls, and to then forward them through to Ciscoworks LMS and successfully displayed on the LMS Admin console? I believe the problem is that syslog-ng's handling of the message changes its contents sufficiently for it not to be recognised as displayable on the LMS console. We do see message traffic arriving at the LMS which gets logged to a file, but it doesn't appear on the GUI console. I have tried the source options no-parse and store-legacy-msghdr with no good effect. Thanks for any responses Paul

1 0

More on simple filtering problem
by Paul Hutton 15 Nov '13

15 Nov '13

Hi all, I pinged this list some time ago, confessing that I couldn't get simple filtering to work. I still haven't had a satisfactory result to the problem that I have been experiencing, so can I rephrase the situation? Does anyone have syslog-ng configured to read source syslog traffic generated by Cisco switches and firewalls, and to then forward them through to Ciscoworks LMS and successfully displayed on the LMS Admin console? I believe the problem is that syslog-ng's handling of the message changes its contents sufficiently for it not to be recognised as displayable on the LMS console. We do see message traffic arriving at the LMS which gets logged to a file, but it doesn't appear on the GUI console. I have tried the source options no-parse and store-legacy-msghdr with no good effect. Thanks for any responses Paul

1 0

insider 2013-11: syslog-ng 3.5 released; Logging to Hadoop; EoL;
by Peter Czanik 14 Nov '13

14 Nov '13

Dear syslog-ng users, This is the 29th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. FEATURED NEWS syslog-ng 3.5 is released ------------------------- Beginning last week syslog-ng 3.5 was finally declared stable, so 3.5.1 was released. Source is available from http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… if you want to compile syslog-ng yourself. Packages and ports for different Linux distributions and FreeBSD are already available from our 3^rd party page: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… New features of 3.5 were detailed here many times, so here are just the headlines: * Redis and Stomp destinations * multiline messages * type hinting For details, check the ChangeLog at http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.5.1/… or algernon's blog at http://asylum.madhouse-project.org/blog/2013/11/06/syslog-ng-3.5.1/ Logging to Hadoop ----------------- Big data is gaining a momentum also in the logging world, so we made some experiments how syslog-ng can log to Hadoop. Even without a dedicated Hadoop destination driver one can already send logs to Hadoop and analyze them using Hadoop tools. For details, check https://tiborbenke.blogs.balabit.com/2013/11/the-syslog-ng-in-the-hadoop-er… syslog-ng EoL policy -------------------- The syslog-ng End of Life policy was updated recently, based on the experiences of syslog-ng stable maintainer, Gergely Nagy. Practically there are four different branches of syslog-ng maintained at any time, representing different level of maturity and support. More details are available at http://asylum.madhouse-project.org/blog/2013/10/22/syslog-ng-eol-dates/ And following the URLs below you can read about how this affects different Linux distributions and FreeBSD: Debian / Ubuntu: https://lists.balabit.hu/pipermail/syslog-ng/2013-November/020884.html openSUSE / SLES: https://lists.balabit.hu/pipermail/syslog-ng/2013- <https://lists.balabit.hu/pipermail/syslog-ng/2013-November/020880.html>November/020880.html <https://lists.balabit.hu/pipermail/syslog-ng/2013-November/020880.html> FreeBSD ports: https://lists.balabit.hu/pipermail/syslog-ng/2013-November/020878.html SHORT NEWS: * Testing syslog-ng PE was never so easy: syslog-ng PE Quick Start Guide: http://czanik.blogs.balabit.com/2013/10/syslog-ng-pe-quick-start-guide/ * syslog-ng git reorganized: https://lists.balabit.hu/pipermail/syslog-ng/2013-November/020875.html * Logging to Loggly using syslog-ng: https://czanik.blogs.balabit.com/2013/11/loggly-and-syslog-ng/ NEW RELEASES * syslog-ng 3.5.1: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-November/000174.… * syslog-ng 3.4.5: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-November/000173.… Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

[Bug 224] New: Segmentation fault in 3.3.7 and 3.4.1
by bugzilla＠bugzilla.balabit.com 13 Nov '13

13 Nov '13

https://bugzilla.balabit.com/show_bug.cgi?id=224 Summary: Segmentation fault in 3.3.7 and 3.4.1 Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 Core dump back trace is #0 0x00007f4020000528 in ?? () #1 0x00007f402a10b1a5 in log_msg_refcache_stop () at logmsg.c:1495 #2 0x00007f402a11760c in log_writer_flush (self=0xa0d460, flush_mode=LW_FLUSH_NORMAL) at logwriter.c:1043 #3 0x00007f402a1177ed in log_writer_work_perform (s=0xa0d460) at logwriter.c:129 #4 0x00007f402a117edb in main_loop_io_worker_job_start (self=0xa0d650) at mainloop.c:371 #5 0x00007f402a1341ca in iv_work_thread_do_work (_thr=0xabdd20) at iv_work.c:118 #6 0x00007f402a13352a in iv_run_tasks (st=0x7f400c40a130) at iv_task.c:48 #7 0x00007f402a13574c in iv_main () at iv_main_posix.c:106 #8 0x00007f402a133fe1 in iv_work_thread (_thr=0xabdd20) at iv_work.c:200 #9 0x00007f402a1361b8 in iv_thread_handler (_thr=0xabddf0) at iv_thread_posix.c:142 #10 0x00007f4028d82851 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f4028ad011d in clone () from /lib64/libc.so.6 some times #2 0x00007f402a11760c in log_writer_flush (self=0xa0d460, flush_mode=LW_FLUSH_NORMAL) at logwriter.c:1043 is #2 0x00007f402a11760c in log_writer_flush (self=0x9b3970, flush_mode=LW_FLUSH_BUFFER) at logwriter.c:1043 of course when this happens all buffers are lost. So much for reliable log tansport :-( -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 8

[Bug 244] New: Pattern beginning with a parser never matches
by bugzilla＠bugzilla.balabit.com 13 Nov '13

13 Nov '13

https://bugzilla.balabit.com/show_bug.cgi?id=244 Summary: Pattern beginning with a parser never matches Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.balabit(a)faxm0dem.org Type of the Report: bug Estimated Hours: 0.0 It would seem the patterns beginning with parsers never match. Example: Consider the following MESSAGE: -- Puppet::Type::Package::ProviderUp2date: file /usr/sbin/up2date-nox does not exist -- The following pattern matches OK: -- <pattern>P@ESTRING:puppet_module: @file @ESTRING:puppet_file: @does not exist</pattern> -- The following pattern does not match: -- <pattern>@ESTRING:puppet_module::: @file @ESTRING:puppet_file: @does not exist</pattern> -- -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

Re: [syslog-ng] syslog-ng Message Deliver Acknowledgment and Actio
by Balazs Scheidler 08 Nov '13

08 Nov '13

On Nov 7, 2013 4:09 PM, "Tamayo, Andres" <Andres.Tamayo(a)viasat.com> wrote: > > Thank you for your feedback. > > > > We are trying to design a log system that protects against log loss, so if any of the failure scenarios that you mentioned occurred, we need to make sure we do not lose any logs. That is the reason why we want to use an audit log file stored in non-volatile memory to store logs that have not been sent to the server yet. However, we are having trouble defining how to know when to delete logs from the file when they are delivered to the server. > > > > I have read a little about RLTP and my understanding is that this protocol is only available for syslog-ng PE edition. We are using syslog-ng OSE because we need to make some minor modifications to the source code. If you could elaborate about the changes, the pe team might be able to work on that to integrate to the pe edition. No promises though, but it'd be useful to know a little more about them. For that reason, we cannot use the RLTP protocol. > > > > Let me ask a few specific questions: > > > > 1) Is there any way to ask the syslog-ng client daemon to send information back to the application that sends messages to it when a message has been successfully delivered to the syslog-ng server daemon (message deliver acknowledgment)? The most reliable would probably be to hand the audit records to syslog-ng as a file, and let syslog-ng track where it's at. syslog-ng keeps counters how many messages are processed by a source, once enough messages are received the files can be removed. The current file position in the current ose release gets updated as soon as the record is queued in the destination's memory based queue. This solution would lose messages when syslogng crashes lacking a disk based queue. Even we had that, it'd have to be resistant to crashes. I'm afraid only the PE edition would solve the problem completely. I am not here to propagate that, but your requirements are way over traditional syslog. The PE team did a lot of work to improve reliability in various failure scenarios effectively converting syslog-ng to a reliable queueing implementation. Bits and pieces are coming to the ose, but it's complex and tricky code, that takes time for me (and the ose team) to take and integrate, and some pieces are proprietary. > > > > 2) I have been looking for a C/C++ API to generate IETF syslog-ng messages but I cannot not find it. I believe that the #include <syslog.h> header is specific for BSD syslog messages. If there is one, where can I find it? I don't know any such implementation for c++, log4j has one. But its really not difficult to write, format the message, and post it using a Unix domain socket, syslog-ng supports that on unix-dgram, so its a single send operation in your c++ app once the message is formatted. > > > > 3) I did a little reading on ampq and zeromp, do you know if I can save messages to non-volatile memory while they are waiting in the queue? I need to make sure that messages are not lost if the system is turned-off or rebooted. These do have disk buffers, I'm not sure what happens if the client crashes. > > > > Thank you for all your help. > > Hope this helps at least somewhat. > > > > From: syslog-ng-bounces(a)lists.balabit.hu [mailto: syslog-ng-bounces(a)lists.balabit.hu] On Behalf Of Balazs Scheidler > Sent: Thursday, November 07, 2013 2:55 AM > To: Syslog-ng users' and developers' mailing list > Subject: Re: [syslog-ng] syslog-ng Message Deliver Acknowledgment and Action > > > > It depends what failure modes you want to protect against. > > There can be > * network connection breaks > * Syslog-ng daemon crashes (client, server) > * audit daemon crashes > > Syslog-ng has most of the infrastructure to carry out (flow control), but some pieces are missing. The PE team did work in this area, they created RLTP for app. Level network acknowledgements that would protect against network and syslogng crashes. Even in that case though the ack information can only be propagated back to your audit app by speaking RLTP. > > Maybe you would need a queueing stack like amqp or 0mq, those tend to provide explicit acks, and there are brokerless solutions as well. > > On Nov 6, 2013 6:15 PM, "Tamayo, Andres" <Andres.Tamayo(a)viasat.com> wrote: > > Hello, > > > > I am new at using syslog-ng and I was hoping someone in the developers mailing list would be able to help me. > > > > I am trying to use a syslog-ng client daemon and a syslog-ng server daemon to implement an audit message system but I cannot find information that would help me configure the daemons for my particular scenario. > > > > Here is my scenario: > > > > I have a separate audit daemon that generates log messages that are written to an audit log file in syslog format. I need the syslog-ng client to read the logs in the audit log file and send them to the syslog-ng server. When the server has received the messages, I need some acknowledgment from the syslog-ng client, so my other audit daemon can remove the submitted log messages from the audit log file (preventing it from reaching maximum capacity). > > > > My scenario does not have to be setup exactly this way if there are better ways to achieve the same result. Basically, I need to keep all logs that have not yet being sent to the syslog-ng server in an audit file. When the messages are delivered to the syslog-ng server, I need to delete them from the audit file. > > > > Can someone tell me if I can achieve this results using syslog-ng daemons? Is there a better way to implement my scenario. > > > > Thank you for your help in advance. > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > >

2 4

syslog-ng 3.4.5 has been released
by devel＠balabit.hu 08 Nov '13

08 Nov '13

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.4.5 SUMMARY : new stable release DATE : Nov 4, 2013 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.4.5) has been released. For latest fixes in the 3.4.x feature branch you are recommended to upgrade to this version. CHANGES: 3.4.5 Mon, 4 Nov 2013 08:54:50 +0100 This is the fourth bug-fix release for the 3.4.x series, with important fixes over the 3.4.4 release. Upgrading to this release is highly recommended. Bugfixes ======== * A race condition in log message reference counting code that sometimes led to crashes was fixed. [#255] * An assertion that happened in certain situations when a subst() rewrite rule was used was fixed. [#256] * A use-after-free error that sometimes happened after a reload, and caused memory corruption was also fixed. [#253] * patterndb was corrected not to create a new instance on reload: this way, the old one is not leaked, and db-parser() does not forget the correlation state, nor its idea of current time on reload. [#257] * The syslog-ng.spec file does not try to install the long-removed ChangeLog file anymore. * A few harmless, but noisy compiler warnings have been eliminated. Features ======== * The JSON parser now understands NULL-typed keys, and ignores them. Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Balazs Scheidler <bazsi(a)balabit.hu> Evan Rempel <erempel(a)uvic.ca> Gergely Nagy <algernon(a)balabit.hu> Gonzalo Paniagua <gonzalo.paniagua+slng1(a)acquia.com> Jose Pedro Oliveira <jpo(a)di.uminho.pt> Michal Privoznik <miso.privoznik(a)gmail.com> Nicolas Szalay <nico(a)rottenbytes.info> DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

2 1

syslog-ng Message Deliver Acknowledgment and Action
by Tamayo, Andres 07 Nov '13

07 Nov '13

Hello, I am new at using syslog-ng and I was hoping someone in the developers mailing list would be able to help me. I am trying to use a syslog-ng client daemon and a syslog-ng server daemon to implement an audit message system but I cannot find information that would help me configure the daemons for my particular scenario. Here is my scenario: I have a separate audit daemon that generates log messages that are written to an audit log file in syslog format. I need the syslog-ng client to read the logs in the audit log file and send them to the syslog-ng server. When the server has received the messages, I need some acknowledgment from the syslog-ng client, so my other audit daemon can remove the submitted log messages from the audit log file (preventing it from reaching maximum capacity). My scenario does not have to be setup exactly this way if there are better ways to achieve the same result. Basically, I need to keep all logs that have not yet being sent to the syslog-ng server in an audit file. When the messages are delivered to the syslog-ng server, I need to delete them from the audit file. Can someone tell me if I can achieve this results using syslog-ng daemons? Is there a better way to implement my scenario. Thank you for your help in advance.

2 2

Debian & Ubuntu packages for syslog-ng
by Gergely Nagy 06 Nov '13

06 Nov '13

As a reminder, and following Peter's FreeBSD & SuSE package news, for our Debian or Ubuntu users, I'm hosting syslog-ng 3.3, 3.4 and 3.5 packages for a wide range of distributions (see below), available at: http://asylum.madhouse-project.org/projects/debian/ syslog-ng 3.3 packages are available for Ubuntu Lucid, Precise, Raring, and Saucy (for both i386 and amd64 architectures) along with Debian Squeeze, Wheezy, Jessie and unstable (i386, amd64 and powerpc architectures). The syslog-ng 3.4 and 3.5 packages are also available for Ubuntu Trusty. The syslog-ng 3.3 package will be maintained for all of these platforms for as long as the 3.3 branch is maintained (and Ubuntu Trusty will be added to the list if and when 3.3.12 gets released). Same goes for 3.4 and 3.5: they'll be maintained until their respective end of life (see my earlier post about that[1]). I will keep adding new Ubuntu and Debian releases as they come out, and remove them shortly after they reach their respective end of lives. [1]: http://asylum.madhouse-project.org/blog/2013/10/22/syslog-ng-eol-dates/ The easiest way to use the repository is to use the interface on the website to select the distribution and components, then copy & paste to /etc/apt/sources.list. But the line to copy follows this pattern: deb http://packages.madhouse-project.org/debian wheezy syslog-ng-3.4 Where the end of the URL is either "debian" or "ubuntu", the release is your release (either the codename, such as "precise" or "wheezy", or version number like "12.04" or "7.0"), and the last part, the component is the syslog-ng branch you want. There is also a syslog-ng and syslog-ng-devel component, which will always follow the currently recommended branches (3.4 and 3.5 at the time of this writing), so if you want to update from one stable version to the other without having to modify your sources.list, I recommend using one of these aliases. A quick and easy way to enable the repository is as follows (as root): # curl https://packages.madhouse-project.org/debian/archive-key.txt | apt-key add - # echo "deb http://packages.madhouse-project.org/$(lsb_release -is | tr '[:upper:]' '[:lower:]') " \ "$(lsb_release -cs | tr '[:upper:]' '[:lower:]') " \ "syslog-ng" \ > /etc/apt/sources.list.d/syslog-ng.list -- |8]

1 0

git repository reorganization
by Balazs Scheidler 06 Nov '13

06 Nov '13

Hi, Until now, we have had separate git repositories for syslog-ng on github, one for each major version. This seemed logical at the time, since we do split maintenance, Algernon is doing the maintenance for older trees, whereas I maintain the latest version. Also, this is how I used to do it on my internal trees. However, this complicates things for people who want to follow syslog-ng development, whenever we create a new major release, they have to follow a different repository. Difficult to track forks, see all releases in one place, etc. Since git is perfectly capable of storing multiple major releases in the same git repository, I think it would be the best to let the multiple repositories idea go and stick to a single repository model in the future. To model this idea, I have created this repository on github, and pushed 3.3, 3.4 and 3.5 HEADs here: https://github.com/balabit/syslog-ng Branch names would be as follows: - master: the latest HEAD where development is happening on the latest release - 3.3/master, 3.4/master, 3.5/master: the HEAD for maintained old major releases - topic branches destined for a specific major release would be named 3.X/<branch>, topic branches for the latest version would omit the version number prefix. - tags indicate releases, they are named after the respective release for example: v3.4.5 I hope this makes things simpler for everyone in the long term. If anyone has feedback, I'd love to hear them now, before announcing this to the general public and update references everywhere. Thanks in advance, -- Bazsi

3 3

syslog-ng packages for openSUSE/SLES
by Peter Czanik 06 Nov '13

06 Nov '13

Hello, I just updated the syslog-ng packages in my openSUSE Build Service home project. As not all dependencies are part of SLES or openSUSE, it is recommended to add repositories by "zypper ar http://to/repo reponame" and install from there as my repositories carry all necessary dependencies as well. https://build.opensuse.org/project/show/home:czanik:syslog-ng33 has syslog-ng 3.3.11 and will be removed as soon as openSUSE 12.2 is EoL (early next year) https://build.opensuse.org/project/show/home:czanik:syslog-ng34 has syslog-ng 3.4.5 and will stay as long as openSUSE 12.3 is supported (Q3 next year) This version will be published as an update for both openSUSE 12.3 and 13.1 in the coming weeks (as it's not a security update, review is slower by the maintenance team) https://build.opensuse.org/project/show/home:czanik:syslog-ng35 has syslog-ng 3.5.1. This version adds redis support. Due to an error in the release process, manual pages are missing, but documentation is available at the usual location: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.5-guid… As these are not official Balabit or SuSE packages, please contact me, if you run into any troubles! Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

[Bug 259] New: udp destination causes seg fault when BUILD_WITH_SSL is not defined
by bugzilla＠bugzilla.balabit.com 06 Nov '13

06 Nov '13

https://bugzilla.balabit.com/show_bug.cgi?id=259 Summary: udp destination causes seg fault when BUILD_WITH_SSL is not defined Product: syslog-ng Version: 3.5.x Platform: Other OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: htsai(a)skytap.com Type of the Report: bug Estimated Hours: 0.0 We ran into this bug when compiling 3.5.0rc1 on top of customized OpenWrt. BUILD_WITH_SSL is not defined on our platform. When a udp destination is defined, the start of syslog-ng failed with seg fault. After debugging, we found the bug is that, in afinet_dd_new_instance (afinet-dest.c:509), AFSocketDestDriver->construct_transport is initialized with afinet_dd_construct_transport, which is defined as afsocket_dd_construct_transport (line 158). When afsocket_dd_construct_transport is called to construct the transport (afsocket-dest.c:166), it calls AFSocketDestDriver->construct_transport (afsocket-dest.h:66), which is again afsocket_dd_construct_transport. The two functions call each other endlessly until the stack is exhausted, hence the seg fault. The bug still exists in 3.5.1 and may potentially affect other destinations using afinet. We didn't check other syslog-ng versions. The bug can be fixed by the following patch. Please review and apply the patch. --- a/modules/afsocket/afinet-dest.c +++ b/modules/afsocket/afinet-dest.c @@ -155,7 +155,7 @@ afinet_dd_construct_transport(AFSocketDe #else -#define afinet_dd_construct_transport afsocket_dd_construct_transport +#define afinet_dd_construct_transport afsocket_dd_construct_transport_method #endif -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

syslog-ng in FreeBSD ports
by Peter Czanik 06 Nov '13

06 Nov '13

Hello, Thanks to Cy of FreeBSD project, syslog-ng in the FreeBSD ports got updated in the past few days. There are multiple syslog-ng ports, here is a list, which version they carry now: sysutils/syslog-ng-devel: this is where all the fun happens, the latest and greatest syslog-ng, right now 3.5.1. It has redis support added, but no manual pages (missing from the release tgz). sysutils/syslog-ng: the "main" syslog-ng port has now 3.4.5 and considered to be the best balance between new features and maturity sysutils/syslog-ng33: the 3.3 line is still available, currently at 3.3.11 for those willing to stay with 3.3 sysutils/syslog-ng2: has the long time EoL 2.0.10, which I plan to remove as soon as all major Linux distributions stopped using it (when migrating to FreeBSD, it's easier to use the same version first and update later) Most likely around the time syslog-ng 3.6 beta is released, the current syslog-ng port will be copied to syslog-ng34, syslog-ng-devel to syslog-ng and syslog-ng-devel will be updated to 3.6. FYI: checking version is easy using FreshPorts: http://www.freshports.org/search.php?query=syslog-ng Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

[Bug 257] New: memory leak on reload
by bugzilla＠bugzilla.balabit.com 04 Nov '13

04 Nov '13

https://bugzilla.balabit.com/show_bug.cgi?id=257 Summary: memory leak on reload Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 We have a complex environment that growns it's memory footprint on every reload. It is completely consistent as long as a reload is not done. A simple configuration does not have this problem. We have: - multiple patterndb uses (4 different databases) - multiple program destinations with different templates - multiple pipe destinations - multiple file destinations Every reload the memory difference is different amount and the difference does not always increase or decrease. For me it is about 13MB every reload. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 17

syslog-ng 3.5.1 has been released
by devel＠balabit.hu 04 Nov '13

04 Nov '13

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.5.1 SUMMARY : new stable release DATE : Nov 4, 2013 ------------------------------------------------------------------------------ DESCRIPTION: The first stable version of syslog-ng Open Source Edition (3.5.1) has been released. For the latest features in syslog-ng, you are encouraged to upgrade to this version. CHANGES: 3.5.1 Mon, 4 Nov 2013 14:49:00 +0100 This is the first stable release in the 3.5 series, adding a number of features compared to 3.4, a result of about eight months of development. This release includes all the fixes of the recent 3.4.5 release, and a host of new features. Bugfixes since 3.5.0rc1 ======================= * A race condition in log message reference counting code that sometimes led to crashes was fixed. [#255] * A use-after-free error that sometimes happened after a reload, and caused memory corruption was also fixed. [#253] * patterndb was corrected not to create a new instance on reload: this way, the old one is not leaked, and db-parser() does not forget the correlation state, nor its idea of current time on reload. [#257] * The syslog-ng.spec file does not try to install the long-removed ChangeLog file anymore. Incompatible changes ==================== * Template escaping was changed in an incompatible way: previously, both the lexer and the template compiler used the '\' character for escaping, which was confusing. The template compiler uses '$$' to escape the '$' char, and '@@' to escape '@'. If a non-numeric value follows '@', a warning will be printed. * The replace() transformation function of value-pairs() was renamed to replace-prefix() to better reflect what it actually does. The old name remains as an obsolete alias, however. * The username() and password() options of the MongoDB destination were removed, they never did anything before, either. Major features since 3.4 ======================== Multi-line support ------------------ A major feature in the 3.5 release is the inclusion of support for multi-line messages, a feature that has been available in syslog-ng PE for a good while. With this, both indented and regexp-based multi-line support becomes available. For further information and examples, see the following blog post: http://asylum.madhouse-project.org/blog/2013/09/05/what-is-cooking-in-syslo… STOMP destination ----------------- Joining the message-queue club, this new destination makes syslog-ng able to send events to any STOMP-capable message queuing server, such as Apache ActiveMQ. For further information about the destination, see this commit message: https://github.com/balabit/syslog-ng-3.5/commit/82d19a4d4ad3f91e8214006f6ea… Redis destination ----------------- Developed during the Google Summer of Code 2013 program, this destination driver makes it possible to easily send commands to a Redis server. For further information and examples, please see the following blog post: http://tichygsoc.blogspot.hu/2013/09/the-road-so-far.html Template type hinting --------------------- While syslog-ng supported sending events to various datastores and queues for a while now (SQL first, MongoDB, JSON, and AMQP later), even if those supported other types of data, syslog-ng only ever sent strings. With template type hinting, it became possible to tell syslog-ng what type a certain template should be, so that the drivers can use that information in whatever way they see fit. This is currently implemented for the MongoDB driver and the $(format-json) template function only. For more information about type hinting and for examples, see the following post: http://asylum.madhouse-project.org/blog/2013/09/05/what-is-cooking-in-syslo… Template options honored everywhere ----------------------------------- Until this release, there were situations where template options were ignored, such as filter expressions that use the comparsion operators, regexp based substitutions, incoming templates for parsers, the new value rewrite rules, SMTP values, some of the new drivers (stomp, amqp), and patterndb context-id. These all honor the global template options now, and per-driver options such as frac-digits and local-time-zone are available for drivers which did not have them before, like MongoDB. Support for unit suffixes in the configuration ---------------------------------------------- You no longer need to remember how many zeros to put after a big number in the syslog-ng configuration file, you can use unit suffixes, such as: log-fifo-size(2GiB) See the following post for more details: http://asylum.madhouse-project.org/blog/2013/09/05/what-is-cooking-in-syslo… The Incubator project --------------------- Alongside the 3.5.1 release, we are pleased to announce the existence of the syslog-ng incubator project, which hosts additional modules and tools not merged into syslog-ng proper. These serve both as examples and as a staging area, but also makes it easier to try out new modules without patching or upgrading your syslog-ng version. The project's homepage is: https://github.com/balabit/syslog-ng-incubator Other features ============== * in-list() filter: this new filter function allows one to easily implement file-based white- and blacklists with a simple syntax: filter f_white { in-list("/path/to/file", value("HOST")); }; * A set of new string-related template functions are available: $(uppercase STRINGS...), $(lowercase STRINGS...) and $(replace-delimiter DELIMITERS NEW-DELIMITER TEXT). The first two do exactly what their names suggest, while the last one replaces all occurrences of any DELIMITERS within TEXT with the NEW-DELIMITER. * There is also a new $(env VARIABLE...) template function, which looks up the given variables in the environment. This is similar to using backticks in the configuration file, however the name of the environment variable with $(env) may contain macros too. * Support for Linux 3.5+'s /dev/kmsg was added, and the system() source will automatically detect whether to use it over /proc/kmsg. * For every correlated message in patterndb, the ${.classifier.context_id} property is automatically set to the context-id attribute of the matching rule. * The build system was completely redone, it is much faster, more reliable, and less verbose by default now. * Several systemd-related enhancements were made, including support for notification-based startup. This also means that when systemd support is compiled in, libsystemd-daemon becomes a new dependency. Known Bugs ========== * The afstreams module is broken, does not compile, and does not work. This will be corrected in a later maintenance release. Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Alexandre Biancalana <biancalana(a)gmail.com> Andras Tim <tia(a)balabit.hu> Anton Koldaev <koldaevav(a)gmail.com> Attila M. Magyar <athos(a)balabit.hu> Attila Nagy <bra(a)fsn.hu> Attila Szalay <sasa(a)balabit.hu> Balazs Scheidler <bazsi(a)balabit.hu> Balint Kovacs <blint(a)balabit.hu> Chris Johnson <chris.johnson3(a)hp.com> Cy Schubert <Cy.Schubert(a)komquats.com> Evan Rempel <erempel(a)uvic.ca> Fabien Wernli <cpan(a)faxm0dem.org> Gergely Nagy <algernon(a)balabit.hu> Gonzalo Paniagua <gonzalo.paniagua+slng1(a)acquia.com> Jose Pedro Oliveira <jpo(a)di.uminho.pt> Laszlo Budai <lbudai(a)balabit.hu> Lucas McLane <lucas(a)clicksecurity.com> Marc Falzon <marc.falzon(a)cloudwatt.com> Martin <bmartin(a)lavabit.com> Michal Privoznik <miso.privoznik(a)gmail.com> Michael Sterrett <mr_bones_(a)gentoo.org> Nicolas Szalay <nico(a)rottenbytes.info> Oscar Muñoz Paul Dann <pdgiddie+balabit(a)gmail.com> Peter Czanik <czanik(a)balabit.hu> Peter Gyongyosi <gyp(a)balabit.hu> Robert Fekete <frobert(a)balabit.hu> Ryan Frederick Sergey Shuman Tamas Pal <folti(a)balabit.hu> Tibor Benke <btibi(a)balabit.hu> Tihamer Petrovics <tihameri(a)gmail.com> Valentijn Sessink <valentijn(a)sessink.nl> Viktor Juhasz <jviktor(a)balabit.hu> Viktor Tusa <tusa(a)balabit.hu> Vincent Brillault <spam(a)lerya.net> DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

[Bug 255] New: segmentation faults often
by bugzilla＠bugzilla.balabit.com 02 Nov '13

02 Nov '13

https://bugzilla.balabit.com/show_bug.cgi?id=255 Summary: segmentation faults often Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: blocker Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: bug Estimated Hours: 0.0 In my setup, which is big, using syslog-ng 3.4.4 I get segmentation faults 2-3 times per minute. 3.4.1 did this once every couple of weeks, and 3.4.2 was worse, but 3.4.4 is completely unusable for me. What would you like from me to aid tracking this down? -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

3 15

patterndb + format_json 100% CPU
by Alexandre Biancalana 27 Oct '13

27 Oct '13

Hi list, Running syslog-ng with self written patterndb and format_json output the process is hitting 100% of CPU looks like it's freezed (infinite loop ?) and nothing is written to the output. Follow the link to syslog-ng.conf, patterndb file and example logfile: syslog-ng.conf - http://pastebin.com/QD6huzA3 httpd.xml (patterndb) - http://pastebin.com/kF9zJHYH all.log - http://pastebin.com/syusGF2H Was possible to reproduce that with 3.4.4 and 3.5.0beta3. Regards, Alexandre

3 10

Question about postfix, multiline and tls
by Clayton Dukes 26 Oct '13

26 Oct '13

Hi Folks, I'm trying to get syslog-ng to send postfix events to a server as a single event, instead of multiple events for the same log message. I've posted the question on stack exchange and thought I was see if anyone here may know the answer: http://stackoverflow.com/questions/19609142/syslog-ng-support-for-postfix-m… ______________________________________________________________ Clayton Dukes ______________________________________________________________

1 0

3.5rc1 packages for openSUSE / SLES
by Peter Czanik 24 Oct '13

24 Oct '13

Hello, If you want to test 3.5rc1 on openSUSE / SLES but afraid to compile it yourself, my unofficial packages are available at https://build.opensuse.org/package/show/home:czanik:syslog-ng35/syslog-ng It received some testing on openSUSE 12.3, but available for a lot other platforms. Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

syslog-ng 3.5.0rc1 has been released
by devel＠balabit.hu 24 Oct '13

24 Oct '13

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.5.0rc1 SUMMARY : new release candidate release DATE : Oct 24, 2013 ------------------------------------------------------------------------------ DESCRIPTION: A new release candidate of syslog-ng Open Source Edition (3.5.0rc1) has been released. For latest fixes in the 3.5.x feature branch you are recommended to upgrade to this version. CHANGES: 3.5.0rc1 Thu, 24 Oct 2013 10:48:51 +0200 This is the first Release Candidate of the syslog-ng 3.5 development branch, with stability and reliability improvements. Being a release candidate, this release should be very close to production ready. Bugfixes ======== * The program() source and destination now properly terminates child processes on reload and shutdown. * The AMQP, SMTP, MongoDB, Redis and STOMP destinations now work correctly across reloads. * A rare race condition in the log message reference cache was corrected, fixing a crash. * The JSON parser now understands null-typed values, and ignores the whole key silently when it encounters one. * Using the subst() rewrite rule when the same source is present in multiple logpaths does not crash anymore. Miscellaneous ============= * The autogen.sh script is included in the distributed tarball. Known Bugs ========== * The afstreams module is broken, does not compile, and does not work. This will be corrected before the final 3.5.1 release. Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Balazs Scheidler <bazsi(a)balabit.hu> Evan Rempel <erempel(a)uvic.ca> Gergely Nagy <algernon(a)balabit.hu> Jose Pedro Oliveira <jpo(a)di.uminho.pt> Michal Privoznik <miso.privoznik(a)gmail.com> Nicolas Szalay <nico(a)rottenbytes.info> Ryan Frederick DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0