- syslog-ng - lists.balabit.hu

More on simple filtering problem
by Paul Hutton 15 Nov '13

15 Nov '13

Hi all, I pinged this list some time ago, confessing that I couldn't get simple filtering to work. I still haven't had a satisfactory result to the problem that I have been experiencing, so can I rephrase the situation? Does anyone have syslog-ng configured to read source syslog traffic generated by Cisco switches and firewalls, and to then forward them through to Ciscoworks LMS and successfully displayed on the LMS Admin console? I believe the problem is that syslog-ng's handling of the message changes its contents sufficiently for it not to be recognised as displayable on the LMS console. We do see message traffic arriving at the LMS which gets logged to a file, but it doesn't appear on the GUI console. I have tried the source options no-parse and store-legacy-msghdr with no good effect. Thanks for any responses Paul

1 0

insider 2013-11: syslog-ng 3.5 released; Logging to Hadoop; EoL;
by Peter Czanik 14 Nov '13

14 Nov '13

Dear syslog-ng users, This is the 29th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. FEATURED NEWS syslog-ng 3.5 is released ------------------------- Beginning last week syslog-ng 3.5 was finally declared stable, so 3.5.1 was released. Source is available from http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… if you want to compile syslog-ng yourself. Packages and ports for different Linux distributions and FreeBSD are already available from our 3^rd party page: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… New features of 3.5 were detailed here many times, so here are just the headlines: * Redis and Stomp destinations * multiline messages * type hinting For details, check the ChangeLog at http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.5.1/… or algernon's blog at http://asylum.madhouse-project.org/blog/2013/11/06/syslog-ng-3.5.1/ Logging to Hadoop ----------------- Big data is gaining a momentum also in the logging world, so we made some experiments how syslog-ng can log to Hadoop. Even without a dedicated Hadoop destination driver one can already send logs to Hadoop and analyze them using Hadoop tools. For details, check https://tiborbenke.blogs.balabit.com/2013/11/the-syslog-ng-in-the-hadoop-er… syslog-ng EoL policy -------------------- The syslog-ng End of Life policy was updated recently, based on the experiences of syslog-ng stable maintainer, Gergely Nagy. Practically there are four different branches of syslog-ng maintained at any time, representing different level of maturity and support. More details are available at http://asylum.madhouse-project.org/blog/2013/10/22/syslog-ng-eol-dates/ And following the URLs below you can read about how this affects different Linux distributions and FreeBSD: Debian / Ubuntu: https://lists.balabit.hu/pipermail/syslog-ng/2013-November/020884.html openSUSE / SLES: https://lists.balabit.hu/pipermail/syslog-ng/2013- <https://lists.balabit.hu/pipermail/syslog-ng/2013-November/020880.html>November/020880.html <https://lists.balabit.hu/pipermail/syslog-ng/2013-November/020880.html> FreeBSD ports: https://lists.balabit.hu/pipermail/syslog-ng/2013-November/020878.html SHORT NEWS: * Testing syslog-ng PE was never so easy: syslog-ng PE Quick Start Guide: http://czanik.blogs.balabit.com/2013/10/syslog-ng-pe-quick-start-guide/ * syslog-ng git reorganized: https://lists.balabit.hu/pipermail/syslog-ng/2013-November/020875.html * Logging to Loggly using syslog-ng: https://czanik.blogs.balabit.com/2013/11/loggly-and-syslog-ng/ NEW RELEASES * syslog-ng 3.5.1: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-November/000174.… * syslog-ng 3.4.5: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-November/000173.… Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

[Bug 224] New: Segmentation fault in 3.3.7 and 3.4.1
by bugzilla＠bugzilla.balabit.com 13 Nov '13

13 Nov '13

https://bugzilla.balabit.com/show_bug.cgi?id=224 Summary: Segmentation fault in 3.3.7 and 3.4.1 Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 Core dump back trace is #0 0x00007f4020000528 in ?? () #1 0x00007f402a10b1a5 in log_msg_refcache_stop () at logmsg.c:1495 #2 0x00007f402a11760c in log_writer_flush (self=0xa0d460, flush_mode=LW_FLUSH_NORMAL) at logwriter.c:1043 #3 0x00007f402a1177ed in log_writer_work_perform (s=0xa0d460) at logwriter.c:129 #4 0x00007f402a117edb in main_loop_io_worker_job_start (self=0xa0d650) at mainloop.c:371 #5 0x00007f402a1341ca in iv_work_thread_do_work (_thr=0xabdd20) at iv_work.c:118 #6 0x00007f402a13352a in iv_run_tasks (st=0x7f400c40a130) at iv_task.c:48 #7 0x00007f402a13574c in iv_main () at iv_main_posix.c:106 #8 0x00007f402a133fe1 in iv_work_thread (_thr=0xabdd20) at iv_work.c:200 #9 0x00007f402a1361b8 in iv_thread_handler (_thr=0xabddf0) at iv_thread_posix.c:142 #10 0x00007f4028d82851 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f4028ad011d in clone () from /lib64/libc.so.6 some times #2 0x00007f402a11760c in log_writer_flush (self=0xa0d460, flush_mode=LW_FLUSH_NORMAL) at logwriter.c:1043 is #2 0x00007f402a11760c in log_writer_flush (self=0x9b3970, flush_mode=LW_FLUSH_BUFFER) at logwriter.c:1043 of course when this happens all buffers are lost. So much for reliable log tansport :-( -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 8

[Bug 244] New: Pattern beginning with a parser never matches
by bugzilla＠bugzilla.balabit.com 13 Nov '13

13 Nov '13

https://bugzilla.balabit.com/show_bug.cgi?id=244 Summary: Pattern beginning with a parser never matches Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.balabit(a)faxm0dem.org Type of the Report: bug Estimated Hours: 0.0 It would seem the patterns beginning with parsers never match. Example: Consider the following MESSAGE: -- Puppet::Type::Package::ProviderUp2date: file /usr/sbin/up2date-nox does not exist -- The following pattern matches OK: -- <pattern>P@ESTRING:puppet_module: @file @ESTRING:puppet_file: @does not exist</pattern> -- The following pattern does not match: -- <pattern>@ESTRING:puppet_module::: @file @ESTRING:puppet_file: @does not exist</pattern> -- -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

Re: [syslog-ng] syslog-ng Message Deliver Acknowledgment and Actio
by Balazs Scheidler 08 Nov '13

08 Nov '13

On Nov 7, 2013 4:09 PM, "Tamayo, Andres" <Andres.Tamayo(a)viasat.com> wrote: > > Thank you for your feedback. > > > > We are trying to design a log system that protects against log loss, so if any of the failure scenarios that you mentioned occurred, we need to make sure we do not lose any logs. That is the reason why we want to use an audit log file stored in non-volatile memory to store logs that have not been sent to the server yet. However, we are having trouble defining how to know when to delete logs from the file when they are delivered to the server. > > > > I have read a little about RLTP and my understanding is that this protocol is only available for syslog-ng PE edition. We are using syslog-ng OSE because we need to make some minor modifications to the source code. If you could elaborate about the changes, the pe team might be able to work on that to integrate to the pe edition. No promises though, but it'd be useful to know a little more about them. For that reason, we cannot use the RLTP protocol. > > > > Let me ask a few specific questions: > > > > 1) Is there any way to ask the syslog-ng client daemon to send information back to the application that sends messages to it when a message has been successfully delivered to the syslog-ng server daemon (message deliver acknowledgment)? The most reliable would probably be to hand the audit records to syslog-ng as a file, and let syslog-ng track where it's at. syslog-ng keeps counters how many messages are processed by a source, once enough messages are received the files can be removed. The current file position in the current ose release gets updated as soon as the record is queued in the destination's memory based queue. This solution would lose messages when syslogng crashes lacking a disk based queue. Even we had that, it'd have to be resistant to crashes. I'm afraid only the PE edition would solve the problem completely. I am not here to propagate that, but your requirements are way over traditional syslog. The PE team did a lot of work to improve reliability in various failure scenarios effectively converting syslog-ng to a reliable queueing implementation. Bits and pieces are coming to the ose, but it's complex and tricky code, that takes time for me (and the ose team) to take and integrate, and some pieces are proprietary. > > > > 2) I have been looking for a C/C++ API to generate IETF syslog-ng messages but I cannot not find it. I believe that the #include <syslog.h> header is specific for BSD syslog messages. If there is one, where can I find it? I don't know any such implementation for c++, log4j has one. But its really not difficult to write, format the message, and post it using a Unix domain socket, syslog-ng supports that on unix-dgram, so its a single send operation in your c++ app once the message is formatted. > > > > 3) I did a little reading on ampq and zeromp, do you know if I can save messages to non-volatile memory while they are waiting in the queue? I need to make sure that messages are not lost if the system is turned-off or rebooted. These do have disk buffers, I'm not sure what happens if the client crashes. > > > > Thank you for all your help. > > Hope this helps at least somewhat. > > > > From: syslog-ng-bounces(a)lists.balabit.hu [mailto: syslog-ng-bounces(a)lists.balabit.hu] On Behalf Of Balazs Scheidler > Sent: Thursday, November 07, 2013 2:55 AM > To: Syslog-ng users' and developers' mailing list > Subject: Re: [syslog-ng] syslog-ng Message Deliver Acknowledgment and Action > > > > It depends what failure modes you want to protect against. > > There can be > * network connection breaks > * Syslog-ng daemon crashes (client, server) > * audit daemon crashes > > Syslog-ng has most of the infrastructure to carry out (flow control), but some pieces are missing. The PE team did work in this area, they created RLTP for app. Level network acknowledgements that would protect against network and syslogng crashes. Even in that case though the ack information can only be propagated back to your audit app by speaking RLTP. > > Maybe you would need a queueing stack like amqp or 0mq, those tend to provide explicit acks, and there are brokerless solutions as well. > > On Nov 6, 2013 6:15 PM, "Tamayo, Andres" <Andres.Tamayo(a)viasat.com> wrote: > > Hello, > > > > I am new at using syslog-ng and I was hoping someone in the developers mailing list would be able to help me. > > > > I am trying to use a syslog-ng client daemon and a syslog-ng server daemon to implement an audit message system but I cannot find information that would help me configure the daemons for my particular scenario. > > > > Here is my scenario: > > > > I have a separate audit daemon that generates log messages that are written to an audit log file in syslog format. I need the syslog-ng client to read the logs in the audit log file and send them to the syslog-ng server. When the server has received the messages, I need some acknowledgment from the syslog-ng client, so my other audit daemon can remove the submitted log messages from the audit log file (preventing it from reaching maximum capacity). > > > > My scenario does not have to be setup exactly this way if there are better ways to achieve the same result. Basically, I need to keep all logs that have not yet being sent to the syslog-ng server in an audit file. When the messages are delivered to the syslog-ng server, I need to delete them from the audit file. > > > > Can someone tell me if I can achieve this results using syslog-ng daemons? Is there a better way to implement my scenario. > > > > Thank you for your help in advance. > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > >

2 4

syslog-ng 3.4.5 has been released
by devel＠balabit.hu 08 Nov '13

08 Nov '13

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.4.5 SUMMARY : new stable release DATE : Nov 4, 2013 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.4.5) has been released. For latest fixes in the 3.4.x feature branch you are recommended to upgrade to this version. CHANGES: 3.4.5 Mon, 4 Nov 2013 08:54:50 +0100 This is the fourth bug-fix release for the 3.4.x series, with important fixes over the 3.4.4 release. Upgrading to this release is highly recommended. Bugfixes ======== * A race condition in log message reference counting code that sometimes led to crashes was fixed. [#255] * An assertion that happened in certain situations when a subst() rewrite rule was used was fixed. [#256] * A use-after-free error that sometimes happened after a reload, and caused memory corruption was also fixed. [#253] * patterndb was corrected not to create a new instance on reload: this way, the old one is not leaked, and db-parser() does not forget the correlation state, nor its idea of current time on reload. [#257] * The syslog-ng.spec file does not try to install the long-removed ChangeLog file anymore. * A few harmless, but noisy compiler warnings have been eliminated. Features ======== * The JSON parser now understands NULL-typed keys, and ignores them. Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Balazs Scheidler <bazsi(a)balabit.hu> Evan Rempel <erempel(a)uvic.ca> Gergely Nagy <algernon(a)balabit.hu> Gonzalo Paniagua <gonzalo.paniagua+slng1(a)acquia.com> Jose Pedro Oliveira <jpo(a)di.uminho.pt> Michal Privoznik <miso.privoznik(a)gmail.com> Nicolas Szalay <nico(a)rottenbytes.info> DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

2 1

syslog-ng Message Deliver Acknowledgment and Action
by Tamayo, Andres 07 Nov '13

07 Nov '13

Hello, I am new at using syslog-ng and I was hoping someone in the developers mailing list would be able to help me. I am trying to use a syslog-ng client daemon and a syslog-ng server daemon to implement an audit message system but I cannot find information that would help me configure the daemons for my particular scenario. Here is my scenario: I have a separate audit daemon that generates log messages that are written to an audit log file in syslog format. I need the syslog-ng client to read the logs in the audit log file and send them to the syslog-ng server. When the server has received the messages, I need some acknowledgment from the syslog-ng client, so my other audit daemon can remove the submitted log messages from the audit log file (preventing it from reaching maximum capacity). My scenario does not have to be setup exactly this way if there are better ways to achieve the same result. Basically, I need to keep all logs that have not yet being sent to the syslog-ng server in an audit file. When the messages are delivered to the syslog-ng server, I need to delete them from the audit file. Can someone tell me if I can achieve this results using syslog-ng daemons? Is there a better way to implement my scenario. Thank you for your help in advance.

2 2

Debian & Ubuntu packages for syslog-ng
by Gergely Nagy 06 Nov '13

06 Nov '13

As a reminder, and following Peter's FreeBSD & SuSE package news, for our Debian or Ubuntu users, I'm hosting syslog-ng 3.3, 3.4 and 3.5 packages for a wide range of distributions (see below), available at: http://asylum.madhouse-project.org/projects/debian/ syslog-ng 3.3 packages are available for Ubuntu Lucid, Precise, Raring, and Saucy (for both i386 and amd64 architectures) along with Debian Squeeze, Wheezy, Jessie and unstable (i386, amd64 and powerpc architectures). The syslog-ng 3.4 and 3.5 packages are also available for Ubuntu Trusty. The syslog-ng 3.3 package will be maintained for all of these platforms for as long as the 3.3 branch is maintained (and Ubuntu Trusty will be added to the list if and when 3.3.12 gets released). Same goes for 3.4 and 3.5: they'll be maintained until their respective end of life (see my earlier post about that[1]). I will keep adding new Ubuntu and Debian releases as they come out, and remove them shortly after they reach their respective end of lives. [1]: http://asylum.madhouse-project.org/blog/2013/10/22/syslog-ng-eol-dates/ The easiest way to use the repository is to use the interface on the website to select the distribution and components, then copy & paste to /etc/apt/sources.list. But the line to copy follows this pattern: deb http://packages.madhouse-project.org/debian wheezy syslog-ng-3.4 Where the end of the URL is either "debian" or "ubuntu", the release is your release (either the codename, such as "precise" or "wheezy", or version number like "12.04" or "7.0"), and the last part, the component is the syslog-ng branch you want. There is also a syslog-ng and syslog-ng-devel component, which will always follow the currently recommended branches (3.4 and 3.5 at the time of this writing), so if you want to update from one stable version to the other without having to modify your sources.list, I recommend using one of these aliases. A quick and easy way to enable the repository is as follows (as root): # curl https://packages.madhouse-project.org/debian/archive-key.txt | apt-key add - # echo "deb http://packages.madhouse-project.org/$(lsb_release -is | tr '[:upper:]' '[:lower:]') " \ "$(lsb_release -cs | tr '[:upper:]' '[:lower:]') " \ "syslog-ng" \ > /etc/apt/sources.list.d/syslog-ng.list -- |8]

1 0

git repository reorganization
by Balazs Scheidler 06 Nov '13

06 Nov '13

Hi, Until now, we have had separate git repositories for syslog-ng on github, one for each major version. This seemed logical at the time, since we do split maintenance, Algernon is doing the maintenance for older trees, whereas I maintain the latest version. Also, this is how I used to do it on my internal trees. However, this complicates things for people who want to follow syslog-ng development, whenever we create a new major release, they have to follow a different repository. Difficult to track forks, see all releases in one place, etc. Since git is perfectly capable of storing multiple major releases in the same git repository, I think it would be the best to let the multiple repositories idea go and stick to a single repository model in the future. To model this idea, I have created this repository on github, and pushed 3.3, 3.4 and 3.5 HEADs here: https://github.com/balabit/syslog-ng Branch names would be as follows: - master: the latest HEAD where development is happening on the latest release - 3.3/master, 3.4/master, 3.5/master: the HEAD for maintained old major releases - topic branches destined for a specific major release would be named 3.X/<branch>, topic branches for the latest version would omit the version number prefix. - tags indicate releases, they are named after the respective release for example: v3.4.5 I hope this makes things simpler for everyone in the long term. If anyone has feedback, I'd love to hear them now, before announcing this to the general public and update references everywhere. Thanks in advance, -- Bazsi

3 3

syslog-ng packages for openSUSE/SLES
by Peter Czanik 06 Nov '13

06 Nov '13

Hello, I just updated the syslog-ng packages in my openSUSE Build Service home project. As not all dependencies are part of SLES or openSUSE, it is recommended to add repositories by "zypper ar http://to/repo reponame" and install from there as my repositories carry all necessary dependencies as well. https://build.opensuse.org/project/show/home:czanik:syslog-ng33 has syslog-ng 3.3.11 and will be removed as soon as openSUSE 12.2 is EoL (early next year) https://build.opensuse.org/project/show/home:czanik:syslog-ng34 has syslog-ng 3.4.5 and will stay as long as openSUSE 12.3 is supported (Q3 next year) This version will be published as an update for both openSUSE 12.3 and 13.1 in the coming weeks (as it's not a security update, review is slower by the maintenance team) https://build.opensuse.org/project/show/home:czanik:syslog-ng35 has syslog-ng 3.5.1. This version adds redis support. Due to an error in the release process, manual pages are missing, but documentation is available at the usual location: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.5-guid… As these are not official Balabit or SuSE packages, please contact me, if you run into any troubles! Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0