March 2014 - syslog-ng - lists.balabit.hu

syslog-ng 3.5.4 has been released
by devel＠balabit.hu 12 Mar '14

12 Mar '14

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.5.4 SUMMARY : new stable release DATE : Mar 12, 2014 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.5.4) has been released. For latest fixes in the 3.5.x feature branch you are recommended to upgrade to this version. CHANGES: 3.5.4 Wed, 12 Mar 2014 13:18:36 +0100 This is the third bug-fix release for the 3.5.x series, with a number of important improvements over the previous release. Upgrading to this version is highly recommended. Bugfixes ======== * The value-pairs code had a serious bug, where under some circumstances (depending on compiler, platform and optimisation flags), it could end up in a live-lock, where the only way out was to forcibly kill the syslog-ng process. * SDATA enterprise IDs are handled better when using value-pairs (MongoDB, $(format-json), etc): the enterprise id is kept together, the possibly dotted numeric ID is not split into sub-structures anymore. * json-parser() can now be used from multiple log paths without crashing the syslog-ng process. * Options that can take a unit-suffixed number now work correctly on 32-bit big-endian platforms. * Using some time-related macros (like R_SEC) in filters do not crash syslog-ng anymore. * The internal copy of libmongo-client was updated to version 0.1.7.1, which fixes an important bug in the safe mode handling code. * The system() source learned to recognise FreeBSD jails, and will not use /dev/klog when running inside one. * The host() and port() options of the MongoDB destination have been marked obsolete (which were documented as such, but the code didn't follow). * The username() and password() options are now mandatory for the AMQP destination, and syslog-ng raises an error if either of them is unset. This is because the destination does not work without them anyway. * The timezone file parser now supports Tzif3 formatted files too. * The $(geoip) template function handles failed look-ups gracefully: it expands to an empty string then, and does not print a warning anymore. * The source code was updated to work with bison 3 and later (only affects people compiling from git, or rebuilding grammar files). * The syslog-ng pkg-config file now has an scldir and a datarootdir variable too, to help third-party packages. Credits ======= syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Andras Mitzki <micek(a)balabit.hu> Andres Tamayo <Andres.Tamayo(a)viasat.com> Balazs Scheidler <bazsi(a)balabit.hu> Daniel Gados <gdani(a)balabit.hu> Evan Rempel <erempel(a)uvic.ca> Fabien Wernli <bugzilla.balabit(a)faxm0dem.org> Gergely Nagy <algernon(a)balabit.hu> Igor Ippolitov <iippolitov(a)gmail.com> Peter Czanik <czanik(a)balabit.hu> Steve Berryman <steve(a)berryman.org.uk> Tibor Benke <btibi(a)balabit.hu> Viktor Tusa <tusa(a)balabit.hu> DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

[Bug 273] New: 3.5.3 segfault on match against some macros
by bugzilla＠bugzilla.balabit.com 12 Mar '14

12 Mar '14

https://bugzilla.balabit.com/show_bug.cgi?id=273 Summary: 3.5.3 segfault on match against some macros Product: syslog-ng Version: 3.5.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 If I include any match against the time related macros, syslog-ng seg faults. Seems that *SEC and UNIXTIME are affected at the very least Very simple conf and all blows up. @version: 3.5 source s_test { internal(); } ; destination d_var_syslog { file("/var/log/syslog.log" owner("root") group("syslogs") perm(0640) ); }; filter f_one { match("1" value("R_SEC") ); }; log { source(s_test); log { filter(f_one); destination(d_var_syslog); }; }; -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 270] New: deadlock with patterndb and format-json ( debug NOT enabled in compilation)
by bugzilla＠bugzilla.balabit.com 12 Mar '14

12 Mar '14

https://bugzilla.balabit.com/show_bug.cgi?id=270 Summary: deadlock with patterndb and format-json (debug NOT enabled in compilation) Product: syslog-ng Version: 3.5.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.balabit(a)faxm0dem.org Type of the Report: bug Estimated Hours: 0.0 Created an attachment (id=90) --> (https://bugzilla.balabit.com/attachment.cgi?id=90) syslog-ng configuration and patterndb file This one was hard to isolate. Using the minimal attached configuration, it's very easy to lock the system down by running the following command (bourne shell): $ i=0; while echo 'sshd: Received disconnect from 1.2.3.4: 11: disconnected by user' > /var/tmp/syslo g-ng.src.fifo; do printf .$i.; i=$(($i+1)); done .0..1..2..3..4..5..6..7..8..9..10..11..12..13..14..15..16..17..18..19..20..21..22..23..24..25..26..27..28 [...] 124..125 [DEADLOCK] After that, syslog-ng is completely locked. command used to run syslog-ng: /sbin/syslog-ng -f /home/faxm0dem/syslog-ng-bug/syslog-ng.conf -R /tmp/syslog-ng.persist -p /tmp/syslog-ng.pid Most importantly, this does *only* happen when --enable-debug is *not* switched during compilation. Moreover, an example pattern in the provided patterndb shows how this can be avoided by renaming the variables. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 7

GSoC Introduction
by Tibor Benke 11 Mar '14

11 Mar '14

Dear All, My name is Tibor Benke, I'm a first year MSc student at Budapest University of Technology and Economics, where I study Computer Science. I heard about GSoC and the most interesting organization was the syslog-ng project. I've professional experience in C/C++, Java, Python and worked with git. Since my childhood I always wanted to be an engineer. After graduating I got a certificate of "IT network installer and operator". Then I went to university, where I specialize in Networks and services, but I concentrate on designing and writing networked applications. My BSc thesis was to write a web based network management application in Python. During the academic years I participated in some competitions. What I'm most proud of is the national EuroSkills 2013 qualifying rounds, where my team earned the sixth place in which I was the Linux specialist. I also have work experience: one summer I patched a network driver in the Linux kernel, then did some research in the field of Hadoop and Big Data. In 2013 I contributed to some open source projects, like Log4j2, Logback and syslog-ng. Along with my studies I lead a group of 60 volunteer students, who operate the network and server of my faculty's dormitory, write games, mobile- and other applications. The project I'm most interested in is "Integration with config management systems". Since I wrote my thesis in Python, so the Salt part shouldn't be so hard. I like to learn new languages, so I'm looking forward to enhance my knowledge by learning Ruby. Regards, Tibor Benke

2 1

[GSOC 2014] Interested in working on "XMPP Destination"
by Shonali Balakrishna 11 Mar '14

11 Mar '14

Hi all, I am Shonali Balakrishna from Bangalore, India. I am going to be starting graduate school(MSc.) at TU Delft this fall and wish to participate in Google Summer of Code 2014 as a part of Syslog-ng. I am an Electronics and Telecommunications engineering graduate(2008-2012) from PESIT, Bangalore and I'm currently working in Oracle India, on automation/scripting for refresh(cloning) tasks in their Cloud Division. I am proficient in programming/scripting in C, Java and Perl. I have hands on experience with working on XML and XMPP through an automation project at work. I have user level familiarity with syslog-ng. Most importantly, I enjoy learning and am always willing to learning something new. I am interested in working on "XMPP Destination" and intend to apply for this project through GSOC 2014. I have been familiarizing myself with Syslog-ng source code in the last couple of weeks. I am working on a project proposal for the same and would like to run it by you for feedback when I'm done. I would be extremely grateful for any pointers or leads to resources you could provide me with, that would help me with this project. Thank you for your time. Best Regards, -- Shonali Balakrishna

2 1

GSoC - ZMQ transport
by Laci Mészáros 11 Mar '14

11 Mar '14

Hello, I am Laszlo Meszaros a student at Szechenyi Istvan University, studying Software Engineering.I have read the GSoC idea list and "ZMQ transport" idea. I think it is a good idea either it is source, destination, transport or all of them. There are use cases when people can use that to make their everyday job easier. (e.g.: salt monitoring, to build scaled systems, or in case of zmq transport use multiple syslog-ng on different nodes)I hope I have most of the skills that is necessary for that, but not all of them. I would like to gather the missing ones during GSoC. (e.g.: I know syslog-ng code base on a basic level, but I would like to get deeper knowledge.) I have programming knowledge in python, C and very minimal in unix shell which I think are enough to complete the project during the summer. So I would like to participate to GSoC as student who undertake to solve ZMQ transport idea. Best Regards, Laci

2 1

GSOC 2014 - ZMQ transport
by Sanidhya Kashyap 11 Mar '14

11 Mar '14

Hello everyone, I am a prospective GSOC applicant and I am quite interested in working on the project idea titled - 'ZMQ transport'. Since, neither was Viktor online nor was Gergely available while I was online, so I thought I should sent a mail on the mailing list. A bit of introduction about me. I am a Masters student studying at IIIT Hyderabad, where I have worked on live migration of virtual machines. There, I have also used ZMQ for one of my sub-projects where I was required to do some distributed data transfer across multiple physical machines. Hoping to hear from you people soon. Thanks, Sanidhya

2 1

Re: [syslog-ng] High availabilty configuration
by Terry Slattery 11 Mar '14

11 Mar '14

Ramesh, My customers are network folks, not sysadmins. Network folks are good at networking and network troubleshooting, but not great at installing and maintaining things like DRBD. So DRBD is not an option. I don't see how you could use rsync with the log folder to make a single data set. The folders/directories become different in a split-brain situation and rsync operates on files and directories. Consider the split-brain case where two syslog-ng servers can't see each other and some network devices can still send to one of the two serves. The syslog-ng data now becomes different, with some devices able to log to one syslog-ng server, but not the other. If you're forwarding syslog data to other consumers (alerting systems, security systems, etc), only one server will typically be able to reach some of these systems. What happens depends on your architecture and the devices involved. This is what makes automatic handling difficult. Failure of the syslog-ng server itself is easy to handle; it is a network failure that creates the split-brain situation that's difficult. -tcs On 3/11/14 2:15 AM, syslog-ng-request(a)lists.balabit.hu wrote: > Date: Mon, 10 Mar 2014 14:10:24 -0400 > From: Ramesh Basukala<basukalaramesh(a)gmail.com> > Subject: Re: [syslog-ng] s High availabilty configuration > To: "Syslog-ng users' and developers' mailing list" > <syslog-ng(a)lists.balabit.hu> > Message-ID: > <CABJn1Wj3TEESuNsE1CcTvHKCBEKAhwOmV6gr6y8ibzR+w+U+MQ(a)mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > Hi Terry, > > Thanks for your suggestion. The reason I am trying to configure high > availability with heartbeat/DRBD is data consistency. I prefer no to have 2 > different sets of data in case one of the syslog server went offline. Do > you think we might be able to rsync the log folder once in a while to make > data consistent? > > Thanks, > RB -- Terry Slattery CCIE# 1026

1 0

High availabilty configuration
by Ramesh Basukala 11 Mar '14

11 Mar '14

Hi, I am running open source version of syslog-ng server, currently I have only one server. I would like to add another server and configure high availability, such that log data will still be available in case my primary server dies. Looking at the documentation, syslog-ng itself does not support high availability configuration and has to be done at Operating System level. I need help setting up high availability, please point me to any resource or documentation to start with. Thanks for the help. -RB

6 7

Re: [syslog-ng] s High availabilty configuration
by Terry Slattery 10 Mar '14

10 Mar '14

My recommendation to customers who need high reliability is to run two instances, each in a separate data center. Configure all gear to send to both servers. If syslog forwarding to other consumers is needed, manually configure it on one of the two servers. Upon a failure, confirmed by a human, the servers are reconfigured for forwarding. Many syslog consumers can't handle more than one feed, thus the need to reconfigure upon a failure. This isn't optimum for a site that wants automatic operation. However, building a system that can properly determine and avoid split-brain failures (where the two systems can't talk with one another, but the network gear can send to them) is an "interesting challenge." It would be easy to build something to generate an alert when the two systems can't talk with one another and make this alert the highest priority. Humans can then step in to diagnose the problem and take the appropriate action. Meanwhile, both systems are logging whatever they receive, which may be the whole feed or a part of a feed, if the network becomes partitioned. -tcs On 3/8/14 2:57 AM, syslog-ng-request(a)lists.balabit.hu wrote: > Date: Fri, 07 Mar 2014 15:50:43 -0500 > From: Ramesh Basukala<basukalaramesh(a)gmail.com> > Subject: [syslog-ng] High availabilty configuration > To:syslog-ng@lists.balabit.hu > Message-ID:<531A3123.5080007(a)gmail.com> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Hi, > > I am running open source version of syslog-ng server, currently I have > only one server. I would like to add another server and configure high > availability, such that log data will still be available in case my > primary server dies. > Looking at the documentation, syslog-ng itself does not support high > availability configuration and has to be done at Operating System level. > > I need help setting up high availability, please point me to any > resource or documentation to start with. > > Thanks for the help. > -RB -- Terry Slattery CCIE# 1026

2 1