July 2012 - syslog-ng - lists.balabit.hu

Following a file to generate one syslog message per appended line
by Terry Burton 17 Jul '12

17 Jul '12

Hi, I am looking to use syslog-ng to follow a file and create one syslog UDP message for each line that is appended to the file. So far I have the following, however this will put multiple lines into a single syslog message when they arrive together: source s_tail_snort { file("/srv/snort/snort.fast" flags(no-parse) ); }; destination to_splunk { udp("143.210.16.141" port(1514) template("$MSG\n")); }; log {source(s_tail_snort); destination(to_splunk); flags(flow-control); }; I have tried adding log_fetch_limit(1) to the source and flush_lines(1) to the destination, but I still get multiple lines per syslog message: source s_tail_snort { file("/srv/snort/snort.fast" flags(no-parse) log_fetch_limit(1) ); }; destination to_splunk { udp("143.210.16.141" port(1514) template("$MSG\n") flush_lines(1)); }; log {source(s_tail_snort); destination(to_splunk); flags(flow-control); }; Am I missing something simple? Thanks, Terry

3 3

syslog-ng Insider - July 2012
by Peter Czanik 12 Jul '12

12 Jul '12

Dear syslog-ng users, This is the 15th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com FEATURED NEWS What's next for syslog-ng ------------------------- Sources for some previously Premium Edition-only features have been available for a while now, but have not been merged into the Open Source Edition yet. As we want to release OSE 3.4 soon, we are unlikely to have the necessary resources to merge all of these. We would like to ask Your opinion on which of these features would you like to see in the upcoming release. Please vote and share your reasoning in the BalaBit community forum at http://communities.balabit.com/balabit/topics/poll_which_of_these_features_… syslog-ng 3.4 alpha3 is released -------------------------------- Summer is here, when usually not much is happening. This is not the case with syslog-ng, where the third alpha of the upcoming version was released. Changes since the previous alpha version include new tag related rewrite rules, a marker option for the JSON parser to ease implementing CEE and many smaller features, like support for Cisco extended timestamp format, ported over from the Premium Edition. Of course there were also many smaller fixes, some of them merged from the 3.3 version. For a complete list, check the announcement at https://lists.balabit.hu/pipermail/syslog-ng-announce/2012-June/000144.html As the changes since v3.3 are less drastic, than were between 3.2 and 3.3, we hope that more people will install it on their test or non critical systems. We hope to gather some feedback, how existing configurations work with syslog-ng 3.4 (so no existing feature was broken), and experiences with the new features. Syslog-ng 3.3.6 is coming ------------------------- This release is a bug-fix release mostly, with a few minor - yet useful - feature enhancements. The most important one is that the patched ivykis syslog-ng 3.3 shipped with until now is no more. We build against upstream ivykis now (still included for convenience, though). Other highlights include much improved systemd support, and an enhancement to the @include feature, and many bugfixes. It will be available in the coming days from http://www.balabit.com/downloads/files?path=/syslog-ng/open-source-edition/… LogZilla 4.0 is released ------------------------ A new major version of LogZilla, previously known as php-syslog-ng, was released last month. Next to its outstanding Cisco network device support, the new release features easier, even unattended installation, enhanced access control and search possibilities. Detailed information about the new release is available at http://www.logzilla.pro/news/releases/4.0 OTHER SHORT NEWS ELSA tested with syslog-ng Premium Edition: http://czanik.blogs.balabit.com/2012/06/elsa-and-syslog-ng-pe/ NEW RELEASES: syslog-ng OSE 3.4 alpha3: https://lists.balabit.hu/pipermail/syslog-ng-announce/2012-June/000144.html syslog-ng PE 4.2.3: https://lists.balabit.hu/pipermail/syslog-ng-announce/2012-July/000145.html syslog-ng OSE 3.3.6: http://www.balabit.com/downloads/files?path=/syslog-ng/open-source-edition/… (once available) ARCHIVE http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

feature request?: monitor open logs for deletion/rename
by Patrick Hemmer 12 Jul '12

12 Jul '12

Something which I think would be an awesome feature would be to monitor open file destinations for the file being renamed or deleted, and then reopen the file in such an event. The benefit of this is that when doing log rotation, you don't have to SIGHUP syslog-ng to make it re-open files. It would also make it so that all destination buffers don't have to be flushed and reopened, just the single file destination. The only downside is portability as not all OSs support the same way of doing this. On linux this can be easily done through inotify, and it looks like the BSD equivalent is kqueue (though I have pretty much no BSD experience). For platforms which there isn't a good method, we could instead fall back to a simple polling. -Patrick

5 9

AUTO: James B Horwath/TheGuardian is out of the office. (returning 07/14/2012)
by James B Horwath 12 Jul '12

12 Jul '12

I am out of the office until 07/14/2012. I am out of the office until Monday July 16, 2012 with limited access to e-mail. If there are Nitro issues please contact Dante Rodino at 212-919-2819 or Sunil Gupta at 610-807-6710. Any other issues please contact Al Limone manager of IT Security Services & Governance. Thanks, Jim Note: This is an automated response to your message "syslog-ng Digest, Vol 87, Issue 11" sent on 7/12/2012 6:00:02. This is the only notification you will receive while this person is away. ----------------------------------------- This message, and any attachments to it, may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are notified that any use, dissemination, distribution, copying, or communication of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately by return e-mail and delete the message and any attachments. Thank you.

1 0

Debugging messages correlation in online mode??
by C. L. Martinez 05 Jul '12

05 Jul '12

Hi all, Is it possible to enable some flag to debug patterndb.xml, contexts defined, variables, etc in online mode when correlation is used?? Thanks.

2 1

syslog-ng 3.4.0alpha3 has been released
by devel＠balabit.hu 05 Jul '12

05 Jul '12

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.4.0alpha3 SUMMARY : new stable release DATE : Jun 30, 2012 ------------------------------------------------------------------------------ DESCRIPTION: A new version of syslog-ng Open Source Edition (3.4.0alpha3) has been released. For latest fixes in the 3.4.x branch you are recommended to upgrade to this version. CHANGES: 3.4.0alpha3 Fri, 29 Jun 2012 15:32:54 +0200 Features: ========= * Added set-tag() and clear-tag() rewrite rules. * Added a marker() option to the json-parser(), which indicates that a message is indeed has a json payload. Used to implement a cee-style payload parser. http://bazsi.blogs.balabit.com/2012/05/cee-prototype-and-a-show-case-for-th… * The system source automatically detectd systemd and defines the proper log socket accordingly. Also, if the system() source is not used, a workaround to use the systemd supplied log socket instead of /dev/log was applied. This kicks in if syslog-ng is running under systemd and its config version is 3.3 or less. * Support for FreeBSD's utmpx database. Featured ported from the Premium Edition ======================================== * Added 12 hour time related macros $AMPM and $HOUR12 * Added $USEC and $MSEC macros for micro and milliseconds respectively. * Added $SYSUPTIME that expands to the time since syslog-ng started in the number of 1/100th of a second (just like in the SNMP sysUptime value). * db-parser(): added field extraction to the matching or program names * Added support for Cisco extended timestamp format, one that includes time synchronity information and sequence numbers. https://github.com/bazsi/syslog-ng-3.4/commit/046f90e76137f31e61412d24b9920… * Added support for mark-mode() that allows the customization how MARK messages are generated. https://github.com/bazsi/syslog-ng-3.4/commit/c38d7ad9a2d40650e243ec29c6b3e… Bugfixes: ========= * Fixed a possible segmentation fault in threaded mode when dns-cache-hosts() is used. * Incorrect encoding of data caused UDP and unix-dgram() receives to be erroneously closed, so no further data would be received. This has been fixed. Incorrect encoding of incoming data causes the connection for connection oriented transports (like tcp() and unix-stream()) to be closed, but the same condition is ignored for datagram oriented transports (like udp() or unix-dgram()). * Accept empty source and destination object in the configuration file. * Fixed a linking issue when using the SMTP destination. * Ignore SIGHUP in the supervisor process. * Fixed fsync() behaviour for destination files. * Fixed a possible NULL deref in db-parser() correllation when context-scope is not defined. Other changes: ============== * Added a --disable-smtp configure option. * Integrated unit test refactorization changes from the BalaBit syslog-ng and SSB teams. * Added systemd alias "syslog" to the unit file. Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Alexander Komyagin <komyagin(a)altell.ru> EgonB <egon(a)local.ee> Evan Rempel <erempel(a)uvic.ca> Imre Lazar <imre(a)balabit.hu> Jose Pedro Oliveira <jpo(a)di.uminho.pt> Mark Ulmer <mark.ulmer(a)apollogrp.edu> Patrick Hemmer Peter Czanik <czanik(a)balabit.hu> Attila Magyar <athos(a)balabit.hu> Attila Nagy <naat(a)balabit.hu> Balazs Scheidler <bazsi(a)balabit.hu> Gergely Nagy <algernon(a)balabit.hu> Juhasz Viktor <jviktor(a)balabit.hu> Peter Gyorko <gyorkop(a)balabit.hu> Viktor Juhasz <jviktor(a)balabit.hu> DOWNLOAD: You can download the source packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

3 6

Followed files that grow while syslog-ng is reloaded
by Francois Durand 04 Jul '12

04 Jul '12

Hi list, I'm using syslog-ng (3.2.4) to follow some nginx log files using these directives: source s_nginx_logs { file("/path/to/nginx/access.log" flags(no-parse)); } When syslog-ng is reloaded, either for its own log files rotation or because a configuration update (which happens regularly when a new site is added), it does not see new data being written in followed files while it is reloading itself, and then complains about file size having changed and retransmit the whole file to my central log file collector. Here is the warning message: syslog-ng[24608]: The current log file has a mismatching size/inode information, restarting from the beginning; state='affile_sd_curpos(/path/to/nginx/access.log) stored_inode='139419', cur_file_inode='139419', stored_size='985114', cur_file_size='997815', raw_stream_pos='1995630' I don't see how to handle that correctly. Any idea ?

4 6

Error trying to restart service
by James McDonald 02 Jul '12

02 Jul '12

After the weekend I came in to run the checks on our logs and noticed nothing new had been written over the weekend, or at least since mid-night last night. When I tried to restart the service, I got an error saying that the "user; user=root" and "group; group=root" could not be verified. I am stuck in a meeting at the moment and was wondering if anyone could offer any advice on how to quickly resolve this, once I get back to my desk? Thanks, James McDonald System Administrator RHCSA, Security+ jmcdonald(a)lce.com

3 6

[Bug 168] New: Solaris 10 make failure after successful configure
by bugzilla＠bugzilla.balabit.com 02 Jul '12

02 Jul '12

https://bugzilla.balabit.com/show_bug.cgi?id=168 Summary: Solaris 10 make failure after successful configure Product: syslog-ng Version: 3.3.x Platform: Other OS/Version: Solaris Status: NEW Severity: blocker Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: david.j.fenske(a)usps.gov Type of the Report: bug Estimated Hours: 0.0 1. Download, gunzip, and de-tar syslog-ng-3.3.4 2. PKG_CONFIG_PATH=/usr/local/lib/pkgconfig 3. LD_LIBRARY_PATH=/usr/local/lib 4. ./configure 5. make make all-recursive Making all in lib make: Fatal error in reader: Makefile, line 993: Unexpected end of line seen Current working directory /djftools/syslog/source/syslog-ng-3.3.4/lib *** Error code 1 The following command caused the error: fail= failcom='exit 1'; \ for f in x $MAKEFLAGS; do \ case $f in \ *=* | --[!k]*);; \ *k*) failcom='fail=yes';; \ esac; \ done; \ dot_seen=no; \ target=`echo all-recursive | sed s/-recursive//`; \ list='lib modules syslog-ng scripts tests doc contrib scl debian tgz2build build'; for subdir in $list; do \ echo "Making $target in $subdir"; \ if test "$subdir" = "."; then \ dot_seen=yes; \ local_target="$target-am"; \ else \ local_target="$target"; \ fi; \ (CDPATH="${ZSH_VERSION+.}:" && cd $subdir && make $local_target) \ || eval $failcom; \ done; \ if test "$dot_seen" = "no"; then \ make "$target-am" || exit 1; \ fi; test -z "$fail" make: Fatal error: Command failed for target `all-recursive' Current working directory /djftools/syslog/source/syslog-ng-3.3.4 *** Error code 1 make: Fatal error: Command failed for target `all' -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 10

Re: [syslog-ng] syslog_ng 3.3.4 make error on HPUX HPPA 11.11
by Rakesh Rajasekharan 02 Jul '12

02 Jul '12

Hello Pal, I do not have all of the patches you listed out , installed on my mahcine. I still, tried to compile it with the other suggestion you had put forward but. still was stuck with the same error. I am trying these patches installed and will give it a try thereafer. Will keep you posted. Thanks, Rakesh. On Fri, Jun 1, 2012 at 3:06 AM, Pal Tamas <folti(a)balabit.hu> wrote: > On Thu, May 31, 2012 at 02:13:28PM +0530, Rakesh Rajasekharan wrote: > > Hello Pal, > > > > I was able to get the PHCO_33282 patch and applied it on my 11.11 > machine. > > PHCO_37940 is fo rthe 11.23 HPUX versions hence can't install that. > > > > # swlist -a patch_state *,c=patch |grep PHCO_33282 > > # PHCO_33282 > > PHCO_33282.C-MIN-32ALIB applied > > PHCO_33282.C-MIN-64ALIB applied > > PHCO_33282.CORE-64SLIB applied > > PHCO_33282.CORE-SHLIBS applied > > PHCO_33282.PAUX-ENG-A-MAN applied > > # > > But, even after applying this, I still get the same errors regarding > pthread. > > > > I am curious to know if I should actually be compiling on this 11.11 > machine at > > the first place. > > I mean, is the syslog-ng 3.3.4 compatible with the 11.11 version. The > previous > > version that I was using on this machine is 3.0.7. > > Is there a place I can check the compatible OS's . > Hi, > > dug into the problem a bit more, syslog-ng OSE 3.3 (probably 3.4 too) > needs some porting work before it can be compiled under HP-UX. The > underlying problem is that syslog-ng has been re-written to be > multi-threaded, modular application in 3.2, from it's original single > threaded monolithic architecure. The rewrite added new dependencies > (ivykis) and other required features from the compilers > (Thread-local storage support), which made porting to older systems a > challenge, although the work has been already done, but not every piece > has been added to syslog-ng OSE from the PE branch and not publicly > documented. > > What I managed to solve for now: > - need the following patches: > PHCO_24402 1.0 libc cumulative header file > patch > PHCO_38279 1.0 libc cumulative patch > PHKL_31918 1.0 fsadm;ACL;locking order;8K > mount;readdir > PHSS_30049 1.0 ld(1) and linker tools > cumulative patch > > - download and install GCC for HPUX[1] and make sure it's location > (/opt/hp-gcc/bin) is in PATH, prepeding every other gcc location you > have. > > - edit configure.in and remove all occurences of -Wl,-whole-archive and > -Wl,-no-whole-archive and regenerate configure with autoconf 2.68. If you > don't have autoconf 2.68, edit directly configure and remove the > strings. > > - add the following to lib/compat.h: > #ifndef CLOCK_MONOTONIC > #define CLOCK_MONOTONIC CLOCK_REALTIME > #endif > > - cd lib/ivykis and compile, build and install the bundled ivykis by hand. > > - re-run your configure command with the following options added: > --with-ivykis=system --disable-mongodb > > Hopefully it compiles properly, but I too tired to check it now. > Good luck, > > [1]: > http://h21007.www2.hp.com/portal/site/dspp/menuitem.863c3e4cbcdc3f3515b49c1… > -- > Pal Tamas/Folti > folti(a)balabit.hu > >

2 2