- syslog-ng - lists.balabit.hu

syslog-ng Insider - November 2011
by Peter Czanik 11 Nov '11

11 Nov '11

Dear syslog-ng users, This is the 8th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> FEATURED NEWS syslog-ng 3.3.2 is about to be released! ---------------------------------------- A new version of syslog-ng is about to be released! There are no new features to announce, but all problems reported since 3.3.1 should be fixed by now! To make it the best syslog-ng ever, please test it to make sure, that all your problems are fixed. Sources are available in git or as a snapshot: * git://git.balabit.hu/bazsi/syslog-ng-3.3 * http://packages.madhouse-project.org/syslog-ng/3.3/3.3.2/syslog-ng-3.3.2-HE… Binary packages are available are available for several Linux distributions: * openSUSE: http://download.opensuse.org/repositories/home:/czanik:/syslog-ng33/ * Debian and Ubuntu: http://asylum.madhouse-project.org/projects/debian/ syslog-ng and CEE ----------------- The latest syslog-ng release, version 3.3 can be used to implement part of the “CEE over syslog” standard. BalaBit’s patterndb <http://www.balabit.com/network-security/syslog-ng/opensource-logging-system…> technology was able to extract information from syslog messages already for a long time. With this release JSON output was added, meaning the extracted information can be output as JSON data. What it means in practice, that syslog-ng is able to parse log messages, and output the extracted fields in the form required by CEE. To see, how it works, check http://czanik.blogs.balabit.com/2011/10/cee-and-syslog-ng/ Development of syslog-ng 3.4 started ------------------------------------ While 3.3 was just released, development of 3.4 is already started. The first version of a JSON parser is already merged ( https://github.com/bazsi/syslog-ng-3.4/commit/e5569687bba2551c89a78faee55bc… ). There are some pending fixes and enhancements, which add boolean, array and nested JSON parsing ( https://github.com/algernon/syslog-ng/commits/feature/3.4/json/parser ). Value-pairs key rewrite is work in progress ( https://github.com/algernon/syslog-ng/commits/feature/3.4/value-pairs/rekey ) and nested JSON output is also planned. The above features among others help us to better support CEE. With key rewriting we could use a “.cee.” prefix in CEE related patterns and rewrite it later. It also makes parsing of messages possible. All the current code is available for testing in Algernon's 3.4 sandbox project: https://github.com/algernon/syslog-ng/tree/sandbox/3.4 To download it, use git: $ git clone -b sandbox/3.4 git@github.com:algernon/syslog-ng <mailto:git@github.com:algernon/syslog-ng> OTHER SHORT NEWS * An interesting article about centralized syslog server in Linux Journal: http://www.linuxjournal.com/content/creating-centralized-syslog-server NEW RELEASES * syslog-ng OSE 3.2.5: http://www.balabit.com/downloads/files/syslog-ng/sources/3.2.5 WHITE PAPERS A longer paper about the “Future of logging tools”, which also provides some background information about HSRL, as used in syslog-ng: http://andrea.blogs.balabit.com/files/2011/10/HSRL_backgrounder_english_fin… ARCHIVE http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

3.3.x conditional rewrite bug
by Martin Holste 11 Nov '11

11 Nov '11

The following will not work in the 3.3.x branch: filter f_rewrite_cisco_program { match('^(%(?:ASA|PIX|FWSM)\-\d\-\d{6}): ([^\n]+)' value("MESSAGE") type("pcre") flags("store-matches" "nobackref")); }; rewrite r_cisco_program { set("$1", value("PROGRAM") condition(filter(f_rewrite_cisco_program))); set("$2", value("MESSAGE") condition(filter(f_rewrite_cisco_program))); }; It will process a few lines and then hang indefinitely. It will process ok if the condition is taken off of the filter, so I am assuming this is a PCRE issue, though I found it made no difference to remove the PCRE requirement and use a simpler matching expression. All works perfectly in 3.2.4.

3 4

Quick filter question
by Lay, James 10 Nov '11

10 Nov '11

Hey all! Real quick...trying to filter OUT firewall hits that have say...169.254. Will this do the trick? filter f_firewall { not program (firewall flags(ignore-case)); and not message("169\.254\.[0-9]+\.[0-9]+"); }; Thanks all. James

4 10

Syslog-ng under cygwin.
by andrea.meconi＠libero.it 10 Nov '11

10 Nov '11

Hello. Does anyone know the step by step to compile syslog-ng under Cygwin? I'm running Cygwin on Windows 2008 R2 64 bit and I need option spoof-source enabled. I'm trying this guide, http://mathieu.carbou.free.fr/wiki/index.php?title=How_to_compile_Libnet_un… without success. Thanks. Regards. Andrea

1 0

[PULL REQ]: scratch-buffers
by Gergely Nagy 09 Nov '11

09 Nov '11

I've added the finishing touches to the scratch buffer patch, and I believe it's ready for a merge. Scratch buffers are a way to store reusable GStrings on a per-thread basis. They're allocated on demand, and once released, they're not free'd, but returned to the pool, so the next time the code requests a scratch buffer, it will reuse an existing one, instead of creating a new one. This reduces memory fragmentation, memory use, among other things. It's also very trivial. I have prepared a branch for both syslog-ng 3.3 and 3.4 too. The 3.3 branch also updates the value-pairs implementation to use scratch buffers (and thus, become thread-safe). The 3.4 branch does this too, and it also updates the json parser similarly. The branches are available as feature/3.3/scratch-buffers[0] and feature/3.4/scratch-buffers[1] in my github repo[2]. [0]: https://github.com/algernon/syslog-ng/commits/feature/3.3/scratch-buffers [1]: https://github.com/algernon/syslog-ng/commits/feature/3.4/scratch-buffers [2]: git://github.com/algernon/syslog-ng.git I'm not posting the patches here, as they'd be pretty much the same, and I've already posted a previous version (and the difference is about 3 lines). Though, if it's easier to review if I post it again, let me know, it's a git send-email away. -- |8]

2 1

Re: [syslog-ng] Quick filter question
by Lay, James 08 Nov '11

08 Nov '11

Excellent...thanks so much Patrick. This is working nicely. At some point in time I may want to add in the firewalls "Deny" statement, but for now this will work. Thanks again. James From: Patrick H. [mailto:syslogng@feystorm.net] Sent: Tuesday, November 08, 2011 11:57 AM To: Lay, James Cc: Syslog-ng users' and developers' mailing list Subject: [Spam] Re: [syslog-ng] Quick filter question Importance: Low That might work, but what the order of operations is on boolean operations (and/or) is, is unclear. In situations like that its always best to explicitly force the order yourself. filter f_firewall { not ( program ("firewall" flags(ignore-case)) and ( message("192\.168\.") or message("169\.254\.") ) ); }; Sent: Tue Nov 08 2011 11:46:35 GMT-0700 (MST) From: Lay, James <james.lay(a)wincofoods.com> <mailto:james.lay@wincofoods.com> To: Patrick H. <syslogng(a)feystorm.net> <mailto:syslogng@feystorm.net> , Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> <mailto:syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Quick filter question Ah...so I need or then yes? filter f_firewall { not ( program ("firewall" flags(ignore-case)) and message("192\.168\.") or message("169\.254\.") ); }; How's that look? James P.S. And thank you :) James From: Patrick H. [mailto:syslogng@feystorm.net] Sent: Tuesday, November 08, 2011 11:40 AM To: Syslog-ng users' and developers' mailing list Cc: Lay, James Subject: Re: [syslog-ng] Quick filter question I'm not sure if semicolons are valid in filter rules, but technically valid or not, they shouldnt be there so try removing them. The filter should look like filter f_firewall { not ( program ("firewall" flags(ignore-case)) and message("192\.168\.") and message("169\.254\.") ); }; Note though, that filter will only trigger if both 192.168. and 169.254. are in the same log entry. Unless that IP address you masked out with "x"s is 169.254 it wont trigger. Sent: Tue Nov 08 2011 11:21:11 GMT-0700 (MST) From: Lay, James <james.lay(a)wincofoods.com> <mailto:james.lay@wincofoods.com> To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> <mailto:syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Quick filter question Hey again all. So...I'm still having issue with this..not sure why. Here's the raw log: Nov 8 11:13:38 x.x.x.x firewall: Deny tcp 20 125 x.x.x.x 192.168.0.15 9517 17777 offset 7 S 3371425811 win 64 And from my syslog-ng.conf filter f_firewall { not ( program ("firewall" flags(ignore-case)); and message("192\.168\."); and message("169\.254\."); ) }; log { source(s_local); filter(f_dumb); filter(f_firewall); destination(d_file); destination(other); }; Any hints as to why these aren't matching? Should I not be \ing the periods? Thanks all. James From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Frank Collette Sent: Tuesday, November 08, 2011 8:36 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Quick filter question filter f_firewall { not ( program("firewall" flags(ignore-case)) and message("169\.254\.[0-9]+\.[0-9]+" value("MESSAGE")); ) }; Thanks, Frank E. Collette IV Technical Services Systems Administrator II Trustmark National Bank Office: 601-208-7517 Fax: 601-208-6105 fcollette(a)trustmark.com From: "Lay, James" <james.lay(a)wincofoods.com> <mailto:james.lay@wincofoods.com> To: <syslog-ng(a)lists.balabit.hu> <mailto:syslog-ng@lists.balabit.hu> Date: 11/08/2011 09:14 AM Subject: [syslog-ng] Quick filter question Sent by: syslog-ng-bounces(a)lists.balabit.hu ________________________________________ Hey all! Real quick...trying to filter OUT firewall hits that have say...169.254. Will this do the trick? filter f_firewall { not program (firewall flags(ignore-case)); and not message("169\.254\.[0-9]+\.[0-9]+"); }; Thanks all. James___________________________________________________________________ ___________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ________________________________________________________________________ ______ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

Documentation errors
by VÁMOS Balázs 08 Nov '11

08 Nov '11

Hi, I have found some mistakes on the documentation page. 1. This link does not work: http://www.balabit.com/support/documentation/syslog-ng-windows-agent-v3.1-g… 2. This link is for version 3.1 but the header says it is 3.2 http://www.balabit.com/sites/default/files/documents/syslog-ng-windows-agen… -- VÁMOS Balázs LOGalyze.com

2 2

Cannot start syslog-ng on Debian
by Hery Fanomezantsoa 07 Nov '11

07 Nov '11

I installed syslog-ng-3.2.4 from source on Debian Lenny and when I start syslog-ng with /usr/local/syslog-ng/sbin/ syslog-ng it shows me this error: Error creating persistent state file; filename='/usr/local/syslog-ng/var/syslog-ng.persist-', error='No such file or directory (2)' Can someone help me please?

2 4

3.2 and 3.3 git experiences
by Peter Czanik 05 Nov '11

05 Nov '11

Hello, I compiled and quickly tested the latest git versions of syslog-ng 3.2 and 3.3. Here are some results: OS X: 3.2 compiles and runs fine once I remove "#define HAVE_ENVIRON 1" from the generated config.h No luck at all compiling 3.3 FreeBSD: tested only 3.3 by slightly editing the sysutils/syslog-ng port. As syslog-ng 3.3 dependencies were already on the machine, it was easy to generate a new tgz release: export LEX=/usr/local/bin/flex ./autogen.sh ./configure gmake dist I copied the resulting tgz to /usr/ports/distfiles, under a new name: syslog-ng_3.3.2.tar.gz and deleted /usr/ports/sysutils/syslog-ng/distinfo, as it contains checksum info for 3.3.1. The Makefile had to be modified only to reflect the version change to 3.3.2. "pkg-plist" also had to be fixed to follow up the change in line "lib/libsyslog-ng-3.3.1.so". The resulting syslog-ng binary worked fine for me with any configuration options set during "make config". The above tests were done on a FreeBSD 8.1 machine. I'll do a quick test on FreeBSD 9 (the current cvs version) later this afternoon, as the Makefile seems to have v9 related fixes: .if ${OSVERSION} >= 900007 @${REINPLACE_CMD} \ -e 's|utmp|utmpx|' -e 's|getutent|getutxent|' \ -e 's|ut_name|ut_user|' \ ${WRKSRC}/configure ${WRKSRC}/modules/afuser/afuser.c \ ${WRKSRC}/lib/utils.c ${WRKSRC}/lib/utils.h .endif Updating my ports tree right now... Linux: I tested 3.2 only with openSUSE 11.4, and it worked for me without any problems. The package is available in the openSUSE build service at https://build.opensuse.org/project/show?project=home%3Aczanik%3Asyslog-ng32 with sources and download links. I tested 3.3 on openSUSE 11.4 and factory (well, actually 12.1 RC2), and also compiled for 11.3 (but did not test the resulting packages). The packages are available in the openSUSE build service at https://build.opensuse.org/project/show?project=home%3Aczanik%3Asyslog-ng33 with sources and download links. Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 3

3.2.5 preparations
by Balazs Scheidler 04 Nov '11

04 Nov '11

Hi, I wanted to publish a last maintenance release from the 3.2 branch before moving on to work on 3.4. I've backported the relevant patches, tests have ran (both on my development computer and a separate build environment), but I'd like to ask anyone running 3.2.x to test it in more production-like environments. The current state is available in git, but I've also uploaded a tarball to: http://people.balabit.hu/bazsi/syslog-ng-3.2.4+20111022+0801.tar.gz These patches were applied since 3.2.4: 4f27ef33af4d84a31f237469e493ddd3834404fe nvtable: implement proper limitation on NVTable size b6136da08fb3421239a1c2dc40b582a5a7a2e78b gprocess: Implement stricter CAP_SYSLOG checking. 7e1a3903c0635a9a5070a6998ebc3f4e1282f33e syslogformat: handle AIX style message forwards properly 304539186bab96254d683b3026a00d5404903252 cfg_persist_config_add_entry(): call destroy notify if persist config is empty 409304c6fdcabbd1c321d57e2bbbd3d412a43cf2 cfg-lexer.c: don't include hidden files when including a directory 7ca0e8d65048b65fe3c0015ffc7ac8d73a234833 timezone format: handle half hours in the negative range properly e9dfe15a804610def3186f104853cc5626adbf53 Copyright fixups 320e4d065df521a61ff000e848f1e6a7a787c616 program-override(): disable the effect of store-legacy-msghdr flag ef4a0ff535f2860036d8901b290b204aca7568fa persist-state: fixed a memory small leak fc91010594fe80ba111020f1bb4e23ef41acfdc6 cfg-lexer: closed a memory leak in token blocks handling 9f2153fdb3d9b9cdc120af85db5e87f93cbcec0b PCRE matcher: free the pointer returned by pcre_study() to close a memory leak 2884c8fa188ec1ab8e09799b9c6d497cf8aa423f blocks: propagate include problems as parser errors c2669ffeeae0825c9de637c04794dcfa3965b3de logmatcher: fixed a possible crash in case a regexp match is applied to a read-only macro a2fdfc9ee73c1def081eed0c554c592fd6bea572 [SCL] set empty default parameter to block ce2d45f63b9b2ee7b3b486e517b92662158c321d logproto: fixed an inbalanced get_state()/put_state() pair 3e9be2596c298e31a3b3478c430d517f5bfe42c9 [lex-rules.am]: rules use $(EGREP) instead of grep -E.(EGREP is -E compliant grep implementati b00fde64d08642eb06e1cc342f1c762a7dde6767 afsocket: Fixed a wrong cast in afsocket_dd_set_keep_alive(). 5490cf73b95a597c8695407c875cf55d23679bb9 [plugin] on AIX syslog-ng try to load .a archive modules e98032d07cdcb3a51fe1a7ddda3d3d24b0985a92 zone_info_parser: move variable declaration to the beginning of the function 88596d2b069bcdc46e207487be398ef31aa0aad9 tlscontext: fixed a possible segfault in certificate validation 160e7ebe5e6bd2ef1b0c0e79563d617c3f6f0065 [SDATA] Allow store sdata section without name or value 8bcab2e198863847036809c765b5415371d34760 afprog: automatically close pipes to child programs when executing external programs cd47492eb7733c443bccac4891efc398ab393a62 [syslog-ng-ctl] Added GLIB check version 81e350e410344e3eb1b79464359aa838507bf56a func_test: improve SQL autodetection e138c672b11eee0b30f5036872f0866dbaa6e255 system(): Bail out on unknown systems, and use a clean environment to generate config. 32650278fb35fb6f82522df54014c8d57f1705aa [persist]: only check the buffer size if no encoding is specified 2dc66883f722709b50e30875f8bb64daf24536ca LogProtoBufferedServer: fixed error handling in case the persist-state cannot be restored a4ff6764e06d7ae9eeaab8b9a89336680362ebb1 gprocess: fixed compilation error on old glibc versions f69ded4f05ddb1afcdf0ffbf3e16f1fbe3d6b53a func_test: make it possible to check whether it makes sense to run a testcase ae0ff59d9a761c2fda8a19b0c05e0e05c59bae57 Use CAP_SYSLOG instead of CAP_SYS_ADMIN, if available. f54686441496ded775877e33ad2c3e0310861b7b logreader flags: fixed segfault on unknown flags e2961e1c738c8b02ca6057ac34415536830dd0e9 add support for installing systemd unit files 2db971fc37471e39f6a8b34595ca23833166831e fixed chain-hostnames() processing 2f214c4f87d944aa28d53e331a67b1fd88d9840f systemd: make sure the acquired fd is in non-blocking mode I intend to wait a couple of days to do the actual release, but I'll proceed in a week or so without feedback too. -- Bazsi

4 12