- syslog-ng - lists.balabit.hu

[PULL REQ]: scratch-buffers
by Gergely Nagy 09 Nov '11

09 Nov '11

I've added the finishing touches to the scratch buffer patch, and I believe it's ready for a merge. Scratch buffers are a way to store reusable GStrings on a per-thread basis. They're allocated on demand, and once released, they're not free'd, but returned to the pool, so the next time the code requests a scratch buffer, it will reuse an existing one, instead of creating a new one. This reduces memory fragmentation, memory use, among other things. It's also very trivial. I have prepared a branch for both syslog-ng 3.3 and 3.4 too. The 3.3 branch also updates the value-pairs implementation to use scratch buffers (and thus, become thread-safe). The 3.4 branch does this too, and it also updates the json parser similarly. The branches are available as feature/3.3/scratch-buffers[0] and feature/3.4/scratch-buffers[1] in my github repo[2]. [0]: https://github.com/algernon/syslog-ng/commits/feature/3.3/scratch-buffers [1]: https://github.com/algernon/syslog-ng/commits/feature/3.4/scratch-buffers [2]: git://github.com/algernon/syslog-ng.git I'm not posting the patches here, as they'd be pretty much the same, and I've already posted a previous version (and the difference is about 3 lines). Though, if it's easier to review if I post it again, let me know, it's a git send-email away. -- |8]

2 1

Re: [syslog-ng] Quick filter question
by Lay, James 08 Nov '11

08 Nov '11

Excellent...thanks so much Patrick. This is working nicely. At some point in time I may want to add in the firewalls "Deny" statement, but for now this will work. Thanks again. James From: Patrick H. [mailto:syslogng@feystorm.net] Sent: Tuesday, November 08, 2011 11:57 AM To: Lay, James Cc: Syslog-ng users' and developers' mailing list Subject: [Spam] Re: [syslog-ng] Quick filter question Importance: Low That might work, but what the order of operations is on boolean operations (and/or) is, is unclear. In situations like that its always best to explicitly force the order yourself. filter f_firewall { not ( program ("firewall" flags(ignore-case)) and ( message("192\.168\.") or message("169\.254\.") ) ); }; Sent: Tue Nov 08 2011 11:46:35 GMT-0700 (MST) From: Lay, James <james.lay(a)wincofoods.com> <mailto:james.lay@wincofoods.com> To: Patrick H. <syslogng(a)feystorm.net> <mailto:syslogng@feystorm.net> , Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> <mailto:syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Quick filter question Ah...so I need or then yes? filter f_firewall { not ( program ("firewall" flags(ignore-case)) and message("192\.168\.") or message("169\.254\.") ); }; How's that look? James P.S. And thank you :) James From: Patrick H. [mailto:syslogng@feystorm.net] Sent: Tuesday, November 08, 2011 11:40 AM To: Syslog-ng users' and developers' mailing list Cc: Lay, James Subject: Re: [syslog-ng] Quick filter question I'm not sure if semicolons are valid in filter rules, but technically valid or not, they shouldnt be there so try removing them. The filter should look like filter f_firewall { not ( program ("firewall" flags(ignore-case)) and message("192\.168\.") and message("169\.254\.") ); }; Note though, that filter will only trigger if both 192.168. and 169.254. are in the same log entry. Unless that IP address you masked out with "x"s is 169.254 it wont trigger. Sent: Tue Nov 08 2011 11:21:11 GMT-0700 (MST) From: Lay, James <james.lay(a)wincofoods.com> <mailto:james.lay@wincofoods.com> To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> <mailto:syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Quick filter question Hey again all. So...I'm still having issue with this..not sure why. Here's the raw log: Nov 8 11:13:38 x.x.x.x firewall: Deny tcp 20 125 x.x.x.x 192.168.0.15 9517 17777 offset 7 S 3371425811 win 64 And from my syslog-ng.conf filter f_firewall { not ( program ("firewall" flags(ignore-case)); and message("192\.168\."); and message("169\.254\."); ) }; log { source(s_local); filter(f_dumb); filter(f_firewall); destination(d_file); destination(other); }; Any hints as to why these aren't matching? Should I not be \ing the periods? Thanks all. James From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Frank Collette Sent: Tuesday, November 08, 2011 8:36 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Quick filter question filter f_firewall { not ( program("firewall" flags(ignore-case)) and message("169\.254\.[0-9]+\.[0-9]+" value("MESSAGE")); ) }; Thanks, Frank E. Collette IV Technical Services Systems Administrator II Trustmark National Bank Office: 601-208-7517 Fax: 601-208-6105 fcollette(a)trustmark.com From: "Lay, James" <james.lay(a)wincofoods.com> <mailto:james.lay@wincofoods.com> To: <syslog-ng(a)lists.balabit.hu> <mailto:syslog-ng@lists.balabit.hu> Date: 11/08/2011 09:14 AM Subject: [syslog-ng] Quick filter question Sent by: syslog-ng-bounces(a)lists.balabit.hu ________________________________________ Hey all! Real quick...trying to filter OUT firewall hits that have say...169.254. Will this do the trick? filter f_firewall { not program (firewall flags(ignore-case)); and not message("169\.254\.[0-9]+\.[0-9]+"); }; Thanks all. James___________________________________________________________________ ___________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ________________________________________________________________________ ______ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

Documentation errors
by VÁMOS Balázs 08 Nov '11

08 Nov '11

Hi, I have found some mistakes on the documentation page. 1. This link does not work: http://www.balabit.com/support/documentation/syslog-ng-windows-agent-v3.1-g… 2. This link is for version 3.1 but the header says it is 3.2 http://www.balabit.com/sites/default/files/documents/syslog-ng-windows-agen… -- VÁMOS Balázs LOGalyze.com

2 2

Cannot start syslog-ng on Debian
by Hery Fanomezantsoa 07 Nov '11

07 Nov '11

I installed syslog-ng-3.2.4 from source on Debian Lenny and when I start syslog-ng with /usr/local/syslog-ng/sbin/ syslog-ng it shows me this error: Error creating persistent state file; filename='/usr/local/syslog-ng/var/syslog-ng.persist-', error='No such file or directory (2)' Can someone help me please?

2 4

3.2 and 3.3 git experiences
by Peter Czanik 05 Nov '11

05 Nov '11

Hello, I compiled and quickly tested the latest git versions of syslog-ng 3.2 and 3.3. Here are some results: OS X: 3.2 compiles and runs fine once I remove "#define HAVE_ENVIRON 1" from the generated config.h No luck at all compiling 3.3 FreeBSD: tested only 3.3 by slightly editing the sysutils/syslog-ng port. As syslog-ng 3.3 dependencies were already on the machine, it was easy to generate a new tgz release: export LEX=/usr/local/bin/flex ./autogen.sh ./configure gmake dist I copied the resulting tgz to /usr/ports/distfiles, under a new name: syslog-ng_3.3.2.tar.gz and deleted /usr/ports/sysutils/syslog-ng/distinfo, as it contains checksum info for 3.3.1. The Makefile had to be modified only to reflect the version change to 3.3.2. "pkg-plist" also had to be fixed to follow up the change in line "lib/libsyslog-ng-3.3.1.so". The resulting syslog-ng binary worked fine for me with any configuration options set during "make config". The above tests were done on a FreeBSD 8.1 machine. I'll do a quick test on FreeBSD 9 (the current cvs version) later this afternoon, as the Makefile seems to have v9 related fixes: .if ${OSVERSION} >= 900007 @${REINPLACE_CMD} \ -e 's|utmp|utmpx|' -e 's|getutent|getutxent|' \ -e 's|ut_name|ut_user|' \ ${WRKSRC}/configure ${WRKSRC}/modules/afuser/afuser.c \ ${WRKSRC}/lib/utils.c ${WRKSRC}/lib/utils.h .endif Updating my ports tree right now... Linux: I tested 3.2 only with openSUSE 11.4, and it worked for me without any problems. The package is available in the openSUSE build service at https://build.opensuse.org/project/show?project=home%3Aczanik%3Asyslog-ng32 with sources and download links. I tested 3.3 on openSUSE 11.4 and factory (well, actually 12.1 RC2), and also compiled for 11.3 (but did not test the resulting packages). The packages are available in the openSUSE build service at https://build.opensuse.org/project/show?project=home%3Aczanik%3Asyslog-ng33 with sources and download links. Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 3

3.2.5 preparations
by Balazs Scheidler 04 Nov '11

04 Nov '11

Hi, I wanted to publish a last maintenance release from the 3.2 branch before moving on to work on 3.4. I've backported the relevant patches, tests have ran (both on my development computer and a separate build environment), but I'd like to ask anyone running 3.2.x to test it in more production-like environments. The current state is available in git, but I've also uploaded a tarball to: http://people.balabit.hu/bazsi/syslog-ng-3.2.4+20111022+0801.tar.gz These patches were applied since 3.2.4: 4f27ef33af4d84a31f237469e493ddd3834404fe nvtable: implement proper limitation on NVTable size b6136da08fb3421239a1c2dc40b582a5a7a2e78b gprocess: Implement stricter CAP_SYSLOG checking. 7e1a3903c0635a9a5070a6998ebc3f4e1282f33e syslogformat: handle AIX style message forwards properly 304539186bab96254d683b3026a00d5404903252 cfg_persist_config_add_entry(): call destroy notify if persist config is empty 409304c6fdcabbd1c321d57e2bbbd3d412a43cf2 cfg-lexer.c: don't include hidden files when including a directory 7ca0e8d65048b65fe3c0015ffc7ac8d73a234833 timezone format: handle half hours in the negative range properly e9dfe15a804610def3186f104853cc5626adbf53 Copyright fixups 320e4d065df521a61ff000e848f1e6a7a787c616 program-override(): disable the effect of store-legacy-msghdr flag ef4a0ff535f2860036d8901b290b204aca7568fa persist-state: fixed a memory small leak fc91010594fe80ba111020f1bb4e23ef41acfdc6 cfg-lexer: closed a memory leak in token blocks handling 9f2153fdb3d9b9cdc120af85db5e87f93cbcec0b PCRE matcher: free the pointer returned by pcre_study() to close a memory leak 2884c8fa188ec1ab8e09799b9c6d497cf8aa423f blocks: propagate include problems as parser errors c2669ffeeae0825c9de637c04794dcfa3965b3de logmatcher: fixed a possible crash in case a regexp match is applied to a read-only macro a2fdfc9ee73c1def081eed0c554c592fd6bea572 [SCL] set empty default parameter to block ce2d45f63b9b2ee7b3b486e517b92662158c321d logproto: fixed an inbalanced get_state()/put_state() pair 3e9be2596c298e31a3b3478c430d517f5bfe42c9 [lex-rules.am]: rules use $(EGREP) instead of grep -E.(EGREP is -E compliant grep implementati b00fde64d08642eb06e1cc342f1c762a7dde6767 afsocket: Fixed a wrong cast in afsocket_dd_set_keep_alive(). 5490cf73b95a597c8695407c875cf55d23679bb9 [plugin] on AIX syslog-ng try to load .a archive modules e98032d07cdcb3a51fe1a7ddda3d3d24b0985a92 zone_info_parser: move variable declaration to the beginning of the function 88596d2b069bcdc46e207487be398ef31aa0aad9 tlscontext: fixed a possible segfault in certificate validation 160e7ebe5e6bd2ef1b0c0e79563d617c3f6f0065 [SDATA] Allow store sdata section without name or value 8bcab2e198863847036809c765b5415371d34760 afprog: automatically close pipes to child programs when executing external programs cd47492eb7733c443bccac4891efc398ab393a62 [syslog-ng-ctl] Added GLIB check version 81e350e410344e3eb1b79464359aa838507bf56a func_test: improve SQL autodetection e138c672b11eee0b30f5036872f0866dbaa6e255 system(): Bail out on unknown systems, and use a clean environment to generate config. 32650278fb35fb6f82522df54014c8d57f1705aa [persist]: only check the buffer size if no encoding is specified 2dc66883f722709b50e30875f8bb64daf24536ca LogProtoBufferedServer: fixed error handling in case the persist-state cannot be restored a4ff6764e06d7ae9eeaab8b9a89336680362ebb1 gprocess: fixed compilation error on old glibc versions f69ded4f05ddb1afcdf0ffbf3e16f1fbe3d6b53a func_test: make it possible to check whether it makes sense to run a testcase ae0ff59d9a761c2fda8a19b0c05e0e05c59bae57 Use CAP_SYSLOG instead of CAP_SYS_ADMIN, if available. f54686441496ded775877e33ad2c3e0310861b7b logreader flags: fixed segfault on unknown flags e2961e1c738c8b02ca6057ac34415536830dd0e9 add support for installing systemd unit files 2db971fc37471e39f6a8b34595ca23833166831e fixed chain-hostnames() processing 2f214c4f87d944aa28d53e331a67b1fd88d9840f systemd: make sure the acquired fd is in non-blocking mode I intend to wait a couple of days to do the actual release, but I'll proceed in a week or so without feedback too. -- Bazsi

4 12

RFC/RFH: JSON status report and plans
by Gergely Nagy 03 Nov '11

03 Nov '11

Hi! To support CEE[1] stuff better, a few things are needed for syslog-ng's JSON support, and I'd like to take this opportunity to both give a headsup on what's still needed, and present a few easy bits that we need, and are very low hanging fruits for aspiring syslog-ng hackers to grab. Whitespace ---------- First of all, for compactness, the JSON we generate should be free of any unneccessary whitespace. At the moment, we generate JSON like this: { "foo": "bar", "baz": "quux" } The whitespace between key and value should be dropped, so should the whitespace at the curly brackets, so that we'd generate the following instead: {"foo":"bar","baz":"quux"} This is fairly trivial to fix, at least when using json-c (and that's the only case we need fixed, for other reasons): * Each json_object struct has a ->_to_json_string, which is used to format the string. * The default implementation puts in whitespace * The fix is: + #include <json_object_private.h> + copy the json_object_object_to_json_string() function from json.c, changing it to not emit extra whitespace + Override the ->to_json_string of the root JSON object in syslog-ng's modules/tfjson/tfjson.c's tf_json_append(), so that it points to the new function that doesn't do whitespace. This is a trivial patch, and I pretty much already wrote it for you. If someone can turn that into a patch, that would be grand! Nested JSON ----------- Another thing that would be very good to have, and not only for JSON, but for the MongoDB driver aswell, is to be able to generate nested objects. So that a key-value pair like "foo.bar"="quux" would end up as {"foo":{"bar":"quux"}} when formatted as JSON. That is, we'd need to split up keys by "." chars, and treat the first piece as the parent object. The harder part is that keys are not ordered within syslog-ng, and there might be conflicts (ie, syslog-ng allows "foo" and "foo.bar" to coexist, but we can't format that as JSON if we split by dots, because foo would then be a string and an object at the same time), which need to be handled one way or the other. My idea was that we could sort the keys in descending order, so that if we have "foo.aaa", "foo.bbb", and "foo", we'd end up with "foo.bbb", "foo.aaa", and "foo" list, split them up at the dots, pack stuff into foo, notice later that we already have a foo object, so a foo string is no-no, and threw a hissy fit (an error, that is). I'm not going to explain the whole algorithm here, but it's not too hard to figure it out. The question in this case is two-fold: anyone up for implementing this (I can provide assistance)? And if not, should this nesting be enabled by default? Should it be an option at all, or the only supported way? (I tend to side with going with nesting and not providing an option to disable it, but I can be persuaded otherwise) JSON parser ----------- Another thing that's reasonably easy to put together, is a JSON parser. Something like the existing CSV and patterndb parsers, except this would parse JSON with the json-c library. I have a few rough ideas, and am happy to provide help if someone feels up to the task of tackling this and developing the feature. =) Other pending issues -------------------- There's also a case for types. It would be beneficial if numbers could be represented as, well, numbers in the generated JSON (and similary, stored as numbers when using the MongoDB destination). This, however, requires some deeper changes, and I'm not yet familiar with that part of the code yet, either. So I'm not going to go into details. Nevertheless, I felt I need to mention this here, so that it won't fall under the table. [1]: http://cee.mitre.org/ -- |8]

1 1

Memory leak on SIGHUP on 3.0.1 and up
by Simon Gauthier 01 Nov '11

01 Nov '11

Hi, I am using syslog-ng 3.01 and I experience memory leak on SIGHUP. This has been seen on a standard Fedora 10 system with syslog-ng 3.0.1 to 3.0.10. the same test leak also with 3.3.1. My syslog-ng.conf file has no rules in it. A simple bash script sending a SIGHUP to the syslog-ng process will leak memory at a fast face. #!/bin/bash while true; do date # $1 is syslog-ng pid sudo /usr/bin/kill -1 $1 done Any idea about, that? Thanks Simon Gauthier

2 2

NFS support
by Omer Faruk SEN 01 Nov '11

01 Nov '11

Hi, Is there a known issue with syslog-ng and NFS (Netapp Storage) share? Can syslog-ng write to NFS storage ? It should write without problem but i wanted to be sure. I have tried to write to NFS storage of syslog message with version 3.2 Regards.

2 1

[Bug 96] New: pdbtool misleading error message
by bugzilla＠bugzilla.balabit.com 01 Nov '11

01 Nov '11

https://bugzilla.balabit.com/show_bug.cgi?id=96 Summary: pdbtool misleading error message Product: syslog-ng Version: 3.2.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: czanik(a)balabit.hu Type of the Report: --- Estimated Hours: 0.0 When trying to test a non-existent file with pdbtool it gives misleading error messages and even calls an external application before failing. czanik@linux-6y8u:~/patterndb> pdbtool test --validate uw-imapd.ceee uw-imapd.ceee: Unable to detect patterndb version, please write the <patterndb> tag on a single line warning: failed to load external entity "/usr/share/syslog-ng/xsd/patterndb-0.xsd" Schemas parser error : Failed to locate the main schema resource at '/usr/share/syslog-ng/xsd/patterndb-0.xsd'. WXS schema /usr/share/syslog-ng/xsd/patterndb-0.xsd failed to compile warning: failed to load external entity "uw-imapd.ceee" uw-imapd.ceee: xmllint returned an error, the executed command was: xmllint --noout --nonet --schema /usr/share/syslog-ng/xsd/patterndb-0.xsd uw-imapd.ceeeError opening classifier configuration file; filename='uw-imapd.ceee', error='No such file or directory (2)' I think, that the proper behavior would be, that pdbtool detect that the file is nonexistent, prints a message about it (without any other misleading messages) and quits without calling external apps. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[PATCH 0/3]: Scratch buffers for syslog-ng 3.4
by Gergely Nagy 31 Oct '11

31 Oct '11

Following this mail will come three patches: * One to implement scratch buffers * Another to convert jsonparser to scratch buffers * And a third to convert value-pairs While the implementation works for me (with threaded(yes) and threaded(no) aswell), and I believe it to be correct, there's one thing I'm not quite sure about. But before I get there, let me try to explain how this simple thing is supposed to work! As Bazsi suggested, there are three methods: scratch_buffer_acquire(), scratch_buffer_release(), and scratch_buffers_reset(). All of them work with a thread-local GTrashStack, because my idea was that instead of trying to keep a list of available and used scratch buffers, we'll just store them in a stack, and leave it up to the scratch buffer users to return the buffers properly. This makes the implementation super simple, at the cost of not holding the hands of the scratch users. I believe that's an acceptable compromise. scratch_buffer_acquire() tries to pop an element from the stack, and if there's none, it creates a new one, and returns that. It also takes care of truncating the string to zero length. scratch_buffer_release() simply pushes an element onto the stack. scratch_buffers_reset() pops all elements from the stack, frees up both the GString contained within them, and the scratch buffer itself. Due to the way GTrashStack works, I introduced a new typedef: ScratchBuffer. The first element is a GTrashStack, which users need not care about, the second is the GString. ScratchBuffers are supposed to be used something like this: void test_scratchbuffer (void) { ScratchBuffer *sb = scratch_buffer_acquire(); g_string_printf(sb->s, "Hello %s!", "World"); /* Do something with sb->s */ scratch_buffer_release (sb); } There's one other trick in the implementation, however, and this is the one I'm unsure about: in order to avoid leaking madly, the thread-local scratch buffers MUST be destroyed when a thread finishes. This must also be done at the lowest level possible, to not litter the code with scratch_buffers_reset() calls. I did this with calling scratch_buffers_reset() at the end of main_loop_io_worker_job_start(), after all the finish callbacks have finished already. My reading of the code was that this is the place where threads are already finished, and they won't be needing scratch buffers anymore, and they should have released all of them aswell. All three patches are available on the feature/3.4/scratch-buffers branch of my github tree[0] too. While this is for 3.4, they should apply cleanly to 3.3 aswell (except for the jsonparser part, of course). [0]: https://github.com/algernon/syslog-ng/tree/feature/3.4/scratch-buffers

2 6

[PATCH 0/1]: json-parser()
by Gergely Nagy 31 Oct '11

31 Oct '11

Following this mail, a bigger patch will be sent, that implements a JSON parser - a fairly simple one, for now. I did not want to explain every little detail in the commit message, so this umbrella mail is sent first, with a few details that cannot be found in the commit message (there will be some overlap, though). This patch (and any further developments I make in the future, until the parser is merged) will be available on the feature/3.3/json/parser branch of my new and shiny git tree[1], as soon as I push it out. [1]: git://github.com/algernon/syslog-ng.git First of all, here's a little code snippet that shows how to use the module: ,----[ syslog-ng.conf ] | @module tfjson | @module jsonparser | | source s_json { tcp(port(12345) flags(no-parse)); }; | destination d_json { | file("/var/log/messages.json" | template("$(format-json --scope dot-nv-pairs)\n")); | }; | parser p_json { json-parser(prefix(".json.")); }; | log { source(s_json); parser(p_json); destination (d_json); }; `---- With this config, we can send logs to this source as follows: ,----[ shell ] | $ echo '{"string": "this is a string!\\n\\nmultiline!", "int": 42, "double": 3.14 }' | nc localhost 12345 `---- And 'lo and behold, the destination looks like this: ,----[ /var/log/messages.json ] | {".json.string":"this is a string!\n\nmultiline!",".json.int":"42",".json.double":"3.140000"} `---- Since syslog-ng stores every value as a string, they will all be converted to strings. At some point in the future, when we can also store types, this information will be transmitted through, and format-json will preserve the types. For now, the parser has only one optional parameter: prefix(). I think you can figure out what it does. I think this is pretty darn cool as it is! Now, lets get down to the gory implementation details, as I'm sure you're all very interested in that! At the core, we have a LogJSONParser structure, which holds the configuration (hi, gchar *prefix!), and two GStrings, that are used over and over again during processing. The reason these two GStrings (serialized.key and serialized.value) are there, is to have as few memory allocations as possible: so we allocate the strings once, and reuse them over and over again. We also have a JSON tokener here, for similar reasons. Parsing itself is delegated to json-c, and all the parser does is iterate over the parsed object's keys, coerce them into strings, prepare a key (based on prefix and the parsed key), and store them in a LogMessage. Then we're done. Yes, really. The rest is boilerplate. Of course, life ain't that simple. For now, there's a bug where the JSON can't be parsed, we honour that with a tasty segmentation fault. This will be fixed at a later time.

2 8

preparations for 3.3.2
by Balazs Scheidler 31 Oct '11

31 Oct '11

Hi, Since there are a nice number of fixes accumulated on the 3.3 tree, I'd like to prepare for the 3.3.2 release. Before doing the actual release though, I'd appreciate any kind of feedback you can give. The current state is in git right now, I'll try to create a timestamped tarball later this weekend. Thanks. -- Bazsi

3 4

[PATCH 1/2] pdbtool: Bail out on errors even if validating only.
by Gergely Nagy 31 Oct '11

31 Oct '11

When run with the --validate flag, and an error is detected, short circuit to the next file, instead of running the rest of the validation with garbled data. Reported-by: Peter Czanik <czanik(a)balabit.hu> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/dbparser/pdbtool.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/modules/dbparser/pdbtool.c b/modules/dbparser/pdbtool.c index 13bc7d4..c2f9117 100644 --- a/modules/dbparser/pdbtool.c +++ b/modules/dbparser/pdbtool.c @@ -694,12 +694,14 @@ pdbtool_test(int argc, char *argv[]) { fprintf(stderr, "%s: Unable to detect patterndb version, please write the <patterndb> tag on a single line\n", argv[arg_pos]); failed_to_validate = TRUE; + continue; } g_snprintf(cmd, sizeof(cmd), "xmllint --noout --nonet --schema %s/patterndb-%d.xsd %s", PATH_XSDDIR, version, argv[arg_pos]); if (system(cmd) != 0) { fprintf(stderr, "%s: xmllint returned an error, the executed command was: %s", argv[arg_pos], cmd); failed_to_validate = TRUE; + continue; } } -- 1.7.7.1

2 2

MSGHDR and rewriting PROGRAM macro
by Patrick H. 31 Oct '11

31 Oct '11

I've got a box running syslog-ng 3.2.4, and I have a rewrite changing the PROGRAM macro. The problem is this doesnt affect the contents of the MSGHDR macro. Additionally if you try to do a rewrite on MSGHDR, it complains that MSGHDR is read-only. I'm inclined to say this is a bug, but wanted to get official opinion on the issue. -Patrick

2 1

Parsing Cisco FWSM with device-id
by Martin Holste 30 Oct '11

30 Oct '11

So I have a customer who has enabled the device-id configuration directive on his FWSM, and that means that instead of this from a normal FWSM: <174>%FWSM-6-302013: Built inbound TCP... it sends this: <174>FWSMHostName %FWSM-6-30203: Built inbound TCP... This means that the program name does not get properly parsed as syslog-ng pushes it into the msg field. If I can't convince the customer to remove the device-id setting, what's the least CPU-intensive way of coping with this so that program and msg are set correctly?

3 4

syslog-ng does not start when a destination hostname is unresolvable
by Heiko Gerstung 29 Oct '11

29 Oct '11

Hi! I am using syslog-ng for my network appliance and came across a problem with unresolved destination hosts. But first of all I have to say that from all the syslog daemons I looked at, syslog-ng really stands out as the most versatile and flexible solution. So, thank you for this great piece of software! In my configuration it is possible to specify a number of remote syslog servers, to which my device forwards all log entries via TCP or UDP/514 in addition to logging everything to local files (/var/log/messages etc.). Now, when a customer enters an unresolvable hostname, syslog-ng does not start at all and therefore the whole system is without a syslog daemon. My current workaround is to try and resolve the configured hostnames myself and only write them to the generated syslog-ng.conf file when this succeeds, but I wonder if there is no configuration option or commandline switch that prevents syslog-ng from completely stopping to work when it cannot resolve a hostname. And, is this really the intended default behavior? On my system (and on almost every other system I assume), syslog-ng starts long before network connectivity is available in order to provide logging services during the boot sequence. I would have expected that, if one of the destinations cannot be resolved, syslog-ng starts without it and periodically tries to resolve it later. I did not find anything about this in the documentation (manpages of syslog-ng and syslog-ng.conf) or in the FAQ, the only thing I found was a bug in bugzilla (https://bugzilla.balabit.com/show_bug.cgi?id=63) which has been filed in 11/2009 and reports that --syntax-only does not catch unresolved _sources_ although this is considered a critical error and prevents syslog-ng from starting. I would assume that this is the same for unresolved destinations. Any comments on this? I know that ntpd tries to resolve configured servers in a separate thread which retries resolving them every once in a while and starts using such a host as soon as it could be resolved. Best Regards, Heiko

2 1

[PATCH] logproto: Fix log_proto_file_writer_flush()'s partial construction.
by Gergely Nagy 29 Oct '11

29 Oct '11

When log_proto_file_writer_flush() encountered a partial write, and tried to construct a self->partial buffer, that contains all the data we didn't write yet, the starting offset computation was... interesting. What we want to copy into self->partial, is whatever is left in the last, partially processed buffer (i0), then append the full contents of the rest. Calculating the starting offset of this first buffer was done in a quite convoluted way, involving the sum of the length of the buffers we already touched, the lenght of the last fully processed buffer, and the total number of bytes already written. This makes my head spin, especially since it can be simplified to substracting the number of bytes to copy from the first buffer from the total length of it. The attached patch simplifies the offset calculation, and it also fixes a subtle bug! Lets assume 5 buffers, the first one 100, the second 200, the third 300, the fourth 400 and the fifth 500 bytes, totalling to 1500 bytes. Lets assume we managed to write out the first three buffers, and half of the fourht: 800 bytes. In this case, rc will be 800, sum will be 1000, i0 will be 3, and ofs will be 200. So far, all is well. But when we try to compute the starting offset within the first partial buffer, we end up with: rc - (sum - buf[i0 - 1].length), which, when values are substituted becomes: 800 - (1000 - 300) = 100! This is obviously not correct: we've already written out half of the fourth buffer, yet, we start copying from the 100th byte instead of the 200th! After the fix, when the starting position is buffer[i0].length - ofs, we get the correct value of 200, and all is well. Reported-by: SZALAY Attila <sasa(a)balabit.hu> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/logproto.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/lib/logproto.c b/lib/logproto.c index bdf9695..5047c98 100644 --- a/lib/logproto.c +++ b/lib/logproto.c @@ -299,7 +299,7 @@ log_proto_file_writer_flush(LogProto *s) /* allocate and copy the remaning data */ self->partial = (guchar *)g_malloc(self->partial_len); ofs = sum - rc; /* the length of the remaning (not processed) chunk in the first message */ - memcpy(self->partial, self->buffer[i0].iov_base + rc - (i0 > 0 ? (sum - self->buffer[i0 - 1].iov_len) : 0), ofs); + memcpy(self->partial, self->buffer[i0].iov_len - ofs, ofs); i = i0 + 1; while (i < self->buf_count) { -- 1.7.7.1

3 5

[PATCH] tfjson: Make $(format-json) have less unneccessary whitespace.
by Gergely Nagy 29 Oct '11

29 Oct '11

When compiling with jsonc, use a custom formatter, so that the generated JSON does not have superflous whitespace between elements, and the only whitespace in the JSON is in the content parts. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/tfjson/tfjson.c | 32 ++++++++++++++++++++++++++++++++ 1 files changed, 32 insertions(+), 0 deletions(-) diff --git a/modules/tfjson/tfjson.c b/modules/tfjson/tfjson.c index 65daadd..eba2147 100644 --- a/modules/tfjson/tfjson.c +++ b/modules/tfjson/tfjson.c @@ -28,7 +28,9 @@ #include "config.h" #ifdef HAVE_JSON_C +#include <printbuf.h> #include <json.h> +#include <json_object_private.h> #endif #ifdef HAVE_JSON_GLIB @@ -54,6 +56,35 @@ tf_json_prepare(LogTemplateFunction *self, LogTemplate *parent, } #if HAVE_JSON_C +static int +tf_json_object_to_string (struct json_object *jso, + struct printbuf *pb) +{ + int i = 0; + struct json_object_iter iter; + sprintbuf (pb, "{"); + + json_object_object_foreachC (jso, iter) + { + gchar *esc; + + if (i) + sprintbuf (pb, ","); + sprintbuf (pb, "\""); + esc = g_strescape (iter.key, NULL); + sprintbuf (pb, esc); + g_free (esc); + sprintbuf (pb, "\":"); + if (iter.val == NULL) + sprintbuf (pb, "null"); + else + iter.val->_to_json_string (iter.val, pb); + i++; + } + + return sprintbuf(pb, "}"); +} + static gboolean tf_json_foreach (const gchar *name, const gchar *value, gpointer user_data) { @@ -72,6 +103,7 @@ tf_json_append(GString *result, ValuePairs *vp, LogMessage *msg) struct json_object *json; json = json_object_new_object(); + json->_to_json_string = tf_json_object_to_string; value_pairs_foreach(vp, tf_json_foreach, msg, 0, json); -- 1.7.7.1

2 3

Re: [syslog-ng] syslog-ng 100% cpu utilization issue
by Jain, Vaibhav (GE Healthcare) 29 Oct '11

29 Oct '11

Hi, I am using RHEL 5.5 and the syslog-ng version present in below link is for EL 6. Can you please share syslog-ng 64 bit for RHEL 5.5 http://download.fedora.redhat.com/pub/epel/6/x86_64/repoview/syslog-ng.h tml Thanks in advance for your support. Regards, Vaibhav -----Original Message----- From: Pillai Pramod-a23020 [mailto:pramod.p_pillai@nsn.com] Sent: Friday, October 28, 2011 6:38 PM To: Jain, Vaibhav (GE Healthcare) Subject: RE: [syslog-ng] syslog-ng 100% cpu utilization issue The client and server uses the same software. The client and server is distinguished by the config file. Pramod -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Jain, Vaibhav (GE Healthcare) Sent: Friday, October 28, 2011 6:32 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng 100% cpu utilization issue Thanks for sharing syslog-ng. Can you please share syslog-ng client and server both ? In the below link only one RPM is there. I don't know whether this is client or server. http://download.fedora.redhat.com/pub/epel/6/x86_64/repoview/syslog-ng.h tml -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Thursday, October 27, 2011 8:20 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng 100% cpu utilization issue On Thu, 2011-10-27 at 16:12 +0200, Gergely Nagy wrote: > "Jain, Vaibhav (GE Healthcare)" <Vaibhav.Jain(a)ge.com> writes: > > > Can you please share the syslog-ng client and server RPMs of > > syslog-ng > > 3.2.4 or above 64 bit for RHEL OS? > > http://download.fedora.redhat.com/pub/epel/6/x86_64/repoview/syslog-ng > .html > > If that's for a too new OS version, then I do not know any ready made > RPMs for older RHEL versions. > syslog-ng binaries are supplied as a part of EPEL (community maintained packages for RHEL) for free and as far as I remember syslog-ng is part of that. ======================================================================== ======= shameless plug follows (don't read if you are not interested in paid offerings) ======================================================================== ======= [Please Scroll down] The Premium Edition of syslog-ng does support a wide range of RHEL versions (starting with RHEL2 to RHEL6). With syslog-ng PE 4.0 it support 46 different OS version / architecture combinations, which are quality checked with thourough automatic testing on each. So if you have a wide variety of platforms, you can check that out. Not to mention that in case you have a problem, you can ask support for help. -- Bazsi ________________________________________________________________________ ______ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ________________________________________________________________________ ______ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

2 3

syslog-ng 100% cpu utilization issue
by Jain, Vaibhav (GE Healthcare) 28 Oct '11

28 Oct '11

Hi, I am getting 100% CPU utilization problem on all the syslog-ng client machines. All the clients are pointing to one syslog-ng server. I don't know why this is happening can you please help me on this. PS:I am using syslog-ng 3.0.1 (open source, 64bit). Thanks, Vaibhav

4 6

[Bug 141] New: Crash while reloading config
by bugzilla＠bugzilla.balabit.com 27 Oct '11

27 Oct '11

https://bugzilla.balabit.com/show_bug.cgi?id=141 Summary: Crash while reloading config Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: eric(a)sevenscale.com Type of the Report: --- Estimated Hours: 0.0 We have experienced multiple instances of crashes on reloading a config with 3.3.1. Here is the backtrace: (gdb) where #0 afsocket_sd_remove_and_kill_connection (self=0x0, connection=0x1d64010) at afsocket.c:343 #1 0x00007f3b91790560 in afsocket_sd_kill_connection_list (list=<value optimized out>) at afsocket.c:367 #2 0x00007f3b9225a4f2 in persist_config_entry_free (self=0x1eb9180) at cfg.c:66 #3 0x0000003b62a2dd3e in ?? () from /lib64/libglib-2.0.so.0 #4 0x0000003b62a2e1ad in g_hash_table_remove_all () from /lib64/libglib-2.0.so.0 #5 0x0000003b62a2e245 in g_hash_table_destroy () from /lib64/libglib-2.0.so.0 #6 0x00007f3b9225a78c in persist_config_free (self=0x35de9e0) at cfg.c:83 #7 0x00007f3b922730d4 in main_loop_reload_config_apply () at mainloop.c:491 #8 0x00007f3b9228f81b in iv_signal_event (_this=0x7f3b924bbee0) at iv_signal.c:80 #9 0x00007f3b9228f3a9 in iv_event_raw_got_event (_this=0x7f3b924bbf08) at iv_event_raw.c:82 #10 0x00007f3b9228cdf6 in iv_run_active_list () at iv_main.c:219 #11 iv_main () at iv_main.c:269 #12 0x00007f3b92272c3e in main_loop_run () at mainloop.c:722 #13 0x00000000004015b1 in main () (gdb) p *connection $5 = {super = {ref_cnt = {counter = 2}, flags = 0, cfg = 0x0, pipe_next = 0x1a2e5b0, queue_data = 0x0, queue = 0, init = 0x7f3b917901b0 <afsocket_sc_init>, deinit = 0x7f3b91790470 <afsocket_sc_deinit>, free_fn = 0x7f3b9178fc60 <afsocket_sc_free>, notify = 0x7f3b91790570 <afsocket_sc_notify>}, owner = 0x0, reader = 0x1d64430, sock = 386, peer_addr = 0x0} (gdb) -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

Latest version of syslog-ng open source 64 bit
by Jain, Vaibhav (GE Healthcare) 27 Oct '11

27 Oct '11

Hi All, I want to download the latest version of syslog-ng (client and server) open source 64 bit. Can you please share with me? Thanks Vaibhav

2 1

Request: GitHub mirror of git bazsi/syslog-ng-3.2.git and bazsi/syslog-ng-3.3.git
by Eric Lindvall 26 Oct '11

26 Oct '11

Hello, GitHub has the ability to setup an automated mirror of a remote git repository to allow for users of GitHub to interact with the code more easily. I would really appreciate having such a mirror setup for syslog-ng to make development easier to track in how I work day-to-day. To get the mirror setup, a free Organization needs to be created for "syslog-ng", and then someone needs to send an email to support(a)github.comasking to setup the mirrors. If this is something that interests anyone, I would be more than willing to help out in getting the process completed. Thanks, Eric

2 2

Cannot install syslog-ng 3.2.4 under cygwin with Oracle support
by Hery Fanomezantsoa 26 Oct '11

26 Oct '11

Hi all, Prior to installing syslog-ng on cygwin, i installed libdbi and libdbi-drivers with oracle instant client on windows. After that, i installed eventlog Finally i tried to install syslog-ng and it failed. Here is my configure: ./configure EVTLOG_CFLAGS="-I/usr/local/eventlog/include/eventlog" \ > EVTLOG_LIBS="-L/usr/local/eventlog/lib -levtlog" \ > GLIB_CFLAGS="-I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include" \ > GLIB_LIBS="-L/usr/lib -lglib-2.0 -lintl" \ > LIBDBI_CFLAGS="-I/usr/local/libdbi/include/dbi-0.8" \ > LIBDBI_LIBS="-L/usr/local/libdbi/lib" \ > --enable-sql I've got this warning: *** Warning: This system can not link to static lib archive /usr/local/eventlog/lib/libevtlog.la. *** I have the capability to make that library automatically link in when *** you link to this library. But I can only do this if you have a *** shared version of the library, which you do not appear to have. *** Warning: linker path does not have real file for library -lrt. *** I have the capability to make that library automatically link in when *** you link to this library. But I can only do this if you have a *** shared version of the library, which you do not appear to have *** because I did check the linker path looking for a file starting *** with librt and none of the candidates passed a file format test *** using a file magic. Last file checked: /lib/librt.a *** Warning: linker path does not have real file for library -lresolv. *** I have the capability to make that library automatically link in when *** you link to this library. But I can only do this if you have a *** shared version of the library, which you do not appear to have *** because I did check the linker path looking for a file starting *** with libresolv and none of the candidates passed a file format test *** using a file magic. Last file checked: /lib/libresolv.a *** Warning: linker path does not have real file for library -ldl. *** I have the capability to make that library automatically link in when *** you link to this library. But I can only do this if you have a *** shared version of the library, which you do not appear to have *** because I did check the linker path looking for a file starting *** with libdl and none of the candidates passed a file format test *** using a file magic. Last file checked: /lib/libdl.a *** The inter-library dependencies that have been dropped here will be *** automatically added whenever a program is linked with this library *** or is declared to -dlopen it. *** Since this library must not contain undefined symbols, *** because either the platform does not support them or *** it was explicitly requested with -no-undefined, *** libtool will only create a static version of it. And this error ../../lib/.libs/libsyslog-ng.a(plugin.o): In function `plugin_load_module': /cygdrive/q/STAGE2011/Soft/syslog-ng-3.2.4/lib/plugin.c:154: undefined reference to `_g_module_symbol' /cygdrive/q/STAGE2011/Soft/syslog-ng-3.2.4/lib/plugin.c:172: undefined reference to `_g_module_build_path' /cygdrive/q/STAGE2011/Soft/syslog-ng-3.2.4/lib/plugin.c:206: undefined reference to `_g_module_open' /cygdrive/q/STAGE2011/Soft/syslog-ng-3.2.4/lib/plugin.c:217: undefined reference to `_g_module_make_resident' /cygdrive/q/STAGE2011/Soft/syslog-ng-3.2.4/lib/plugin.c:219: undefined reference to `_g_module_symbol' /cygdrive/q/STAGE2011/Soft/syslog-ng-3.2.4/lib/plugin.c:221: undefined reference to `_g_module_error' /cygdrive/q/STAGE2011/Soft/syslog-ng-3.2.4/lib/plugin.c:210: undefined reference to `_g_module_error' /cygdrive/q/STAGE2011/Soft/syslog-ng-3.2.4/lib/plugin.c:146: undefined reference to `_g_module_open' collect2: ld returned 1 exit status make[5]: *** [pdbtool.exe] Error 1 make[5]: Leaving directory `/cygdrive/q/STAGE2011/Soft/syslog-ng-3.2.4/modules/dbparser' make[4]: *** [all-recursive] Error 1 make[4]: Leaving directory `/cygdrive/q/STAGE2011/Soft/syslog-ng-3.2.4/modules/dbparser' make[3]: *** [all] Error 2 make[3]: Leaving directory `/cygdrive/q/STAGE2011/Soft/syslog-ng-3.2.4/modules/dbparser' make[2]: *** [all-recursive] Error 1 make[2]: Leaving directory `/cygdrive/q/STAGE2011/Soft/syslog-ng-3.2.4/modules' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/cygdrive/q/STAGE2011/Soft/syslog-ng-3.2.4' make: *** [all] Error 2 What do I miss? Thanks in advance.

2 3