- syslog-ng - lists.balabit.hu

Errors building syslog-ng-3.3.7 under FreeBSD 8.3
by C. L. Martinez 08 Nov '12

08 Nov '12

Hi all, I am trying to build syslog-ng 3.3.7 in a FreeBSD 8.3 i386 host, but make returns me these errors: syslog-ng Open Source Edition 3.3.7 configured Compiler options: compiler : gcc -std=gnu99 compiler options : -g -O2 -Wall -pthread -D_REENTRANT -I/usr/local/include/glib-2.0 -I/usr/local/include/eventlog -I/usr/local/include -I/usr/local/include -I$(top_srcdir)/lib/ivykis/lib/include -I$(top_builddir)/lib/ivykis/lib/include -I$(top_srcdir)/lib/ivykis/modules/include -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 linker flags : prefix : /data/soft/syslog-ng linking mode : dynamic __thread keyword : yes Submodules: ivykis : internal libmongo-client : internal Features: Debug symbols : no GCC profiling : no Memtrace : no IPV6 support : yes spoof-source support : yes tcp-wrapper support : yes Linux capability support : no PCRE support : yes Env wrapper support : no systemd support : no (unit dir: none) Modules: Module search path : /data/soft/syslog-ng/lib/syslog-ng Default module list : affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat Sun STREAMS support (module): no SSL support (module) : yes SQL support (module) : no PACCT module (EXPERIMENTAL) : no MongoDB destination (module): yes JSON support (module) : no (using no) root@fbsd02:/tmp/u/syslog-ng-3.3.7# make make all-recursive Making all in lib "Makefile", line 1101: Need an operator make: fatal errors encountered -- cannot continue *** Error code 1 Stop in /tmp/u/syslog-ng-3.3.7. *** Error code 1 Stop in /tmp/u/syslog-ng-3.3.7 My package list: bison-2.5.1,1 A parser generator from FSF, (mostly) compatible with Yacc compat6x-i386-6.4.604000.200810_3 A convenience package to install the compat6x libraries eventlog-0.2.12 Replacement of the simple syslog() API providing structure flex-2.5.37 Fast lexical analyzer generator gamin-0.1.10_4 A file and directory monitoring system gettext-0.18.1.1 GNU gettext package gio-fam-backend-2.28.8_1 FAM backend for GLib\'s GIO library glib-2.28.8_4 Some useful routines of C programming (current stable versi gmake-3.82_1 GNU version of 'make' utility libee-0.3.2 An event expression library inspired by CEE libestr-0.1.2 A library for some string essentials libiconv-1.14 A character set conversion library libnet11-1.1.2.1_4,1 A C library for creating IP packets libsysinfo-0.0.2 GNU libc's sysinfo port for FreeBSD libtool-2.4.2 Generic shared library support script m4-1.4.16_1,1 GNU m4 openssl-1.0.1_4 SSL and crypto library pcre-8.31 Perl Compatible Regular Expressions library perl-5.14.2_2 Practical Extraction and Report Language pkgconf-0.8.8 pkg-config compatible utility which does not depend on glib python27-2.7.3_3 An interpreted object-oriented programming language

2 3

XML entities in PatternDB
by Martin Holste 08 Nov '12

08 Nov '12

I'm trying to find a way to use non-printable characters to delimit messages to avoid problems with escaping message delimiters. However, I'm having trouble representing any special characters in the patterndb.xml file. Specifically, I'd like to use characters like those found here: http://en.wikipedia.org/wiki/List_of_XML_and_HTML_character_entity_referenc… as U+200C. Trying to use the XML entity name, &zwnj; results in an error: "Entity name \'zwnj\' is not known." Is there a place that lists what entities are allowed in the XML? Thanks, Martin

2 1

Please help me on this error --- syslog-ng[4687]: Connection failed; error='Connection refused (111)', time_reopen='10' I don't see where is the problem
by Jian Zhu 05 Nov '12

05 Nov '12

Hi, Anyone know what does this mean? please help Oct 4 17:02:07 loghost syslog-ng[4687]: Connection failed; error='Connection refused (111)', time_reopen='10' I don't see where is the problem -- *Thanks & Best Regards* *James Zhu* *Mobile**:* *+86 18**6 8482 2818* *Email: *zhujian0805(a)gmail.com

2 2

syslog-ng 3.3.7 has been released
by devel＠balabit.hu 31 Oct '12

31 Oct '12

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.3.7 SUMMARY : new stable release DATE : Oct 30, 2012 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.3.7) has been released. For latest fixes in the 3.3.x feature branch you are recommended to upgrade to this version. CHANGES: 3.3.7 Wed, 31 Oct 2012 10:00:00 +0200 Highlights ========== This release is a bug-fix release, correcting a handful of issues discovered after the previous version was tagged. Bugfixes ======== * The bundled ivykis has been updated, fixing the following issues: * Fixed a Solaris-specific issue relating to TCP sources [#190] * Fixed a spinlock issue, triggered on at least FreeBSD [#193] * Workaround an issue with kqueue() on /dev/klog [#201] * The file source was corrected to properly handle character devices. This, and the ivykis update fixes a CPU spinning issue on FreeBSD. [#201] * The sun-streams module had a file descriptor leak, which has been corrected too. [#151] * The glob-based configuration file inclusion was fixed to behave similarly to including a whole directory, to include files alphabetically. [#191] The @include mechanism was also updated to not fail silently, but report an error in certain cases (such as permission errors, or missing files in case of an explicit, non-glob include). [#209] * Fixed a crash when trying to display the available modules in debug mode. [#189] * Fixed the building of afsocket-notls, so that it is correctly built without TLS support. [#188] * Fix compilation without spoof-source. [#192] * Avoid a feedback loop when emitting debug (and trace) messages. [#208] * Various minor fixes around the build system. Features ======== * The stats will now list the filename of unix domain sockets. [#195] Credits ======= syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Anton Koldaev <koldaevav(a)gmail.com> Attila Nagy <bra(a)fsn.hu> Balazs Scheidler <bazsi(a)balabit.hu> Cy Schubert <cy(a)FreeBSD.org> Dave Reisner <dreisner(a)archlinux.org> Evan Rempel <erempel(a)uvic.ca> Gergely Nagy <algernon(a)balabit.hu> Jose Pedro Oliveira <jpo(a)di.uminho.pt> Lennert Buytenhek <buytenh(a)wantstofly.org> Marvin Nipper <marvin.nipper(a)stream.com> Michael Hocke <michael.hocke(a)nyu.edu> Peter Czanik <czanik(a)balabit.hu> Tamas Pal <folti(a)balabit.hu> Yorick Peterse DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

3 2

[Bug 203] New: patterndb rule tag can not have a hyphen in the class attribute
by bugzilla＠bugzilla.balabit.com 28 Oct '12

28 Oct '12

https://bugzilla.balabit.com/show_bug.cgi?id=203 Summary: patterndb rule tag can not have a hyphen in the class attribute Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 Just seems odd that the rule tag can not have a hyphen in the class attribute. The error is element rule: Schemas validity error : Element 'rule', attribute 'class': [facet 'pattern'] The value 'sm-mta' is not accepted by the pattern '[a-zA-Z0-9_\.]+'. I would expect (and prefer) that the pattern should be '[a-zA-Z0-9_/\.+:-]' <ruleset name="sm-mta" id="RS-8e86ff39-2db4-48b6-bbe0-f980d9afebf0"> <pattern>sm-mta</pattern> <rules> <rule id="8e86ff39-2db4-48b6-bbe0-f980d9afebf0" class="sm-mta" provider="UVic"> <patterns> <pattern>whatever pattern</pattern> </patterns> </rule> </rules> </ruleset"> will reproduce this error. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

6 16

syslog-ng running help
by saeed salim 28 Oct '12

28 Oct '12

dear all, am new to the list and i want a help . i installed syslog-ng on ubuntu 11.10 when i run the application ,i got this error message ( Excess of number arguments ) any help? Thanks

3 7

[Bug 210] New: syslog-ng drops its capabilities before reading the config files
by bugzilla＠bugzilla.balabit.com 27 Oct '12

27 Oct '12

https://bugzilla.balabit.com/show_bug.cgi?id=210 Summary: syslog-ng drops its capabilities before reading the config files Product: syslog-ng Version: 3.3.x Platform: All OS/Version: All Status: ASSIGNED Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: algernon(a)balabit.hu ReportedBy: algernon(a)balabit.hu Type of the Report: --- Estimated Hours: 0.0 When reading config files, syslog-ng should be running with full capabilities (or at least enough to read root owned 0600 files), whether that happens during initial startup or on reload. (Via Yorick Peterse) -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 209] New: @include can silently fail
by bugzilla＠bugzilla.balabit.com 27 Oct '12

27 Oct '12

https://bugzilla.balabit.com/show_bug.cgi?id=209 Summary: @include can silently fail Product: syslog-ng Version: 3.3.x Platform: All OS/Version: All Status: ASSIGNED Severity: major Priority: unspecified Component: syslog-ng AssignedTo: algernon(a)balabit.hu ReportedBy: algernon(a)balabit.hu Type of the Report: --- Estimated Hours: 0.0 With the glob-based @includes, we can silently ignore files that could not be opened, which can very well hide problems. Case in point: /etc/syslog-ng/conf.d/ with 0750 mode and a single file within it, syslog-ng.conf has @include "/etc/syslog-ng/conf.d/some-file.conf" In this case, we never see some-file.conf being tried, even though it is explicitly included (it still goes through the glob code), and glob() returns GLOB_NOMATCH, with errno set to EPERM. In this case, we should abort. An @include should only succeed if there was nothing to include (in case of a globby include), or if everything went well. That we can't read a file, is an error, and should be treated as such. (Via Yorick Peterse on twitter) -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 206] New: program_override() and host_override() do not work in 3.4
by bugzilla＠bugzilla.balabit.com 27 Oct '12

27 Oct '12

https://bugzilla.balabit.com/show_bug.cgi?id=206 Summary: program_override() and host_override() do not work in 3.4 Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: ASSIGNED Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: algernon(a)balabit.hu ReportedBy: algernon(a)balabit.hu Type of the Report: --- Estimated Hours: 0.0 The *Options simplification change (d93d35857e3acac5eb59a9735d8947dedf43a511) broke host_override () & program_override (), because they host_override_len and program_override_len properties are no longer initialized to -1, but to 0, so when setting an override, a value with 0 length will be set, emptying the variable instead of setting it to a new value. This should be fairly easy to fix, this bug is a reminder for myself to not forget it. But if anyone happens to get around to submit a patch before I get there, so much the better! -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[PATCH 0/2]: afamqp: AMQP destination driver
by Gergely Nagy 27 Oct '12

27 Oct '12

The two patches that will follow implement an AMQP destination driver for syslog-ng, contributed by Attila Nagy <bra(a)fsn.hu>, with a few minor enhancements and porting to syslog-ng 3.4 by myself. The patch is also available on the feature/3.4/amqp[1] and feature/3.3/amqp[2] branches of my git repository, for syslog-ng 3.4 and 3.3 respectively. Functionally, they're exactly the same, but all further improvements and fixes will go to the 3.4 branch only, the 3.3 branch is there for testing and historical purposes only, and will be removed sometime after the driver gets merged to 3.4. Now, I won't explain in detail what AMQP is, let it suffice that it is a messaging protocol, of which RabbitMQ[3] is an implementation of. Interested people can read the available documentation, there's plenty! For now, lets concentrate on how to take the driver for a test drive! Once rabbitmq is installed (on debian and derivatives of recent versions it is as simple as apt-get install rabbitmq-server), all one needs to do, is configure syslog-ng to send messages, and grab a client that can look at the queue. For testing purposes, there's a simple one at https://gist.github.com/3859756 - you'll need a couple of python modules for that. What it does, is it sets up a queue named 'test', and binds it to the 'syslog' exchange (the default used by the AMQP driver), and starts consuming messages. It prints both headers and the payload (if any, by default, there is no payload). What one needs to know about the driver, is that it does not declare any queues: it's the responsibility of the admin to set queues up, and route messages as they see fit. This can be done on the RabbitMQ web admin interface (again, see the docs about that!), among other things, or programmatically, like the script above does. Hopefully, this is enough to get one started! If not, I plan to write a more detailed blog post about the topic in the not too distant future. There are plans to write an accompanying source driver too, but work has not started on that front yet. Rest assured though, it will come. I'd like to thank Attila again for contributing the driver, and answering my silly questions during review - both are much appreciated! [1]: https://github.com/algernon/syslog-ng/tree/feature/3.4/amqp [2]: https://github.com/algernon/syslog-ng/tree/feature/3.3/amqp [3]: http://www.rabbitmq.com/

2 3

[PATCH] logreader: When following a file, treat non-regular files specially
by Gergely Nagy 27 Oct '12

27 Oct '12

When following a file with follow-freq(), and it is not a regular file, but something for which stat(2) always returns 0 size (such as the /dev/klog character device on FreeBSD), log_reader_io_follow_file() never went into the process input branch, because the current position (0) was always at EOF (also 0). To break out of this unfortunate situation, if the file to read from is not a regular file, go into the process input loop anyway: at worst, it will do an expensive noop. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/logreader.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/logreader.c b/lib/logreader.c index eb52330..1e2ea5e 100644 --- a/lib/logreader.c +++ b/lib/logreader.c @@ -276,7 +276,7 @@ log_reader_io_follow_file(gpointer s) evt_tag_int("size", st.st_size), NULL); - if (pos < st.st_size) + if (pos < st.st_size || !S_ISREG(st.st_mode)) { /* we have data to read */ log_reader_io_process_input(s); -- 1.7.10.4

2 1

Collect log from beginning incase central server's log file gets deleted.
by sagar naravane 27 Oct '12

27 Oct '12

Hi, Below is my setup (see the config details). Issue i am facing is : Lets say on central servers apache logs are collected till 2:00 PM for current date. Incase if i delete the log file on central server, a new file gets created and starts collecting the log after the delete time ie 2:00 PM. What i want to achieve is that in such an event where the log files gets deleted.. the new log-file will collect the log from the beginning of the day ie 12:00 AM. Is this possible ? *************** client config *********************** @include "scl.conf" options { keep_hostname(yes); log_fifo_size(100000) }; rewrite r_setfilename { set( "$FILE_NAME", value(".SDATA.file(a)18372.4.name") ); }; rewrite r_apache_basepath_www-dev01 { subst( "/apps/log/apache/www-dev01/", "", value(".SDATA.file(a)18372.4.name") type("string") flags("prefix") ); }; source s_apache_www-dev01 { file("/apps/log/apache/www-dev01/*.log" program_override("apache/www/dev01") flags(no-parse) recursive(yes) ); }; destination d_server { syslog("x.x.x.x" transport("tcp") port(514)); }; log { source(s_apache_www-dev01); rewrite(r_setfilename); rewrite(r_apache_basepath_www-dev01); destination(d_server); }; *************** Server Config ************ @include "scl.conf" options { create_dirs (yes); dir_perm(0755); dir_group(60030); dir_owner(wwwtools); perm(0755); owner(wwwtools); group(60030); keep_hostname(yes); threaded(yes); log_msg_size(5000000); }; source s_network { syslog(ip(0.0.0.0) port(514) transport("tcp") max-connections(200)); }; filter f_apache_www-dev01 { program('apache/www/dev01'); }; destination d_central { file("/apps/log/syslog-ng/${PROGRAM}/${YEAR}${MONTH}${DAY}/${HOST}/${. SDATA.file(a)18372.4.name}" template("${MSGONLY}\n") ); }; log { source(s_network); filter(f_apache_www-dev01); destination(d_central); }; -- Regards, Sagar Naravane

3 3

[Bug 194] New: Can't open libsyslog-ng-crypto. so during module loading on Fedora
by bugzilla＠bugzilla.balabit.com 27 Oct '12

27 Oct '12

https://bugzilla.balabit.com/show_bug.cgi?id=194 Summary: Can't open libsyslog-ng-crypto.so during module loading on Fedora Product: syslog-ng Version: 3.3.x Platform: All OS/Version: Linux Status: ASSIGNED Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: algernon(a)balabit.hu ReportedBy: algernon(a)balabit.hu CC: algernon(a)balabit.hu Type of the Report: --- Estimated Hours: 0.0 This clone is to track the library path issue separately. +++ This bug was initially created as a clone of Bug #188 +++ Linux distro: Fedora 17 x86_64 Syslog-ng source tarball: syslog-ng-3.3.5.90-20120810-v3.3.5-43-g19f6c.tar.gz we have encountered a couple of problems when running syslog-ng-3.3.5.90-20120810 under Fedora 17 x86_64: 1) ld.so library path 2) undefined symbol: tls_context_setup_session Steps to reproduce the problem: 1) just run "/usr/sbin/syslog-ng --version" ---------- # /usr/sbin/syslog-ng --version syslog-ng 3.3.5.90 Installer-Version: 3.3.5.90 Revision: Compile-Date: Aug 17 2012 22:09:52 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat,afsql Error opening plugin module; module='afsocket', error='libsyslog-ng-crypto.so: cannot open shared object file: No such file or directory' Error opening plugin module; module='afsocket-notls', error='//usr/lib64/syslog-ng/libafsocket-notls.so: undefined symbol: tls_context_setup_session' Error opening plugin module; module='afsocket-tls', error='libsyslog-ng-crypto.so: cannot open shared object file: No such file or directory' Available-Modules: dummy,afsql,tfjson,dbparser,csvparser,convertfuncs,afuser,basicfuncs,afmongodb,confgen,syslogformat,affile,afprog Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on ---------- The library path problem can be solved by creating the file "/etc/ld.so.conf.d/syslog-ng.conf" containing the syslog-ng module directory ( /usr/lib64/syslog-ng in this particular case ). Running syslog-ng under strace without touching the ld.so library paths produces: ---------- # # strace -e trace=file /usr/sbin/syslog-ng --version ... access("//usr/lib64/syslog-ng/libafsocket.so", F_OK) = 0 stat("//usr/lib64/syslog-ng/libafsocket.so", {st_mode=S_IFREG|0755, st_size=103560, ...}) = 0 open("//usr/lib64/syslog-ng/libafsocket.so", O_RDONLY|O_CLOEXEC) = 4 open("/lib64/tls/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) open("/lib64/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) open("/usr/lib64/tls/x86_64/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) stat("/usr/lib64/tls/x86_64", 0x7fff0f81fcb0) = -1 ENOENT (No such file or directory) open("/usr/lib64/tls/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) stat("/usr/lib64/tls", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 open("/usr/lib64/x86_64/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) stat("/usr/lib64/x86_64", 0x7fff0f81fcb0) = -1 ENOENT (No such file or directory) open("/usr/lib64/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) stat("/usr/lib64", {st_mode=S_IFDIR|0555, st_size=49152, ...}) = 0 Error opening plugin module; module='afsocket', error='libsyslog-ng-crypto.so: cannot open shared object file: No such file or directory' ... ----------- Note that all syslog-ng modules with the exception of libsyslog-ng-crypto.so are loaded from the syslog-ng modules directory. Output of "syslog-ng --version" after creating the file "/etc/ld.so.conf.d/syslog-ng.conf" ---------- # /usr/sbin/syslog-ng --version syslog-ng 3.3.5.90 Installer-Version: 3.3.5.90 Revision: Compile-Date: Aug 17 2012 22:09:52 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat,afsql Error opening plugin module; module='afsocket-notls', error='//usr/lib64/syslog-ng/libafsocket-notls.so: undefined symbol: tls_context_setup_session' Available-Modules: dummy,afsocket,afsql,tfjson,dbparser,csvparser,convertfuncs,afuser,basicfuncs,afmongodb,afsocket-tls,confgen,syslogformat,affile,afprog Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on ---------- /jpo -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 6

Syslog-ng OSE multiline messages support
by Daniel Josue Rodriguez Paiz 25 Oct '12

25 Oct '12

Hello, I have one question, does syslog-ng OSE support multiline parsing logs? i have one applications that send multiline messages and syslog-ng save the log of the first line only. Thanks for your support in adavnce, Daniel

3 4

[Bug 207] New: HOST macro contains different data inside db_parser than in the config
by bugzilla＠bugzilla.balabit.com 23 Oct '12

23 Oct '12

https://bugzilla.balabit.com/show_bug.cgi?id=207 Summary: HOST macro contains different data inside db_parser than in the config Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 When using the $HOST macro in the config file of syslog-ng the macro contains source@hostname type of data. When using the $HOST macro inside the db_parser the macro contains just the hostname. This is confusing. I am not how fix this. I would not like to see the removal of the "source@" portion, but I do want to have the raw hostname available. Perhaps having a macro of $SOURCE that would have "source@hostname" and make the $HOST have just the hostname. The FULLHOST will still have the chain of hosts, including the $SOURCE, just like it has now. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

3 5

Please help: Can not figure out why syslog-ng ose 3.2.2 server doesn't receive syslog client messages over udp/11514 on 64-bit RHEL 5
by Mehdi Mortezai 19 Oct '12

19 Oct '12

Hello, I am new to syslog-ng. I have set up a syslog-ng ose 3.2.2 server and configured syslog on 64-bit RHEL 5 client servers to send all messages to the syslog-ng server over udp/11514 (client's syslog.conf: *.* @syslog-ngIP:11514) but no packet is received. When I change the port on the syslog-ng server and on the client's syslog from udp/11514 to udp/514 all works. Verified that firewalls passing packets for both udp/514 and 11514 on the server and RHEL client and the port is not used by any other app. I cannot figure out why the logs are not happening when changing the port to udp/11514 from default? Is there any global options I should set or any change to my source statements? Here are the global options and sources definitions: *options* { time_reopen (10); long_hostnames (off); owner("root"); group("root"); perm(0600); dir_perm(0755); use_dns (yes); use_fqdn (no); create_dirs (yes); keep_hostname (no); }; *source:* ########### Define Sources *### Internal * source s_local_sys { file ("/proc/kmsg" program_override("kernel: ")); unix-stream ("/dev/log"); internal();}; *### External or remote:* # For Server Groups source s_RHEL5_udp { udp(port(11514));}; *# For all other single hosts:* source s_remote_udp { udp(port(514));}; Thanks in advance for your help. Medi

1 0

GeoIP template function
by Csaba Major 19 Oct '12

19 Oct '12

Hi, I have a system where I have a lot of logs with IP addresses (typically firewall logs), and the requirement was to also include the country name in the logs. I found that a geoip based template function would be an easy way for this, so I created a PoC for that. It works pretty well, however some improvements and performance measurement would be necessary. The created patch is attached (based on 3.3.5), some may find it useful :) (It requires libgeoip, and a geoip database, of course) Regards, Csaba

4 6

Can't get kernel messages on FreeBSD
by Tomoki BENIYA 19 Oct '12

19 Oct '12

Hi all. I installed syslog-ng3.3.6_3 using ports on FreeBSD9.0-RELEASE(i386). I can't get kernel messages by default configuration. ex. em0: link state changed to DOWN em0: link state changed to UP But, I can see them using 'cat /dev/klog' after stoppedsyslog-ng. So, I think syslog-ng can't get kermel messages from /dev/klog. How to get kernel messages using syslog-ng? Please give me a hint or something! * /etc/rc.conf syslogd_enable="NO" syslog_ng_enable="YES" * /usr/local/etc/syslog-ng.conf source s_kern { file("/dev/klog"); }; filter f_kern { facility(kern); }; destination d_kern { file("/var/log/kern.log"); }; log { source(s_kern); filter(f_kern); destination(d_kern); }; -- Tomoki BENIYA <beniya(a)bit-isle.co.jp>

2 2

syslog-ng 3.05 receive but not write on filesystem
by FIORENZI ALESSANDRO 17 Oct '12

17 Oct '12

Hi, I am experiencing e problem that havent have before. Are 3 years that all seems to work fine with syslog-ng 3.05. recently I have find that some of my servers sends syslog messages to syslog server, Syslog server interface recive it, syslog-ng elaborate it but do not write on file system. I have configured syslog-ng server to log message in the following manner /opt/logs/$YEAR/$MONTH/$DAY/SERVER/$HOST/messages What I have done: - Listen on eth4 to look for udp syslog messages coming form server A - Try to access via ssh to server A - I have seen udp packet from A to syslog-server. Than I have investigate on syslog-ng - I have get the pid of syslog-ng and make a strace -p pid -o prova.txt - Try to access via ssh to server A (to let server a send udp syslog message to syslog server) - Stop strace - Look on prova.txt for IP of server A and find It! But I cannot do more, can someone help me? Alessandro Fiorenzi Prima di stampare, pensa all'ambiente ** Think about the environment before printing ________________________________ Il presente messaggio, inclusi gli eventuali allegati, ha natura aziendale e potrebbe contenere informazioni confidenziali e/o riservate. Chiunque lo ricevesse per errore, ? pregato di avvisare tempestivamente il mittente e di cancellarlo. E' strettamente vietata qualsiasi forma di utilizzo, riproduzione o diffusione non autorizzata del contenuto di questo messaggio o di parte di esso. Pur essendo state assunte le dovute precauzioni per ridurre al minimo il rischio di trasmissione di virus, si suggerisce di effettuare gli opportuni controlli sui documenti allegati al presente messaggio. Non si assume alcuna responsabilit? per eventuali danni o perdite derivanti dalla presenza di virus. Per lo svolgimento delle attivit? di investimento nel Regno Unito, la societ? ? autorizzata da Banca d'Italia ed ? soggetta alla vigilanza limitata della Financial Services Authority. Maggiori informazioni in merito ai poteri di vigilanza della Financial Services Authority sono a disposizione previa richiesta.. Nel Regno Unito Intesa Sanpaolo S.p.A. opera attraverso la filiale di Londra, sita in 90 Queen Street, London EC4N 1SA, registrata in Inghilterra & Galles sotto No.FC016201, Branch No.BR000036 In osservanza dei requisito imposti dal Internal Revenue Service (Agenzia delle Entrate degli Stati Uniti), qualunque discussione relativa a temi di natura fiscale contenuta in questo messaggio o nei suoi allegati non e' intesa ne' e' stata scritta per essere utilizzata, ne' puo' essere utilizata per (i) evitare l'imposizione di gravami fiscali secondo il codice tributario vigente negli Stati Uniti o (ii) per promuovere, sollecitare o raccomandare una operazione finanziaria o altra transazione indirizzata ad un altro destinatario. *** ________________________________ This email (including any attachment) is a corporate message and may contain confidential and/or privileged and/or proprietary information. If you have received this email in error, please notify the sender immediately, do not use or share it and destroy this email. Any unauthorised use, copying or disclosure of the material in this email or of parts hereof (including reliance thereon) is strictly forbidden. We have taken precautions to minimize the risk of transmitting software viruses but nevertheless advise you to carry out your own virus checks on any attachment of this message. We accept no liability for loss or damage caused by software viruses. For the conduct of investment business in the UK, the Company is authorised by Banca d'Italia and subject to limited regulation in the UK by the Financial Services Authority. Details about the extent of our regulation by the Financial Services Authority are available from us on request. In the UK Intesa Sanpaolo S.p.A. operates through its London Branch, located at 90 Queen Street, London EC4N 1SA. Registered in England & Wales under No.FC016201, Branch No.BR000036 To comply with requirements imposed by the IRS, we inform you that any discussion of U.S. federal tax issues contained herein (including any attachments) was not intended or written to be used, and cannot be used by you, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending any transaction or matter addressed herein to another party.

2 1

Debian package news for syslog-ng 3.3 and 3.4
by Gergely Nagy 16 Oct '12

16 Oct '12

Hi! I've been providing syslog-ng 3.3 packages for a while[1], more than a year, actually! And since I announced my intention to package syslog-ng 3.4 in January[2], I believe it's time to give an update on where things stand, and what to expect from the repository in the coming weeks, months. There are two important news I want to share: - Ubuntu Natty (11.04) support will soon be discontinued. Official support for the platform ends this month, the repository will not see any further updates. I plan to completely remove it by December, but the most recent package already built (3.3.7RC2) will be available until then. (Though, since building for Natty is easy, I might end up uploading 3.3.7 final, before the nail hits the coffin.) - syslog-ng 3.4 packages are coming, built from my sandbox branch, which is a collection of everything that's intended to reach the master branch, and are in already usable state. These include: an amqp destination, indented-multiline and linux 3.5+ /dev/kmsg support, along with a host of bugfixes. The location of the repository will be: deb http://packages.madhouse-project.org/$VENDOR $RELEASE syslog-ng-devel Where $VENDOR is debian or ubuntu, and release is the particular OS release (wheezy, precise, etc). Initially packages will be provided for Ubuntu Oneiric, Precise and Quantal, along with Debian Squeeze, Wheezy and unstable. Ubuntu Lucid and Natty were dropped from the list compared to 3.3. Natty due to reaching its end of life, and Lucid because backporting all the required bits and pieces to it took up more time than all the rest combined. If there's a significant interest in syslog-ng 3.4 packages for Ubuntu Lucid, please let me know. A mass of demanding users can always convince me that the pain of backporting to lucid is worth it. The syslog-ng 3.3.7RC2 packages are already in the repositories, 3.4.0alpha3 + patches is building as I write this, and it's very likely they'll hit my archive tomorrow latest. The syslog-ng-devel component is already created, and I'll soon start populating it with the required dependencies on platforms that need them. I've yet to figure out how to change my repository page[3] to allow convenient selection of the various bits and pieces, so the information there may lag behind for a little while. [1]: http://article.gmane.org/gmane.comp.syslog-ng/11900 [2]: http://article.gmane.org/gmane.comp.syslog-ng/12990 [3]: http://asylum.madhouse-project.org/projects/debian/ Finally, I'd like to share a few interesting statistics, a result of a few greps through my server logs: Since August 20, 2011, there have been 119 unique downloaders of the Debian packages, 614 downloads in total (spanning all versions from 3.3.0beta2 up until 3.3.6.91). For Ubuntu, these numbers are 452 unique downloaders, and 1563 downloads in total. I'm quite happy about those numbers, these debs are by far the most downloaded things I have created in the past decade (not counting one particular package in the Debian archive I try to forget). :) So a huge "Thank you!" goes out to everyone who uses these packages! -- |8]

1 1

syslog-ng 3.3.7 RC2 (3.3.6.91) released
by Gergely Nagy 15 Oct '12

15 Oct '12

Hi! While the 3.3.7 RC1 looked promising, a few more issues turned up meanwhile, so I am releasing an RC2 aswell. With the credits section of the NEWS file being longer than the list of fixes, my hope is that the 3.3.7 final release (due october 31) will not be much more than a version bump. As usual, testing this release candidate and reporting any and all issues found with it - be those regressions or anything else - would be greatly appreciated! The current state of the NEWS file is included below, and if anyone wants to track how the release is progressing and what needs to be fixed still before the final release, see the 3.3.7 target milestone on bugzilla: https://bugzilla.balabit.com/buglist.cgi?query_format=advanced&target_miles… The release is available from the following sources: GitHub ------ The release is tagged as v3.3.6.91 on the official repository at git://github.com/balabit/syslog-ng-3.3.git A detailed list of commits & changes is available via the compare view: https://github.com/balabit/syslog-ng-3.3/compare/v3.3.6.90...v3.3.6.91 Tarball ------- A make dist tarball built from the same tag is available from my package repository: http://packages.madhouse-project.org/syslog-ng/3.3/3.3.6.91/syslog-ng-3.3.6… And finally, the NEWS file from the release: 3.3.6.91 (3.3.7 RC2) Mon, 08 Oct 2012 13:00:00 +0200 Bugfixes ======== * The bundled ivykis has been updated, the new version contains a workaround for an issue specific to FreeBSD 9 only. * The file source was corrected to properly handle character devices. This, and the ivykis update fixes a CPU spinning issue on FreeBSD. [#201] * The sun-streams module had a file descriptor leak, which has been corrected too. [#151] * Various minor fixes around the build system. Features ======== * The stats will now list the filename of unix domain sockets. [#195] Credits ======= syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Attila Nagy <bra(a)fsn.hu> Balazs Scheidler <bazsi(a)balabit.hu> Evan Rempel <erempel(a)uvic.ca> Gergely Nagy <algernon(a)balabit.hu> Lennert Buytenhek <buytenh(a)wantstofly.org> Michael Hocke <michael.hocke(a)nyu.edu> Peter Czanik <czanik(a)balabit.hu> -- |8]

3 6

[PATCH (3.4)] json: The test should not depend on the environment's timezone
by Gergely Nagy 13 Oct '12

13 Oct '12

The $(format-json) test was failing when a timezone was set in the environment that did not match the tests expectations. To fix this, explicitly set the timezone to UTC. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/json/tests/test_json.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/modules/json/tests/test_json.c b/modules/json/tests/test_json.c index f188b7f..15d038b 100644 --- a/modules/json/tests/test_json.c +++ b/modules/json/tests/test_json.c @@ -7,7 +7,7 @@ test_format_json(void) { assert_template_format("$(format-json MSG=$MSG)", "{\"MSG\":\"árvíztűrőtükörfúrógép\"}"); assert_template_format_with_context("$(format-json MSG=$MSG)", "{\"MSG\":\"árvíztűrőtükörfúrógép\"}{\"MSG\":\"árvíztűrőtükörfúrógép\"}"); - assert_template_format("$(format-json --scope rfc3164)", "{\"PROGRAM\":\"syslog-ng\",\"PRIORITY\":\"err\",\"PID\":\"23323\",\"MESSAGE\":\"árvíztűrőtükörfúrógép\",\"HOST\":\"bzorp\",\"FACILITY\":\"local3\",\"DATE\":\"Feb 11 19:58:35\"}"); + assert_template_format("$(format-json --scope rfc3164)", "{\"PROGRAM\":\"syslog-ng\",\"PRIORITY\":\"err\",\"PID\":\"23323\",\"MESSAGE\":\"árvíztűrőtükörfúrógép\",\"HOST\":\"bzorp\",\"FACILITY\":\"local3\",\"DATE\":\"Feb 11 18:58:35\"}"); assert_template_format("$(format-json msg.text=$MSG msg.id=42 host=bzorp)", "{\"msg\":{\"text\":\"árvíztűrőtükörfúrógép\",\"id\":\"42\"},\"host\":\"bzorp\"}"); assert_template_format("$(format-json msg.text.str=$MSG msg.text.len=42 msg.id=42 host=bzorp)", "{\"msg\":{\"text\":{\"str\":\"árvíztűrőtükörfúrógép\",\"len\":\"42\"},\"id\":\"42\"},\"host\":\"bzorp\"}"); @@ -17,6 +17,8 @@ int main(int argc G_GNUC_UNUSED, char *argv[] G_GNUC_UNUSED) { app_startup(); + putenv("TZ=UTC"); + tzset(); init_template_tests(); plugin_load_module("json-plugin", configuration, NULL); -- 1.7.10.4

1 0

[PATCH (3.4)] json-format: Unbreak $(format-json)
by Gergely Nagy 13 Oct '12

13 Oct '12

The $(format-json) template function was misbehaving under certain conditions, namely where we had a sub-object, a simple value, and another object starting right after: in this case, no comma was placed between the simple value and the name of the new sub-object. The fix changes how comma placement is done completely: it is no longer a mix of prefix_data & user_data magic, but all state is kept in the user data, with the _obj_start(), _obj_end() and _value() functions setting it as appropriate: - When starting a new object, if we need a comma, we'll put one, and turn the comma flag off (the first item within an object won't need a comma). - When ending an object, we never add a comma, but the next item will need the comma flag set. - When adding a value, if the comma flag is set, we add one, and in the end, set the flag ourselves, as the next one will need a comma too. This is much simpler than the previous hackery, and also results in correct JSON in every situation. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/json/format-json.c | 27 +++++---------------------- modules/json/tests/test_json.c | 2 ++ 2 files changed, 7 insertions(+), 22 deletions(-) diff --git a/modules/json/format-json.c b/modules/json/format-json.c index 4f0f01c..e17bcb0 100644 --- a/modules/json/format-json.c +++ b/modules/json/format-json.c @@ -118,14 +118,8 @@ tf_json_obj_start(const gchar *name, gpointer user_data) { json_state_t *state = (json_state_t *)user_data; - gboolean need_comma = FALSE; - if (prefix_data) - need_comma = GPOINTER_TO_INT(*prefix_data); - else - need_comma = state->need_comma; - - if (need_comma) + if (state->need_comma) g_string_append_c(state->buffer, ','); if (name) @@ -133,13 +127,11 @@ tf_json_obj_start(const gchar *name, g_string_append_c(state->buffer, '"'); g_string_append_escaped(state->buffer, name); g_string_append(state->buffer, "\":{"); - state->need_comma = TRUE; } else g_string_append_c(state->buffer, '{'); - if (prefix_data) - *prefix_data=GINT_TO_POINTER(0); + state->need_comma = FALSE; return FALSE; } @@ -152,11 +144,10 @@ tf_json_obj_end(const gchar *name, { json_state_t *state = (json_state_t *)user_data; - if (prev_data) - *prev_data = GINT_TO_POINTER(1); - g_string_append_c(state->buffer, '}'); + state->need_comma = TRUE; + return FALSE; } @@ -164,18 +155,10 @@ static gboolean tf_json_value(const gchar *name, const gchar *prefix, const gchar *value, gpointer *prefix_data, gpointer user_data) { - gboolean need_comma = FALSE; json_state_t *state = (json_state_t *)user_data; - if (prefix_data) - need_comma = GPOINTER_TO_INT(*prefix_data); - else - need_comma = state->need_comma; - - if (need_comma) + if (state->need_comma) g_string_append_c(state->buffer, ','); - else if (prefix_data) - *prefix_data = GINT_TO_POINTER(1); g_string_append_c(state->buffer, '"'); g_string_append_escaped(state->buffer, name); diff --git a/modules/json/tests/test_json.c b/modules/json/tests/test_json.c index f188b7f..2a88c63 100644 --- a/modules/json/tests/test_json.c +++ b/modules/json/tests/test_json.c @@ -11,6 +11,8 @@ test_format_json(void) assert_template_format("$(format-json msg.text=$MSG msg.id=42 host=bzorp)", "{\"msg\":{\"text\":\"árvíztűrőtükörfúrógép\",\"id\":\"42\"},\"host\":\"bzorp\"}"); assert_template_format("$(format-json msg.text.str=$MSG msg.text.len=42 msg.id=42 host=bzorp)", "{\"msg\":{\"text\":{\"str\":\"árvíztűrőtükörfúrógép\",\"len\":\"42\"},\"id\":\"42\"},\"host\":\"bzorp\"}"); + assert_template_format("$(format-json kernel.SUBSYSTEM=pci kernel.DEVICE.type=pci kernel.DEVICE.name=0000:02:00.0 MSGID=801 MESSAGE=test)", + "{\"kernel\":{\"SUBSYSTEM\":\"pci\",\"DEVICE\":{\"type\":\"pci\",\"name\":\"0000:02:00.0\"}},\"MSGID\":\"801\",\"MESSAGE\":\"test\"}"); } int -- 1.7.10.4

1 0

[RFC (3.4) 0/5]: multiline & linux 3.5+ /dev/kmsg support
by Gergely Nagy 13 Oct '12

13 Oct '12

The next few patches gradually introduce two - in my opinion - interesting features for syslog-ng 3.4: support for one kind of multiline messages, where continuation lines start with whitespace; and support for the /dev/kmsg format introduced in linux 3.5. Neither implementation is complete yet, there are subtle issues with both (see later), but they're at a stage where taking them to a test drive would be useful, and code review wouldn't hurt either. The patches are also available on the feature/3.4/indented-multiline[1] branch of my git repository[2]. The branch may get rebased in the future, I do not make any promise of keeping it fast-forwardable. [1]: https://github.com/algernon/syslog-ng/commits/feature/3.4/indented-multiline [2]: git://github.com/algernon/syslog-ng.git linux 3.5 /dev/kmsg =================== The linux 3.5+ /dev/kmsg support is the easier to test: you only need a 3.5+ kernel, and when using the system() source, things will magically work! This includes parsing the relative timestamps in kmsg messages and also parsing any additional key-value pairs. This means that if we had an input line like this: ,---- | 6,802,65338577;ATL1E 0000:02:00.0: eth0: NIC Link is Up <100 Mbps Full Duplex> | SUBSYSTEM=pci | DEVICE=+pci:0000:02:00.0 `---- Then we'll get back something like this: ,---- | { | "FILE_NAME" : "/dev/kmsg", | "MSGID" : "802", | "HOST" : "luthien", | "DATE" : "Oct 13 16:53:44", | "FACILITY" : "kern", | "SOURCEIP" : "127.0.0.1", | "TAGS" : ".source.s_kmsg", | "kernel" : { | "SUBSYSTEM" : "pci", | "DEVICE" : { | "name" : "0000:02:00.0", | "type" : "pci" | } | }, | "PRIORITY" : "info", | "HOST_FROM" : "luthien", | "MESSAGE" : "ATL1E 0000:02:00.0: eth0: NIC Link is Up <100 Mbps Full Duplex>", | "PROGRAM" : "kernel" | } `---- For the above, the following template was used: $(format-json --scope selected-macros --scope nv-pairs --key .kernel.* --shift 1) (Mind you, program_override() is currently broken in 3.4, but that is unrelated to the /dev/kmsg support) Any name-value pairs the kernel supplies will be put into variables of the same name, prefixed with ".kernel." (so SUBSYSTEM becomes .kernel.SUBSYSTEM), and the value of DEVICE= will be further parsed, based on rules written down in the appropriate kernel docs. Indented multiline ================== The indented multiline support enhances the file() and tcp() sources with the ability to read multiline records (if the indented-multiline flag is set), where continuation lines start with whitespace. Such output is produced by the $(indent-multi-line) template function. Usage: source s_ml { tcp(port(12345) flags(indented-multiline, no-parse)); }; Then, you can test it like this: (cat <<EOF This is the first line - second - third EOF ) | nc localhost 12345 Using a JSON output, this would result in: {"MESSAGE":"This is the first line\n - second\n - third"} A record can be terminated in two ways: by another line that does not begin with whitespace, or by reacing EOF. This means that if netcat didn't close the connection, we wouldn't get the test record until another line arrives. Unfortunately, there's not much I can do about this limitation, but if anyone has a good idea how to work around or fix it, please let me know! Known issues ============ The /dev/kmsg handling is not reload/restart-safe: if syslog-ng gets reloaded or restarted, it will read the whole lot of /dev/kmsg over again, for example. There are possibly other subtle issues too, but all of those will be ironed out before I submit the patches for merging. The indented-multiline support has a more serious limitation: if data needs to be split (which happens in some circumstances when the incoming data is bigger than the internal message size (8192 by default)), it is not correctly reassembled, making indented-multiline less than useful for some tasks.

1 5

[Bug 202] New: pdbtool dump does not recognize the -p option
by bugzilla＠bugzilla.balabit.com 13 Oct '12

13 Oct '12

https://bugzilla.balabit.com/show_bug.cgi?id=202 Summary: pdbtool dump does not recognize the -p option Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 Man page states dump [options] Display the RADIX tree built from the patterns. This shows how are the patterns represented in syslog-ng and it might also help to track down pattern-matching problems. The dump utility can dump the tree used for matching the PROGRAM or the MSG parts. --debug or -d Enable debug/diagnostic messages on stderr. --pdb or -p Name of the pattern database file to use. --program or -P Displays the RADIX tree built from the patterns belonging to the $PROGRAM application. --program-tree or -T Display the $PROGRAM tree. --verbose or -v Enable verbose messages on stderr. Example and sample output: But the command does localhost% pdbtool dump -P fscheck -p patterndb.xml Error parsing command line arguments: Unknown option -p localhost% pdbtool dump -P fscheck --pdb patterndb.xml Error parsing command line arguments: Unknown option --pdb -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2