October 2012 - syslog-ng - lists.balabit.hu

[Bug 206] New: program_override() and host_override() do not work in 3.4
by bugzilla＠bugzilla.balabit.com 27 Oct '12

27 Oct '12

https://bugzilla.balabit.com/show_bug.cgi?id=206 Summary: program_override() and host_override() do not work in 3.4 Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: ASSIGNED Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: algernon(a)balabit.hu ReportedBy: algernon(a)balabit.hu Type of the Report: --- Estimated Hours: 0.0 The *Options simplification change (d93d35857e3acac5eb59a9735d8947dedf43a511) broke host_override () & program_override (), because they host_override_len and program_override_len properties are no longer initialized to -1, but to 0, so when setting an override, a value with 0 length will be set, emptying the variable instead of setting it to a new value. This should be fairly easy to fix, this bug is a reminder for myself to not forget it. But if anyone happens to get around to submit a patch before I get there, so much the better! -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[PATCH 0/2]: afamqp: AMQP destination driver
by Gergely Nagy 27 Oct '12

27 Oct '12

The two patches that will follow implement an AMQP destination driver for syslog-ng, contributed by Attila Nagy <bra(a)fsn.hu>, with a few minor enhancements and porting to syslog-ng 3.4 by myself. The patch is also available on the feature/3.4/amqp[1] and feature/3.3/amqp[2] branches of my git repository, for syslog-ng 3.4 and 3.3 respectively. Functionally, they're exactly the same, but all further improvements and fixes will go to the 3.4 branch only, the 3.3 branch is there for testing and historical purposes only, and will be removed sometime after the driver gets merged to 3.4. Now, I won't explain in detail what AMQP is, let it suffice that it is a messaging protocol, of which RabbitMQ[3] is an implementation of. Interested people can read the available documentation, there's plenty! For now, lets concentrate on how to take the driver for a test drive! Once rabbitmq is installed (on debian and derivatives of recent versions it is as simple as apt-get install rabbitmq-server), all one needs to do, is configure syslog-ng to send messages, and grab a client that can look at the queue. For testing purposes, there's a simple one at https://gist.github.com/3859756 - you'll need a couple of python modules for that. What it does, is it sets up a queue named 'test', and binds it to the 'syslog' exchange (the default used by the AMQP driver), and starts consuming messages. It prints both headers and the payload (if any, by default, there is no payload). What one needs to know about the driver, is that it does not declare any queues: it's the responsibility of the admin to set queues up, and route messages as they see fit. This can be done on the RabbitMQ web admin interface (again, see the docs about that!), among other things, or programmatically, like the script above does. Hopefully, this is enough to get one started! If not, I plan to write a more detailed blog post about the topic in the not too distant future. There are plans to write an accompanying source driver too, but work has not started on that front yet. Rest assured though, it will come. I'd like to thank Attila again for contributing the driver, and answering my silly questions during review - both are much appreciated! [1]: https://github.com/algernon/syslog-ng/tree/feature/3.4/amqp [2]: https://github.com/algernon/syslog-ng/tree/feature/3.3/amqp [3]: http://www.rabbitmq.com/

2 3

[PATCH] logreader: When following a file, treat non-regular files specially
by Gergely Nagy 27 Oct '12

27 Oct '12

When following a file with follow-freq(), and it is not a regular file, but something for which stat(2) always returns 0 size (such as the /dev/klog character device on FreeBSD), log_reader_io_follow_file() never went into the process input branch, because the current position (0) was always at EOF (also 0). To break out of this unfortunate situation, if the file to read from is not a regular file, go into the process input loop anyway: at worst, it will do an expensive noop. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/logreader.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/logreader.c b/lib/logreader.c index eb52330..1e2ea5e 100644 --- a/lib/logreader.c +++ b/lib/logreader.c @@ -276,7 +276,7 @@ log_reader_io_follow_file(gpointer s) evt_tag_int("size", st.st_size), NULL); - if (pos < st.st_size) + if (pos < st.st_size || !S_ISREG(st.st_mode)) { /* we have data to read */ log_reader_io_process_input(s); -- 1.7.10.4

2 1

Collect log from beginning incase central server's log file gets deleted.
by sagar naravane 27 Oct '12

27 Oct '12

Hi, Below is my setup (see the config details). Issue i am facing is : Lets say on central servers apache logs are collected till 2:00 PM for current date. Incase if i delete the log file on central server, a new file gets created and starts collecting the log after the delete time ie 2:00 PM. What i want to achieve is that in such an event where the log files gets deleted.. the new log-file will collect the log from the beginning of the day ie 12:00 AM. Is this possible ? *************** client config *********************** @include "scl.conf" options { keep_hostname(yes); log_fifo_size(100000) }; rewrite r_setfilename { set( "$FILE_NAME", value(".SDATA.file(a)18372.4.name") ); }; rewrite r_apache_basepath_www-dev01 { subst( "/apps/log/apache/www-dev01/", "", value(".SDATA.file(a)18372.4.name") type("string") flags("prefix") ); }; source s_apache_www-dev01 { file("/apps/log/apache/www-dev01/*.log" program_override("apache/www/dev01") flags(no-parse) recursive(yes) ); }; destination d_server { syslog("x.x.x.x" transport("tcp") port(514)); }; log { source(s_apache_www-dev01); rewrite(r_setfilename); rewrite(r_apache_basepath_www-dev01); destination(d_server); }; *************** Server Config ************ @include "scl.conf" options { create_dirs (yes); dir_perm(0755); dir_group(60030); dir_owner(wwwtools); perm(0755); owner(wwwtools); group(60030); keep_hostname(yes); threaded(yes); log_msg_size(5000000); }; source s_network { syslog(ip(0.0.0.0) port(514) transport("tcp") max-connections(200)); }; filter f_apache_www-dev01 { program('apache/www/dev01'); }; destination d_central { file("/apps/log/syslog-ng/${PROGRAM}/${YEAR}${MONTH}${DAY}/${HOST}/${. SDATA.file(a)18372.4.name}" template("${MSGONLY}\n") ); }; log { source(s_network); filter(f_apache_www-dev01); destination(d_central); }; -- Regards, Sagar Naravane

3 3

[Bug 194] New: Can't open libsyslog-ng-crypto. so during module loading on Fedora
by bugzilla＠bugzilla.balabit.com 27 Oct '12

27 Oct '12

https://bugzilla.balabit.com/show_bug.cgi?id=194 Summary: Can't open libsyslog-ng-crypto.so during module loading on Fedora Product: syslog-ng Version: 3.3.x Platform: All OS/Version: Linux Status: ASSIGNED Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: algernon(a)balabit.hu ReportedBy: algernon(a)balabit.hu CC: algernon(a)balabit.hu Type of the Report: --- Estimated Hours: 0.0 This clone is to track the library path issue separately. +++ This bug was initially created as a clone of Bug #188 +++ Linux distro: Fedora 17 x86_64 Syslog-ng source tarball: syslog-ng-3.3.5.90-20120810-v3.3.5-43-g19f6c.tar.gz we have encountered a couple of problems when running syslog-ng-3.3.5.90-20120810 under Fedora 17 x86_64: 1) ld.so library path 2) undefined symbol: tls_context_setup_session Steps to reproduce the problem: 1) just run "/usr/sbin/syslog-ng --version" ---------- # /usr/sbin/syslog-ng --version syslog-ng 3.3.5.90 Installer-Version: 3.3.5.90 Revision: Compile-Date: Aug 17 2012 22:09:52 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat,afsql Error opening plugin module; module='afsocket', error='libsyslog-ng-crypto.so: cannot open shared object file: No such file or directory' Error opening plugin module; module='afsocket-notls', error='//usr/lib64/syslog-ng/libafsocket-notls.so: undefined symbol: tls_context_setup_session' Error opening plugin module; module='afsocket-tls', error='libsyslog-ng-crypto.so: cannot open shared object file: No such file or directory' Available-Modules: dummy,afsql,tfjson,dbparser,csvparser,convertfuncs,afuser,basicfuncs,afmongodb,confgen,syslogformat,affile,afprog Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on ---------- The library path problem can be solved by creating the file "/etc/ld.so.conf.d/syslog-ng.conf" containing the syslog-ng module directory ( /usr/lib64/syslog-ng in this particular case ). Running syslog-ng under strace without touching the ld.so library paths produces: ---------- # # strace -e trace=file /usr/sbin/syslog-ng --version ... access("//usr/lib64/syslog-ng/libafsocket.so", F_OK) = 0 stat("//usr/lib64/syslog-ng/libafsocket.so", {st_mode=S_IFREG|0755, st_size=103560, ...}) = 0 open("//usr/lib64/syslog-ng/libafsocket.so", O_RDONLY|O_CLOEXEC) = 4 open("/lib64/tls/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) open("/lib64/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) open("/usr/lib64/tls/x86_64/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) stat("/usr/lib64/tls/x86_64", 0x7fff0f81fcb0) = -1 ENOENT (No such file or directory) open("/usr/lib64/tls/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) stat("/usr/lib64/tls", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 open("/usr/lib64/x86_64/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) stat("/usr/lib64/x86_64", 0x7fff0f81fcb0) = -1 ENOENT (No such file or directory) open("/usr/lib64/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) stat("/usr/lib64", {st_mode=S_IFDIR|0555, st_size=49152, ...}) = 0 Error opening plugin module; module='afsocket', error='libsyslog-ng-crypto.so: cannot open shared object file: No such file or directory' ... ----------- Note that all syslog-ng modules with the exception of libsyslog-ng-crypto.so are loaded from the syslog-ng modules directory. Output of "syslog-ng --version" after creating the file "/etc/ld.so.conf.d/syslog-ng.conf" ---------- # /usr/sbin/syslog-ng --version syslog-ng 3.3.5.90 Installer-Version: 3.3.5.90 Revision: Compile-Date: Aug 17 2012 22:09:52 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat,afsql Error opening plugin module; module='afsocket-notls', error='//usr/lib64/syslog-ng/libafsocket-notls.so: undefined symbol: tls_context_setup_session' Available-Modules: dummy,afsocket,afsql,tfjson,dbparser,csvparser,convertfuncs,afuser,basicfuncs,afmongodb,afsocket-tls,confgen,syslogformat,affile,afprog Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on ---------- /jpo -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 6

Syslog-ng OSE multiline messages support
by Daniel Josue Rodriguez Paiz 25 Oct '12

25 Oct '12

Hello, I have one question, does syslog-ng OSE support multiline parsing logs? i have one applications that send multiline messages and syslog-ng save the log of the first line only. Thanks for your support in adavnce, Daniel

3 4

[Bug 207] New: HOST macro contains different data inside db_parser than in the config
by bugzilla＠bugzilla.balabit.com 23 Oct '12

23 Oct '12

https://bugzilla.balabit.com/show_bug.cgi?id=207 Summary: HOST macro contains different data inside db_parser than in the config Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 When using the $HOST macro in the config file of syslog-ng the macro contains source@hostname type of data. When using the $HOST macro inside the db_parser the macro contains just the hostname. This is confusing. I am not how fix this. I would not like to see the removal of the "source@" portion, but I do want to have the raw hostname available. Perhaps having a macro of $SOURCE that would have "source@hostname" and make the $HOST have just the hostname. The FULLHOST will still have the chain of hosts, including the $SOURCE, just like it has now. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

3 5

Please help: Can not figure out why syslog-ng ose 3.2.2 server doesn't receive syslog client messages over udp/11514 on 64-bit RHEL 5
by Mehdi Mortezai 19 Oct '12

19 Oct '12

Hello, I am new to syslog-ng. I have set up a syslog-ng ose 3.2.2 server and configured syslog on 64-bit RHEL 5 client servers to send all messages to the syslog-ng server over udp/11514 (client's syslog.conf: *.* @syslog-ngIP:11514) but no packet is received. When I change the port on the syslog-ng server and on the client's syslog from udp/11514 to udp/514 all works. Verified that firewalls passing packets for both udp/514 and 11514 on the server and RHEL client and the port is not used by any other app. I cannot figure out why the logs are not happening when changing the port to udp/11514 from default? Is there any global options I should set or any change to my source statements? Here are the global options and sources definitions: *options* { time_reopen (10); long_hostnames (off); owner("root"); group("root"); perm(0600); dir_perm(0755); use_dns (yes); use_fqdn (no); create_dirs (yes); keep_hostname (no); }; *source:* ########### Define Sources *### Internal * source s_local_sys { file ("/proc/kmsg" program_override("kernel: ")); unix-stream ("/dev/log"); internal();}; *### External or remote:* # For Server Groups source s_RHEL5_udp { udp(port(11514));}; *# For all other single hosts:* source s_remote_udp { udp(port(514));}; Thanks in advance for your help. Medi

1 0

GeoIP template function
by Csaba Major 19 Oct '12

19 Oct '12

Hi, I have a system where I have a lot of logs with IP addresses (typically firewall logs), and the requirement was to also include the country name in the logs. I found that a geoip based template function would be an easy way for this, so I created a PoC for that. It works pretty well, however some improvements and performance measurement would be necessary. The created patch is attached (based on 3.3.5), some may find it useful :) (It requires libgeoip, and a geoip database, of course) Regards, Csaba

4 6

Can't get kernel messages on FreeBSD
by Tomoki BENIYA 19 Oct '12

19 Oct '12

Hi all. I installed syslog-ng3.3.6_3 using ports on FreeBSD9.0-RELEASE(i386). I can't get kernel messages by default configuration. ex. em0: link state changed to DOWN em0: link state changed to UP But, I can see them using 'cat /dev/klog' after stoppedsyslog-ng. So, I think syslog-ng can't get kermel messages from /dev/klog. How to get kernel messages using syslog-ng? Please give me a hint or something! * /etc/rc.conf syslogd_enable="NO" syslog_ng_enable="YES" * /usr/local/etc/syslog-ng.conf source s_kern { file("/dev/klog"); }; filter f_kern { facility(kern); }; destination d_kern { file("/var/log/kern.log"); }; log { source(s_kern); filter(f_kern); destination(d_kern); }; -- Tomoki BENIYA <beniya(a)bit-isle.co.jp>

2 2