- syslog-ng - lists.balabit.hu

AUTO: James B Horwath/TheGuardian is out of the office. (returning 07/05/2011)
by James B Horwath 25 Jun '11

25 Jun '11

I am out of the office until 07/05/2011. I am out of the office until Wednesday, July 6, 2011 with limited access to e-mail. If there are Nitro issues please contact Dante Rodino at 212-919-2819 or Sunil Gupta at 610-807-6710. Any other issues please contact Christopher Clarke manager of Security Operations Thanks, Jim Note: This is an automated response to your message "syslog-ng Digest, Vol 74, Issue 34" sent on 6/25/11 6:00:02. This is the only notification you will receive while this person is away. ----------------------------------------- This message, and any attachments to it, may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are notified that any use, dissemination, distribution, copying, or communication of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately by return e-mail and delete the message and any attachments. Thank you.

1 0

Disk based buffering for syslog-ng v3.2.4 OSE
by Yu Watanabe 24 Jun '11

24 Jun '11

Hello all. I would like to ask question to all. Has any one implemented a disk based buffering logic in syslog-ng? Since the log_disk_fifo_size has been deprecated, I am considering to implementing with other way. In the manual v3.2, I found couple of examples with flow control but none with the disk buffering. It would be helpful if anyone can share their thoughts about this. Thanks, Yu

3 3

[Bug 125] New: [systemd] refuses stop request from systemctl on socket activated start
by bugzilla＠bugzilla.balabit.com 24 Jun '11

24 Jun '11

https://bugzilla.balabit.com/show_bug.cgi?id=125 Summary: [systemd] refuses stop request from systemctl on socket activated start Product: syslog-ng Version: 3.2.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: d(a)falconindy.com Type of the Report: bug Estimated Hours: 0.0 I'm seeing some unexpected behavior from syslog-ng when it's started by way of systemd's socket activation. The visible issue is that the process is ignoring a stop request from systemctl will will lead to systemd issuing a kill -9. An strace of the syslog-ng process when the signal is sent is as follows: 0x6e9ed0, 8192, 0, 0x7fff35fe0360, 0x7fff35fe03ec) = ? ERESTARTSYS (To be restarted) --- {si_signo=SIGTERM, si_code=SI_USER, si_pid=1, si_uid=0, si_value={int=1, ptr=0x1}} (Terminated) --- --- {si_signo=SIGCONT, si_code=SI_USER, si_pid=1, si_uid=0, si_value={int=1, ptr=0x1}} (Continued) --- rt_sigreturn(0xf) = -1 EINTR (Interrupted system call) recvfrom(3, And eventually: +++ killed by SIGKILL +++ It seems to ignore the SIGTERM. It does continue to receive messages and log them (checked via logger(1)). After the daemon is SIGKILL'ed, if I start it and then try to stop, it ends normally: ) = ? ERESTART_RESTARTBLOCK (To be restarted) --- {si_signo=SIGTERM, si_code=SI_USER, si_pid=1, si_uid=0, si_value={int=1, ptr=0x1}} (Terminated) --- --- {si_signo=SIGCONT, si_code=SI_USER, si_pid=1, si_uid=0, si_value={int=1, ptr=0x1}} (Continued) --- rt_sigreturn(0xf) = -1 EINTR (Interrupted system call) gettimeofday({1308428877, 84197}, NULL) = 0 gettimeofday({1308428877, 84385}, NULL) = 0 close(6) = 0 gettimeofday({1308428877, 84763}, NULL) = 0 poll([{fd=3, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6, events=POLLIN}], 3, 0) = 1 ([{fd=6, revents=POLLNVAL}]) gettimeofday({1308428877, 85143}, NULL) = 0 gettimeofday({1308428877, 85332}, NULL) = 0 lseek(7, 0, SEEK_END) = 3013 write(7, "Jun 18 16:27:57 kweemoo syslog-n"..., 87) = 87 lseek(8, 0, SEEK_END) = 175609 write(8, "Jun 18 16:27:57 kweemoo syslog-n"..., 87) = 87 gettimeofday({1308428877, 86379}, NULL) = 0 poll([{fd=3, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6, events=POLLIN}], 3, 0) = 1 ([{fd=6, revents=POLLNVAL}]) gettimeofday({1308428877, 86757}, NULL) = 0 gettimeofday({1308428877, 86946}, NULL) = 0 lseek(7, 0, SEEK_END) = 3100 write(7, "Jun 18 16:27:57 kweemoo syslog-n"..., 81) = 81 lseek(8, 0, SEEK_END) = 175696 write(8, "Jun 18 16:27:57 kweemoo syslog-n"..., 81) = 81 close(3) = 0 close(5) = 0 close(7) = 0 close(8) = 0 unlink("/var/run/syslog-ng.pid") = 0 exit_group(0) = ? If if disable the syslog-ng.service from automatically starting at boot and start it manually, it appears to behave properly. The service file I use is: ------------------- [Unit] Description=System Logger Daemon [Service] Sockets=syslog.socket ExecStartPre=/bin/systemctl stop systemd-kmsg-syslogd.service ExecStart=/usr/sbin/syslog-ng -F [Install] WantedBy=multi-user.target ------------------- Additional Info: * version: 3.2.4 * build options: --prefix=/usr --sysconfdir=/etc/syslog-ng --libexecdir=/usr/lib --localstatedir=/var/lib/syslog-ng --enable-tcp-wrapper --with-pidfile-dir=/var/run --disable-spoof-source --enable-systemd * Distro: Arch Linux * Kernel: 2.6.39.1 -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

ISO 8601 format in syslog-ng
by Pramod Pillai 24 Jun '11

24 Jun '11

Hi Need your help to understand ISO 8601 format settings on the syslog-ng server. On the syslog-ng client log the time stamp is shown as "2011-06-15T23:42:26+00:00" On the syslog-ng server log the time stamp is printed as "2011-06-15T23:42:26+05:30" The client uses $ISODATE in the template The server is using $S_ISODATE in the template As per my understanding the time stamp on the server should be "2011-06-16T05:12:26+05:30" . Is my understaning of ISO 8601 format is incorrect ? or my setting in the syslog-ng config file is wrong ? Regards Pramod

2 1

[PATCH (3.4)] affile: Fix NULL dereference when opening the destination fails.
by Gergely Nagy 24 Jun '11

24 Jun '11

Viktor Tusa discovered an error in the affile destination driver: if opening the destination failed (for example, due to insufficient permissions), next was set to NULL, and later on, at the end of affile_dd_open_writer() tried to dereferrence it. A simple workaround is to return NULL in case next is NULL. Reported-by: Viktor Tusa <tusa(a)balabit.hu> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/affile/affile.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/modules/affile/affile.c b/modules/affile/affile.c index 87767aa..57ccb7a 100644 --- a/modules/affile/affile.c +++ b/modules/affile/affile.c @@ -1052,6 +1052,9 @@ affile_dd_open_writer(gpointer args[]) } } } + if (!next) + return NULL; + next->queue_pending = TRUE; /* we're returning a reference */ return &next->super; -- 1.7.2.5

1 1

Twelve days of Log: A short animation
by Gergely Nagy 23 Jun '11

23 Jun '11

Hi! A day or two ago, I had a rush of inspiration, and ended up doing something fairly pointless (yet, at least in my opinion, funny): I took twelve days worth of my logs, and using Gource[0], I built a little animation out of them: http://vimeo.com/25483686 There's still a lot of knobs to turn and tweak, but I thought I'd share a short demo, before I finish up this part of my log visualisation project. The fun thing is, that one does not need anything else to make these kind of animations, but syslog-ng and gource. Pretty much any recent version (possibly even 3.0) should work out of the box! I wrote a little HOWTO on it, which is available on my blog at http://algernon.blogs.balabit.com/2011/06/twelve-days-of-log/. 0: https://code.google.com/p/gource/ -- |8]

2 2

Turning off TCP connection logs
by JP Vossen 22 Jun '11

22 Jun '11

This is probably totally obvious, but it's not jumping out at me in the admin guide or Google. How can I turn off the TCP connection logs like: Jun 22 01:40:47 centos5 syslog-ng[31034]: Syslog connection accepted; fd='10', client='AF_INET(192.168.1.148:46400)', local='AF_INET(0.0.0.0:514)' [...] Jun 22 01:40:47 centos5 syslog-ng[31034]: Syslog connection closed; fd='10', client='AF_INET(192.168.1.148:46400)', local='AF_INET(0.0.0.0:514)' I assume these are coming from the syslog-ng internal() source, but I'm not seeing how to turn only those off without affecting other internal syslog-ng messages. What am I being totally dense about and missing? Thanks, JP ----------------------------|:::======|------------------------------- JP Vossen, CISSP |:::======| http://bashcookbook.com/ My Account, My Opinions |=========| http://www.jpsdomain.org/ ----------------------------|=========|------------------------------- "Microsoft Tax" = the additional hardware & yearly fees for the add-on software required to protect Windows from its own poorly designed and implemented self, while the overhead incidentally flattens Moore's Law.

2 1

New on syslog-ng
by Josu Lazkano 22 Jun '11

22 Jun '11

Hello list, I am new on this list. I start working on a IT department and I need to update and check the actual syslog system. We have more than 50 Ubuntu servers, locals and remotes, on each servers there is rsyslog installed and configured this way: $ cat /etc/rsyslog.d/99-rsyslog.conf auth.*,authpriv.* @logserver kern.warn @logserver kern.err @logserver mail.* @logserver There is server (logserver) with syslog-ng to manage and save all logs. It receive on UDP port without secure, this the source section on syslog-ng configuration: source s_all { internal(); unix-stream("/dev/log"); file("/proc/kmsg" log_prefix("kernel: ")); }; source logs_externs{ udp(); }; I have some questions about how to secure it: 1. How can I secure the logs on the net? I must use TCP to secure? 2. I have lot of data (5-10GB at week) to store, which is the best method to manage it? Logrotate? Scripts? I want to move logs to NAS monthly. 3. Is it a good idea to make files for each host and services? Something like: /var/log/host1/auth.log, /var/log/host2/auth.log, /var/log/host1/mail.log, /var/log/host2/mail.log These are my newby questions, thanks for all your help and best regards.

3 4

Having some compile errors on ubuntu lucid-amd64
by Mike Gracy 21 Jun '11

21 Jun '11

I get the same error when compiling the latest stable also: libtool: link: gcc -std=gnu99 -Wno-pointer-sign -g -O2 -Wall -o .libs/pdbtool pdbtool-pdbtool.o -Wl,--export-dynamic -pthread libsyslog-ng-patterndb.a ../../lib/.libs/libsyslog-ng.so -lnsl /usr/lib/ libgmodule-2.0.so /usr/lib/libgthread-2.0.so -lrt /usr/lib/libglib-2.0.so/usr/lib/libevtlog.so -ldl -lssl -lcrypto -pthread libsyslog-ng-patterndb.a(libsyslog_ng_patterndb_a-patterndb.o): In function `pattern_db_set_time': /home/mgracy/ftp/syslog-ng/syslog-ng-3.2.3/modules/dbparser/patterndb.c:1379: undefined reference to `evt_tag_long' libsyslog-ng-patterndb.a(libsyslog_ng_patterndb_a-patterndb.o): In function `pattern_db_timer_tick': /home/mgracy/ftp/syslog-ng/syslog-ng-3.2.3/modules/dbparser/patterndb.c:1352: undefined reference to `evt_tag_long' libsyslog-ng-patterndb.a(libsyslog_ng_patterndb_a-patterndb.o): In function `pattern_db_expire_entry': /home/mgracy/ftp/syslog-ng/syslog-ng-3.2.3/modules/dbparser/patterndb.c:1315: undefined reference to `evt_tag_long' collect2: ld returned 1 exit status make[5]: *** [pdbtool] Error 1 make[5]: Leaving directory `/home/mgracy/ftp/syslog-ng/syslog-ng-3.2.3/modules/dbparser' make[4]: *** [all-recursive] Error 1 make[4]: Leaving directory `/home/mgracy/ftp/syslog-ng/syslog-ng-3.2.3/modules/dbparser' make[3]: *** [all] Error 2 make[3]: Leaving directory `/home/mgracy/ftp/syslog-ng/syslog-ng-3.2.3/modules/dbparser' make[2]: *** [all-recursive] Error 1 make[2]: Leaving directory `/home/mgracy/ftp/syslog-ng/syslog-ng-3.2.3/modules' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/home/mgracy/ftp/syslog-ng/syslog-ng-3.2.3' make: *** [all] Error 2 I am trying to compile to a deb. I didn't have a problem with libevtlog.

2 2

[RFC]: MongoDB destination plans (for 3.4 and beyond)
by Gergely Nagy 21 Jun '11

21 Jun '11

Hi! I've been working on a few mongodb destination related features recently, and I thought I'll ask for comments here, before I proceed further, to see if my ideas can be improved, and if there's anyone actually interested in the stuff I play with (mostly out of sheer curiosity; I'll scratch my own itches even if noone else has similar needs :P). Before I go further, let me introduce the current mongodb destination features, available in the syslog-ng 3.3 branch: * We can connect to a single MongoDB server per destination, which we expect to be the master (or a standalone server). * We can send logs to it, structured in various interesting ways via value-pairs(). ...aaand that's about it. Pretty bare bones, but it's enough for a lot of stuff. Now, I plan to extend this in two ways: first, it will be possible to connect to a ReplicaSet, which is basically a set of MongoDB servers that replicate from a master. The advantage of this, is that if the master goes down, one of the secondaries will automatically take over, and the mongodb driver will reconnect to it automatically aswell. In case that fails too, the destination driver will fall back to queuing within syslog-ng, and retry after a configured interval. Another advantage will be the use of "safe mode", which when turned on, will verify that the message could be inserted into a MongoDB collection, and it will not be ACKed on the syslog-ng side until it is. After a number of retries, it can be dropped along with an appropriate log message, so that these won't fill up the queue. (This, of course, would be optional, with never dropping messages by default, unless the internal queue is full.) So we could have a destination configured like this: d_mongo { mongodb( servers("10.0.10.1:27017" "10.0.10.2:27017" "10.0.10.3:27017") safe-mode(on) ); }; This would connect to 10.0.10.1 on port 27017 by default, and if that becomes inaccessible, it'd retry with .2 and .3, in that order. If any of them listed other servers as part of their replicaset, the driver would retry with those aswell. It would also turn on safe-mode, which are a bunch of extra checks to ensure that data arrived to MongoDB safe and sound. We'd get more reliability this way, and with safe-mode, better data safety aswell. Now, the question is: is there anything else that may be worth adding? If anyone here used MongoDB, is there something else you'd like to see added to the mongodb destination? (I could add GridFS support aswell, but so far, I haven't found an acceptable use-case for that yet.) -- |8]

2 2