October 2008 - syslog-ng - lists.balabit.hu

Destination program behavior
by Gaetan Lord 13 Oct '08

13 Oct '08

Hi, I try to define a log parser. The idea is to monitor every lines for specific pattern and run action command. My config lines are destination syslog_parser { program("/usr/local/bin/test_syslog"); }; log { source(src); destination(syslog_parser); }; To test syslog-ng, I wrote a basic shell script, which only contain /bin/echo "This is a test" | /usr/bin/wall and exit This should issue a "wall" command on every lines written in the syslog I test it with the command "logger", and this do not work. I have the following syslog-ng error when I start it. Oct 10 13:04:00 minime1 syslog-ng[5470]: io.c: do_write: write() failed (errno 32), Broken pipe Oct 10 13:04:00 minime1 syslog-ng[5470]: pkt_buffer::do_flush(): Error flushing data I'm currently running version 1.6 from SLES10, syslog-ng-1.6.8-20.18 for IA64 I red the documentation provided by the package, and saw the following NOTE: the program is executed once at startup, and kept running until SIGHUP or exit. The reason is to prevent starting up a large number of programs for messages, which would imply an easy DoS. My script is meant to run once and exit. Could it be the problem. Is there any way to achieve what I want. I'm aware about the DoS, but in this case, there it's not a problem. Thank you -- Spike Milligan - "All I ask is the chance to prove that money can't make me happy."

2 1

Does syslog-ng support triggers?
by Luís Miguel Silva 11 Oct '08

11 Oct '08

Hello everyone, I was wondering if syslog-ng supports triggers (based on keywords OR time events). Ie: i want to be able to call an application if syslog-ng detects the same log message came from the same host x times in y minutes. We are suffering a lot of attacks against our webmail servers and would like to use this to try and trigger an alarm against brute force connections. Thanks in advance, -- Luís Miguel Ferreira da Silva Qualidade e Segurança CICA - FEUP GSM: +351 912671471

3 3

Re: [syslog-ng] Log to postgres
by ml＠bortal.de 09 Oct '08

09 Oct '08

Hello List, i got syslog-ng compiled and now i get those errors: (is there maybe a table dump for postgres i could use?) What do there errors mean? How can i debug them? -------------------------------------------------------------------- ~# syslog-ng -F -v -f /etc/syslog-ng.conf -e syslog-ng starting up; version='2.1.1' Database thread started; Syslog connection accepted; from='AF_INET(<some-ip>:51708)', to='AF_INET(<some-ip>:514)' Initializing destination file writer; template='/var/log/messages', filename='/var/log/messages' Syslog connection accepted; from='AF_INET(<some-ip>:54521)', to='AF_INET(<some-ip>:514)' Syslog connection accepted; from='AF_UNIX(anonymous)', to='AF_UNIX(/dev/log)' EOF occurred while reading; fd='26' Closing log reader fd; fd='26' Syslog connection accepted; from='AF_UNIX(anonymous)', to='AF_UNIX(/dev/log)' EOF occurred while reading; fd='26' Closing log reader fd; fd='26' Syslog connection accepted; from='AF_INET(<some-ip>:54887)', to='AF_INET(<some-ip>:514)' EOF occurred while reading; fd='26' Closing log reader fd; fd='26' Syslog connection accepted; from='AF_INET(<some-ip>:64789)', to='AF_INET(<some-ip>:514)' Syslog connection accepted; from='AF_UNIX(anonymous)', to='AF_UNIX(/dev/log)' Syslog connection accepted; from='AF_UNIX(anonymous)', to='AF_UNIX(/dev/log)' EOF occurred while reading; fd='27' Closing log reader fd; fd='27' EOF occurred while reading; fd='28' Closing log reader fd; fd='28' Syslog connection accepted; from='AF_INET(<some-ip>:4622)', to='AF_INET(<some-ip>:514)'

4 3

(no subject)
by n r 08 Oct '08

08 Oct '08

hi every body My syslog-ng server is on debian eth. I woudl like to know how can I push on my server: - my client /var/log/dkpg.log, /varl/log/samba/ "host".log, /var/log/samba/wtmp and utmp directory ? thanks nr _________________________________________________________________ Appelez vos amis de PC à PC -- C'EST GRATUIT http://get.live.com/messenger/overview

2 3

syslog-ng not working
by Ramprasad 08 Oct '08

08 Oct '08

Hi list, I have the following configuration in /etc/syslog-ng/syslog-ng.conf file on OpenSuSE 11.0 64 bit system. # cat /etc/syslog-ng/syslog-ng.conf options { long_hostnames(off); sync(0); perm(0640); stats(3600); }; source src { internal(); unix-stream("/dev/log"); unix-dgram("/dev/log"); }; destination d_filepath_10 { file("/tmp/syslog" fsync(yes)); }; filter f_filter_10 {level(debug .. emerg); }; log { source(src); filter(f_filter_10); destination(d_filepath_10); }; I don't see any log messaged being logged to /tmp/syslog file. Though, I restarted the syslog daemon, there is not effect. I went through some syslog-ng examples available on internet such as the one below, but still I don't see syslog-ng logging to the specified file - /tmp/syslog # cat /etc/syslog-ng/syslog-ng.conf options { keep_hostname(yes); long_hostnames(off); sync(1); log_fifo_size(1024); }; source src { unix-stream("/dev/log"); internal(); udp(); tcp(port(5140) keep-alive(yes)); }; destination syslog { file("/var/log/syslog"); }; destination debug { file("/var/log/debug"); }; destination messages { file("/tmp/syslog"); }; filter f_messages { level(info .. warn) and not facility(auth, authpriv, cron, daemon, mail, news); }; log { source(src); filter(f_messages); destination(messages); }; Please help us on this. Thanks & Regards, Ramprasad B

1 0

Re: [syslog-ng] Reload does not free up RAM
by Evan Rempel 07 Oct '08

07 Oct '08

Balazs Scheidler wrote: > On Mon, 2008-10-06 at 11:38 -0700, Evan Rempel wrote: >> Balazs Scheidler wrote: >>> On Sun, 2008-10-05 at 13:49 -0700, Evan Rempel wrote: >>>> Balazs Scheidler wrote: >>>>>>>>>> We have cases where a syslog-ng buffers quite a bit of data (2GB or so) before >>>>>>>>>> it is able to flush the data to the destination. Currently, the only way to "free up" that RAM >>>>>>>>>> is to restart syslog-ng, which means that some log messages are lost during the restart >>>>>>>>>> window. >>>>> Both PE and OSE frees up disk/memory when possible. If they don't, then >>>>> that's probably a leak. >> No, it isn't a memory leak as the process can be months old and only grows when >> it buffers messages. >> >>>>> So if your syslog-ng process is 2GB in size, and you didn't specify an >>>>> overly large log_fifo_size(), it's probably because of a leak. >> >> >>>> I have a large log_fifo_size() but the process starts small, grows when it needs >>>> to buffer, and does NOT get small again when the destination is available. >>> Hmm... that might be related to the malloc implementation. Which >>> platform are you using? >>> >>>>>> 2. The buffer space should be thrown away only after the message >>>>>> has been delivered to the destination. >>>>> I see, this is what should happen now. >> >> >>>> This does not appear to be happening. >>>> I am using OSE 2.0.8 Have there been memory fixes/changes since then? >>> Not that I know of. >>> >>>> Is there a way to get buffer count statistics out? How many messages are >>>> in each fifo buffer? >>> In the experimental 3.0 branch, yes there is. In current 2.0, I'm afraid >>> there's none. >>> >>> Is there a limit on the growth of syslog-ng? I mean if it gets large >>> once and then empties its buffers, it stays large, but does it start >>> growing again, or it stays steadily at the size it was before? >> I have not specifically tested this scenario, but we do track processes >> and memory usage, and it appears that once the space is allocated, it does >> not grow unless a larger set of messages gets buffered. >> >> >>> Again, the platform matters a lot, how the local malloc implementation >>> takes care about fragmentation, whether it shrinks the process once >>> memory is freed. >> This is running on Redhat AS 3 (2.4 kernel) and Redhat AS 4 (2.6 kernel) >> >> On the Redhat 4 system the glibc packages installed are >> >> glibc-common-2.3.4-2.36 >> glibc-devel-2.3.4-2.36 >> glibc-headers-2.3.4-2.36 >> glib2-2.4.7-1 >> glibc-kernheaders-2.4-9.1.100.EL >> > > hmm.. this version should be all right, at least in my experience glibc > tries to release memory back to the system once the application freed it > using free(). > > Maybe it is some kind of memory fragmentation issue. How easy is this to > reproduce? Does it require months of running, or you have a simple > recipe to create it? We are encountering this fairly easily in production, so I should be able to verify a recipe. It will take me a couple of days to "get to it", but will report back. -- Evan Rempel erempel(a)uvic.ca Senior Programmer Analyst 250.721.7691 University Systems, University of Victoria

1 0

syslog-ng dropping packet
by Francis Lacoste 07 Oct '08

07 Oct '08

Hello all, I'm experiencing packet losses with syslog-ng, I know that there's some tweak that can be done to kernel to handle a bigger buffer... Anyone could tell me what can I tweak with sysctl so it will stop dropping ? Thanks. ------- Francis Lacoste

3 4

Re: [syslog-ng] Reload does not free up RAM
by Evan Rempel 07 Oct '08

07 Oct '08

Balazs Scheidler wrote: > On Sun, 2008-10-05 at 13:49 -0700, Evan Rempel wrote: >> Balazs Scheidler wrote: >>>>>>>> We have cases where a syslog-ng buffers quite a bit of data (2GB or so) before >>>>>>>> it is able to flush the data to the destination. Currently, the only way to "free up" that RAM >>>>>>>> is to restart syslog-ng, which means that some log messages are lost during the restart >>>>>>>> window. >>> Both PE and OSE frees up disk/memory when possible. If they don't, then >>> that's probably a leak. No, it isn't a memory leak as the process can be months old and only grows when it buffers messages. >>> >>> So if your syslog-ng process is 2GB in size, and you didn't specify an >>> overly large log_fifo_size(), it's probably because of a leak. >> >> I have a large log_fifo_size() but the process starts small, grows when it needs >> to buffer, and does NOT get small again when the destination is available. > > Hmm... that might be related to the malloc implementation. Which > platform are you using? > >>>> 2. The buffer space should be thrown away only after the message >>>> has been delivered to the destination. >>> I see, this is what should happen now. >> >> This does not appear to be happening. >> I am using OSE 2.0.8 Have there been memory fixes/changes since then? > > Not that I know of. > >> Is there a way to get buffer count statistics out? How many messages are >> in each fifo buffer? > > In the experimental 3.0 branch, yes there is. In current 2.0, I'm afraid > there's none. > > Is there a limit on the growth of syslog-ng? I mean if it gets large > once and then empties its buffers, it stays large, but does it start > growing again, or it stays steadily at the size it was before? I have not specifically tested this scenario, but we do track processes and memory usage, and it appears that once the space is allocated, it does not grow unless a larger set of messages gets buffered. > Again, the platform matters a lot, how the local malloc implementation > takes care about fragmentation, whether it shrinks the process once > memory is freed. This is running on Redhat AS 3 (2.4 kernel) and Redhat AS 4 (2.6 kernel) On the Redhat 4 system the glibc packages installed are glibc-common-2.3.4-2.36 glibc-devel-2.3.4-2.36 glibc-headers-2.3.4-2.36 glib2-2.4.7-1 glibc-kernheaders-2.4-9.1.100.EL > The problem is, that if the local malloc implementation is wrong the > only thing I could do is to try to use a different allocator (e.g. use > GNU malloc on your platform), but that's not a simple change. Understood. -- Evan Rempel

2 1

syslog-ng 2.1: upstream libdbi does NOT provide pkgconfig .pc file
by Wolfram Schlich 07 Oct '08

07 Oct '08

Hi, syslog-ng 2.1 depends on libdbi >= 0.8.0 with --enable-sql. Unfortunately, it assumes that a pkg-config .pc file for libdbi is in place which the upstream libdbi just does not include. It seems that there is a balabit-modified version of libdbi available (libdbi8_0.8.2bb2-2.tar.gz) which was patched to include a dbi-0.8.pc file. What's the reason for that? Was upstream asked to include that .pc file in their regular releases? What other changes were made to the upstream libdbi to form the balabit version? This is kinda maintenance and dependency hell so far :( Please clear up the situation so I can take care of building a Gentoo package for 2.1 *including* SQL support :) Thanks! -- Regards, Wolfram Schlich <wschlich(a)gentoo.org> Gentoo Linux * http://dev.gentoo.org/~wschlich/

2 4

Regexp question
by Mártha Csaba 06 Oct '08

06 Oct '08

Hello I use this regexp: match ="FlashLog: ([0-9a-zA-Z]+)-([0-9a-zA-Z]+)-([0-9]{4})-([0-9]{2})- ([0-9]{2})-([0-9]{2})-([0-9]{2})-([0-9a-zA-Z]+)"); and this df_out: destination df_out { file("/var/log/client/flashlog/${1}/${8}_.txt"); }; I have /var/log/client/flashlog/_.txt file. What is the problem? Best Regards: csaba

1 0