- syslog-ng - lists.balabit.hu

tcp destination unavailable ... what happens ?
by Ali EL MOUSSAWI 11 Oct '04

11 Oct '04

Hi all, Could you please tell me what can happen if the destination tcp driver is unavailable ? I made the test with a wrong IP address in destination on my machine running syslog-ng, my software generates messages every minutes, and these messages are redirected to the destination tcp driver. The only think I saw is that in /var/adm/messages logs, the following message was repeated every 4 minutes or so ... Oct 11 10:26:39 met-ixb syslog-ng[216]: Error connecting to remote host AF_INET(192.168.192.168:514), reattempting in 10 seconds Is there any risk for my machine if the destination tcp driver is no longer available ? Will there be any buffer over size or something like that ? Or ressources over load ? Thanks a lot Ali

1 0

Re: [syslog-ng]syslog-ng loses messages
by Aaron watkins 07 Oct '04

07 Oct '04

Dave, > * What are the numbers of lines that get dropped in each of those tests? If I kill (-TERM or -9) the receiving server while messages are not being written to the input pipe, I will lose message 0 and message 1. When I kill -TERM while writing to the input pipe: Test 1: Messages 27->34 are lost (8 in total) Test 2: Messages 24->31 are lost (8 in total) When I kill -9 while writing to the input pipe: Test 1: Messages 35->38 are lost (4 in total) Test 2: Messages 23->29 are lost (7 in total) > * What happens in your test if you change machine2 fifo's to 0? I presume you mean the log_fifo_size option... When I set this to 0, the following messages got through: Iteration 1: Message 0,1,14,28 Iteration 2: Message 0,1,12,21,32 Iteration 3: Message 0,1,12,21,30,39 That is to say, in every iteration, around 37 messages were lost. And this was with the system operating normally (ie. without killing the machine2's syslog-ng). Any ideas? Thanks, Aaron > > On Fri, 1 Oct 2004 12:38:56 +0100, Aaron watkins > <aaron.j.watkins(a)gmail.com> wrote: > > Hi there, > > > > I have been doing some stress testing of syslog-ng as a possible > > facilitator of event messaging for use on our network. I have set up a > > system as below: > > > > Event Process > > | > > \_/ > > FIFO/pipe > > | > > _____\_/_______ > > [ ] > > [ Machine 1 ] > > [ running ] > > [ syslog-ng ] > > [______________] > > | > > \_/ > > TCP socket > > | > > _____\_/_______ > > [ ] > > [ Machine 2 ] > > [ running ] > > [ syslog-ng ] > > [______________] > > | > > \_/ > > FIFO/pipe > > | > > \_/ > > cat > > > > Or to describe it, I have a process writing to a pipe on Machine 1, > > which syslog-ng then forwards via a socket to syslog-ng running on > > Machine 2, which then outputs to another pipe, which I am catting. > > > > My event process is in a while(true) loop, simply writing a number of > > messages to the pipe and then sleeping for 1 second. > > > > The problem I have discovered is that if I kill (-TERM or -9) > > syslog-ng running on Machine 2, some messages are lost before it > > starts buffering them in Machine 1. If I kill it while my event > > process is sleeping and then restart it, the first two messages (once > > the event process restarts transmitting) will be lost. Alternatively, > > if I kill it while my event process is writing data (and syslog-ng on > > Machine 1 is transmitting over the socket), normally about five > > messages will be lost. > > > > Is this a recognised problem and is there a way to work around this? > > Obviously, I would like to be able to get all of my messages through > > (up to the buffer limit) without having to reprocess the messages from > > the local event copy file. > > > > I am running this system on Solaris 5.8. > > > > The code for my event process is: > > > > #!/bin/sh > > while [ 1 ] > > do > > count=0 > > while [ $count -le 40 ] > > do > > echo "${count}: Hello World" >> event_pipe > > count=`expr ${count} + 1` > > done > > sleep 1 > > done > > > > My Machine 1 config file: > > > > options { > > sync(0); # Sends message as soon as it's received > > time_reopen(10); # Wait 10 before reopening closed connections > > log_fifo_size(500); # The number of messages that can go in the output queue > > create_dirs(yes); # Whether to create destination directories, if required > > }; > > > > source src_internalEvents { > > internal(); > > }; > > > > source src_InputPipe { > > pipe( "/tmp/event_pipe" ); > > }; > > > > destination dst_localLog { > > file( "/data/gen/log/nievent_all.log" > > template("$YEAR-$MONTH-$DAY--$HOUR:$MIN:$SEC ($HOST:$PROGRAM): > > $MSG\n" )); }; > > > > destination dst_localEventCopy { > > file( "/data/gen/log/nievent_copy.log" > > template("$MSG\n") ); > > }; > > > > destination dst_OutputSocket { > > tcp( "imgwrkfl-inp" port(1515) ); > > }; > > > > log { > > source( src_InputPipe ); > > destination( dst_localLog ); > > destination( dst_localEventCopy ); > > destination( dst_OutputSocket ); > > }; > > > > log { > > source( src_internalEvents ); > > source( src_InputPipe ); > > destination( dst_localLog ); > > flags( fallback ); > > }; > > > > For Machine 2: > > > > options { > > sync(0); # Sends message as soon as it's received > > time_reopen(10); # Wait 10 before reopening closed connections > > log_fifo_size(500); # The number of messages that can go in the output queue > > create_dirs(yes); # Whether to create destination directories, if required > > }; > > > > source src_internalEvents { > > internal(); > > }; > > > > source src_InputSocket { > > tcp( port(1515) ); > > }; > > > > destination dst_localLog { > > file( "/data/gen/log/nievent_all.log" > > template( "$YEAR-$MONTH-$DAY--$HOUR:$MIN:$SEC ($HOST:$PROGRAM): $MSG\n" ) ); > > }; > > > > destination dst_localEventCopy { > > file( "/data/gen/log/nievent_copy.log" > > template( "$MSG\n") ); > > }; > > > > destination dst_OutputPipe { > > pipe( "/tmp/prepress/nievents_pipe" > > template( "$MSG\n") ); > > }; > > > > log { > > source( src_InputSocket ); > > destination( dst_localLog ); > > destination( dst_localEventCopy ); > > destination( dst_OutputPipe ); > > }; > > > > log { > > source( src_internalEvents ); > > source( src_InputSocket ); > > destination( dst_localLog ); > > flags( fallback ); > > }; > > > > Thanks in advance for your help. > > > > Aaron Watkins > > _______________________________________________ > > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > > https://lists.balabit.hu/mailman/listinfo/syslog-ng > > Frequently asked questions at http://www.campin.net/syslog-ng/faq.html > > > > >

2 1

perl syslog::parse problem
by lists＠mhcsoftware.de 07 Oct '04

07 Oct '04

hi, i've switched to syslog-ng and now mailgraph stoped working. i've read that there is a patch requiered for syslog::parse but i can't find it anywhere. can some one please point me to this patch ? TIA matthias -- Matthias Henze matthias(a)mhcsoftware.de Use PGP!! http://www.mhcsoftware.de/MatthiasHenze.asc - - - - - - - - - - - - - - - - - - - - - - - - - - - - MHC SoftWare GmbH voice: +49-(0)9533-92006-0 Fichtera 17 fax: +49-(0)9533-92006-6 96274 Itzgrund/Germany e-Mail: info(a)mhcsoftware.de - - - - - - - - - - - - - - - - - - - - - - - - - - - -

1 0

syslog-ng loses messages
by Aaron watkins 01 Oct '04

01 Oct '04

Hi there, I have been doing some stress testing of syslog-ng as a possible facilitator of event messaging for use on our network. I have set up a system as below: Event Process | \_/ FIFO/pipe | _____\_/_______ [ ] [ Machine 1 ] [ running ] [ syslog-ng ] [______________] | \_/ TCP socket | _____\_/_______ [ ] [ Machine 2 ] [ running ] [ syslog-ng ] [______________] | \_/ FIFO/pipe | \_/ cat Or to describe it, I have a process writing to a pipe on Machine 1, which syslog-ng then forwards via a socket to syslog-ng running on Machine 2, which then outputs to another pipe, which I am catting. My event process is in a while(true) loop, simply writing a number of messages to the pipe and then sleeping for 1 second. The problem I have discovered is that if I kill (-TERM or -9) syslog-ng running on Machine 2, some messages are lost before it starts buffering them in Machine 1. If I kill it while my event process is sleeping and then restart it, the first two messages (once the event process restarts transmitting) will be lost. Alternatively, if I kill it while my event process is writing data (and syslog-ng on Machine 1 is transmitting over the socket), normally about five messages will be lost. Is this a recognised problem and is there a way to work around this? Obviously, I would like to be able to get all of my messages through (up to the buffer limit) without having to reprocess the messages from the local event copy file. I am running this system on Solaris 5.8. The code for my event process is: #!/bin/sh while [ 1 ] do count=0 while [ $count -le 40 ] do echo "${count}: Hello World" >> event_pipe count=`expr ${count} + 1` done sleep 1 done My Machine 1 config file: options { sync(0); # Sends message as soon as it's received time_reopen(10); # Wait 10 before reopening closed connections log_fifo_size(500); # The number of messages that can go in the output queue create_dirs(yes); # Whether to create destination directories, if required }; source src_internalEvents { internal(); }; source src_InputPipe { pipe( "/tmp/event_pipe" ); }; destination dst_localLog { file( "/data/gen/log/nievent_all.log" template("$YEAR-$MONTH-$DAY--$HOUR:$MIN:$SEC ($HOST:$PROGRAM): $MSG\n" )); }; destination dst_localEventCopy { file( "/data/gen/log/nievent_copy.log" template("$MSG\n") ); }; destination dst_OutputSocket { tcp( "imgwrkfl-inp" port(1515) ); }; log { source( src_InputPipe ); destination( dst_localLog ); destination( dst_localEventCopy ); destination( dst_OutputSocket ); }; log { source( src_internalEvents ); source( src_InputPipe ); destination( dst_localLog ); flags( fallback ); }; For Machine 2: options { sync(0); # Sends message as soon as it's received time_reopen(10); # Wait 10 before reopening closed connections log_fifo_size(500); # The number of messages that can go in the output queue create_dirs(yes); # Whether to create destination directories, if required }; source src_internalEvents { internal(); }; source src_InputSocket { tcp( port(1515) ); }; destination dst_localLog { file( "/data/gen/log/nievent_all.log" template( "$YEAR-$MONTH-$DAY--$HOUR:$MIN:$SEC ($HOST:$PROGRAM): $MSG\n" ) ); }; destination dst_localEventCopy { file( "/data/gen/log/nievent_copy.log" template( "$MSG\n") ); }; destination dst_OutputPipe { pipe( "/tmp/prepress/nievents_pipe" template( "$MSG\n") ); }; log { source( src_InputSocket ); destination( dst_localLog ); destination( dst_localEventCopy ); destination( dst_OutputPipe ); }; log { source( src_internalEvents ); source( src_InputSocket ); destination( dst_localLog ); flags( fallback ); }; Thanks in advance for your help. Aaron Watkins

2 1

syslog-ng to MySQL question
by Ron Clark 01 Oct '04

01 Oct '04

Hello all, This may be off topic, but I would be very appreciative of any help. I am using the sqlsyslogd file from http://www.frasunek.com/sources/security/sqlsyslogd/. But when it reads from the logs coming into syslog, on the time stamp field, it adds some text in <> signs, and does not fully capture the time stamp. Does anyone know what these are? And more important, how does one get the script to get fully imput the time stamp in the syslogs? Does anyone have any tricks to the sqlsyslog.c file that might fix this? If this is not the correct forum, then please forward me to the right person or place. Thank you in advance, Ron Clark actual syslog: Sep 30 16:03:05 system1 /kernel: mysql entry: | 15468 | <118>Sep 30 18: | system1 | psmintr |

2 1

/proc/kmsg and chrooted syslog-ng
by Michael D. (Mick) Bauer 01 Oct '04

01 Oct '04

One more ignorance-betraying question about Syslog-ng chroot-jails: what should one do about /proc/kmsg? Maybe "touch /var/logjail/proc/kmsg; ln -s /var/logjail/proc/kmsg /proc/kmsg"? And do I take it that, per another recent posting, one should similarly create a new dev/log in the jail and link to it from /dev/log? I'd like to post a revised Syslog-ng chroot jail procedure later this week. Thanks to the two of you who've replied so far! Mick /-------------------------------------------------\ | Michael D. (Mick) Bauer | | Security Editor, Linux Journal | | Dir. of Value-Subtracted Svcs., Wiremonkeys.org | \-------------------------------------------------/

3 2

syslog-ng to MySQL question
by Ron Clark 01 Oct '04

01 Oct '04

Hello all, This may be off topic, but I would be very appreciative of any help. I am using the sqlsyslogd file from http://www.frasunek.com/sources/security/sqlsyslogd/. But when it reads from the logs coming into syslog, on the time stamp field, it adds some text in <> signs, and does not fully capture the time stamp. Does anyone know what these are? And more important, how does one get the script to get fully imput the time stamp in the syslogs? Does anyone have any tricks to the sqlsyslog.c file that might fix this? If this is not the correct forum, then please forward me to the right person or place. Thank you in advance, Ron Clark actual syslog: Sep 30 16:03:05 system1 /kernel: mysql entry: | 15468 | <118>Sep 30 18: | system1 | psmintr |

1 0

Loghost can't see its own syslogs
by Doug Hanks 30 Sep '04

30 Sep '04

Hello, I have an interesting problem. I have a loghost with the servername of FFLOGSVR. Everything works wonderfully except that FFLOGSVR isn't logging its own logs. When I perform the command "logger FFLOGSVR" I do not see entry in my syslog. I have attached my syslog-ng.conf for review. Thank you in advance, -- - Doug Hanks = dhanks(at)gmail(dot)com # Syslog-ng (new generation) configuration file # syslog.conf by Doug Hanks 9/25/04 # Global options options { time_reopen (10); log_fifo_size (1000); long_hostnames (on); use_dns (yes); use_fqdn (no); create_dirs(yes); dns_cache(yes); keep_hostname(yes); }; # Define where the logs come from - udp and /dev/log only source gateway { unix-stream("/dev/log"); internal(); udp(port(514)); }; # Define some filters so we can break down the logs by facility filter f_auth { facility(auth); }; filter f_syslog { not facility(authpriv, mail); }; filter f_cron { facility(cron); }; filter f_daemon { facility(daemon); }; filter f_kern { facility(kern); }; filter f_mail { facility(mail); }; filter f_user { facility(user); }; filter f_debug { not facility(auth, authpriv, news, mail); }; filter f_messages { level(info..warn) and not facility(auth, authpriv, mail, news); }; filter f_emergency { level(emerg); }; filter f_info { level(info); }; filter f_notice { level(notice); }; filter f_warn { level(warn); }; filter f_crit { level(crit); }; filter f_err { level(err); }; # LogWatch filter f_logwatch { match("su") or match("ssh") or match("sudo"); }; # SU filter f_su_0 { match("su: from [a-zA-Z0-9]+ to") or # AIX match("'su [a-zA-Z0-9]+' succeeded for") or # Solaris match(": su : \+"); # HPUX }; filter f_su_1 { match("su: BAD SU from") or # AIX match("'su [a-zA-Z0-9]+' failed for") or # Solaris match(": su : \-"); # HPUX }; # SSH filter f_ssh_0 { match("Accepted [a-zA-Z0-9]+ for") or # AIX, Solaris, and HPUX match("[a-zA-Z0-9]+ authentication for user [a-zA-Z0-9]+ accepted"); # SAP }; filter f_ssh_1 { match("Failed [a-zA-Z0-9]+ for") or # AIX, Solaris, and HPUX match("[a-zA-Z0-9]+ authentication for user [a-zA-Z0-9]+ failed"); # SAP }; # SUDO filter f_sudo_0 { match("sudo:"); }; filter f_sudo_1 { match("sudo:") and ( match("command not allowed") or match("user NOT in sudoers") ); }; # Define containers where you want to stuff the logs destination localhost { file("/usr/logs/syslog/messages.all"); }; destination loghost { file("/usr/logs/HOSTS/$HOST/messages" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes) ); }; # Program template for alerts destination fifo_alert { program("/usr/local/sbin/email-syslog-alert" template("$HOST:$PROGRAM:$MSG\n") template-escape(yes) ); }; # SU log destination su_0 { file("/usr/logs/syslog/messages.su0"); }; destination su_1 { file("/usr/logs/syslog/messages.su1"); }; # SSH logs destination ssh_0 { file("/usr/logs/syslog/messages.ssh0"); }; destination ssh_1 { file("/usr/logs/syslog/messages.ssh1"); }; # SUDO logs destination sudo_0 { file("/usr/logs/syslog/messages.sudo0"); }; destination sudo_1 { file("/usr/logs/syslog/messages.sudo1"); }; # LogWatch destination logwatch { file("/usr/logs/syslog/logwatch.log"); }; # Regular syslogs destination authlog { file("/usr/logs/syslog/messages.auth"); }; destination cron { file("/usr/logs/syslog/messages.cron"); }; destination daemon { file("/usr/logs/syslog/messages.daemon"); }; destination kern { file("/usr/logs/syslog/messages.kern"); }; destination lpr { file("/usr/logs/syslog/messages.lpr"); }; destination user { file("/usr/logs/syslog/messages.user"); }; destination mail { file("/usr/logs/syslog/messages.mail"); }; destination debug { file("/usr/logs/syslog/messages.debug"); }; destination messages { file("/usr/logs/syslog/messages"); }; destination console { file("/dev/console"); }; # Bring the sources, filters (if any) together and shove it in a log (read destination) log { source(gateway); destination(localhost); }; log { source(gateway); destination(loghost); }; # FIFO alert log { source(gateway); filter(f_su_1); destination(fifo_alert); }; log { source(gateway); filter(f_ssh_1); destination(fifo_alert); }; log { source(gateway); filter(f_sudo_1); destination(fifo_alert); }; # Log SU to roots log { source(gateway); filter(f_su_0); destination(su_0); }; log { source(gateway); filter(f_su_1); destination(su_1); }; # Log SSH stuff log { source(gateway); filter(f_ssh_0); destination(ssh_0); }; log { source(gateway); filter(f_ssh_1); destination(ssh_1); }; # Log SUDO stuff log { source(gateway); filter(f_sudo_0); destination(sudo_0); }; log { source(gateway); filter(f_sudo_1); destination(sudo_1); }; # A master log of SSH, SUDO and SU for LogWatch log { source(gateway); filter(f_logwatch); destination(logwatch); }; # Break down the logs further log { source(gateway); filter(f_auth); destination(authlog); }; log { source(gateway); filter(f_cron); destination(cron); }; log { source(gateway); filter(f_daemon); destination(daemon); }; log { source(gateway); filter(f_kern); destination(kern); }; log { source(gateway); filter(f_mail); destination(mail); }; log { source(gateway); filter(f_user); destination(user); }; log { source(gateway); filter(f_debug); destination(debug); }; log { source(gateway); filter(f_messages); destination(messages); }; log { source(gateway); filter(f_emergency); destination(console); };

2 2

RE: [syslog-ng]Corrupted messages in log
by Dmitri Smirnov 28 Sep '04

28 Sep '04

Thanks a lot, It fixed the problem! Dmitri -----Original Message----- From: Balazs Scheidler [mailto:bazsi@balabit.hu] Sent: Monday, September 27, 2004 1:30 PM To: syslog-ng(a)lists.balabit.hu Subject: RE: [syslog-ng]Corrupted messages in log Hi, I've found the problem, this is syslog-ng 1.6.3 specific, you should upgrade to at least 1.6.4, or apply the patch in the mailing list archive around 2004 May.

1 0

RE: [syslog-ng]Corrupted messages in log
by Dmitri Smirnov 27 Sep '04

27 Sep '04

An example of corrupted message (one line in syslog file instead of six): Sep 27 02:01:09 loghost local1.err MSWTRS [5572]: [ user.error] Error: 256 Couldn't get W3SVC11/ex040926.log from XXX Sep 27 02:01:11 loghost local1.err MSWTRS [5572]: [ user.error] Error: 256 Couldn't get W3SVC15/ex040926.log from XXX Sep 27 02:01:15 loghost local1.err MSWTRS [5572]: [ user.error] Error: 256 Couldn't get W3SVC20/ex040926.log from XXX Sep 27 02:01:15 loghost local1.err MSWTRS [5572]: [ user.error] Error: 256 Couldn't get W3SVC22/ex040926.log from XXX Sep 27 02:01:30 loghost local1.err MSWTRS [5572]: [ user.error] Error: 256 Couldn't get W3SVC7/ex040926.log from XXX Sep 27 02:02:00 REMOTEHOST/REMOTEHOST user.warning loader: [ID 702911 user.warning] Autoload has been locked for over an hour I just realized that I haven't seen two messages from remote hosts contatenated. It only happens with messages generated on loghost (server where syslog-ng is installed) and messages from remote hosts (sun-stream() plus udp()). Platform is Solaris 8 with 117350-06. Dmitri -----Original Message----- From: Balazs Scheidler [mailto:bazsi@balabit.hu] Sent: Monday, September 27, 2004 3:24 AM To: syslog-ng(a)lists.balabit.hu Subject: RE: [syslog-ng]Corrupted messages in log On Sun, 2004-09-26 at 23:44, Dmitri Smirnov wrote: > Thanks, Loic, > > some important parts of config, skipping filters: > > # > source local { sun-streams("/dev/log"); internal(); udp(); }; > > options { > use_fqdn(yes); > use_dns(yes); > dns_cache(yes); > keep_hostname(yes); > chain_hostnames(no); > bad_hostname("^5.*"); > sync(0); > stats(0); > log_fifo_size(1024); > log_msg_size(2048); > use_time_recvd(yes); > dns_cache_expire(36000); > dns_cache_expire_failed(3600); > dns_cache_size(10000); > > }; > > > destination syslog { file("/var/log/syslog" owner(root) group(other) > perm(0644) template("$DATE $FULLHOST $FACILITY.$PRIORITY $MESSAGE\n") > template_escape(no)); }; > > log { source(local); filter(filter1_not); filter(filter2_not); > filter(filter3_not); filter(filter4_not); filter(filter5_not); > destination(syslog); }; In what way are messages corrupted? You said they are concatenated, but could you post an example? It would also be important to check which syslog-ng parts are used, e.g. the message path as it is received from the network. (udp source, sun-stream source) It would also be useful to verify whether it was mangled on the syslog-ng host itself, or it was already mangled before. BTW: it is known that certain kernel messages on Linux might get corrupted, because of the kernel ring-buffer overflow, increasing the ring buffer size can be used to mitigate (but not solve) the problem. -- Bazsi _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

2 1

Missing remote logs
by fRANz 27 Sep '04

27 Sep '04

I've a syslogd that send his log remotly over a vpn tunnel to an openbsd3.4 with syslog-ng(port syslog-ng-1.5.24.tgz). I see a new log on syslogd, I see the packet go into vpn tunnel but syslog-ng not log it immediately, indeed, I not receive all logs, someone only. Can I miss something in syslog-ng config?! This is my syslog-ng.conf I'd like a realtime remote logging: I necessarily transport between 2 syslog-ng in tcp instead udp?! Thank you very much, fRANz # Copyright (c) 1999 anonymous # Copyright (c) 1999 Balazs Scheidler # $Id: syslog-ng.conf.sample,v 1.2 1999/11/15 12:30:41 bazsi Exp $ # options { long_hostnames(off); sync(0); }; source src { unix-stream("/dev/log"); internal(); }; source net { udp(ip(172.16.13.3)); }; destination authlog { file("/var/log/auth.log"); }; destination syslog { file("/var/log/syslog"); }; destination cron { file("/var/log/cron.log"); }; destination daemon { file("/var/log/daemon.log"); }; destination kern { file("/var/log/kern.log"); }; destination lpr { file("/var/log/lpr.log"); }; destination user { file("/var/log/user.log"); }; destination uucp { file("/var/log/uucp.log"); }; destination mail { file("/var/log/mail.log"); }; destination mailinfo { file("/var/log/mail.info"); }; destination mailwarn { file("/var/log/mail.warn"); }; destination mailerr { file("/var/log/mail.err"); }; destination newscrit { file("/var/log/news/news.crit"); }; destination newserr { file("/var/log/news/news.err"); }; destination newsnotice { file("/var/log/news/news.notice"); }; destination debug { file("/var/log/debug"); }; destination messages { file("/var/log/messages"); }; destination console { usertty("root"); }; destination console_all { file("/dev/tty12"); }; destination xconsole { pipe("/dev/xconsole"); }; destination sshd { file("/var/log/secure"); }; destination gate-1 { file("/var/log/gate-1"); }; filter f_auth { facility(auth); }; filter f_authpriv { facility(auth, authpriv); }; filter f_syslog { not facility(authpriv, mail); }; filter f_cron { facility(cron); }; filter f_daemon { facility(daemon); }; filter f_kern { facility(kern); }; filter f_lpr { facility(lpr); }; filter f_mail { facility(mail); }; filter f_user { facility(user); }; filter f_uucp { facility(cron); }; #filter f_ppp { facility(ppp); }; filter f_news { facility(news); }; filter f_debug { not facility(auth, authpriv, news, mail); }; filter f_messages { level(info..warn) and not facility(auth, authpriv, mail, news); }; filter f_emergency { level(emerg); }; filter f_info { level(info); }; filter f_notice { level(notice); }; filter f_warn { level(warn); }; filter f_crit { level(crit); }; filter f_err { level(err); }; filter f_sshd { program(sshd); }; filter f_gate-1 { host("172.16.13.254"); }; log { source(src); filter(f_authpriv); destination(authlog); }; log { source(src); filter(f_syslog); destination(syslog); }; log { source(src); filter(f_cron); destination(cron); }; log { source(src); filter(f_daemon); destination(daemon); }; log { source(src); filter(f_kern); destination(kern); }; log { source(src); filter(f_lpr); destination(lpr); }; log { source(src); filter(f_mail); destination(mail); }; log { source(src); filter(f_user); destination(user); }; log { source(src); filter(f_uucp); destination(uucp); }; log { source(src); filter(f_mail); filter(f_info); destination(mailinfo); }; log { source(src); filter(f_mail); filter(f_warn); destination(mailwarn); }; log { source(src); filter(f_mail); filter(f_err); destination(mailerr); }; log { source(src); filter(f_news); filter(f_crit); destination(newscrit); }; log { source(src); filter(f_news); filter(f_err); destination(newserr); }; log { source(src); filter(f_news); filter(f_notice); destination(newsnotice); }; log { source(src); filter(f_debug); destination(debug); }; log { source(src); filter(f_messages); destination(messages); }; log { source(src); filter(f_emergency); destination(console); }; #log { source(src); filter(f_ppp); destination(ppp); }; log { source(src); destination(console_all); }; log { source(src); filter(f_sshd); destination(sshd); }; log { source(net); filter(f_gate-1); destination(gate-1); }; #

1 0

RE: [syslog-ng]Corrupted messages in log
by Dmitri Smirnov 27 Sep '04

27 Sep '04

Thanks, Loic, some important parts of config, skipping filters: # source local { sun-streams("/dev/log"); internal(); udp(); }; options { use_fqdn(yes); use_dns(yes); dns_cache(yes); keep_hostname(yes); chain_hostnames(no); bad_hostname("^5.*"); sync(0); stats(0); log_fifo_size(1024); log_msg_size(2048); use_time_recvd(yes); dns_cache_expire(36000); dns_cache_expire_failed(3600); dns_cache_size(10000); }; destination syslog { file("/var/log/syslog" owner(root) group(other) perm(0644) template("$DATE $FULLHOST $FACILITY.$PRIORITY $MESSAGE\n") template_escape(no)); }; log { source(local); filter(filter1_not); filter(filter2_not); filter(filter3_not); filter(filter4_not); filter(filter5_not); destination(syslog); }; Dmitri -----Original Message----- From: Loïc Minier [mailto:lool+syslog@via.ecp.fr] Sent: Wednesday, September 22, 2004 3:19 AM To: syslog-ng(a)lists.balabit.hu Subject: Re: [syslog-ng]Corrupted messages in log Dmitri Smirnov <Dmitri.Smirnov(a)fusepoint.com> - Tue, Sep 21, 2004: > syslog-ng 1.6.3, Solaris 8 with latest patches. > Destination to file using templates, network and local sources, > multiple filters. > Started to receive multiple syslog messages from different hosts > concatenated in one big syslog message .. Post your configuration, did you use unix-stream or unix-dgram in your sources? or sun-streams? -- Loïc Minier <lool(a)dooz.org> _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

2 1

Which signal does reload sends ?
by Ali EL MOUSSAWI 24 Sep '04

24 Sep '04

Hi all Can you please tell me what happens when syslog-ng reloads (i.e. receives HUP signal) ? I need to know what signal does it send to the destination programs, and is it a good practice to catch this signal. In fact I don't want my destination program to die at reload, or at least I want to serialize his data structures before it dies, in order to retrieve it when it goes back. On the other hand, if a message arrives from an UDP source while syslog-ng is reloading, will it be lost ? Why the keep alive option is only for tcp source ? Thanks a lot for any help Ali

3 2

blow up a flea
by bedo.sandor＠ln.matav.hu 23 Sep '04

23 Sep '04

Hi All! I and my co-workers have had a trip around "STATS: dropped N" log messages in the last few days, and I am made to share my experience with you. I'm administrating a central log collector, archiver, retriever and analyser service. Of course its heart is syslog-ng. Syslog-ng stores the incoming data in separate files for each recognized subsystems in separate directories for every host. For example you can think about subsystems like cron, postfix, sshd, sendmail, exim, apache, su and sudo, netfilter, ipchains, kernel messages, grsec or execlog for special kernels, System, Security, Application, IIS, ActiveDirectory logs by a Windows source... The analyser part of the system runs a few scripts on the files containing yesterday arrived log messages, for example eximstats, or so. I have written a syslog-ng log analyser script just for fun. One step of the analyser is to collect those evil reputed "STATS: dropped N" messages and draw a graph, that shows the rate of dropped messages based on the last days crop. It was a little bit frustrating to see that graph for a few intranet log forwarder machines, because of more than 30 million lost lines per day! First I couldent believe! "Is that value measured in bytes?? Maybe in bits?" - that was the first idea of one of my co-worker, but the answer was crushing: it is measured in log lines! There have had plenty of agonized crack at the problem: increase the buffer size (certainly this was a silly idea), verify the network interfaces, drivers, cables (the idea was good and shed light on other problemes nevertheless solved the original), plans to upgrade to gigabit ethernet, eliminate every network devices and use only cross utp cables, use more computers for the analysis diffused on the intranet, etc., etc. After all there was an answer message on this list with a little mistake. It suggested that the number in the log lines can only be from tcp destinations. That's wrong! It can arise from other destination too. For example from program(), pipe() and maybe more... and there come the propitiation: f*ckin' debug pipes!!!! We have a local practice of pipe destinations. Chroot-ed environments often have a /var/log/messages pipe inside, and the syslog-ng running in the base system sends the data quarried from the jail back to that pipe for debugging reasons. If you are acting in the jail, you can "cat" the pipe and see the related events without exiting to the base system (where the data is stored in regular files). On the log forwarder machines there are pipes for debugging the incoming tcp and udp log traffic (for example it's nice to explore and locate unknown log sources, to see that remote logging is on, or to identify dummy systems using udp protocol). Pipes seemd to be practical because they didn't eat up the disk space even so ther was a point to access the log stream if it needed. Our solution is an ugly workaround. I have written a little C code that opens a pipe with O_NONBLOCK and sends data from standard input to the pipe. Because of non-blocking io syslog-ng cannot detect the data loss. So pipe destinations are replaced to program(".../sbin/pipecat .../var/log/pipe") destinations to eliminate false scaled syslog-ng statistics. (It is a pity that program("/bin/cat > /var/log/pipe"); generates the same wrong statistic, so my pipecat seems the one and only solution. Do You have any better idea?) I'm sorry if I was boring! I think promlems, their solutions and others experiences can be of service to anybody. Let these be a lesson for all of you out there! :) -- Have a nice day! bSanyI

1 0

SIGHUP not working (Plz Help)
by Shahid Shafi 23 Sep '04

23 Sep '04

Hi guys, I am in slight trouble. I installed syslog-ng, configured my stuff and everything is working fine. I am using newsyslog to rotate logs file every 2 MB. newsyslog is invoked via cron and send HUP to the syslog-ng after rotating. File is rotated and compressed but the new file never got any log data again. I then had to restart the syslog-ng process and everyhting works normally after that. Can somebody help here? I am using syslogng 1.6.5 with libol 0.3.14 thx Shahid

2 2

RE: [syslog-ng]problem with different timezones
by ARGEXT-HIDALGO, FRANCISCO 23 Sep '04

23 Sep '04

>the problem with this is that bufferring happens at a later stage, after >the message is already formatted. Maybe you could achieve the same by >putting the received timestamp at the receiving syslog-ng. No, that's just what I need, to know when originally was recieved from the router, before it's buffered. So the $S_DATE and $R_DATE macros are the solution I was looking for, thank you very much. >-- >Bazsi _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

1 0

Re: [syslog-ng]Problems with syslog-ng remote logging on Sun/Linu x environment!
by Loïc Minier 23 Sep '04

23 Sep '04

Jörgen Ekstedt <jorgen.ekstedt(a)hiq.se> - Thu, Sep 23, 2004: > I tried to look up if the connection is beeing made with netstat and > tcpdump, and at the first glance it seem to be the problem. Now I am not > that kind of network guru to really say that IS doesnt connect. I suggest saving the output of tcpdump/snoop to a file and open it with Ethereal. Regards, -- Loïc Minier <lool(a)dooz.org>

1 0

problem with different timezones
by ARGEXT-HIDALGO, FRANCISCO 23 Sep '04

23 Sep '04

I have the following situation: I have several logging servers in different timezones and these send all their information with TCP to a central logging server. What I don't understand is that when I recieve a log in the central log server I recieve the exact same line that appears in the log server that sent the log. The date isnt updated no matter if I use or not the use_time_recvd() option. The only solution I found was to use a template with the $DATE macro. That would be my first doubt, why use_time_recvd() doesn't work? My second doubt is, is there any macro for the time sent?, so I can have in the same log (on the central logging) the time sent and the time received, so I know if there was any delay. For example if the link went down and they were retained in the original server for a while. (the servers would be sync'ed in time) Thank you.

2 1

unsubscribe
by Wyner, Peter 23 Sep '04

23 Sep '04

1 0

AIX log workaround
by bedo.sandor＠ln.matav.hu 23 Sep '04

23 Sep '04

Hi, In an earlier version of syslog-ng's changelog I have read about a patch that workarounds the AIX's "Message from xy:" and "Forwarded from xy:" idiocy. In the version 1.6.x it seems to be not fixed. I store my lod data in separate files with the name containing $HOST, but 1.6.x substitues $HOST to the string "Message" by the AIX log lines. Am I something misanderstanding? Had that patch gone about somthing else? Is there any possibility to separate my hosts log data? Thanks, bSanyI

2 1

STATS: dropped N
by bedo.sandor＠ln.matav.hu 23 Sep '04

23 Sep '04

Hi, Could You tell me what is the measure exactly of the number at the end of the "STATS: dropped" log lines? I have really great numbers found... Thanks, bSanyI

3 2

Problems with syslog-ng remote logging on Sun/Linux environment!
by Jörgen Ekstedt 22 Sep '04

22 Sep '04

Hi! I have a loghos that is running RedHat9 and everything work just fine. except for one Sun we have in the network I have configured syslog-ng to include support for Doors and sun-streams,and it logs everything well to its local logfiles. I have configured it to send the logs via tcp to my loghost, but nothing is beeing sent? Anyone got any good hint ? BR Jorgen.E

2 1

Corrupted messages in log
by Dmitri Smirnov 22 Sep '04

22 Sep '04

syslog-ng 1.6.3, Solaris 8 with latest patches. Destination to file using templates, network and local sources, multiple filters. Started to receive multiple syslog messages from different hosts concatenated in one big syslog message .. Can be a result of new filters or increased load on syslog. Did anyone see anything like this? Dmitri

2 1

STATS: dropped 0
by Angelita Soares de Camargos 21 Sep '04

21 Sep '04

Hello everybody. I installed the syslog-ng 1.6.5, storing the logs in mysql 4.0.21 and it seems to be working fine, but sometimes I get this message at the log: "STATS: dropped 0". Reading older messages I understood that some log that did not match any of my filters and was flushed. Well, how can I know what message is it. Below, a part of my syslog-ng.conf: filter f_emerg { level(emerg); }; filter f_kern_info {facility(kern) and not priority(debug); }; filter f_user_info {facility(user) and not priority(debug); }; filter f_mail_info {facility(mail) and not priority(debug); }; filter f_daemon_info {facility(daemon) and not priority(debug); }; filter f_auth_info {facility(auth) and not priority(debug); }; filter f_cron_info {facility(cron) and not priority(debug); }; filter f_err {priority(err..emerg); }; filter f_messages{ not priority(debug); }; thanks in advance, Angel.

2 2

Message retension period
by Jim Mozley 20 Sep '04

20 Sep '04

I have a remote server running syslog-ng collecting messages from network devices that are local to it and transferring them to a central server using TCP. This is to avoid messages being lost when UDP is used a a network problem occurs. All works as it should and I have tested breaking the link between the local collection station and the central server; the messages are passed to the central server when the link is restored. I had a question regarding how long the remote server will hold on to the messages when it cannot pass them to the central server. Is this a time dependent parameter, memory limit or hard coded somewhere? I couldn't find a destination driver configuration parameter for this in the documentation. Jim Mozley

2 9