March 2002 - syslog-ng - lists.balabit.hu

[PATCH] Unixtime Macros
by Gert Menke 29 Mar '02

29 Mar '02

Hi! The following patch adds the Macros $UNIXTIME, $S_UNIXTIME and $R_UNIXTIME. They give you the time in the internal unix time format (seconds since 1970 or so). Since it is useful for me, it is probably useful for others, too. This patch is against syslog-ng-1.5.14 with my sourceip127-patch (which I posted on 05.Feb.2002), but will also work with the plain syslog-ng-1.5.14 sources (with an offset of -3 lines). Balazs, please include this patch in the next release if you find it useful. Greetings Gert

1 0

templates...
by Dave Patterson 29 Mar '02

29 Mar '02

Hello, has anyone had any luck /w templates <to configure the way syslog-ng writes its log> I'd like to call a template file containing the information if possible. this is my current config: template {date '+%b %e %T' hostname PROGRAM:} destination d_program {file("/var/syslog-ng/$HOST/$PROGRAM/$MONTH_$DAY" perm(0644) compress (yes) template(template)) }; I'm not haveing any luck at all,... any help is apprecated, Thanks Dave Patterson

2 1

RE: [syslog-ng]syslog-ng pix help
by John Coke 28 Mar '02

28 Mar '02

Are you still having problems? I have my pixen logging, I might be able to help. -----Original Message----- From: Steve [mailto:steve_slog_ml@timko.com] Sent: Thursday, March 28, 2002 1:49 PM To: syslog-ng(a)lists.balabit.hu Subject: Re: [syslog-ng]syslog-ng pix help ok now I got pix.log file collecting info but It seems the pix is sending syslog out to err so messages and console is still logging everything which is ugly.. what am I donig wrong? I tried filter f_not_pix { host(regexp to match host); }; that didn't work.. -steve On Thu, 28 Mar 2002, Steve wrote: > ok I got logging in debug mode working on the pix but now everything is > coming to console and messages.. > > I'm trying to get it just to log to pix.log filename.. > > nothing there.. > > -s > > > On Thu, 28 Mar 2002, Steve wrote: > > > Hello, > > > > I saw this post: > > > > http://lists.balabit.hu/pipermail/syslog-ng/2001-November/002760.html > > > > and I'm kinda having a simular problem, but I can't even get logging to > > work. At one point I had the pix logs going to console... > > > > > > Now I got nothing.. > > > > Heres what I got in syslog-ng.conf > > > > destination pix { file("/var/log/pix.log"); }; > > # > > # host filters > > # > > filter f_pix { host(10.1.5.3); }; > > log { source(src); filter(f_pix); destination(pix); }; > > > > > > Does it matter where you put this lines? > > > > thanks for the help if u can.. > > > > > > oh yea > > > > pix has this enabled: > > > > pixfirewall# show loggi > > Syslog logging: enabled > > Timestamp logging: disabled > > Standby logging: disabled > > Console logging: disabled > > Monitor logging: disabled > > Buffer logging: disabled > > Trap logging: disabled > > History logging: disabled > > > > along with > > > > ogging on > > no logging timestamp > > no logging standby > > no logging console > > no logging monitor > > no logging buffered > > no logging trap > > no logging history > > logging facility 20 > > logging queue 512 > > logging host inside 10.1.5.222 > > > > > > > > thanks for any help > > > > -steve > > > > > > > > _______________________________________________ > > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > > https://lists.balabit.hu/mailman/listinfo/syslog-ng > > > > > _______________________________________________ > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > https://lists.balabit.hu/mailman/listinfo/syslog-ng > _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng

1 0

syslog-ng pix help
by Steve 28 Mar '02

28 Mar '02

Hello, I saw this post: http://lists.balabit.hu/pipermail/syslog-ng/2001-November/002760.html and I'm kinda having a simular problem, but I can't even get logging to work. At one point I had the pix logs going to console... Now I got nothing.. Heres what I got in syslog-ng.conf destination pix { file("/var/log/pix.log"); }; # # host filters # filter f_pix { host(10.1.5.3); }; log { source(src); filter(f_pix); destination(pix); }; Does it matter where you put this lines? thanks for the help if u can.. oh yea pix has this enabled: pixfirewall# show loggi Syslog logging: enabled Timestamp logging: disabled Standby logging: disabled Console logging: disabled Monitor logging: disabled Buffer logging: disabled Trap logging: disabled History logging: disabled along with ogging on no logging timestamp no logging standby no logging console no logging monitor no logging buffered no logging trap no logging history logging facility 20 logging queue 512 logging host inside 10.1.5.222 thanks for any help -steve

1 2

Feature request: Default template
by Gert Menke 28 Mar '02

28 Mar '02

Hello, Balazs! I need a configuration option for a default template which is used whenever I don't specify one explicitly. If you don't have the time for implementing it in the near future I would very much appreciate any hints on how to do it myself. Thank you! Greetings, Gert

1 0

Re: [syslog-ng]DNS caching
by Ted_Rule＠flextech.co.uk 21 Mar '02

21 Mar '02

A couple of things... > You also mention "syslog-ng blocks on DNS queries, so enabling DNS may lead > to a Denial of Service attack." in your documentation. Does this mean that > syslog messages which are received by the NIC, while syslog-ng performs a > synchronous DNS lookup, are stored in the kernels receive buffer or are > dropped? My belief is that your presumption is indeed correct, and that the kernel will store such inbound UDP messages in the local socket buffer ( usually of the order of 10's of kbytes big ) whilst performing gethostbyaddr() or equivalent. Reworking any such daemon to be multi-threaded enough to use the re-entrant version gethostbyaddr_r() instead, and thereby allow only the given source socket whence the original syslog message arrived to be blocked pending the DNS response, is something I would imagine syslog-ng has not yet been encoded to perform. However, I have come across a even worse problem with syslog ( non-NG syslogd that is ) and DNS, namely a terrifying full blown hard deadlock. This for Linux kernel 2.2.19 + latest RedHat syslogd ( with Unix DGRAM sockets preferred ) + local bind 8.2.3 server, whilst acting as remote syslog receiver. The problem only appeared to arise under extreme conditions, once in a blue moon. After months of tortuous attempts at debugging the problem, the issue appears to be that syslogd ( as I said this is not syslog-ng but it might yet apply which is why I raise the issue here ) deadlocks when it attempts to perform gethostbyaddr() ( and hence res_query() ) to localhost:53, whilst at exactly the same instant, the DNS server sitting on localhost:53 perform a syslog operation in respect of something/anything/perhaps completely unrelated, ( such as logging a Zone Transfer complete ). At this point syslogd/gethostbyddr() are blocked waiting on the DNS server, and the DNS server is blocked waiting on syslogd(). Note that this flies in the face of what one might deduce might happen, namely that gethostbyaddr() times out waiting for localhost:53 to reply ( because the DNS server is temporarily deadlocked waiting for syslogd itself ), and works round the deadlock by performing the DNS query against the secondary DNS server in /etc/resolv.conf. Once this deadlock has occurred, any given process which potentially uses syslog deadlocks on the next occasion it attempts to use syslog(). Over a matter of minutes or less, the whole system deadlocks on syslogd, BUT snmpd ( which very rarely syslogs ) and ping ( which is unlikely to syslog unless something in ipchains or similar performs a printk() ) continue to function Ok. Thus simplistic monitoring of the host indicates nothing untoward via SNMP / ping but end-user experience is deadly. The best solution I have so far is to ensure syslogd never uses a local DNS server, but instead always uses a remote DNS server to perform the lookups. Under these conditions, any such deadlock appears to be recoverable, as gethostbyaddr() never blocks waiting on the local DNS server. Whilst using nscd improves the situation - make it less likely to occur - and including a small DNS cache within the syslogd process itself, make it less likely still, neither workround completely fixes the potential deadlock. And of course even trying to debug for such a deadlock is fraught with difficulties. I feel the really proper solution where one strongly desires to run a local DNS server, as intimated above, is to multi-thread the syslog daemon, use a fully threaded resolver library, and only block the socket the message arrived on whilst performing the gethostbyaddr_() - and I haven't had enough strong coffee all year to face attempting that nightmare rewrite! As to why the deadlock really occurs - i.e. why gethostbyaddr() doesn't simply timeout, fall through to backup DNS server, and thereby release the deadlock - I have no idea. Maybe it somehow got fixed in Kernel 2.4? Again, testing the theory is tricky. If anyone has further ideas on releasing the deadlock, I'd be only too happy to hear from you. By the by, I've been looking into migrating to syslog-ng for other reasons anyway, mainly due to its more fine-grained filtering structure, and despite some minor reservations I'm happily impressed with what I've seen. Many thanks to all involved in its development so far. Ted Rule, Flextech Television. *************************************************************************************************** This E-mail message, including any attachments, is intended only for the person or entity to which it is addressed, and may contain confidential information. If you are not the intended recipient, any review, retransmission, disclosure, copying, modification or other use of this E-mail message or attachments is strictly forbidden. If you have received this E-mail message in error, please contact the author and delete the message and any attachments from your computer. You are also advised that the views and opinions expressed in this E-mail message and any attachments are the author's own, and may not reflect the views and opinions of FLEXTECH Television Limited. ***************************************************************************************************

1 0

Re: [syslog-ng]DNS caching
by Michael Renner 21 Mar '02

21 Mar '02

At 09:01 20.03.2002 +1000, you wrote: >On Tue, 19 Mar 2002 at 10:24am (+0100), Balazs Scheidler wrote: >[...] > > > > > can you check this patch, whether it fixes your problem ? > > > > diff -u -r1.37 cfgfile.c > > --- cfgfile.c 2001/09/03 16:42:23 1.37 > > +++ cfgfile.c 2002/03/19 09:23:55 > >[...] > >Thank you... it appears to be working as advertised now. When running with >NSCACHE_DEBUG enabled we get lots of messages like... > >.... which looks good. CPU usage for the syslog process has dropped from >~80% to ~40% (hazzah!) and the named process that was doing local caching >has dropped from ~10% to almost nill. Dear Bazsi, Thanks, also works flawless here, I get hardly any hits on my dnscache. Btw. the default option of dns_cache is "on", maybe you should document this or change it to "no", otherwise people who upgrade to newer versions (with the fixed dns_cache) may be confused by the "new" behaviour of syslog-ng. You also mention "syslog-ng blocks on DNS queries, so enabling DNS may lead to a Denial of Service attack." in your documentation. Does this mean that syslog messages which are received by the NIC, while syslog-ng performs a synchronous DNS lookup, are stored in the kernels receive buffer or are dropped? best regards, -- Michael Rennner Junior System Engineer Inode Telekommunikationsdienstleistungs GmbH - http://www.inode.at support(a)inode.at, Tel.: 05 9999-0, Fax.: 05 9999-2699

2 1

position dependencie of rules and "flags(final);"
by Volker Apelt 20 Mar '02

20 Mar '02

Hello, Are syslog rules position dependend? Is "flags(final);" position dependend? Is example #1 different from #2 in what it logs? Theory: # example 1 log { source; filter1; destination1; } log { source; filter1; filter2; destination2; flags(final); } # example 2 log { source; filter1; filter2; destination2; flags(final); } log { source; filter1; destination1; } Let's say, that filter2 matches only a sub set of the messages caught by filter1 and define: set1 := messages that matched by filter1 set2 := messages that matched by filter2 setLeftover := set1 without messages of set2 which means set1 == setLeftover + set2 In example #2 destination1 receives the messages of setLeftover and destination2 receives set2. But if rules+final are position dependend, then destination1 in example #1 will log _all_ messages of source1, _including_ those of filter2. So, destination1 receives the messages of set1 If they are not position dependend both destinations will receive the same message sets as in example #2. Is syslog-ng implemented in a way to Or as a practical example from my configuration: Will this configuration do what it is intended to? ## in pseudo code of what I am trying to do. if ( is it from "ssh" ) { if ( is loglevel equal or worse than "error" ){ send it to D_udp_network and D_tty_all } send it to D_ssh final ; # ie: dont send it any place else } ## I'll try to solve that with two different rules, one with a flag(final) and one without. I try to log all messages of ssh into one file and all messages of ssh above a certain log level (error or worse) shall be send to another destination (network + alert tty), too. All messages caught by the second rule must be logged to the first destination and No message from ssh shall be logged any place else, and I don't want to write a filter(f_no_ssh); into each and every log{} rule. # A message from ssh with loglevel emergency If syslog rules are postion dependend #### options { long_hostnames(off); sync(0); }; source S_local { unix-stream("/dev/log"); file("/proc/kmsg"); internal();}; ## destination D_udp_network{ udp( 192.168.51.2 port(514)); }; destination D_ssh { file("/var/log/ssh" perm(0640) owner(root) group(log)); }; ## filter l_normal_or_worse { level(normal..emerg); }; filter l_err_or_worse { level(err..emerg); }; ## filter f_ssh { program("ssh.*") or program("su"); }; ##### TWO RULES accepting messages from ssh ##### first without "final", second with "final" ## ## selected ssh messages of level error or worse ## send critical ssh messages to all tty's log { source(S_local); filter(f_ssh); filter(l_err_or_worse); destination( D_udp_network ); destination(D_tty_net); }; ## secure login / ssh / su ... ## shall store all messages (normal ... emerg) of ssh log { source(S_local); filter(f_ssh); filter(l_normal_or_worse); destination(D_ssh); flags(final); }; What kind of output on D_ssh and D_udp_network should I expect if there are these two messages? ## example messages #loglevel message [normal] datum host sshd[550]: some normal message of sshd [error] datum host sshd[550]: some error output of sshd Thank you, Volker Apelt -- Volker Apelt volker_apelt .@. yahoo.de (remove the dots, please) Dipl. Chem. +49 6172 31126

2 3

template option for tcp destination
by Nam 20 Mar '02

20 Mar '02

Does the template() option work for tcp destination? Help docs only list the option for file destination, but it also works with pipe. I would like to be able to apply formatting on the output for data inputing into a vendor product without having to use netcat (feeding from a piped destination). Nam __________________________________________________________ Get your FREE personalized e-mail at http://www.canada.com

2 1

DNS caching
by Matthew Melvin 19 Mar '02

19 Mar '02

Hello, I'm loath to ask a question right off the blocks but I wasn't able to find too much in the archives about this... I'm using syslog-ng 1.5.14 and am trying to get the 'dns_cache(yes);' option to work for me. When compiled with -DNSCACHE_DEBUG I get this message to start with... initialising nscache: 1007 entries, 3600/60 expiration. ... but no messages to suggest nscache_lookup() or any of the other cache functions are ever being called. Is there more to this option than meets the eye? Other recent discussions have talked about using djb's dnscache or nscd to do this instead and questioned the wisdom of using caching internal to syslog-ng. Should I not be using this feature? Does it, in fact, work at all? M. -- WebCentral Pty Ltd Australia's #1 Internet Web Hosting Company Level 1, 96 Lytton Road. Network Operations - Systems Engineer PO Box 4169, East Brisbane. phone: +61 7 3249 2500 Queensland, Australia. pgp key id: 0x900E515F

2 2