- syslog-ng - lists.balabit.hu

DOMAIN Name Variable
by Breitman, Jason 15 Mar '06

15 Mar '06

I am using the below line in my syslog-ng configuration file. destination maillog { file("/var/log/clients/$YEAR/$MONTH/$HOST/mail.log"); }; I am looking to add a variable for the domain name and have been unable to find one. Can someone help me out? Thank you. Jason Breitman jason.breitman(a)blackrock.com THE INFORMATION CONTAINED IN THIS MESSAGE AND ANY ATTACHMENT MAY BE PRIVILEGED, CONFIDENTIAL, PROPRIETARY OR OTHERWISE PROTECTED FROM DISCLOSURE. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, copying or use of this message and any attachment is strictly prohibited. If you have received this message in error, please notify us immediately by replying to the message and permanently delete it from your computer and destroy any printout thereof.

6 8

RE: [syslog-ng] solaris UDP loss
by SOLIS, ALEX 14 Mar '06

14 Mar '06

What exactly is Garbage collecting? I don't get those in my logs. Alex ________________________________ From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Cary, Kim Sent: Tuesday, March 14, 2006 12:30 PM To: syslog-ng(a)lists.balabit.hu Subject: [syslog-ng] solaris UDP loss Finally got around to reading the docs and applying the recommendations. FIRST - thanks much! NOW - my results: # ndd /dev/udp udp_recv_hiwat 65536 # ndd /dev/udp udp_max_buf 1073741824 # date ; netstat -s | grep udpInOverflows Tue Mar 14 10:15:16 udpInCksumErrs = 0 udpInOverflows =677996405 # date ; netstat -s | grep udpInOverflows Tue Mar 14 10:15:19 udpInCksumErrs = 0 udpInOverflows =677996571 # date ; netstat -s | grep udpInOverflows Tue Mar 14 10:15:22 udpInCksumErrs = 0 udpInOverflows =677996726 I'm pretty sure this is just the Firewall sending more than 55M lines of connection/deny logs over UDP. IF I was to switch the FW over to TCP logging from UDP, would this be a better solution? BTW, here are a few words from syslog-ng: Mar 14 09:32:32 syslog-ng[20685]: STATS: dropped 0 Mar 14 09:42:32 syslog-ng[20685]: STATS: dropped 0 Mar 14 09:52:32 syslog-ng[20685]: STATS: dropped 0 Mar 14 10:02:32 syslog-ng[20685]: STATS: dropped 0 Mar 14 10:04:53 syslog-ng[20685]: Garbage collecting while busy... Mar 14 10:04:58 syslog-ng[20685]: Objects alive: 310, garbage collected: 2999695 Mar 14 10:12:32 syslog-ng[20685]: STATS: dropped 0 Mar 14 10:16:22 syslog-ng[20685]: Garbage collecting while idle... Mar 14 10:16:24 syslog-ng[20685]: Objects alive: 318, garbage collected: 1129970 Mar 14 10:22:32 syslog-ng[20685]: STATS: dropped 0 and its config: source s_udp_inetpix { udp(ip(192.168.0.1)); }; On Mar 7, 2006, at 3:01 PM, syslog-ng-request(a)lists.balabit.hu wrote: | 1) Am I reading that loss right?? Probably, you might however want to snoop on the interface to see what kind of udp packets come on your interface. | 2) Any tips from Solaris/syslog-ng tuners would be appreciated! udp_max_buf does not set the queue length of the udp socket, which by the way can have a different value for each socket... You could have a look at: http://sunsolve.sun.com/search/document.do?assetkey=1-30-3218-1 basically: increasing udp_max_buf without increasing udp_recv_hiwat has no meaning. Furthermore, you can increase you socket buffer that way up to 64k (Solaris 8 & 9), if you want to increase it further up you must use the setsockopt call (up to udp_max_buf which has a maximum value of 1GB). Here is the official SUN documentation regarding this: http://docs.sun.com/app/docs/doc/817-0404/6mg74vsb5?a=view#gbtag Now regarding your packet loss issue. I would increase udp_recv_hiwat -> 65536 udp_max_buf -> 1073741824 (you will never get here anyway) Then I would try to play with syslog-ng config: log_fifo_size, log_iw_size and log_fetch_limit. But here I'd appreciate a syslog-ng expert to step in and tell us what to do more preceisely. This e-mail contains Omaha Public Power District's confidential and proprietary information and is for use only by the intended recipient. Unless explicitly stated otherwise, this e-mail is not a contract offer, amendment, nor acceptance. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.

2 1

RE: [syslog-ng] DOMAIN Name Variable
by Breitman, Jason 14 Mar '06

14 Mar '06

To clarify what I want servername.domainname.com or servername.city.domainame.com I believe the first name is the server name and everything else is the domain name. ================================================ #!/bin/sh if [ $# != 1 ] ; then echo "USAGE: $0 fqdn" exit 1 fi FQDN=$1 DOMAINNAME=`echo ${FQDN} | cut -d. -f2,3,4,5` echo ${DOMAINNAME} ================================================ I would like to put all servers in the same domain under one directory. This will allow me to allow access to different user groups based on domain name. Also, If I wanted, I can leave out the host name and have one big file for each domain name. -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Rob Munsch Sent: Tuesday, March 14, 2006 4:03 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] DOMAIN Name Variable oh yes, of course i forgot. we certainly wouldn't want the US to stick .us on the end of their domains and create the impression of a global standard or anything, sigh... David Anderson wrote: >Rob, > >Monday, March 13, 2006, 11:47:52 PM, you wrote: > > > >>that would be the TLD, yes: but i got the impression jason means >>'domain' more like xyz.com foo.edu how.de >> >> > > > >>so therefore 'starting with the string before the last dot.' >> >> > >That doesn't work, as many domains outside of the US have a 3-part >domain, as in xyz.co.uk. > > > -- Rob Munsch Solutions For Progress IT _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html THE INFORMATION CONTAINED IN THIS MESSAGE AND ANY ATTACHMENT MAY BE PRIVILEGED, CONFIDENTIAL, PROPRIETARY OR OTHERWISE PROTECTED FROM DISCLOSURE. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, copying or use of this message and any attachment is strictly prohibited. If you have received this message in error, please notify us immediately by replying to the message and permanently delete it from your computer and destroy any printout thereof.

2 1

RE: [syslog-ng] solaris UDP loss
by SOLIS, ALEX 14 Mar '06

14 Mar '06

Just in case: If the firewall is a Cisco PIX you might want to reconsider. If the PIX looses its ability to send TCP messages to its loghost then it denies all traffic until loghost connectivity is restored. ________________________________ From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Cary, Kim Sent: Tuesday, March 14, 2006 12:30 PM To: syslog-ng(a)lists.balabit.hu Subject: [syslog-ng] solaris UDP loss Finally got around to reading the docs and applying the recommendations. FIRST - thanks much! NOW - my results: # ndd /dev/udp udp_recv_hiwat 65536 # ndd /dev/udp udp_max_buf 1073741824 # date ; netstat -s | grep udpInOverflows Tue Mar 14 10:15:16 udpInCksumErrs = 0 udpInOverflows =677996405 # date ; netstat -s | grep udpInOverflows Tue Mar 14 10:15:19 udpInCksumErrs = 0 udpInOverflows =677996571 # date ; netstat -s | grep udpInOverflows Tue Mar 14 10:15:22 udpInCksumErrs = 0 udpInOverflows =677996726 I'm pretty sure this is just the Firewall sending more than 55M lines of connection/deny logs over UDP. IF I was to switch the FW over to TCP logging from UDP, would this be a better solution? BTW, here are a few words from syslog-ng: Mar 14 09:32:32 syslog-ng[20685]: STATS: dropped 0 Mar 14 09:42:32 syslog-ng[20685]: STATS: dropped 0 Mar 14 09:52:32 syslog-ng[20685]: STATS: dropped 0 Mar 14 10:02:32 syslog-ng[20685]: STATS: dropped 0 Mar 14 10:04:53 syslog-ng[20685]: Garbage collecting while busy... Mar 14 10:04:58 syslog-ng[20685]: Objects alive: 310, garbage collected: 2999695 Mar 14 10:12:32 syslog-ng[20685]: STATS: dropped 0 Mar 14 10:16:22 syslog-ng[20685]: Garbage collecting while idle... Mar 14 10:16:24 syslog-ng[20685]: Objects alive: 318, garbage collected: 1129970 Mar 14 10:22:32 syslog-ng[20685]: STATS: dropped 0 and its config: source s_udp_inetpix { udp(ip(192.168.0.1)); }; On Mar 7, 2006, at 3:01 PM, syslog-ng-request(a)lists.balabit.hu wrote: | 1) Am I reading that loss right?? Probably, you might however want to snoop on the interface to see what kind of udp packets come on your interface. | 2) Any tips from Solaris/syslog-ng tuners would be appreciated! udp_max_buf does not set the queue length of the udp socket, which by the way can have a different value for each socket... You could have a look at: http://sunsolve.sun.com/search/document.do?assetkey=1-30-3218-1 basically: increasing udp_max_buf without increasing udp_recv_hiwat has no meaning. Furthermore, you can increase you socket buffer that way up to 64k (Solaris 8 & 9), if you want to increase it further up you must use the setsockopt call (up to udp_max_buf which has a maximum value of 1GB). Here is the official SUN documentation regarding this: http://docs.sun.com/app/docs/doc/817-0404/6mg74vsb5?a=view#gbtag Now regarding your packet loss issue. I would increase udp_recv_hiwat -> 65536 udp_max_buf -> 1073741824 (you will never get here anyway) Then I would try to play with syslog-ng config: log_fifo_size, log_iw_size and log_fetch_limit. But here I'd appreciate a syslog-ng expert to step in and tell us what to do more preceisely. This e-mail contains Omaha Public Power District's confidential and proprietary information and is for use only by the intended recipient. Unless explicitly stated otherwise, this e-mail is not a contract offer, amendment, nor acceptance. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.

1 0

Re: syslog-ng Digest, Vol 11, Issue 20
by dwarnock＠harbrook.net 14 Mar '06

14 Mar '06

I am on holiday on Wednesday 15th March returning on Thursday 16th. If your query is urgent please contact the office on +44 (0)1423 508060 or Neil Harvey on +44 (0)15395 64734. Regards David Warnock

2 1

solaris UDP loss
by Cary, Kim 14 Mar '06

14 Mar '06

Finally got around to reading the docs and applying the recommendations. FIRST - thanks much! NOW - my results: # ndd /dev/udp udp_recv_hiwat 65536 # ndd /dev/udp udp_max_buf 1073741824 # date ; netstat -s | grep udpInOverflows Tue Mar 14 10:15:16 udpInCksumErrs = 0 udpInOverflows =677996405 # date ; netstat -s | grep udpInOverflows Tue Mar 14 10:15:19 udpInCksumErrs = 0 udpInOverflows =677996571 # date ; netstat -s | grep udpInOverflows Tue Mar 14 10:15:22 udpInCksumErrs = 0 udpInOverflows =677996726 I'm pretty sure this is just the Firewall sending more than 55M lines of connection/deny logs over UDP. IF I was to switch the FW over to TCP logging from UDP, would this be a better solution? BTW, here are a few words from syslog-ng: Mar 14 09:32:32 syslog-ng[20685]: STATS: dropped 0 Mar 14 09:42:32 syslog-ng[20685]: STATS: dropped 0 Mar 14 09:52:32 syslog-ng[20685]: STATS: dropped 0 Mar 14 10:02:32 syslog-ng[20685]: STATS: dropped 0 Mar 14 10:04:53 syslog-ng[20685]: Garbage collecting while busy... Mar 14 10:04:58 syslog-ng[20685]: Objects alive: 310, garbage collected: 2999695 Mar 14 10:12:32 syslog-ng[20685]: STATS: dropped 0 Mar 14 10:16:22 syslog-ng[20685]: Garbage collecting while idle... Mar 14 10:16:24 syslog-ng[20685]: Objects alive: 318, garbage collected: 1129970 Mar 14 10:22:32 syslog-ng[20685]: STATS: dropped 0 and its config: source s_udp_inetpix { udp(ip(192.168.0.1)); }; On Mar 7, 2006, at 3:01 PM, syslog-ng-request(a)lists.balabit.hu <mailto:syslog-ng-request@lists.balabit.hu> wrote: | 1) Am I reading that loss right?? Probably, you might however want to snoop on the interface to see what kind of udp packets come on your interface. | 2) Any tips from Solaris/syslog-ng tuners would be appreciated! udp_max_buf does not set the queue length of the udp socket, which by the way can have a different value for each socket... You could have a look at: http://sunsolve.sun.com/search/document.do?assetkey=1-30-3218-1 <http://sunsolve.sun.com/search/document.do?assetkey=1-30-3218-1> basically: increasing udp_max_buf without increasing udp_recv_hiwat has no meaning. Furthermore, you can increase you socket buffer that way up to 64k (Solaris 8 & 9), if you want to increase it further up you must use the setsockopt call (up to udp_max_buf which has a maximum value of 1GB). Here is the official SUN documentation regarding this: http://docs.sun.com/app/docs/doc/817-0404/6mg74vsb5?a=view#gbtag <http://docs.sun.com/app/docs/doc/817-0404/6mg74vsb5?a=view#gbtag> Now regarding your packet loss issue. I would increase udp_recv_hiwat -> 65536 udp_max_buf -> 1073741824 (you will never get here anyway) Then I would try to play with syslog-ng config: log_fifo_size, log_iw_size and log_fetch_limit. But here I'd appreciate a syslog-ng expert to step in and tell us what to do more preceisely.

1 0

Message loss (probably) within syslog-ng
by Vincent Haverlant 14 Mar '06

14 Mar '06

Hi, I get a kind of message loss trouble like in some previous message with the subject "remote logging not reliable", but in his case, the remote logging was done other tcp and this was pointed as the probable cause of message loss. In my case only udp is involved. Description of the infra: About 2500 unix hosts sending logs via their original syslog daemon (Solaris or RedHat). They are set up that way: ===== kern.info @logserver *.error;user.none @logserver auth.info @logserver ===== The central syslog-ng (1.9.9) server is a solaris 8 host configured that way: ===== options { time_reopen (1); time_reap(600); stats_freq(60); log_fifo_size (25000000); keep_hostname (yes); long_hostnames (no); use_dns (yes); dns_cache (yes); dns_cache_size(3000); use_fqdn (no); # utilisation du nom court de la machine owner("root"); # Logs owner group("sys"); # Logs group owner perm(0755); dir_owner("sysexplo"); # Directory Owner dir_group("sys"); # Directpry Group dir_perm(0775); # Directory Perm create_dirs (yes); use_time_recvd(yes); # gc_idle_threshold(1000); # gc_busy_threshold(100000); }; # # Configuration directives for remote logs # source lan { udp (port(514)); }; destination hostfiles { file("/projets/SYS/sysexplo/syslogdata/$YEAR$MONTH$DAY.logremote/$HOST" owner("sysexplo") group("sys") perm(0755) template("$ISODATE $HOST $MSG\n") ); }; log { source(lan); destination(hostfiles); flags(final); }; # #local logs # source localmsg { sun-stream("/dev/log" door("/etc/.syslog_door")); internal(); }; destination syslog { file("/var/log/syslog"); }; destination authlog { file("/var/adm/authlog"); }; destination messages { file("/var/adm/messages"); }; # filters to mimic traditional Solaris logging filter f_mail { facility(mail); }; filter f_auth { level(info) and facility(auth, authpriv); }; filter f_not_mail { not facility(mail); }; log { source(localmsg); filter(f_auth); destination(authlog); }; log { source(localmsg); filter(f_mail); destination(syslog); }; log { source(localmsg); filter(f_not_mail); destination(messages); }; log { source(localmsg); destination(hostfiles); flags(final); }; # Also save logs from local host # ===== This generates between 3 to 8 GiB logs per day on the logserver. Everything went fine until we wanted to check a specific event on a specific host which generates something like 10 lines of auth.info logs for which only 5 lines were saved on the log server. It appeared that dome packets were randomly ignored. But not sure if it was a network issue (udp) or syslog-ng we produced the following tool to dump what arrives on the udp socket: ==== syslog-dump.c #include <sys/types.h> #include <sys/socket.h> #include <netinet/in.h> #include <sys/errno.h> #define BUFFLEN 2047 int main (int argc, char **argv) { int syslog_socket; int syslog_port=514; struct sockaddr_in syslog_addr; unsigned char syslog_buffer[BUFFLEN+1]; int len,ret; /* create socket */ syslog_socket=socket(PF_INET, SOCK_DGRAM, 0); if (syslog_socket==-1) { printf("Error creating socket"); exit(1); } bzero(&syslog_addr,sizeof(syslog_addr)); syslog_addr.sin_family = AF_INET; syslog_addr.sin_addr.s_addr = htonl(INADDR_ANY); syslog_addr.sin_port = htons(syslog_port); ret=bind(syslog_socket, (struct sockaddr *) &syslog_addr, sizeof(syslog_addr)); if (ret==-1) { perror("Error binding to socket"); exit(1); } while (len=read(syslog_socket, syslog_buffer, BUFFLEN)){ printf("%s\n",syslog_buffer); bzero(syslog_buffer, BUFFLEN+1); } } ==== We run this instead of syslog-ng for a few minutes and produces som 20000 lines of log on 2 hosts in 20 seconds using logger. During the tests I also kept receiving the normal messages from my 2500 hosts in addition to my test messages. I tested it for more than half an hour and never lost any message using my syslog-dump program (something like 20 consecutive tests). The next half hour I repeated the test with syslog-ng. Unfortunately I lost around 10% to 20% of my test messages every time. I started with syslog-ng 1.6.9 but upgraded to 1.9.9 to be sure I had all the improvements. Has anybody an idea of where my packets get lost, is there a tunning only solution, should I start looking at the code to find my lost packets ? I'd appreciate a small explanation of syslog-ng internal if that is the case :) Aside from that I would like to say that I'm quite happy with syslog-ng features. Regards, Vincent.

4 10

Syslog-ng & RedHat
by syslogyk 14 Mar '06

14 Mar '06

Hi! I tried to work with syslog-ng-1.9.9 on Redhat Linux AS 4. Sources (eventlog and syslog-ng) were taken here: http://www.balabit.com/downloads/syslog-ng/1.9/src/ I walked through configure, make and make install successfully. Simple syslog-ng.conf was placed to /usr/local/etc When I run /usr/local/sbin/syslog-ng, I get the following error: fsync() does not work yet; fsync() does not work yet; Error binding socket; error='No such file or directory (2)', addr='AF_UNIX(/var/lib/named/dev/log)' Error initializing source driver; source='src' Could you please give me a hint about what was a reason of such error? Anyway, I can't tell I'm interested in last version only. So, could you please advice me syslog-ng version that worked successfully on RedHat Linux AS 4? Thank you in advance.

3 5

Remote hosts logging as loopback, not hostname
by Rob Munsch 14 Mar '06

14 Mar '06

Hello, Knowing next t'nothing about openssl, stunnel, and syslog-ng, i've gotten a central loghost going with one remote client. Poking into MySQL is via: INSERT INTO logs (host, facility, priority ... VALUES ( '$HOST','$FACILITY','$PRIORITY'... )etc. with all the $MACROS you'd expect. However. The messages from the remote host always seem to appear as 127.0.0.1! My question is: is it because of something i am or am not doing in syslog? I am not logging remote messages to any local logs, something that only dawned on me after i had it all running. Instead, I am doing this: # Sending to the MySQL table log { source(s_all); destination(d_mysql); }; # ... as well as remote client messages log { source(stunnel); destination(d_mysql); }; which, i realize, may inspire laughing. or not. It seemed like a good idea at the time, as the idea was to have every individual hosts' logs be 'personal,' and the database be the amalgam. I am starting to suspect stunnel, however, as well: source stunnel { tcp(ip(127.0.0.1) <<=== ! port(514) max-connections(10) ); }; is THAT why remote messages are being stamped with a $HOST of the loopback address? Is it a combination of both factors, neither, something else entirely...? I'm prepared to go bug the stunnel list too, if necessary :D. Thanks! Robert -- Rob Munsch Solutions For Progress IT

3 5

Why does this not work?
by David Anderson 14 Mar '06

14 Mar '06

Hi, I am trying to lessen the amount in my messages log. I have logs coming in from my router which I can successfully put into their own log using the following filters. Unfortunately this also drops the info into messages as well: filter f_messages { not facility(news, mail) and not filter(f_iptables); }; filter f_router { host(vigor); }; If I try to prevent the additions to the message log using the following filters I get the error shown below. filter f_messages { not facility(news, mail) and not filter(f_iptables, f_router); }; filter f_router { host(vigor); }; benden:~ # /etc/init.d/syslog restart Shutting down syslog services done Starting syslog servicessyntax error at 76 Parse error reading configuration file, exiting. (line 76) startproc: exit status of parent of /sbin/syslog-ng: 1 failed Can anyone explain what I am doing wrong please? -- Best regards, David mailto:dma@pern.co.uk

3 3