January 2007 - syslog-ng - lists.balabit.hu

RE: [syslog-ng] syslog configuration
by Padmanabhan, Rajeesh (GE Healthcare) 16 Jan '07

16 Jan '07

I am talking about the evenlog libraries, which is the support libraries for Syslog-ng. Thanks for responding. -Rajeesh _____ From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Hari Sekhon Sent: Tuesday, January 16, 2007 5:16 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog configuration please specify, are you talking about manually importing event logs from windows to syslog-ng or making windows log to your central logserver or something else, programmatic or what? -h Hari Sekhon Padmanabhan, Rajeesh (GE Healthcare) wrote: Hi All, How do I link eventlog libraries to Syslog-ng manually? Thanks, Rajeesh _____ _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

1 0

syslog-ng looping problem
by Jeff Gates 15 Jan '07

15 Jan '07

Hello, I'm currently running Fedora 6. I installed eventlog-0.2.6+20070115 and syslog-ng-2.0.1+20070115 and started syslog-ng and all goes well. Then if I do a logger command: logger [test:test123] this works but if I run logger [status:test123] syslog-ng goes onto a loop and fills up my disk. This only happens when resolv.conf is configured but the DNS server not up. I can reproduce it by setting the resolv.conf to: domain maxsp.com #nameserver 192.168.1.53 #nameserver 63.198.110.86 And you can see in the syslog-ng.conf where I'm using the Status in the filter. ]# cat syslog-ng.conf options { sync (0); time_reopen (10); log_fifo_size (10000); long_hostnames (on); use_dns (no); use_fqdn (yes); create_dirs (no); keep_hostname (yes); }; # Input source for syslog source src { internal(); udp(); tcp(); unix-stream("/dev/log"); # file("/proc/kmsg" log_prefix("kernel: ")); }; # Standard Unix syslog bucketes destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog"); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_mlal { usertty("*"); }; filter f_filter1 { facility(kern); }; filter f_filter2 { level(info) and not (facility(mail) or facility(authpriv) or facility(cron)); }; filter f_filter3 { facility(authpriv); }; filter f_filter4 { facility(mail); }; filter f_filter5 { level(emerg); }; filter f_filter6 { facility(uucp) or (facility(news) and level(crit)); }; filter f_filter7 { facility(local7); }; filter f_filter8 { facility(cron); }; #log { source(src); filter(f_filter1); destination(d_cons); }; #log { source(src); filter(f_filter2); destination(d_mesg); }; log { source(src); filter(f_filter3); destination(d_auth); }; log { source(src); filter(f_filter4); destination(d_mail); }; log { source(src); filter(f_filter5); destination(d_mlal); }; log { source(src); filter(f_filter6); destination(d_spol); }; log { source(src); filter(f_filter7); destination(d_boot); }; log { source(src); filter(f_filter8); destination(d_cron); }; # Filter for useful log data before sending upstream filter f_apps { match("[Application:Activescan]") or match("[Application:Expert]") or match("[Application:BCAmonitor]") ; }; filter f_alert_message { match("Type:High") or match("Type:Medium") or match("\\[Status:") or match("\\[status:") or match("\\[zippy:") ; }; # Full log messages file destination std { file("/var/log/messages" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes) ); }; log { source(src); destination(std); }; ## set up logging to loghost destination loghost { tcp("logsink.maxsp.com" port(514)); }; # send to loghost log { source(src); filter(f_apps); filter(f_alert_message); destination(loghost); }; Thanks, Jeff

1 0

RE: [syslog-ng] Forwarding + Spoofing = Errors & Dropped Packets?
by Ivey, Chris 15 Jan '07

15 Jan '07

Thanks Bazsi. I finally got to the bottom of this, quite on accident. On a whim (since nothing else was working), I changed the destination for forwarding from hostnames to IP addresses. After I stopped and restarted the syslog-ng service, all worked well. I noticed in all my ltrace outputs that syslog-ng was performing a LOT of DNS queries when spoofing was on and we were using hostnames as the targets. There is not a local DNS server with this syslog-ng server, so the queries were taking quite a bit of time to come back (50-60 ms). Once I made the change to IP addresses, everything worked much better. I have now set my other syslog-ng server to use IP addresses instead of hostnames for forwarding as well. Whoever runs the syslog-ng FAQ on campin.net may need to know that if you have performance issues, switching your targets to IP addresses instead of hostnames may clear up some issues. Can you explain to me, though, why the spoofing needs to do so many nslookups? Why can it not cache the results of the first query? Does the application block waiting for DNS queries to come back? That may need to be addressed.... Thanks! Chris Ivey Affiliated Computer Services Enterprise Management Integration Services Infrastructure Management Senior Analyst 1120 Celebration Blvd. Celebration, FL 34747 (321) 939-6540 Office 8296-6540 Disney Tie-Line (407) 557-1072 Cell 162*326905*19 Nextel Direct Connect chris.ivey(a)acs-inc.com "When you find yourself in a hole, the best thing to do is stop digging!" -- Nick Stokes -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Sunday, January 14, 2007 9:47 AM To: Syslog-ng users' and developers' mailing list Subject: RE: [syslog-ng] Forwarding + Spoofing = Errors & Dropped Packets? On Thu, 2007-01-11 at 10:24 -0600, Ivey, Chris wrote: > Bazsi, > Thanks for your help thus far. After MUCH trial and error > with enabling and disabling things in syslog-ng.conf, and watching > "netstat -an" output after each attempt, I have narrowed this issue > down to something to do with spoofing. From my starting > syslog-ng.conf file, if I do NOTHING but turn off spoofing, the Recv-Q > buffer stops filling and all is happy. If I turn on spoofing, those > buffers start filling again. I used tkdiff to compare my 2 > syslog-ng.conf files side-by-side, and the only differences between > the 2 are the lines where the forward-to hosts are defined and where > the destinations are defined (the bad server has lines commented, the > good one does not). Lsof on both machines for syslog-ng shows the > same libraries opened and such. So, the question now is what happens > with spoofing that would cause all these issues? What does spoofing > do in the background that my server is unhappy about? Any insight > would be greatly appreciated! As the server is "unhappy", it is probably some kind of spoof filtering, e.g. the default route of the server points to a different interface, and the server drops the packets because of this. -- Bazsi _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

2 1

Migration
by max makli 15 Jan '07

15 Jan '07

Hi My php-syslog-ng 2.8 is stable, it's possible to migrate to the new version 2.9?? THNAKS _________________________________________________________________ Windows Live Spaces : créez votre blog à votre image ! http://www.windowslive.fr/spaces

1 0

RE: [syslog-ng] Forwarding + Spoofing = Errors & Dropped Packets?
by Ivey, Chris 14 Jan '07

14 Jan '07

Bazsi, Thanks for your help thus far. After MUCH trial and error with enabling and disabling things in syslog-ng.conf, and watching "netstat -an" output after each attempt, I have narrowed this issue down to something to do with spoofing. From my starting syslog-ng.conf file, if I do NOTHING but turn off spoofing, the Recv-Q buffer stops filling and all is happy. If I turn on spoofing, those buffers start filling again. I used tkdiff to compare my 2 syslog-ng.conf files side-by-side, and the only differences between the 2 are the lines where the forward-to hosts are defined and where the destinations are defined (the bad server has lines commented, the good one does not). Lsof on both machines for syslog-ng shows the same libraries opened and such. So, the question now is what happens with spoofing that would cause all these issues? What does spoofing do in the background that my server is unhappy about? Any insight would be greatly appreciated! Chris Ivey Affiliated Computer Services Enterprise Management Integration Services Infrastructure Management Senior Analyst 1120 Celebration Blvd. Celebration, FL 34747 chris.ivey(a)acs-inc.com "When you find yourself in a hole, the best thing to do is stop digging!" -- Nick Stokes -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Thursday, January 11, 2007 8:06 AM To: Syslog-ng users' and developers' mailing list Subject: RE: [syslog-ng] Forwarding + Spoofing = Errors & Dropped Packets? On Wed, 2007-01-10 at 13:31 -0600, Ivey, Chris wrote: > We took DNS out of the config, and had no change. How do we go about > seeing if we are blocking on /proc/kmsg? You need to identify the bottleneck, there are various tools for this. Check CPU/disk usage, vmstat, maybe even strace syslog-ng and check where it is spending its time. If everything looks normal, you might need to increase the UDP receive buffer size. What is your message rate? -- Bazsi _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

2 1

Logging HTTP and HTTPS requests in one file
by Phusion 12 Jan '07

12 Jan '07

I am trying to log HTTP and HTTPS requests in one file per virtual host on a server and need some help. Currently, it only logs HTTP access requests and HTTP error requests. It doesn't pick up the SSL access requests or the SSL error requests. I will include some parts of the config files. syslog-ng.conf --------------------- filter fd_httpd_test.abc.com { (match("logger:") and match("test.abc.com")) or (match("httpd[\[0-9]+\]") and match("/websites/abc/test")); }; filter f_terms { not match("httpd[\[0-9]+\]") and not match("logger:"); }; The f_terms filter is for a catchall filter that catches things not specified to go to a certain file. httpd.conf -------------- ErrorLog syslog LogFormat "%v:%p %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" commonvhost CustomLog "|/usr/bin/logger" commonvhost <VirtualHost *:80> ServerName test.abc.com ServerAdmin root@localhost DocumentRoot /websites/abc/test Redirect / https://test.abc.com/ </VirtualHost> <VirtualHost *:443> ServerName test.abc.com ServerAdmin root@localhost DocumentRoot /websites/abc/test </VirtualHost> ssl.conf ------------ ErrorLog syslog LogFormat "%v:%p %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" commonvhost CustomLog "|/usr/bin/logger" commonvhost Let me know when the problem lies. Phusion

1 0

Re: Question about regular expressions
by G.W. Haywood 12 Jan '07

12 Jan '07

Hi there, On Fri, 12 Jan 2007 Phusion wrote: > httpd[....] and /websites/abc/test > ... > > I have tried the following. > > filter fd_httpd_test.abc.com { match("httpd[\[0-9]+\]") and ... Isn't filter fd_httpd_test.abc.com { program("httpd") and ... }; what you're looking for there? -- 73, Ged.

2 1

Help with startup of syslog-ng
by PATEL, Leena, GBM 12 Jan '07

12 Jan '07

Hi I'm trying to start syslog-ng for use with splunk but get the following error and am not sure how the address is already in use. I want to configure syslog-ng so that it receives files remotely via udp. [root@lonrs00253 init.d]# service syslog-ng stop Stopping Kernel Logger: [ OK ] [root@lonrs00253 init.d]# service syslog-ng start Starting syslog-ng: Error binding socket; addr='AF_INET(0.0.0.0:514)', error='Address already in use (98)' Error initializing source driver; source='remote' Starting Kernel Logger: [ OK ] My syslog-ng config file is below which i've configured for splunk using the splunk forums. Cheers leena ---------------------------------------------------------------------------- ---------------------------------------------------- options { sync (0); time_reopen (10); log_fifo_size (1000); long_hostnames (on); use_dns (yes); use_fqdn (yes); create_dirs (yes); group (logs); dir_group (logs); perm (0640); dir_perm (0750); keep_hostname (yes); }; source s_sys { unix-stream("/dev/log"); internal(); pipe("/proc/kmsg"); }; source remote { udp(); }; destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog"); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_mlal { usertty("*"); }; destination splunk { pipe("/var/log/syslog-ng/syslog_fifo"); }; destination hosts { file("/var/log/syslog-ng/hosts/$HOST/messages"); }; filter f_filter1 { facility(kern); }; filter f_filter2 { level(info) and not (facility(mail) or facility(authpriv) or facility(cron)); }; filter f_filter3 { facility(authpriv); }; filter f_filter4 { facility(mail); }; filter f_filter5 { level(emerg); }; filter f_filter6 { facility(uucp) or (facility(news) and level(crit)); }; filter f_filter7 { facility(local7); }; filter f_filter8 { facility(cron); }; log { source(s_sys); filter(f_filter1); destination(d_cons); }; log { source(s_sys); filter(f_filter2); destination(d_mesg); }; log { source(s_sys); filter(f_filter3); destination(d_auth); }; log { source(s_sys); filter(f_filter4); destination(d_mail); }; log { source(s_sys); filter(f_filter5); destination(d_mlal); }; log { source(s_sys); filter(f_filter6); destination(d_spol); }; log { source(s_sys); filter(f_filter7); destination(d_boot); }; log { source(s_sys); filter(f_filter8); destination(d_cron); }; log { source(remote); destination(splunk); }; log { source(remote); destination(hosts); }; Regards Leena Patel | Royal Bank of Scotland Global Banking & Markets | E-Commerce | Desk: +44 (0) 207 085 0692 *********************************************************************************** The Royal Bank of Scotland plc. Registered in Scotland No 90312. Registered Office: 36 St Andrew Square, Edinburgh EH2 2YB. Authorised and regulated by the Financial Services Authority This e-mail message is confidential and for use by the addressee only. If the message is received by anyone other than the addressee, please return the message to the sender by replying to it and then delete the message from your computer. Internet e-mails are not necessarily secure. The Royal Bank of Scotland plc does not accept responsibility for changes made to this message after it was sent. Whilst all reasonable care has been taken to avoid the transmission of viruses, it is the responsibility of the recipient to ensure that the onward transmission, opening or use of this message and any attachments will not adversely affect its systems or data. No responsibility is accepted by The Royal Bank of Scotland plc in this regard and the recipient should carry out such virus and other checks as it considers appropriate. Visit our websites at: www.rbs.com www.rbsgc.com www.rbsmarkets.com ***********************************************************************************

1 0

Question about regular expressions
by Phusion 12 Jan '07

12 Jan '07

I need some help in my config file using regular expressions. This filter is for http requests from Apache. I am trying to get this filter to catch the following two types of messages. httpd[....] and /websites/abc/test logger: and test.abc.com I have tried the following. filter fd_httpd_test.abc.com { match("httpd[\[0-9]+\]") and match("/websites/abc/test") or match("logger:") and match("test.abc.com"); }; I also tried this. filter fd_httpd_test.abc.com { match("httpd[\[0-9]+\]*/websites/abc/test") or match("logger:*test.abc.com"); }; Let me know. Thanks. Phusion

2 1

RE: [syslog-ng] Forwarding + Spoofing = Errors & Dropped Packets?
by Ivey, Chris 11 Jan '07

11 Jan '07

We took DNS out of the config, and had no change. How do we go about seeing if we are blocking on /proc/kmsg? Thanks! Chris Ivey Affiliated Computer Services Enterprise Management Integration Services Infrastructure Management Senior Analyst 1120 Celebration Blvd. Celebration, FL 34747 chris.ivey(a)acs-inc.com "When you find yourself in a hole, the best thing to do is stop digging!" -- Nick Stokes -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Wednesday, January 10, 2007 1:10 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Forwarding + Spoofing = Errors & Dropped Packets? On Wed, 2007-01-10 at 07:30 -0600, Ivey, Chris wrote: > We are having a REALLY weird issue with syslog-ng that I need to > request some assistance with resolving. It has to do with forwarding > and spoofing. If I go into syslog-ng.conf and enable forwarding to my > 3 remote servers along with spoofing, it causes issues on the server. > First, the Recv-Q fills to capacity (as seen in "netstat -a | grep > syslog"). Once that buffer fills, we start seeing "packet receive > errors" (as seen in "netstat -su"). We have an INORDINATE amount of > these errors (about 45%). Observe: syslog-ng is busy doing something and it causes not to read the UDP receive buffers in a timely manner. Can you check: * syslog-ng is not blocking on DNS * syslog-ng is not blocking on /proc/kmsg or something else. -- Bazsi _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

2 1