June 2005 - syslog-ng - lists.balabit.hu

RE: [syslog-ng] Resolving Hostnames for Syslog Source IPs
by Sawall, Christopher L 06 Jun '05

06 Jun '05

Thanks for the suggestions. It seems that about 10 out of 50 of the devices did not have the proper PTR records. Adding this has resolved my problem. Thanks, chris -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Valdis.Kletnieks(a)vt.edu Sent: Wednesday, June 01, 2005 3:53 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Resolving Hostnames for Syslog Source IPs On Wed, 01 Jun 2005 09:33:58 PDT, Jarrod Manzer said: > I had this same problem. I resolved it by logging by IP and then doing > reverse DNS lookups with a script and creating symbolic links to those > IP based directories. The end result was people who like to use IP or > DNS were happy. Gotta make sure your reverse is set up properly though. > > But I never did find out why syslog-ng couldn't resolve the same names > that the host command on the same box could. The most common cause for things like this is semi-borked DNS that *appears* to work, but in fact is subtly misconfigured. A few things to check: 1) Take the IP address, and look up the PTR, which should give you a hostname (this is where most 'host' commands stop). Then actually check that hostname in the DNS, and make sure the IP is listed in an A record (some resolvers do this additional sanity checking). 2) You may have a "lame delegation". Look at the SOA and NS entries for your DNS zones, both PTR and A, and double-check that all machines listed in NS are in fact serving up correct data for the zones (a quick double-check is if all the DNS servers show the correct zone serial number in the SOA). It often happens that if there are multiple NS records, a daemon will "lock in" on asking one NS first, and if it returns an authoritative NXDOMAIN because it's a lame delegation, the daemon won't ask other NS. However, when you use the 'host' command, it may check some *other* NS entry first and magically appear to work. 3) Double-check /etc/resolv.conf to make sure it points 'nameserver' entries at DNS servers that pass the sanity checks in (1) and (2).... ******************************* The information contained in this message may be privileged and/or confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. Note that any views or opinions presented in this message are solely those of the author and do not necessarily represent those of Ameren. All emails are subject to monitoring and archival. Finally, the recipient should check this message and any attachments for the presence of viruses. Ameren accepts no liability for any damage caused by any virus transmitted by this email. If you have received this in error, please notify the sender immediately by replying to the message and deleting the material from any computer. Ameren Corporation *******************************

1 0

Syntax error in default config?
by J. Meub 04 Jun '05

04 Jun '05

Hello, i'm new to syslog-ng and recently compiled and installed it (version 1.9.4) on my system (LFS, Kernel 2.6.11.7). The problem i have is that even with the default configuiration file it claims to have syntax errors. The lines causing the errors are: destination user { file("/var/log/user.log"); }; and filter f_auth { facility(auth); }; Whereas the first problem is generated due to the fact the the identifier is called "user". If i call it "user_" it works fine. With the second line the problem is "auth" because facility(authpriv) is accepted. But why not auth?! I'm really at a loss with that. I hope someone here has an idea what might be the problem. Thanks in advance. Regards, J. Meub

3 10

syslog-ng anon patch
by micah milano 03 Jun '05

03 Jun '05

Hello, The attached patch comes from http://dev.riseup.net/patches/syslog-ng what it does is provide a simple filter to strip out unwanted regular expressions from logs, as well as an IP alias that enables you to strip out IP addresses from your logs. This patch has been applied to the Debian package of syslog-ng, I am writing here to let people know about it and submit it for consideration into syslog-ng. >From the README: This patch adds the capability to syslog-ng that allows you to strip out any given regexp or all IP addresses from log messages before they are written to disk. The goal is to give the system administrator the means to implement site logging policies, by allowing them easy control over exactly what data they retain in their logfiles, regardless of what a particular daemon might think is best. Background: Data retention has become a hot legal topic for ISPs and other Online Service Providers (OSPs). There are many instances where it is preferable to keep less information on users than is collected by default on many systems. In the United States it is not currently required to retain data on users of a server, but you may be required to provide all data on a user which you have retained. OSPs can protect themselves from legal hassles and added work by choosing what data they wish to retain. >From "Best Practices for Online Service Providers" (http://www.eff.org/osp) As an intermediary, the OSP finds itself in a position to collect and store detailed information about its users and their online activities that may be of great interest to third parties. The USA PATRIOT Act also provides the government with expanded powers to request this information. As a result, OSP owners must deal with requests from law enforcement and lawyers to hand over private user information and logs. Yet, compliance with these demands takes away from an OSP's goal of providing users with reliable, secure network services. In this paper, EFF offers some suggestions, both legal and technical, for best practices that balance the needs of OSPs and their users' privacy and civil liberties. Rather than scrubbing the information you don't want in logs, this patch ensures that the information is never written to disk. Also, for those daemons which log through syslog facilities, this patch provides a convenient single configuration to limit what you wish to log. Here are some related links: Best Practices for Online Service Providers http://www.eff.org/osp http://www.eff.org/osp/20040819_OSPBestPractices.pdf EPIC International Data Retention Page http://www.epic.org/privacy/intl/data_retention.html Working Paper on Usage Log Data Management (from Computer, Freedom, and Privacy conference) http://cryptome.org/usage-logs.htm How to use it This patch adds the filter "strip". For example: filter f_strip {strip(<regexp>);}; This will strip out all matches of the regular expression on logs to which the filter is applied and replaces all matches with the fixed length four dashes ("----"). In place of a regular expression, you can put "ips", which will replace all internet addresses with 0.0.0.0 <http://0.0.0.0>. For example: filter f_strip {strip(ips);}; You can alter what the replacement strings are by using replace: replace(ips,"0.0.0.0 <http://0.0.0.0>") <--- this is the same as strip(ips) replace(<regex>,"----") <--- this is the same as strip(regex) We provide a debian package of 1.6.7 with this patch added (the repository is http://deb.riseup.net/debian unstable main), or you can retrieve the patch yourself from http://dev.riseup.net/websvn/listing.php?repname=syslog-ng-anon&path=%2F&sc… apply it with: # patch -p1 < syslog-ng-anon.diff

6 7

Re: Assessing reliability - did we get all messages?
by Cary, Kim 02 Jun '05

02 Jun '05

Thanks to Mike for the tip on checking whether we're having problems taking on our UDP data from our solaris 9 syslog server. On Jun 1, 2005, at 1:52 PM, syslog-ng-request(a)lists.balabit.hu wrote: > For Solaris: > netstat -s -P udp > I am pretty sure that "udpInErrors" will represent packets dropped from > teh UDP buffer, but am unsure.. > > I don't think I would use syslog over TCP from a PIX for the exact > reason > you said..the PIX will stop until the syslog server is ready..instead, > maybe just increase your UDP buffer size. Depending on what else your > syslog server is doing..crank it up to 64MBytes or more..RAM is pretty > cheap. Here's what it looks like (uptime like 2 days): netstat -s -P udp UDP udpInDatagrams =101638801 udpInErrors = 0 udpOutDatagrams = 8151 udpOutErrors = 0 ndd -get /dev/udp udp_max_buf 262144 ndd -get /dev/udp udp_recv_hiwat 8192 Since this system will only get more logs sent to it, I think increasing these parameters might be good. On consultation with: http://docs.sun.com/app/docs/doc/806-7009/6jftnqskc?a=view ndd -set /dev/udp udp_recv_hiwat 65536 ndd -get /dev/udp udp_max_buf 524288 And I'm considering adding these to startup. However, I'm wondering, since apps can set their own buffers, should I even worry about this? I'm not enough of a C reader to find this out from the source code. Does syslog-ng accept the defaults or set its own buffers? Am I on the right track here? The implementation seems very solid so far, just planning for the future. Kim (thanks to Bill for his test idea as well)

1 0

Shiming Yang/OCIO_CorpDataSecurity/Planning_Office/HQ_for_the_Americas/BTMNA is out of the office.
by Shiming Yang 02 Jun '05

02 Jun '05

I will be out of the office starting 06/02/2005 and will not return until 06/07/2005. I will respond to your message when I return. If there is problem about Proxy or IBM web hosting facility, please call my backup Will at 8380. ----------------------------------------- The information contained in this electronic mail message, and any and all accompanying documents, constitutes confidential information. If you are not the intended recipient of this information, any disclosure, copying, distribution, or the taking of any action in reliance on it is strictly prohibited. If you received this information in error, please notify the sender immediately and destroy this communication. Messages sent via this medium may be subject to delays and/or unauthorized alteration. Neither The Bank of Tokyo-Mitsubishi, Ltd. nor any of its affiliates shall be held liable for the contents of this message.

1 0

Resolving Hostnames for Syslog Source IPs
by Sawall, Christopher L 01 Jun '05

01 Jun '05

I used to only log by IP and was fine with it. But with a recent acquistion and to make things easier, I turned on the use of DNS by Syslog-NG. However, it doesn't appear to resolve every source host. For example, I am sending syslogs from about 50 sets of Cisco PIX firewalls, 5 VPN Concentrators and 2 Cisco Cache Engines. Only about 80% of the PIX firewalls are being resolved when they show up in the syslog file, 2 of the concentrators and none of the cache engines. I have verified that all of the firewalls are in DNS and PINGable from the syslog server itself. I'm not sure what else to check. I saw an old post from September 2003, but I don't think it's the same issue. This appears to be a consistent problem, some hosts are always resolved and some are not. Below are the options that are set in my syslog-ng.conf file. options { sync (1); time_reopen (10); log_fifo_size (2048); long_hostnames (off); dns_cache(yes); use_dns (yes); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source src { unix-stream("/dev/log"); internal(); udp(ip(0.0.0.0) port (514)); }; Let me know if any other info would be benenficial. Thanks, Chris Sawall, GSEC, GSNA Ameren Information Security ******************************* The information contained in this message may be privileged and/or confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. Note that any views or opinions presented in this message are solely those of the author and do not necessarily represent those of Ameren. All emails are subject to monitoring and archival. Finally, the recipient should check this message and any attachments for the presence of viruses. Ameren accepts no liability for any damage caused by any virus transmitted by this email. If you have received this in error, please notify the sender immediately by replying to the message and deleting the material from any computer. Ameren Corporation *******************************

3 2

RE: [syslog-ng] Resolving Hostnames for Syslog Source IPs
by Sawall, Christopher L 01 Jun '05

01 Jun '05

Only problem is I'm letting Syslog-NG create a new file for each host, so Syslog-NG needs to be able to resolve the host before creating the file. This is a very odd problem. Chris -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Jarrod Manzer Sent: Wednesday, June 01, 2005 11:34 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Resolving Hostnames for Syslog Source IPs I had this same problem. I resolved it by logging by IP and then doing reverse DNS lookups with a script and creating symbolic links to those IP based directories. The end result was people who like to use IP or DNS were happy. Gotta make sure your reverse is set up properly though. But I never did find out why syslog-ng couldn't resolve the same names that the host command on the same box could. Sawall, Christopher L wrote: >I used to only log by IP and was fine with it. But with a recent >acquistion and to make things easier, I turned on the use of DNS by >Syslog-NG. However, it doesn't appear to resolve every source host. > >For example, I am sending syslogs from about 50 sets of Cisco PIX >firewalls, 5 VPN Concentrators and 2 Cisco Cache Engines. Only about >80% of the PIX firewalls are being resolved when they show up in the >syslog file, 2 of the concentrators and none of the cache engines. I >have verified that all of the firewalls are in DNS and PINGable from >the syslog server itself. > >I'm not sure what else to check. I saw an old post from September >2003, but I don't think it's the same issue. This appears to be a >consistent problem, some hosts are always resolved and some are not. > >Below are the options that are set in my syslog-ng.conf file. > >options { sync (1); > time_reopen (10); > log_fifo_size (2048); > long_hostnames (off); > dns_cache(yes); > use_dns (yes); > use_fqdn (no); > create_dirs (no); > keep_hostname (yes); > }; > >source src { unix-stream("/dev/log"); internal(); udp(ip(0.0.0.0) port >(514)); }; > >Let me know if any other info would be benenficial. > >Thanks, >Chris Sawall, GSEC, GSNA >Ameren >Information Security > > > > >******************************* >The information contained in this message may be privileged and/or >confidential and >protected from disclosure. If the reader of this message is not the intended recipient, >or an employee or agent responsible for delivering this message to the intended recipient, >you are hereby notified that any dissemination, distribution or copying of this >communication is strictly prohibited. Note that any views or opinions presented in this >message are solely those of the author and do not necessarily represent those of Ameren. >All emails are subject to monitoring and archival. Finally, the recipient should check >this message and any attachments for the presence of viruses. Ameren accepts no liability >for any damage caused by any virus transmitted by this email. If you have received this in >error, please notify the sender immediately by replying to the message and deleting the >material from any computer. Ameren Corporation >******************************* > > > >_______________________________________________ >syslog-ng maillist - syslog-ng(a)lists.balabit.hu >https://lists.balabit.hu/mailman/listinfo/syslog-ng >Frequently asked questions at http://www.campin.net/syslog-ng/faq.html > > > -- .Jarrod Manzer. .Network Team. .Go Daddy Group. .(480) 366 3631. .AIM: nejarrod. _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html ******************************* The information contained in this message may be privileged and/or confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. Note that any views or opinions presented in this message are solely those of the author and do not necessarily represent those of Ameren. All emails are subject to monitoring and archival. Finally, the recipient should check this message and any attachments for the presence of viruses. Ameren accepts no liability for any damage caused by any virus transmitted by this email. If you have received this in error, please notify the sender immediately by replying to the message and deleting the material from any computer. Ameren Corporation *******************************

1 0

Assessing reliability - did we get all messages?
by Cary, Kim 01 Jun '05

01 Jun '05

Hi folks, I'm trying to figure out whether we're getting all messages from a PIX into the mysql database. I've done some tests and everything syslog-ng is writing to disk is going to our db on the application server. Now I need to figure out if syslog-ng is getting everything (or nearly) from the PIX. Specifically my questions are: - Does 'garbage collecting while busy' imply I'm dropping packets? - Any suggestions for checking my UDP buffers on the syslog server? - I'd like to use TCP from the PIX > syslog-ng. Does that work? Its a little hard to test as this is a production system on our internet gateway. My understanding is that Pix logging via TCP will STOP the transmission of packets if the syslog server goes away, so, we may not want to do that with a single syslog server. Our setup is like this: Application server (mysql log database & web-based search interface - MacOS X Panther) ^ TCP ^ Syslog server (perl script (fisq.pl) parses input and sends via perl dbi) ^fifo pipe^ Syslog server (syslog-ng 1.6.4 from sunfreeware, Solaris 9 1.5G ) > file on local disk ^UDP^ Pix 520 The network run for the PIX > Syslog server link is dedicated 100Mbps full duplex. We get about 20Gb of syslog messages a day from that source. Here's what syslog-ng says about itself: May 31 22:27:11 logsrvr syslog-ng[347]: STATS: dropped 0 May 31 22:37:11 logsrvr syslog-ng[347]: STATS: dropped 0 May 31 22:45:14 logsrvr syslog-ng[347]: Garbage collecting while busy... May 31 22:45:17 logsrvr syslog-ng[347]: Objects alive: 240, garbage collected: 2999765 May 31 22:47:11 logsrvr syslog-ng[347]: STATS: dropped 0 May 31 22:57:11 logsrvr syslog-ng[347]: STATS: dropped 0 May 31 23:04:21 logsrvr syslog-ng[347]: Garbage collecting while idle... May 31 23:04:22 logsrvr syslog-ng[347]: Objects alive: 240, garbage collected: 665611 May 31 23:07:11 logsrvr syslog-ng[347]: STATS: dropped 0 May 31 23:17:11 logsrvr syslog-ng[347]: STATS: dropped 0 Here are our tuning options from syslog-ng.conf: options { use_fqdn(yes); sync(0); keep_hostname(yes); chain_hostnames(no); create_dirs(yes); log_fifo_size(100000); gc_busy_threshold(3000000); gc_idle_threshold(300); }; BTW, syslog-ng is 'the bomb'. I love what we can do with it more easily than stock syslog. Kim

3 2