- syslog-ng - lists.balabit.hu

Character Conversion Question
by James B Horwath 06 Feb '10

06 Feb '10

I am using SNARE to send messages to syslog-ng (2.0.9). There are fields in the messages separated by a TAB (0x9). When syslog-ng writes this out to a file it convert the hexadecimal (0x9) to \011 (literally slash, zero, one, one). This makes the lines hard to read. How can I either convert the TAB to a space or preserve the TAB in the output. Preserving the TAB would be my preference. I did not see anything in the documentation except writing a regex to convert the character. Thanks in advance, Jim ----------------------------------------- This message, and any attachments to it, may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are notified that any use, dissemination, distribution, copying, or communication of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately by return e-mail and delete the message and any attachments. Thank you.

2 1

Help with spoofing hostname
by PAUL WILLIAMSON 06 Feb '10

06 Feb '10

Thanks for the suggestions yesterday. As usual, there are multiple ways to solve the problem, each of them equally easy! Now on to my next issue... We have a product called Symantic SIM (Security Information Manager) that is on the receiving end of some forwarded messages. I have the keep_hostname(yes) option enabled, and when our SIM gets the message, the originating hostname is in the message. The problem is that is seems like the SIM is detecting that the message is coming from my loghost where Syslog-ng is installed, and tagging every message like it's from that instead of the actual host. We've been over the config with their engineers and our security department, and this is what we got back from Symantec today. "The hostname is available in the message, so it looks like that part is working. The problem is this, the message the SSIM sees is: <TIME> <Syslog Server IP> <Message> The SSIM then puts the syslog server IP as the source and destination of the host. Essentially the message needs to be sent to the SSIM as: <TIME> <Originating device Source IP> <Message> To do this, the message will need to be spoofed." So, I have two questions: 1. Can the messages be spoofed? 2. Does anyone else use this product and would be willing to share configs (of either syslog-ng or SSIM). Thanks, Paul ************************************ This email may contain privileged and/or confidential information that is intended solely for the use of the addressee. If you are not the intended recipient or entity, you are strictly prohibited from disclosing, copying, distributing or using any of the information contained in the transmission. If you received this communication in error, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. This communication may contain nonpublic personal information about consumers subject to the restrictions of the Gramm-Leach-Bliley Act and the Sarbanes-Oxley Act. You may not directly or indirectly reuse or disclose such information for any purpose other than to provide the services for which you are receiving the information. There are risks associated with the use of electronic transmission. The sender of this information does not control the method of transmittal or service providers and assumes no duty or obligation for the security, receipt, or third party interception of this transmission. ************************************

2 1

syslog-ng MSGHDR
by Mij 06 Feb '10

06 Feb '10

Dear syslog-ng folks, I am the maintainer of sshguard, see http://www.sshguard.net . Sshguard can be interfaced with syslog-ng. Multiple users of syslog-ng recently reported that switching to 3.x required a configuration change for preserving the original logging format, see https://sourceforge.net/mailarchive/forum.php?thread_name=EE040D72-0185-41E… https://sourceforge.net/mailarchive/forum.php?thread_name=DA2160C1-09A0-475… We reflected the reports by updating the setup docs to contain a block for the 2.x version and one for 3.x , see http://www.sshguard.net/docs/setup/getlogs/syslog-ng/ However, this change is not apparent in your documentation or changelogs, and other users reported that with even more recent versions, the "old format" is again the correct one. Can you clarify what is the intended template for producing entry tags of the classic format "Jan 21 12:54:09 examplehost proftpd[18965]: applmsg" in the different versions? thanks

2 3

Re: [syslog-ng] Apache + syslog-ng w/o logger?
by Brian Donaldson 06 Feb '10

06 Feb '10

Thanks for the suggestions Jim & Lance. I'm running CentOS 5 using the default logger. I did try what Lance suggested: restarting syslog-ng w/ log_msg_size set to 65K in the options section (both the client and server). However, echo'ing a 8192 character string to logger resulted in several fragments. Guess it's on to Jim's suggestion... On Fri, Feb 5, 2010 at 3:00 AM, <syslog-ng-request(a)lists.balabit.hu> wrote: > Send syslog-ng mailing list submissions to > syslog-ng(a)lists.balabit.hu > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.balabit.hu/mailman/listinfo/syslog-ng > or, via email, send a message with subject or body 'help' to > syslog-ng-request(a)lists.balabit.hu > > You can reach the person managing the list at > syslog-ng-owner(a)lists.balabit.hu > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of syslog-ng digest..." > > > Today's Topics: > > 1. Re: config file question (Rory Toma) > 2. Apache + syslog-ng w/o logger? (Brian Donaldson) > 3. Re: Apache + syslog-ng w/o logger? (Jim Hendrick) > 4. Re: Apache + syslog-ng w/o logger? (Lance Laursen) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 04 Feb 2010 16:52:52 -0800 > From: Rory Toma <rory(a)ooma.com> > Subject: Re: [syslog-ng] config file question > To: Zolt?n Pallagi <pzolee(a)balabit.hu> > Cc: Syslog-ng users' and developers' mailing list > <syslog-ng(a)lists.balabit.hu> > Message-ID: <4B6B6BE4.30109(a)ooma.com> > Content-Type: text/plain; charset="iso-8859-1" > > On 2/2/10 4:49 AM, Zolt?n Pallagi wrote: > > Hi Rory, > > > > Will you tell me the current naming of your log files? I don't know > > the real reason for many log files, but as Robert said, perhaps, there > > are another easier solutions to avoid it. > > > We maintain one log file per client per day, in directory buckets of > 2000 clients. >

2 1

What testing has been done for the new features in open source version of syslog-ng version 3.x
by Janani Janakiraman 06 Feb '10

06 Feb '10

I am trying to judge the level of confidence for the latest stable open-source version of syslog-ng version 3.0.5. Where can I find the test results of the tests that have been run again version 3.0.5 of syslog-ng? I want to find out if tests have been run to test the new features in Version 3.x specifically support for the new IETF-syslog protocol standard ( RFC5424 ). Do any of the major Linux distros support version 3.x of syslog-ng? Thanks, Janani

3 4

Re-post: What testing has been done for the new features in open source version of syslog-ng version 3.x
by Jeremy M. Guthrie 06 Feb '10

06 Feb '10

Sorry, I didn't see an answer. Just wanted to re-post to reconfirm: I have an issue where what is being written to file is different than what I have being written to disk. I don't know if the macro is broken, my understanding is wrong, or I am just using the wrong macro altogether. My understanding is roughly as follows: PROGRAM The name of the program sending the message. PID The PID of the program sending the message. MSG or MESSAGE Message contents including the program name and pid. MSGONLY Message contents without the program name. 1. The first thing I ran into: Example messages on file: Jan 25 08:34:15 XXXXXXXXXX sshd[7209]: service name is :: sshd Same example message in SQL: Jan 25 08:34:15 XXXXXXXXXX service name is :: sshd template("INSERT INTO tsyslog (host, facility, severity, date, time, message) VALUES ( '$HOST','$FACILITY_NUM','$LEVEL_NUM','$YEAR-$MONTH-$DAY','$HOUR:$MIN: $SEC', '$MSG');\n" ) 2. Then what I tried to get it closer.... Example messages on file: Jan 25 08:40:18 XXXXXXXXXX sshd[7209]: service name is :: sshd Same example message in SQL: Jan 25 08:40:18 XXXXXXXXXX sshd: service name is :: sshd template("INSERT INTO tsyslog (host, facility, severity, date, time, message) VALUES ( '$HOST','$FACILITY_NUM','$LEVEL_NUM','$YEAR-$MONTH-$DAY','$HOUR:$MIN: $SEC', '$PROGRAM: $MSG');\n" ) 3. Then what I tried to get even closer: Example messages on file: Jan 25 08:53:15 XXXXXXXXXX sshd[7209]: service name is :: sshd Same example message in SQL: Jan 25 08:53:15 XXXXXXXXXX sshd[7209]: service name is :: sshd template("INSERT INTO tsyslog (host, facility, severity, date, time, message) VALUES ( '$HOST','$FACILITY_NUM','$LEVEL_NUM','$YEAR-$MONTH-$DAY','$HOUR:$MIN: $SEC', '$PROGRAM[$PID]: $MSG');\n" ) However, this broke messages coming from Cisco devices... Jan 25 08:55:10 FIREWALL %FWSM-3-710003[]: tcp access denied by ACL from 188.23.180.240/3826 to outside:172.26.54.65/445 The message above should be: Jan 25 08:55:10 FIREWALL %FWSM-3-710003: tcp access denied by ACL from 188.23.180.240/3826 to outside:172.26.54.65/445 Notice the above '[]' after the FWSM code? I am running 3.0.5 syslog-ng. Ideally I'd get both: Jan 25 08:55:10 FIREWALL %FWSM-3-710003: tcp access denied by ACL from 188.23.180.240/3826 to outside:172.26.54.65/445 AND Jan 25 08:53:15 XXXXXXXXXX sshd[7209]: service name is :: sshd Is the MSG/MESSAGE macro broken? What am I missing? -- -------------------------------------------------- Jeremy M. Guthrie jeremy.guthrie(a)cdw.com Managed Cisco Security Services Hosting and Managed Services Technical Architect Phone: 608-298-1061 CDW Fax: 608-288-3007 5520 Research Park Drive NOC: 608-298-1102 Madison, WI 53711 NOC Email: hmshelp(a)cdw.com

2 1

config file question
by Rory Toma 05 Feb '10

05 Feb '10

OK, I have a question. I've found that it is good to run several syslog-ng processes when dealing with tens of thousands of log files. So, I'd like to receive everything in one syslog-ng process, and forward them, via localhost, to about a hundred or so different syslog-ng processes on the same machine. Is there a good way to write a (log; filter; dest;) line that would round-robin between a bunch of destinations in a single line?

4 6

syslog-ng dropping events - update
by Terry Slattery 03 Feb '10

03 Feb '10

I did a lot of research last week on network tuning for Centos 5.3, the OS on which syslog-ng is showing missed events. I also have a RH Enterprise 3 system that doesn't show event loss. The RHE3 system is forwarding events to the Centos-based system that is dropping events. I compare the logs from the RHE3 system with the logs on the Centos to determine if loss is occurring. Here's what I did: ==== Mods to syslog-ng.conf ==== Enable dns cache options in syslog-ng.conf: use_dns (yes); dns_cache (yes); dns_cache_size (2000); dns_cache_expire (86400); Set so_rcvbuf() option in syslog-ng.conf: udp(so_rcvbuf(1024000)); The resulting config: @version: 3.0 #Default configuration file for syslog-ng. # # For a description of syslog-ng configuration file directives, please read # the syslog-ng Administrator's guide at: # # http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html # options { time_reopen (10); log_fifo_size (100000); long_hostnames (off); use_dns (yes); dns_cache (yes); dns_cache_size (2000); dns_cache_expire (86400); use_fqdn (no); create_dirs (yes); keep_hostname (yes); }; ###### # sources source s_local { # message generated by Syslog-NG internal(); # standard Linux log source (this is the default place for the syslog() # function to send logs to) unix-stream("/dev/log"); # messages from the kernel file("/proc/kmsg" program_override("kernel: ")); }; # Set the UDP receive buffer to 1M. source s_net { udp(so_rcvbuf(1024000)); tcp(); #syslog(); }; ###### # destinations destination d_messages { file("/var/log/messages"); }; ###### # Log local and network events log { source(s_local); source(s_net); destination(d_messages); }; ==== Mods to sysctl.conf ==== Modify kernel buffer parameters in /etc/sysctl.conf: # Increase the maximum total buffer-space allocatable net.ipv4.tcp_mem = 8388608 12582912 16777216 net.ipv4.udp_mem = 8388608 12582912 16777216 # Increase the maximum read-buffer space allocatable net.ipv4.tcp_rmem = 8192 87380 16777216 net.ipv4.udp_rmem_min = 16384 # Increase the maximum write-buffer-space allocatable net.ipv4.tcp_wmem = 8192 65536 16777216 net.ipv4.udp_wmem_min = 16384 # Increase the maximum and default receive socket buffer size net.core.rmem_max=16777216 net.core.rmem_default=87380 #Set maximum number of packets, queued on the INPUT side, when the interface receives packets faster than kernel can process them. Default was 1000. net.core.netdev_max_backlog = 2500 # Increase the max number of connections. Default was 128. # I think this is only for tcp and one web site recommended increasing # it for heavily used web servers. net.core.somaxconn=1024 # Default to 20480 net.core.optmem_max = 1024000 # Drop packets that look like they are spoofed. Default = 1. # 0 = don't check; # 1 = drop packets that as sourced at a directly connected interface # but were input from another interface; # 2 = drop any packets that look spoofed. net.ipv4.conf.default.rp_filter = 0 net.ipv4.conf.eth0.rp_filter = 0 Then restart sysctl: sysctl -p ==== How to check packet loss on Linux ==== [Look for the section titled UDP in netstat output. I like to preceed the netstat command with a date command and save both to a file as a record for later comparisons.] # date Wed Feb 3 16:33:55 EST 2010 # netstat -us ... Udp: 24264777 packets received 2831 packets to unknown port received. 14262 packet receive errors 24538489 packets sent ... # date Wed Jan 27 11:21:54 EST 2010 # netstat -us ... Udp: 24227388 packets received 470 packets to unknown port received. 10644 packet receive errors 24503312 packets sent ... Note the increase in UDP receive errors. The packet rate is pretty low on these systems, peaking at a few packets per second and averages well below a packet per second. ====== Other thoughts ===== The RHE3 system has been forwarding log messages using the address spoofing feature. I turned it off yesterday afternoon and the logs show no packet loss since then. Unfortunately, I did not do a netstat at that time, so I can't tell if the kernel knows of any loss. I also did not have eth0/rp_filter = 0 until today, figuring that the default/rp_filter = 0 parameter would cover it, but decided that it wouldn't hurt to have both set. Interestingly, the RHE3 system has both values set to 1 and it does not drop any events (that I can detect - it certainly is getting more events than the Centos system). I'm about to load RHE5 on a different server, forward events to it and see if it drops events. -tcs -- Terry Slattery CCIE# 1026

1 0

Third Party Application Logs under syslog-ng.conf -solaris 10
by manik tripathi 03 Feb '10

03 Feb '10

Hi, We have oracle siebel Apps , that generates bunch of Logs under a "dir" and we wanted to feed the logs under that dir to Centralized Syslog-ng server. Is there a sample syslog-ng for solaris for doing wildcard dir or wildcard logs files.... Thanks _________________________________________________________________ Hotmail: Trusted email with Microsoft’s powerful SPAM protection. https://signup.live.com/signup.aspx?id=60969

2 1

syslog-ng v2.09
by Clayton Dukes 02 Feb '10

02 Feb '10

Hey guys, I have a customer that is running v2.09 (with security patches). We need them to forward logs to our syslog-ng server but need to keep the original device as the source name. I'm assuming this is what they need: keep_hostname(yes); chain_hostnames(no); Can anyone confirm that v2.09 supports this? Is there anything else needed to accomplish this in v2.09? -- ______________________________________________________________ Clayton Dukes ______________________________________________________________

2 3

syslog-ng poll
by Robert Fekete 01 Feb '10

01 Feb '10

Dear syslog-ng Users, As you might have already seen, we have recently restructured the webpage of syslog-ng Open Source Edition to resemble more to the webpages of our other products. We have also added some new information (e.g., references to third-party packages, etc.). We would also like to ask regularly your opinion about syslog-ng and its various features using anonymous, quick-to-complete polls. As a starter, we would like to know which version of syslog-ng do you most (meaning, on most of your hosts). If you have a few free seconds, please vote in the sidebox of the http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/ page. Also, any comments or would like to suggest questions to future polls is welcome. Regards, Robert

1 0

Logs not identical.
by Andy Loughran 29 Jan '10

29 Jan '10

Guys, I have had some issues with my httpd server, therefore had to enable some Debug Logging. This is in a different format to the rest of the logs. I have had the httpd.conf set logging to syslog:local1, and had the following in place to parse the logs log { source(s_sys); fillter(f_fac1); destination(d_logserver); }; that worked fine. However, I noticed it wasn't logging my Apache2::Resource debugs, here's an example of the two types of logs: [Fri Jan 29 16:00:30 2010] [debug] prefork.c(991): AcceptMutex: sysvsem (default: sysvsem) Apache2::Resource: PID 30247 attempting to set `DATA'=400:500 ...ok the top line is logged, but the second line isn't. I found this out, by setting httpd.conf to log to /var/log/httpd/error.log instead of syslog. For some reason syslog isn't getting/is cutting out the Apache2:: lines. So, I thought it prudent to get the httpd.conf to log to the file, and then get syslog-ng to read from this file. source s_httpd_err { file("/var/log/httpd/error.log" follow_freq(1) flags(no-parse)); }; I then put that straight into a separate file, with no filters. destination d_httpd_err { file("/logs/httpd_error.log"); }; log { source(s_httpd_err); destination(d_httpd_err); }; However, the destination is missing the Apache2:: logs - but has the normal formatted logs. Is there a filter still being applied, even though I don't specify one? Let me know if you need more information to help me solve my problem. Thanks, Andy

1 0

now on to my next problem
by Rory Toma 28 Jan '10

28 Jan '10

I'm getting a lot of dropped messages. The messages are getting to my syslog-ng server, but when I do an strace, I have: accept(3, 0x7fff5783dc10, [285598969948537856]) = -1 EAGAIN (Resource temporarily unavailable) Here is the config: @version: 3.0 options { flush_lines (3); time_reopen (10); log_fifo_size (1000000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); dir_perm (0755); keep_hostname (yes); ts_format("iso"); }; are there some other params I should be adjusting?

2 2

pattern database : first feedback and thoughts
by Christophe Brocas 27 Jan '10

27 Jan '10

Hello, I have start to play with pattern database. It is a very good starting point to do alerting when syslog-ng finds some logs belonging to (for example) security class. 1. Bug The current online pattern db snapshot provides the system-apache2.xml file. This file is not in a xml format but is just a set of log messages (it seems). 2. Class Can you tell us on what kind of criteria is used ti choose the class value (for example : system vs security class) of a particular log entry ? 3. Pattern database contribution / management / delivering Despite a big effort from Balabit (thanks for it !) to deliver an impressive first release (8000 log messages, 200 applications), I am sure that a community effort is the only way to maintain a good quality level of this database. You release the pattern database under the CC by-nc-sa licence. It is a very good way to start a community effort. In order to build a community around pattern database, I think some interesting points may be : - Having a dedicated DNS name and website (logpattern.org is available). Balabit can be of course the main sponsor of the website. But to be able to attract people and even more other companies for database contribution, I think all the communication / site management would have to rely on individuals and not on Balabit company. - Providing open log entry submission process through multi ways (web form, (D)VCS or bugreport system) - Providing a regular (each week, month or three months for ex.) database snapshot release cycle. If you want some help for the website, I can be involved :-) Thank you very much this pattern database effort :-) Christophe -- Christophe Brocas CNAMTS/DDSI/DSEC 12, allées Haussmann 33300 Bordeaux fixe : +33 (0)5.57.85.53.55 mobile : +33 (0)6.77.05.19.01 fax : +33 (0)5.56.39.84.48 keyid : 0x237E9DB2 ***************************************************** "Le contenu de ce courriel et ses eventuelles pièces jointes sont confidentiels. Ils s'adressent exclusivement à la personne destinataire. Si cet envoi ne vous est pas destiné, ou si vous l'avez reçu par erreur, et afin de ne pas violer le secret des correspondances, vous ne devez pas le transmettre à d'autres personnes ni le reproduire. Merci de le renvoyer à l'émetteur et de le détruire. Attention : L'Organisme de l'émetteur du message ne pourra être tenu responsable de l'altération du présent courriel. Il appartient au destinataire de vérifier que les messages et pièces jointes reçus ne contiennent pas de virus. Les opinions contenues dans ce courriel et ses éventuelles pièces jointes sont celles de l'émetteur. Elles ne reflètent pas la position de l'Organisme sauf s'il en est disposé autrement dans le présent courriel." ******************************************************

2 1

trouble shoot program() function
by Sal Polifemo 26 Jan '10

26 Jan '10

Hi everyone. I am trying to send mail notifications using the syslog-ng program() function and it's not working. To verify the filter works I use the same filter to log to a file and the data is going to the file correctly. When I run the script manually "grep ASA-5-111008 log_file | /usr/local/bin/cisco-cfg.pl" mail is generated. I added code to write to a file and that did not work. I am including the syslog-ng.cfg snippet and the cisco-cfg.pl code. Any ideas how I can trouble shoot why the email is not being sent from syslog-ng Thanks for you help syslog-ng.conf =========== source external { udp(port(514)); tcp(ip(0.0.0.0) port(5000) max-connections(300)); }; destination d_cisco_config{ program ("/usr/local/bin/cisco-cfg.pl"); }; destination test { file ("/var/log/HOSTS/test"); }; filter f_firewall_config{ match("ASA-6-605005") or match("ASA-5-111008"); }; log { #log to file to test that filter is working source(external); filter(f_firewall_config); destination(test); }; log { source (external); filter (f_firewall_config); destination(d_cisco_config); }; /usr/local/bin/cisco-cfg.pl ================ #!/usr/bin/perl -n use strict; use warnings; my $d_month = ""; my $d_day = ""; my $d_hour = ""; my $d_min = ""; my $d_sec = ""; my $host = ""; my $user = ""; my $command = ""; my $user_pc = ""; my $interface = ""; my $title = ""; my $to = ""; my $from = ""; my $subject = ""; if (/ASA\-5\-111008/) { /(\w{1,3})\s*(\d{1,2})\s(\d{2})\d{2})\d{2}).*?(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}).*?'(.*?)'.*?'(.*?)'/; $d_month =$1; $d_day = $2; $d_hour = $3; $d_min = $4; $d_sec = $5; $host = $6; $user =$7; $command = $8; $to = 'user(a)mail.com'; $title = "$host Firewall config alert"; $from = 'root(a)cadis.net'; open( MAIL, "|/usr/sbin/sendmail -t"); print MAIL "EOT"; ## Mail Header print MAIL "To: $to\n"; print MAIL "From: $from\n"; print MAIL "Subject: Firewall config change on $host\n\n"; ## mail body print MAIL "Host: $host\n"; print MAIL "Edited by: $user\n"; print MAIL "Command issued: $command\n"; print MAIL "Log stamp: $d_month $d_day $d_hour:$d_min.$d_sec\n"; print MAIL "\n\n"; print MAIL "Log message:\n"; print MAIL "$_\n"; print MAIL "EOT"; close( MAIL ); }

2 1

need help with a match filter
by Rory Toma 25 Jan '10

25 Jan '10

I'm trying to dynamically create lognames and directories form the host name that is sent. My hosts' names are of the form foo_<mac address> I want to create a destination that would be /logs/foo_<first 9 digits of mac address>/<date>/foo_<mac_address>.log Here is what I tried so far, and it does not work, but is close. Can someone suggest a config that might work? Thx. I can do this individually, but having 100+ filters and 100+ log lines in the config really slows things down. #filter for foo filter foo_filter { match('host("foo_[0-9A-F]\{9\}$")' flags("store-matches")); }; # foo destinations destination foo_dest { file("/logs/$0/$R_YEAR$R_MONTH$R_DAY/$HOST-$R_YEAR$R_MONTH$R_DAY.log" owner(root) group(root) perm(0444) template("$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC $MSG\n") template_escape(no)); };

1 1

Messages dropped...
by Pontius, Brian D CIV NAVSISA 25 Jan '10

25 Jan '10

I apologize for what seems to be repost of a rather similar problem but I having looked through the archives and unable to find answers. I am running syslog-ng 3.0.4 on Solaris 10 x86 (64bit). I have about 200 hosts, all running over udp. I have 1 heavy hitter, which is my firewall. I puts about 1500 messages a minute. It seems that syslog-ng is able to handle this amount of traffic but I am having trouble figuring out why I can't seem to make it work that way. I started to notice that I was dropping udp packets by running netstat -s |grep udpInOverflows. I tweaked the udp buffers by setting them to their max ndd -set /dev/udp udp_max_buf 1073741824 ndd -set /dev/udp udp_recv_hiwt 65536 I was still losing packets until I started to tweek my syslog-ng.conf and added the so_rcvbuf entries. The problem is, the logfiles do not reflect that all of the messages are making it. I only know this because the firewall is also logging to another standalone solaris server running standard syslogd and the syslog-ng's firewall's logs are still only getting 1/3 of the logs. Here is my syslog-ng.conf @version:3.0 # syslog-ng configuration file. # # This should behave pretty much like the original syslog on SunOS. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # # 20000925 gb(a)sysfive.com # Global Options options { flush_lines (2000); time_reopen (10); time_sleep (50); log_fifo_size (100000); log_fetch_limit (50000); chain_hostnames (yes); use_dns (yes); use_fqdn (no); keep_hostname (yes); dns_cache (yes); dns_cache_size (2000); dns_cache_expire(87600); create_dirs (yes); owner(splunk); group(Unix_users); dir_group(Unix_users); perm(0640); dir_perm(0750); stats_level(1); }; ######################################################## # Sources # Sun Remote port source s_sun { udp(ip("xxx.xxx.xxx.101") port(514) so_rcvbuf(1073741824)); }; # Hp Remote port source s_hp { udp(ip("xxx.xxx.xxx.102") port(514) so_rcvbuf(1073741824)); }; # Other Remote port source s_other { udp(ip("xxx.xxx.xxx.103") port(514)); }; # ESX Remote port source s_esx { udp(ip("xxx.xxx.xxx.104") port(514) so_rcvbuf(1073741824)); }; # Linux Remote port source s_linux { udp(ip("xxx.xxx.xxx.105") port(514) so_rcvbuf(1073741824)); }; # Switch Remote port source s_switch { udp(ip("xxx.xxx.xxx.106") port(514) so_rcvbuf(1073741824) ); }; source s_local { internal(); }; ######################################################## # Destinations # Sun destinations destination d_sun_cons { file("/shares/logfiles/syslog/sun/$HOST/console"); }; destination d_sun_mesg { file("/shares/logfiles/syslog/sun/$HOST/messages"); }; destination d_sun_syslog { file("/shares/logfiles/syslog/sun/$HOST/syslog"); }; destination d_sun_auth { file("/shares/logfiles/syslog/sun/$HOST/auth"); }; destination d_sun_audit { file("/shares/logfiles/syslog/sun/$HOST/audit.log"); }; # HP destinations destination d_hp_cons { file("/shares/logfiles/syslog/hp/$HOST/console"); }; destination d_hp_mail { file("/shares/logfiles/syslog/hp/$HOST/mail.log"); }; destination d_hp_syslog { file("/shares/logfiles/syslog/hp/$HOST/syslog.log"); }; # SAN destinations destination d_switch { file("/shares/logfiles/syslog/switches/$HOST/syslog.log"); }; # Other destinations destination d_other { file("/shares/logfiles/syslog/other_devices/$HOST/syslog.log"); }; # ESX destinations destination d_esx_cons { file("/shares/logfiles/syslog/esx/$HOST/console"); }; destination d_esx_mesg { file("/shares/logfiles/syslog/esx/$HOST/messages"); }; destination d_esx_mail { file("/shares/logfiles/syslog/esx/$HOST/maillog"); }; destination d_esx_auth { file("/shares/logfiles/syslog/esx/$HOST/secure"); }; destination d_esx_cron { file("/shares/logfiles/syslog/esx/$HOST/cron.log"); }; # Linux destinations destination d_linux_cons { file("/shares/logfiles/syslog/linux/$HOST/console"); }; destination d_linux_mesg { file("/shares/logfiles/syslog/linux/$HOST/messages"); }; destination d_linux_mail { file("/shares/logfiles/syslog/linux/$HOST/maillog"); }; destination d_linux_auth { file("/shares/logfiles/syslog/linux/$HOST/secure"); }; destination d_linux_cron { file("/shares/logfiles/syslog/linux/$HOST/cron.log"); }; destination d_local { file("/shares/logfiles/syslog/syslog-ng.log"); }; ######################################################## # Filtering # Sun Remote Filtering filter f_sun_mesg { level(err) or level(crit) or (facility(kern)) or (facility(daemon)) or (facility(mail) and level(crit)); }; filter f_sun_syslog { facility(mail); }; filter f_sun_auth { facility(auth); }; filter f_sun_audit { facility(13); }; filter f_sun_cons { level(crit) or level(alert) or (facility (auth)) or level(emerg); }; # Hp Remote Filtering filter f_hp_mail { facility(mail); }; filter f_hp_syslog { level(info); }; filter f_hp_cons { level(alert) or level(emerg); }; # SAN Remote Filtering #filter f_san { level(info); }; # ESX Remote Filtering filter f_esx_cons { level(emerg); }; filter f_esx_mesg { level(info) or facility(auth) or facility(local6) or facility(local5) or facility(local7) or (facility(mail) and level(crit)); }; filter f_esx_auth { facility(auth); }; filter f_esx_mail { facility(mail); }; filter f_esx_cron { facility(cron); }; # Linux Remote Filtering filter f_linux_cons { level(emerg); }; filter f_linux_mesg { level(info) or facility(auth) or facility(local6) or facility(local5) or facility(local7) or (facility(mail) and level(crit)); }; filter f_linux_auth { facility(auth); }; filter f_linux_mail { facility(mail); }; filter f_linux_cron { facility(cron); }; ######################################################## # Logging # Sun Remote Logging log { source(s_sun); filter(f_sun_cons); destination(d_sun_cons); flags(flow-control); }; log { source(s_sun); filter(f_sun_mesg); destination(d_sun_mesg); flags(flow-control); }; log { source(s_sun); filter(f_sun_syslog); destination(d_sun_syslog); flags(flow-control); }; log { source(s_sun); filter(f_sun_auth); destination(d_sun_auth); flags(flow-control); }; log { source(s_sun); filter(f_sun_audit); destination(d_sun_audit); flags(flow-control); }; # Hp Remote Logging log { source(s_hp); filter(f_hp_mail); destination(d_hp_mail); flags(flow-control); }; log { source(s_hp); filter(f_hp_syslog); destination(d_hp_syslog); flags(flow-control); }; log { source(s_hp); filter(f_hp_cons); destination(d_hp_cons); flags(flow-control); }; # SAN log { source(s_switch); destination(d_switch); flags(flow-control); }; # SAN log { source(s_other); destination(d_other); flags(flow-control); }; # ESX Remote Logging log { source(s_esx); filter(f_esx_cons); destination(d_esx_cons); flags(flow-control); }; log { source(s_esx); filter(f_esx_mesg); destination(d_esx_mesg); flags(flow-control); }; log { source(s_esx); filter(f_esx_mail); destination(d_esx_mail); flags(flow-control); }; log { source(s_esx); filter(f_esx_auth); destination(d_esx_auth); flags(flow-control); }; log { source(s_esx); filter(f_esx_cron); destination(d_esx_cron); flags(flow-control); }; # Linux Remote Logging log { source(s_linux); filter(f_linux_cons); destination(d_linux_cons); flags(flow-control); }; log { source(s_linux); filter(f_linux_mesg); destination(d_linux_mesg); flags(flow-control); }; log { source(s_linux); filter(f_linux_mail); destination(d_linux_mail); flags(flow-control); }; log { source(s_linux); filter(f_linux_auth); destination(d_linux_auth); flags(flow-control); }; log { source(s_linux); filter(f_linux_cron); destination(d_linux_cron); flags(flow-control); }; log { source(s_local); destination(d_local); }; Thanks for any help you can offer. Brian Pontius

2 2

problem with server
by Rory Toma 23 Jan '10

23 Jan '10

I am using syslog-ng-3.0.4-1.rhel on a CentOS-5.4 system. I am using tls, and have a setup as below. Here is what happens. It logs fine. However, if I reboot my client that is sending logs, it no longer works until I restart the syslog-ng server. What do I need to do here? @version: 3.0 options { flush_lines (3); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); dir_perm (0755); keep_hostname (yes); ts_format("iso"); }; source telo { tcp( port(80) tls( key_file("/export/tls/key.pem") cert_file("/export/tls/cert.pem") peer_verify(optional-untrusted)) ); }; # Myx destinations destination myx_dest_0000 { file("/logs/myx_008161000/$R_YEAR$R_MONTH$R_DAY/$HOS T-$R_YEAR$R_MONTH$R_DAY.log" owner(root) group(root) perm(0644) template("$YEAR- $MONTH-$DAY $HOUR:$MIN:$SEC $MSG\n") template_escape(no)); }; filter myx_filter_0000 { host("myx_001861000[0-9A-F]\{3\}$"); }; log { source(telo); filter(myx_filter_0000); destination(myx_dest_0000); };

2 6

Re: [syslog-ng] Messages dropped...
by Terry Slattery 22 Jan '10

22 Jan '10

> I started to notice that I was dropping udp packets... I'm seeing the same thing on a Centos 5.3 based system. I suspect something in the kernel or its interface to user space. My other syslog-ng systems are not dropping messages. -tcs -- Terry Slattery CCIE# 1026

1 0

Log all msgs but a specific filter
by PAUL WILLIAMSON 21 Jan '10

21 Jan '10

I need to log all messages except for a specific filter. I have the filter defined and these messages are getting logged to a separate file. However, I have one "main" log file where I do all my other processing. How can I include that filter in my destination so that it drops those specific messages from getting logged? Thanks, Paul ************************************ This email may contain privileged and/or confidential information that is intended solely for the use of the addressee. If you are not the intended recipient or entity, you are strictly prohibited from disclosing, copying, distributing or using any of the information contained in the transmission. If you received this communication in error, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. This communication may contain nonpublic personal information about consumers subject to the restrictions of the Gramm-Leach-Bliley Act and the Sarbanes-Oxley Act. You may not directly or indirectly reuse or disclose such information for any purpose other than to provide the services for which you are receiving the information. There are risks associated with the use of electronic transmission. The sender of this information does not control the method of transmittal or service providers and assumes no duty or obligation for the security, receipt, or third party interception of this transmission. ************************************

3 2

question about pattern database version 1
by Guillaume Rousse 20 Jan '10

20 Jan '10

Hello list. The pattern database format 1 seems to force a mandatory program pattern, wich doesn't exists for some messages, suchas kernel oops and memory exhaustion issues: Jan 19 15:18:24 localhost Call Trace: Jan 19 15:18:24 localhost ntpd invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0 It seems to be corrected in version 2: rulesets can be defined without program pattern where these act as a fail-back rule sets if no program pattern is matching or when message does not have program part I guess that means I'm can't do anything to catch those problems as long as syslog-ng 3.1 OSE is released, right ? -- BOFH excuse #330: quantum decoherence

2 1

filtering messages by context
by Guillaume Rousse 20 Jan '10

20 Jan '10

Hello list. We use hardware-based SLB for our critical services (ldap, radius, dns, ...). It means we have probes testing our services every X seconds, polluting our logs. It's quite easy to catch those traces when they consist of a single message, as for our kerberos servers Jan 20 11:17:02 avron1 kdc[477]: connection closed before end of data after 0 bytes from IPv4:10.202.11.254 to tcp/0 Jan 20 11:17:04 avron1 kdc[477]: connection closed before end of data after 0 bytes from IPv4:10.202.11.254 to tcp/0 Jan 20 11:17:06 avron1 kdc[477]: connection closed before end of data after 0 bytes from IPv4:10.202.11.254 to tcp/0 This simple setup is enough to drop them: destination d_drop { } filter f_slb_kerberos_probe { message("connection closed before end of data"); }; log { source(s_sys); filter(f_kerberos); filter(f_slb_kerberos_probe); destination(d_drop); flags(final); }; However, when those traces actually consist of two messages, as in our LDAP servers, it's a bit more difficult: Jan 20 11:23:04 avron1 slapd[13802]: conn=68089 fd=51 ACCEPT from IP=10.202.11.254:39428 (IP=10.202.11.8:389) Jan 20 11:23:04 avron1 slapd[13802]: conn=68089 fd=51 closed (connection lost) Jan 20 11:23:06 avron1 slapd[13802]: conn=68090 fd=51 ACCEPT from IP=10.202.11.254:39434 (IP=10.202.11.8:389) Jan 20 11:23:06 avron1 slapd[13802]: conn=68090 fd=51 closed (connection lost) I can filter out the first message, using the probe IP adress as criteria, but not the second one: filter f_slb_ldap_probe { message("ACCEPT from IP=10.202.11.254"); }; log { source(s_sys); filter(f_ldap); filter(f_slb_ldap_probe); destination(d_drop); flags(final); }; is there any way to catch the connection id in the first message (68090), so as to filter out any following one refering to the same connection ? -- BOFH excuse #152: My pony-tail hit the on/off switch on the power strip.

2 1

Remove double timestamping
by Marty Sørensen 20 Jan '10

20 Jan '10

Hello .... New user to syslog-ng but still hoping someone can help me with a small config example Im forwarding syslog from my syslog-ng but when it arrives it has double timestamps/hostname: -- Jan 19 11:02:58 cut-hostname 10.229.5.2 32176: Jan 19 11:02:57: %SFF8472-5-THRESHOLD_VIOLATION -- I have addet this info my config: options { use_time_recvd(yes); keep_hostname(no); } also included: destination udp_forward { udp("<my ip>"); }; template fjern { template("$MSGONLY\n"); }; destination cisco_cupe { file("/var/log/cisco_cupe" template("fjern")); }; log { source(s_all); filter(local5); destination(cisco_cupe); destination(udp_forward); }; Cheers Martin

3 5

md5sum or sha1sum files not found .
by Mr. James W. Laferriere 19 Jan '10

19 Jan '10

Hello All , When going to ... http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.0.4/… I am not finding either a md5sum or sha1sum (or greater) file for the contents . Is this due to decission not to or a oversight ? Tia , JimL -- +------------------------------------------------------------------+ | James W. Laferriere | System Techniques | Give me VMS | | Network&System Engineer | 3237 Holden Road | Give me Linux | | babydr(a)baby-dragons.com | Fairbanks, AK. 99709 | only on AXP | +------------------------------------------------------------------+

5 15

Dear Syslog-Ng, If you can't find in google, try JUSTDIAL.COM
by Vadiraj 18 Jan '10

18 Jan '10

Dear Syslog-Ng,I strongly recommend this website www.justdial.com. It's a world class local search service & I've always found anything I've ever wanted.You can find info on any company, product, or service in over 240 cities in India.You can also call them up 24x7, on phone (69999999), a local call in 240 Indian cities.Ask for anything, you'll get the info on the phone and/or by SMS within 30 secs, and this service is at no cost!For a change, it's an original Indian idea and an Indian company with world class service, and with a vision to spread all over the world.Be a proud Indian and forward this to every Indian you know.Best Wishes,VadirajClick Here to unsubscribe.

1 0