- syslog-ng - lists.balabit.hu

trouble shoot program() function
by Sal Polifemo 26 Jan '10

26 Jan '10

Hi everyone. I am trying to send mail notifications using the syslog-ng program() function and it's not working. To verify the filter works I use the same filter to log to a file and the data is going to the file correctly. When I run the script manually "grep ASA-5-111008 log_file | /usr/local/bin/cisco-cfg.pl" mail is generated. I added code to write to a file and that did not work. I am including the syslog-ng.cfg snippet and the cisco-cfg.pl code. Any ideas how I can trouble shoot why the email is not being sent from syslog-ng Thanks for you help syslog-ng.conf =========== source external { udp(port(514)); tcp(ip(0.0.0.0) port(5000) max-connections(300)); }; destination d_cisco_config{ program ("/usr/local/bin/cisco-cfg.pl"); }; destination test { file ("/var/log/HOSTS/test"); }; filter f_firewall_config{ match("ASA-6-605005") or match("ASA-5-111008"); }; log { #log to file to test that filter is working source(external); filter(f_firewall_config); destination(test); }; log { source (external); filter (f_firewall_config); destination(d_cisco_config); }; /usr/local/bin/cisco-cfg.pl ================ #!/usr/bin/perl -n use strict; use warnings; my $d_month = ""; my $d_day = ""; my $d_hour = ""; my $d_min = ""; my $d_sec = ""; my $host = ""; my $user = ""; my $command = ""; my $user_pc = ""; my $interface = ""; my $title = ""; my $to = ""; my $from = ""; my $subject = ""; if (/ASA\-5\-111008/) { /(\w{1,3})\s*(\d{1,2})\s(\d{2})\d{2})\d{2}).*?(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}).*?'(.*?)'.*?'(.*?)'/; $d_month =$1; $d_day = $2; $d_hour = $3; $d_min = $4; $d_sec = $5; $host = $6; $user =$7; $command = $8; $to = 'user(a)mail.com'; $title = "$host Firewall config alert"; $from = 'root(a)cadis.net'; open( MAIL, "|/usr/sbin/sendmail -t"); print MAIL "EOT"; ## Mail Header print MAIL "To: $to\n"; print MAIL "From: $from\n"; print MAIL "Subject: Firewall config change on $host\n\n"; ## mail body print MAIL "Host: $host\n"; print MAIL "Edited by: $user\n"; print MAIL "Command issued: $command\n"; print MAIL "Log stamp: $d_month $d_day $d_hour:$d_min.$d_sec\n"; print MAIL "\n\n"; print MAIL "Log message:\n"; print MAIL "$_\n"; print MAIL "EOT"; close( MAIL ); }

2 1

need help with a match filter
by Rory Toma 25 Jan '10

25 Jan '10

I'm trying to dynamically create lognames and directories form the host name that is sent. My hosts' names are of the form foo_<mac address> I want to create a destination that would be /logs/foo_<first 9 digits of mac address>/<date>/foo_<mac_address>.log Here is what I tried so far, and it does not work, but is close. Can someone suggest a config that might work? Thx. I can do this individually, but having 100+ filters and 100+ log lines in the config really slows things down. #filter for foo filter foo_filter { match('host("foo_[0-9A-F]\{9\}$")' flags("store-matches")); }; # foo destinations destination foo_dest { file("/logs/$0/$R_YEAR$R_MONTH$R_DAY/$HOST-$R_YEAR$R_MONTH$R_DAY.log" owner(root) group(root) perm(0444) template("$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC $MSG\n") template_escape(no)); };

1 1

Messages dropped...
by Pontius, Brian D CIV NAVSISA 25 Jan '10

25 Jan '10

I apologize for what seems to be repost of a rather similar problem but I having looked through the archives and unable to find answers. I am running syslog-ng 3.0.4 on Solaris 10 x86 (64bit). I have about 200 hosts, all running over udp. I have 1 heavy hitter, which is my firewall. I puts about 1500 messages a minute. It seems that syslog-ng is able to handle this amount of traffic but I am having trouble figuring out why I can't seem to make it work that way. I started to notice that I was dropping udp packets by running netstat -s |grep udpInOverflows. I tweaked the udp buffers by setting them to their max ndd -set /dev/udp udp_max_buf 1073741824 ndd -set /dev/udp udp_recv_hiwt 65536 I was still losing packets until I started to tweek my syslog-ng.conf and added the so_rcvbuf entries. The problem is, the logfiles do not reflect that all of the messages are making it. I only know this because the firewall is also logging to another standalone solaris server running standard syslogd and the syslog-ng's firewall's logs are still only getting 1/3 of the logs. Here is my syslog-ng.conf @version:3.0 # syslog-ng configuration file. # # This should behave pretty much like the original syslog on SunOS. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # # 20000925 gb(a)sysfive.com # Global Options options { flush_lines (2000); time_reopen (10); time_sleep (50); log_fifo_size (100000); log_fetch_limit (50000); chain_hostnames (yes); use_dns (yes); use_fqdn (no); keep_hostname (yes); dns_cache (yes); dns_cache_size (2000); dns_cache_expire(87600); create_dirs (yes); owner(splunk); group(Unix_users); dir_group(Unix_users); perm(0640); dir_perm(0750); stats_level(1); }; ######################################################## # Sources # Sun Remote port source s_sun { udp(ip("xxx.xxx.xxx.101") port(514) so_rcvbuf(1073741824)); }; # Hp Remote port source s_hp { udp(ip("xxx.xxx.xxx.102") port(514) so_rcvbuf(1073741824)); }; # Other Remote port source s_other { udp(ip("xxx.xxx.xxx.103") port(514)); }; # ESX Remote port source s_esx { udp(ip("xxx.xxx.xxx.104") port(514) so_rcvbuf(1073741824)); }; # Linux Remote port source s_linux { udp(ip("xxx.xxx.xxx.105") port(514) so_rcvbuf(1073741824)); }; # Switch Remote port source s_switch { udp(ip("xxx.xxx.xxx.106") port(514) so_rcvbuf(1073741824) ); }; source s_local { internal(); }; ######################################################## # Destinations # Sun destinations destination d_sun_cons { file("/shares/logfiles/syslog/sun/$HOST/console"); }; destination d_sun_mesg { file("/shares/logfiles/syslog/sun/$HOST/messages"); }; destination d_sun_syslog { file("/shares/logfiles/syslog/sun/$HOST/syslog"); }; destination d_sun_auth { file("/shares/logfiles/syslog/sun/$HOST/auth"); }; destination d_sun_audit { file("/shares/logfiles/syslog/sun/$HOST/audit.log"); }; # HP destinations destination d_hp_cons { file("/shares/logfiles/syslog/hp/$HOST/console"); }; destination d_hp_mail { file("/shares/logfiles/syslog/hp/$HOST/mail.log"); }; destination d_hp_syslog { file("/shares/logfiles/syslog/hp/$HOST/syslog.log"); }; # SAN destinations destination d_switch { file("/shares/logfiles/syslog/switches/$HOST/syslog.log"); }; # Other destinations destination d_other { file("/shares/logfiles/syslog/other_devices/$HOST/syslog.log"); }; # ESX destinations destination d_esx_cons { file("/shares/logfiles/syslog/esx/$HOST/console"); }; destination d_esx_mesg { file("/shares/logfiles/syslog/esx/$HOST/messages"); }; destination d_esx_mail { file("/shares/logfiles/syslog/esx/$HOST/maillog"); }; destination d_esx_auth { file("/shares/logfiles/syslog/esx/$HOST/secure"); }; destination d_esx_cron { file("/shares/logfiles/syslog/esx/$HOST/cron.log"); }; # Linux destinations destination d_linux_cons { file("/shares/logfiles/syslog/linux/$HOST/console"); }; destination d_linux_mesg { file("/shares/logfiles/syslog/linux/$HOST/messages"); }; destination d_linux_mail { file("/shares/logfiles/syslog/linux/$HOST/maillog"); }; destination d_linux_auth { file("/shares/logfiles/syslog/linux/$HOST/secure"); }; destination d_linux_cron { file("/shares/logfiles/syslog/linux/$HOST/cron.log"); }; destination d_local { file("/shares/logfiles/syslog/syslog-ng.log"); }; ######################################################## # Filtering # Sun Remote Filtering filter f_sun_mesg { level(err) or level(crit) or (facility(kern)) or (facility(daemon)) or (facility(mail) and level(crit)); }; filter f_sun_syslog { facility(mail); }; filter f_sun_auth { facility(auth); }; filter f_sun_audit { facility(13); }; filter f_sun_cons { level(crit) or level(alert) or (facility (auth)) or level(emerg); }; # Hp Remote Filtering filter f_hp_mail { facility(mail); }; filter f_hp_syslog { level(info); }; filter f_hp_cons { level(alert) or level(emerg); }; # SAN Remote Filtering #filter f_san { level(info); }; # ESX Remote Filtering filter f_esx_cons { level(emerg); }; filter f_esx_mesg { level(info) or facility(auth) or facility(local6) or facility(local5) or facility(local7) or (facility(mail) and level(crit)); }; filter f_esx_auth { facility(auth); }; filter f_esx_mail { facility(mail); }; filter f_esx_cron { facility(cron); }; # Linux Remote Filtering filter f_linux_cons { level(emerg); }; filter f_linux_mesg { level(info) or facility(auth) or facility(local6) or facility(local5) or facility(local7) or (facility(mail) and level(crit)); }; filter f_linux_auth { facility(auth); }; filter f_linux_mail { facility(mail); }; filter f_linux_cron { facility(cron); }; ######################################################## # Logging # Sun Remote Logging log { source(s_sun); filter(f_sun_cons); destination(d_sun_cons); flags(flow-control); }; log { source(s_sun); filter(f_sun_mesg); destination(d_sun_mesg); flags(flow-control); }; log { source(s_sun); filter(f_sun_syslog); destination(d_sun_syslog); flags(flow-control); }; log { source(s_sun); filter(f_sun_auth); destination(d_sun_auth); flags(flow-control); }; log { source(s_sun); filter(f_sun_audit); destination(d_sun_audit); flags(flow-control); }; # Hp Remote Logging log { source(s_hp); filter(f_hp_mail); destination(d_hp_mail); flags(flow-control); }; log { source(s_hp); filter(f_hp_syslog); destination(d_hp_syslog); flags(flow-control); }; log { source(s_hp); filter(f_hp_cons); destination(d_hp_cons); flags(flow-control); }; # SAN log { source(s_switch); destination(d_switch); flags(flow-control); }; # SAN log { source(s_other); destination(d_other); flags(flow-control); }; # ESX Remote Logging log { source(s_esx); filter(f_esx_cons); destination(d_esx_cons); flags(flow-control); }; log { source(s_esx); filter(f_esx_mesg); destination(d_esx_mesg); flags(flow-control); }; log { source(s_esx); filter(f_esx_mail); destination(d_esx_mail); flags(flow-control); }; log { source(s_esx); filter(f_esx_auth); destination(d_esx_auth); flags(flow-control); }; log { source(s_esx); filter(f_esx_cron); destination(d_esx_cron); flags(flow-control); }; # Linux Remote Logging log { source(s_linux); filter(f_linux_cons); destination(d_linux_cons); flags(flow-control); }; log { source(s_linux); filter(f_linux_mesg); destination(d_linux_mesg); flags(flow-control); }; log { source(s_linux); filter(f_linux_mail); destination(d_linux_mail); flags(flow-control); }; log { source(s_linux); filter(f_linux_auth); destination(d_linux_auth); flags(flow-control); }; log { source(s_linux); filter(f_linux_cron); destination(d_linux_cron); flags(flow-control); }; log { source(s_local); destination(d_local); }; Thanks for any help you can offer. Brian Pontius

2 2

problem with server
by Rory Toma 23 Jan '10

23 Jan '10

I am using syslog-ng-3.0.4-1.rhel on a CentOS-5.4 system. I am using tls, and have a setup as below. Here is what happens. It logs fine. However, if I reboot my client that is sending logs, it no longer works until I restart the syslog-ng server. What do I need to do here? @version: 3.0 options { flush_lines (3); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); dir_perm (0755); keep_hostname (yes); ts_format("iso"); }; source telo { tcp( port(80) tls( key_file("/export/tls/key.pem") cert_file("/export/tls/cert.pem") peer_verify(optional-untrusted)) ); }; # Myx destinations destination myx_dest_0000 { file("/logs/myx_008161000/$R_YEAR$R_MONTH$R_DAY/$HOS T-$R_YEAR$R_MONTH$R_DAY.log" owner(root) group(root) perm(0644) template("$YEAR- $MONTH-$DAY $HOUR:$MIN:$SEC $MSG\n") template_escape(no)); }; filter myx_filter_0000 { host("myx_001861000[0-9A-F]\{3\}$"); }; log { source(telo); filter(myx_filter_0000); destination(myx_dest_0000); };

2 6

Re: [syslog-ng] Messages dropped...
by Terry Slattery 22 Jan '10

22 Jan '10

> I started to notice that I was dropping udp packets... I'm seeing the same thing on a Centos 5.3 based system. I suspect something in the kernel or its interface to user space. My other syslog-ng systems are not dropping messages. -tcs -- Terry Slattery CCIE# 1026

1 0

Log all msgs but a specific filter
by PAUL WILLIAMSON 21 Jan '10

21 Jan '10

I need to log all messages except for a specific filter. I have the filter defined and these messages are getting logged to a separate file. However, I have one "main" log file where I do all my other processing. How can I include that filter in my destination so that it drops those specific messages from getting logged? Thanks, Paul ************************************ This email may contain privileged and/or confidential information that is intended solely for the use of the addressee. If you are not the intended recipient or entity, you are strictly prohibited from disclosing, copying, distributing or using any of the information contained in the transmission. If you received this communication in error, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. This communication may contain nonpublic personal information about consumers subject to the restrictions of the Gramm-Leach-Bliley Act and the Sarbanes-Oxley Act. You may not directly or indirectly reuse or disclose such information for any purpose other than to provide the services for which you are receiving the information. There are risks associated with the use of electronic transmission. The sender of this information does not control the method of transmittal or service providers and assumes no duty or obligation for the security, receipt, or third party interception of this transmission. ************************************

3 2

question about pattern database version 1
by Guillaume Rousse 20 Jan '10

20 Jan '10

Hello list. The pattern database format 1 seems to force a mandatory program pattern, wich doesn't exists for some messages, suchas kernel oops and memory exhaustion issues: Jan 19 15:18:24 localhost Call Trace: Jan 19 15:18:24 localhost ntpd invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0 It seems to be corrected in version 2: rulesets can be defined without program pattern where these act as a fail-back rule sets if no program pattern is matching or when message does not have program part I guess that means I'm can't do anything to catch those problems as long as syslog-ng 3.1 OSE is released, right ? -- BOFH excuse #330: quantum decoherence

2 1

filtering messages by context
by Guillaume Rousse 20 Jan '10

20 Jan '10

Hello list. We use hardware-based SLB for our critical services (ldap, radius, dns, ...). It means we have probes testing our services every X seconds, polluting our logs. It's quite easy to catch those traces when they consist of a single message, as for our kerberos servers Jan 20 11:17:02 avron1 kdc[477]: connection closed before end of data after 0 bytes from IPv4:10.202.11.254 to tcp/0 Jan 20 11:17:04 avron1 kdc[477]: connection closed before end of data after 0 bytes from IPv4:10.202.11.254 to tcp/0 Jan 20 11:17:06 avron1 kdc[477]: connection closed before end of data after 0 bytes from IPv4:10.202.11.254 to tcp/0 This simple setup is enough to drop them: destination d_drop { } filter f_slb_kerberos_probe { message("connection closed before end of data"); }; log { source(s_sys); filter(f_kerberos); filter(f_slb_kerberos_probe); destination(d_drop); flags(final); }; However, when those traces actually consist of two messages, as in our LDAP servers, it's a bit more difficult: Jan 20 11:23:04 avron1 slapd[13802]: conn=68089 fd=51 ACCEPT from IP=10.202.11.254:39428 (IP=10.202.11.8:389) Jan 20 11:23:04 avron1 slapd[13802]: conn=68089 fd=51 closed (connection lost) Jan 20 11:23:06 avron1 slapd[13802]: conn=68090 fd=51 ACCEPT from IP=10.202.11.254:39434 (IP=10.202.11.8:389) Jan 20 11:23:06 avron1 slapd[13802]: conn=68090 fd=51 closed (connection lost) I can filter out the first message, using the probe IP adress as criteria, but not the second one: filter f_slb_ldap_probe { message("ACCEPT from IP=10.202.11.254"); }; log { source(s_sys); filter(f_ldap); filter(f_slb_ldap_probe); destination(d_drop); flags(final); }; is there any way to catch the connection id in the first message (68090), so as to filter out any following one refering to the same connection ? -- BOFH excuse #152: My pony-tail hit the on/off switch on the power strip.

2 1

Remove double timestamping
by Marty Sørensen 20 Jan '10

20 Jan '10

Hello .... New user to syslog-ng but still hoping someone can help me with a small config example Im forwarding syslog from my syslog-ng but when it arrives it has double timestamps/hostname: -- Jan 19 11:02:58 cut-hostname 10.229.5.2 32176: Jan 19 11:02:57: %SFF8472-5-THRESHOLD_VIOLATION -- I have addet this info my config: options { use_time_recvd(yes); keep_hostname(no); } also included: destination udp_forward { udp("<my ip>"); }; template fjern { template("$MSGONLY\n"); }; destination cisco_cupe { file("/var/log/cisco_cupe" template("fjern")); }; log { source(s_all); filter(local5); destination(cisco_cupe); destination(udp_forward); }; Cheers Martin

3 5

md5sum or sha1sum files not found .
by Mr. James W. Laferriere 19 Jan '10

19 Jan '10

Hello All , When going to ... http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.0.4/… I am not finding either a md5sum or sha1sum (or greater) file for the contents . Is this due to decission not to or a oversight ? Tia , JimL -- +------------------------------------------------------------------+ | James W. Laferriere | System Techniques | Give me VMS | | Network&System Engineer | 3237 Holden Road | Give me Linux | | babydr(a)baby-dragons.com | Fairbanks, AK. 99709 | only on AXP | +------------------------------------------------------------------+

5 15