- syslog-ng - lists.balabit.hu

config with many rules
by Vasiliy G Tolstov 19 Sep '09

19 Sep '09

Hello. I'm new in this list, and may be my quistion answered already, but i can't founded any solutions: Problem: I have many filters, like this filter ff_auth { program(sshd) or program(dropbear) or facility(auth, authpriv); }; filter ff_cron { facility(cron); }; filter ff_daemon { facility(daemon); }; filter ff_user { facility(user); }; filter ff_debug { not facility(auth, authpriv, news, mail); }; filter ff_scm { program(git-daemon); }; filter ff_db { program(postgres) or program (mysql); }; filter ff_mail { facility(mail); }; filter ff_ftp { program(vsftpd) or program(proftpd); }; filter ff_rsync { program(rsync); }; filter ff_snmp { program(snmpd); }; filter ff_named { program(named); }; filter fl_alert { level(alert); }; filter fl_crit { level(crit); }; filter fl_err { level(err); }; filter fl_warn { level(warn); }; filter fl_notice { level(notice); }; filter fl_info { level(info); }; filter fl_debug { level(debug); }; If i want to add new filter to select kernel messages, that does not contain already defained data filter ff_kernel { facility(kern) and not match(firewall) and not match(mail) and not match(snmp) and not match(vsftpd) and not match(firewall); .....}; I think - not best solution to write all filters in not match and not program. Solution: Can You provide some help, how can i reduce unneded strings, may be not like ff_db and not like ff_debug ? P.S. Sorry if my english is bad and i can't fully discribe problem. -- Vasiliy G Tolstov <v.tolstov(a)selfip.ru> Selfip.Ru

2 2

Keeping hostname when forwarding
by PAUL WILLIAMSON 18 Sep '09

18 Sep '09

I have keep_hostname(yes) set in my global options, and no other mention of it anywhere else in my config. However, when I look at the remote system's logs, they are all tagged with the ip address of my syslog-ng host. I'm just trying to forward Cisco syslogs from my machine to another one, and using a filter for local7 is working great. I just need to figure out why the actual host is not being detected. Thanks, Paul ************************************ This email may contain privileged and/or confidential information that is intended solely for the use of the addressee. If you are not the intended recipient or entity, you are strictly prohibited from disclosing, copying, distributing or using any of the information contained in the transmission. If you received this communication in error, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. This communication may contain nonpublic personal information about consumers subject to the restrictions of the Gramm-Leach-Bliley Act and the Sarbanes-Oxley Act. You may not directly or indirectly reuse or disclose such information for any purpose other than to provide the services for which you are receiving the information. There are risks associated with the use of electronic transmission. The sender of this information does not control the method of transmittal or service providers and assumes no duty or obligation for the security, receipt, or third party interception of this transmission. ************************************

1 0

syslog-ng v3.0.4 stops receiving remote logs
by Mike Duncan 18 Sep '09

18 Sep '09

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 We are running into an issue using syslog-ng v3.0.3 and v3.0.4. After about 1minute or so, all external/remote hosts can no longer log to the logging server (syslog-ng). We run udp only and have something around 300-400 hosts logging to the server. The local logging seems to be okay even after the remote logging ceases. We are not getting any indication from syslog-ng (logs or stdout) that there is an issue unless we run it in debug mode. Firstly, we tried to ensure the rmem and wmem were set to handle everything. We were seeing some 30million packet receive errors, so bumped up the settings from 8MB to 16MB. This did not help resolve the issue. Secondly, after looking through mailing lists and various posts online, we thought the issue to be the 0byte-udp-packet bug as mentioned here: https://bugzilla.balabit.com/show_bug.cgi?id=32. When running v3.0.3 in debug mode with trace enabled, we saw the EOF error messages which lead us to download v3.0.4, apply the patch and compile. Unfortunately, we are still having the same issue as before but no EOF error messages this time. ulimit is set to unlimited and at most, we only get 30-40 fd's opened. So, we are at a loss and hoping to get some input from here. Here are some details...(sorry for wrapping issues)... netstat -su snippet =========================== Udp: 259879531 packets received 709190 packets to unknown port received. 29579106 packet receive errors 328899408 packets sent RcvbufErrors: 29579106 netstat -anp snippet =========================== udp 16776712 0 0.0.0.0:514 0.0.0.0:* - sysctl snippet =========================== net.core.rmem_default = 16777216 net.core.rmem_max = 16777216 net.core.wmem_default = 16777216 net.core.wmem_max = 16777216 syslog-ng.conf snippets =========================== options { chain_hostnames(0); time_reopen(10); time_reap(360); log_fifo_size(2048); create_dirs(yes); group(adm); perm(0640); dir_perm(0755); use_dns(yes); #log_msg_size(2048); stats_freq(0); bad_hostname("^gconfd$"); }; source s_all { internal(); unix-stream("/dev/log"); file("/proc/kmsg" program_override("kernel: ")); udp(ip(0.0.0.0) port(514)); }; #example remote logging destination df_windows { file("/var/log/windows.log"); }; filter f_windows { program("MSWinEventLog"); }; log { source(s_all); filter(f_windows); destination(df_windows); flags(final); }; #another example destination df_networkdevice { file("/var/log/networkdevice.log"); }; filter f_2e { host("2e[0-9]") or host("2eg[0-9]"); }; #2nd floor East log { source(s_all); filter(f_2e); destination(df_networkdevice); flags(final); }; uname -a ============================= Linux logalot 2.6.24-24-server #1 SMP Fri Jul 24 23:27:55 UTC 2009 i686 GNU/Linux ps ax | grep syslog ============================= 2201 ? S 0:00 supervising syslog-ng 2202 ? Ss 0:00 /opt/syslog-ng/sbin/syslog-ng --no-caps Are we running into a limit of some sort or a bug or more likely -- are we missing something? Any help is appreciated. If you need more information, just let me know. Thanks. - -- Mike Duncan ISSO, Application Security Specialist Government Contractor with STG, Inc. NOAA :: National Climatic Data Center -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkqzyA4ACgkQnvIkv6fg9hZdiQCgkO6wW8yo9AC49f7E+sEQxD7J nH4AmwYNG/zsLJugvy+apqrueG+UR4TS =Y6Ol -----END PGP SIGNATURE-----

2 1

the match() filter without the use of the value() option is deprecated
by Jeremy M. Guthrie 18 Sep '09

18 Sep '09

Can anyone shed light on what the the the below error message means? I cannot find anything on what value() is and why my syntax is wrong. I am migrating a older configuration to 3.0.4. WARNING: the match() filter without the use of the value() option is deprecated and hinders performance, please update your configuration; -- -------------------------------------------------- Jeremy M. Guthrie jeremy.guthrie(a)cdw.com Managed Cisco Security Services Hosting and Managed Services Technical Architect Phone: 608-298-1061 CDW Fax: 608-288-3007 5520 Research Park Drive NOC: 608-298-1102 Madison, WI 53711 NOC Email: hmshelp(a)cdw.com

2 1

dropped log?
by fedora fedora 18 Sep '09

18 Sep '09

Hello, i am testing syslog-ng on a server which will be used to collect logs from different devices on our network, at this point the server is not busy at all and no outside device is sending syslog to it, i enabled stats_freq() in the global config last night and just now i notice there are 5k logs got dropped since then, see below Aug 14 08:32:59 test syslog-ng[14059]: Log statistics; dropped='pipe(/dev/xconsole)=5093', processed='center(queued)=2795', processed='center(received)=705', So what exactly does "dropped='pipe(/dev/xconsole)=5093'" mean? The load of this server is not heavy at all, i guess most of these messages are firewall logs, why so many logs got dropped? Thanks!

5 8

Re: [syslog-ng] Python help
by chris packham 16 Sep '09

16 Sep '09

On Wed, 2009-09-16 at 11:03 +0530, Jain, Vaibhav (GE Healthcare) wrote: > Hi, > > I am using a python parser for parsing the incoming message from > syslog-ng. In my log messages some fields are optional and some are > not.I want to know how to handle these optional parameters in the > python regual expression. > Your question is about python not syslog so you might have more luck from a python forum/mailing list. Having said that, the standard way of making a group optional in a regex is to add ? after it e.g. \d(\w+)? means a single digit followed by and optional string of word characters. Here's a small snippet I whipped up based on your example data. --- import re; data = ['Sep 15 22:23:37 ecis007 codeSystemName=5 UserID="Admin"', 'Sep 15 22:23:37 ecis007 UserID="Admin"'] r = re.compile (r'(\w+)\s+(\d+)\s+(\d\d:\d\d:\d\d)\s+([^\s]+)\s+(codeSystemName=(\d+)\s+)?(.*)') for d in data: print "d="+d m = r.match(d); if m: print repr(m.groups()) --- # python test.py d=Sep 15 22:23:37 ecis007 codeSystemName=5 UserID="Admin" ('Sep', '15', '22:23:37', 'ecis007', 'codeSystemName=5 ', '5', 'UserID="Admin"') d=Sep 15 22:23:37 ecis007 UserID="Admin" ('Sep', '15', '22:23:37', 'ecis007', None, None, 'UserID="Admin"')

1 0

Python help
by Jain, Vaibhav (GE Healthcare) 16 Sep '09

16 Sep '09

Hi, I am using a python parser for parsing the incoming message from syslog-ng. In my log messages some fields are optional and some are not.I want to know how to handle these optional parameters in the python regual expression. for example: sample log :-> log='Sep 15 22:23:37 ecis007 codeSystemName=5 UserID="Admin"' regular expression :-> aregex=re.compile(r"(?P<destmonth>\S+) (?P<devicedate>\S+) (?P<devtime>\S+) (?P<devicename>\S+) codeSystemName=(?P<EventIDcodeSystemName>.*)") if I run the following python command then i can get all the parameters r=aregex.match(log2) but if "codeSystemName=5" is a optional parameters and it is not in some messages then my parser is not able to parse the message. log='Sep 15 22:23:37 ecis007 UserID="Admin' let me know how to write the regex for optional parameters ? Regards, Vaibhav

2 1

Syslog-ng open source version -- program_override option
by Jain, Vaibhav (GE Healthcare) 15 Sep '09

15 Sep '09

Hi, I want to pass the source file name with each syslog-ng message. I am using program_override option for this. But program_override option overrides the program name field value present in the syslog-ng messages. source source_sys_log { file("/root/log/syslog.log" program_override("syslog.log")); } Let me know how to pass source file name in the syslog-ng message? I am using open source version of syslog-ng. Regards Vaibhav

4 8

Getting error when checking syntax
by PAUL WILLIAMSON 14 Sep '09

14 Sep '09

When I run ./syslog-ng --syntax, I get the following: ld.so.1: syslog-ng: fatal: relocation error: file syslog-ng: symbol g_option_context_new: referenced symbol not found Killed As suggested, I installed syslog-ng 3.0.4 from SFW. This is a Solaris 10 SPARC physical machine. Thanks, Paul ************************************ This email may contain privileged and/or confidential information that is intended solely for the use of the addressee. If you are not the intended recipient or entity, you are strictly prohibited from disclosing, copying, distributing or using any of the information contained in the transmission. If you received this communication in error, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. This communication may contain nonpublic personal information about consumers subject to the restrictions of the Gramm-Leach-Bliley Act and the Sarbanes-Oxley Act. You may not directly or indirectly reuse or disclose such information for any purpose other than to provide the services for which you are receiving the information. There are risks associated with the use of electronic transmission. The sender of this information does not control the method of transmittal or service providers and assumes no duty or obligation for the security, receipt, or third party interception of this transmission. ************************************

3 3

Running two syslog-ng instances
by Pedro Rafael Alves Simoes 12 Sep '09

12 Sep '09

Hello, I need to run two syslog-ng instances on a machine. I'm going to receive two heavy "streams" of events. The two syslog-ng's will have different configurations files. My question is: Assuming that just one syslog-ng isn't enough, this is the best way to do this? There is any drawback? Is a virtualized system (for example Xen on the same machine) a better option (isolating the syslog-ng instances and configurations)? Thank you, Pedro Simões

2 1