- syslog-ng - lists.balabit.hu

intentionally dropping messages
by Jan Schaumann 03 Jun '09

03 Jun '09

Hello, I'm trying to drop a certain set of messages. Following http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch07s08.html I notice that if I do *not* include a 'destination' in the 'log' statement, then a very large number of messages that should *not* be dropped are in fact dropped. As soon as I add a 'destination' to the log statement, it filters the correct messages. That is: destination d_dev_null { file("/dev/null" perm(0666) ); }; filter demo_debugfilter { (level(debug..notice) and facility(local0); }; log { source(s_udp); filter(demo_debugfilter); flags(final); }; Appears to falsely drop a number of messages that were of local1.info. As soon as I change the 'log' directive to be: log { source(s_udp); filter(demo_debugfilter); destination(d_dev_null); flags(final); }; it correctly filters only level0.{debug,info,notice}. This happens on a host with a large number of messages per second, but not on a host that receives very little traffic. I can't make much sense of this and was hoping somebody on this list could... -Jan

2 2

Re: [syslog-ng] db-parser issues
by Jakub Jankowski 03 Jun '09

03 Jun '09

On 2009-06-02, Robert Fekete wrote: > I think the problem is that you define a pattern in your xml for the program > as well: >> <program name='logger'> >> <pattern>param</pattern> > The <pattern> tag overrides the <program name>, so instead of logger, > syslog-ng thinks this rule applies to messages that have "param" in their > PROGRAM field. In addition to what you've quoted above, I also tried: 1. <program name='logger'> <pattern>logger</pattern> 2. <program name='somethingirrelevant'> <pattern>logger</pattern> and even without <pattern>: 3. <program name='logger'> All of these give me the same result - no match. Any other suggestions? Cheers, Jakub. -- Jakub Jankowski|shasta@toxcorp.com|http://toxcorp.com/ GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D

2 1

Roadmap for syslog-ng OSE 4.0
by Balazs Scheidler 03 Jun '09

03 Jun '09

Dear syslog-ng users, The roadmap for syslog-ng OSE has been updated and published at: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The same page also describes important release policy changes, thus it is recommended reading for all syslog-ng users. A related blog post can be found here: http://bazsi.blogs.balabit.com/2009/05/syslog-ng-40-roadmap-plus-release.ht… -- Bazsi

1 0

plz help
by dreamywoman 02 Jun '09

02 Jun '09

Hi All last week i have installed php-syslog-ng-2.5.1 and it works and i can see all logs in my web but after week i have decided to change the version of php-syslog-ng to php-syslog-ng-2.9.8j.tgz but after web installation i decided to change the perivious version (php-syslog-ng-2.5.1 ) after that i have problems i can get logs in other file like /var/log/catchall but i cant get anything in my php-syslog-ng in my web i use tcpdump i can get logs but i cant see anything in my web what should i do? source s_remote { unix-stream("/dev/log"); internal(); udp(); }; destination d_mysql { pipe("/tmp/mysql.pipe " template("INSERT INTO logs (host, facility, priority, level, tag, date, time, program, msg) VALUES ('$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY', '$HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n") template-escape(yes)); }; log { source(s_remote); destination(d_mysql); }; what is my problem in installation?

1 0

iptables logging in a separate logfile works not properly
by Ralf Heidenreich 02 Jun '09

02 Jun '09

Hello, I have the following line in iptables: iptables -A INPUT -j LOG --log-prefix "INPUT: " In syslog-ng (/etc/syslog-ng/syslog-ng.conf) I have the following config: ############################################# options { sync (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source s_sys { file ("/proc/kmsg" log_prefix("kernel: ")); unix-stream ("/dev/log"); internal(); # udp(ip(0.0.0.0) port(514)); }; destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" sync(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_kern { file("/var/log/messages"); }; destination d_mlal { usertty("*"); }; # direct iptables logs into /var/log/firewall.log destination iptables { file("/var/log/firewall.log"); }; filter iptables { match("INPUT:"); }; log { source(s_sys); filter(iptables); destination(iptables); }; filter noiptables { not match("INPUT:"); }; log { source(s_sys); filter(noiptables); destination(d_kern); }; filter f_kernel { facility(kern); }; filter f_default { level(info..emerg) and not (facility(mail) or facility(authpriv) or facility(cron)); }; filter f_auth { facility(authpriv); }; filter f_mail { facility(mail); }; filter f_emergency { level(emerg); }; filter f_news { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_boot { facility(local7); }; filter f_cron { facility(cron); }; #log { source(s_sys); filter(f_kernel); destination(d_cons); }; log { source(s_sys); filter(f_kernel); destination(d_kern); }; log { source(s_sys); filter(f_default); destination(d_mesg); }; log { source(s_sys); filter(f_auth); destination(d_auth); }; log { source(s_sys); filter(f_mail); destination(d_mail); }; log { source(s_sys); filter(f_emergency); destination(d_mlal); }; log { source(s_sys); filter(f_news); destination(d_spol); }; log { source(s_sys); filter(f_boot); destination(d_boot); }; log { source(s_sys); filter(f_cron); destination(d_cron); }; ################################################################ iptables loggs to /var/log/firewall.log, but also to /var/log/messages. A line in messages is: May 29 14:13:55 hostname kernel: INPUT: IN=eth0 OUT= MAC=00:16:17:6b:34:bf:00:02:85:12:c9:40:08:00 SRC=source-ip DST=server-ip LEN=104 TOS=0x00 PREC=0x00 TTL=56 ID=38612 PROTO=TCP SPT=26787 DPT=122 WINDOW=1032 RES=0x00 ACK PSH URGP=0 What I am doing wrong. with kind regards Ralf

2 3

rotate logs in recursive directories
by Reaky Rok 01 Jun '09

01 Jun '09

Hi friends I have a problem with syslog-ng in my server, I setup it successfuly in RHEL5 and It waorking without any problems, I make it a central log server for all servers in my company, The problem is that I orgnized it for each server bu date and facility like that $HOST/$YEAR.$MONTH.$DAY/$FACILITY.log which makes a lot of recursive directory for each server, Now I'm facing a problem to rotate it as I don't know how to make the logrotate rotate all this files in the recursive directories, Could you help me please, This is my /etc/logrotate.d/syslog-ng /var/log/syslog-ng { daily rotate 30 compress missingok notifempty create 0600 root root } Thanks & Best Regards Bassam Muhammad _________________________________________________________________ Windows Live™: Keep your life in sync. Check it out! http://windowslive.com/explore?ocid=TXT_TAGLM_WL_t1_allup_explore_012009

1 0

db-parser issues
by Jakub Jankowski 30 May '09

30 May '09

Hi, I'm having hard time getting db-parser functionality to work. This is my syslog-ng config (omitted irrelevant stuff; some lines may be split due to line wrappings in my MUA): @version: 3.0 source s_sys { file ("/proc/kmsg"); unix-stream ("/dev/log" flags(no-multi-line)); internal(); }; destination d_test { file("/var/log/test.log" template("$HOUR:$MIN:$SEC $TZ $HOST [$LEVEL] ${.classifier.class} ${MSGONLY} ${FOO.BAR}\n") template_escape(no)); }; parser p_test { db-parser(file(/etc/syslog-ng/test.xml)); }; filter f_test { program("logger" type(string)); }; log { source(s_sys); filter(f_test); parser(p_test); destination(d_test); }; My pattern db (test.xml) is as simple as: <patterndb version='1' pub_date='2009-05-16'> <program name='logger'> <pattern>param</pattern> <rules> <rule provider='tester' id='666' class='security'> <description>Test rule</description> <pattern>param1=@IPV4:FOO.BAR@</pattern> </rule> </rules> </program> </patterndb> Syslog-ng has no trouble loading it on startup, as suggested by these lines in /var/log/messages: May 30 00:17:37 atest1 syslog-ng[8681]: Log pattern database reloaded; file='/etc/syslog-ng/test.xml', version='1', pub_date='2009-05-16' May 30 00:17:37 atest1 syslog-ng[8681]: syslog-ng starting up; version='3.0.2' I'm testing db-parser, by issuing: $ echo param1=1.2.3.4 param2=abcde | logger -i $ And this is what I get in /var/log/test.log: 00:17:38 +02:00 atest1 [notice] unknown param1=1.2.3.4 param2=abcde Clearly not what I wanted. Seems like my parser doesn't match anything, as ${FOO.BAR} is empty. Can you shed some light on my problem? Am I doing something obviously wrong? Db-parser functionality is neat, but it lacks documentation - I base on Balázs' presentation and some blog posts only. Thanks in advance, Jakub -- Jakub Jankowski|shasta@toxcorp.com|http://toxcorp.com/ GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D

1 0

Re: [syslog-ng] syslog-ng 3.0.2 issues: Extraneous colon (:) in forwarded log; two instances of syslog-ng
by Marvin Nipper 29 May '09

29 May '09

Sorry for a second message, but I've just noticed a third issue with 3.0.2. Same test scenario as already described. Here's a messaged logged on my production 2.0.9 server, and also forwarded to my test server (I've removed some of the trailing noise): May 29 16:15:09 fwmil01 [LOG_WARNING] ipsrd[213]: igmp_recv_leave_group: <truncated.....> When it arrives on my test server, and a 2.0.9 daemon catches it (and writes it to a file), it looks exactly as above, but when 3.0.2 catches it, this is the result: May 29 16:15:09 fwmil01 ipsrd[213]: igmp_recv_leave_group: <truncated.....> In short, the Severity information has been stripped. Again, in looking at the 3.0 docs, I'm not clear as to any change that would cause this to be stripped out. (It might not be that big a deal to lose that, except that there are subsequent filters in my system that are looking for that "translated severity" as part of their search.) Anyway, again, I would appreciate any input as to why 3.x would be doing this, and what configuration change might restore the original content. THANKS for your help. _____________________________________________ From: Marvin Nipper Sent: Friday, May 29, 2009 12:08 PM To: 'syslog-ng(a)lists.balabit.hu' Subject: syslog-ng 3.0.2 issues: Extraneous colon (:) in forwarded log; two instances of syslog-ng Hi. I'm been running syslog-ng 2.0.9 for eons, and wanted to get 3.0.2 in place, but have run into two issues. First, the prerequisite info: This is on Solaris 10 x86, and I'm running the eventlog-0.2.9 component. It has all compiled cleanly. I'm using the same script to initialize syslog-ng as I was using before. In the conf file, added in some value() parameters on my filters, in accordance with the 3.0 "warnings", and added in the @version directive, but other than that, it's essentially the same (reasonably simple) conf file I was using with my 2.0.9 version of the code. For reference, I've appended my conf file at the end of the email. "Generally" syslog-ng, seems to start-up fine, and does feed the output files that I've indicated in the conf file. If I didn't "look too closely", I would have thought it was all fine, but... My problems: As part of my test environment, I'm forwarding data from another (production 2.0.9) server, to my test server, so that I can see how things are working. Here are some (truncated) messages, showing both the original message (as it looks in the output file on the 2.0.9 server), and the second message being what gets written to the file by the 3.0.2 component: Original: May 29 09:15:05 10.132.240.25 id=firewall sn=0006B10B2E88 <truncated> Received and written by 3.0.2: May 29 09:15:05 10.132.240.25 id=firewall: sn=0006B10B2E88 <truncated> So the "bug" (I assume it to be a bug) is that "colon" that is now sitting just behind the word "firewall". When my test server is running a 2.0.9 daemon, these two lines match, but the 3.0.2 daemon has inserted that colon. I can only assume that this is tied to a bug in the filtering activity, because if you look at my conf file, you will see these lines: filter SonicWallNoise { match("id=firewall" value("MSG")) and filter(SonicWallMsgs); }; Bottom line, I am doing a filter that happens to "match on" that exact "firewall" string, and it seems like more than just coincidence that this is the exact location where that unwanted colon now appears. It seems like that filtering process has somehow injected that character into the output. (Maybe I'm just missing some "new functionality or feature" in the 3.0 docs, that would intentionally cause this?) Anyway... I'm obviously looking for any input as to why this is different, and/or how to fix it (or even just work-around the problem until it is fixed). And secondly, as my subject line implies, when I run my "/etc/init.d/syslog-ng start" scripting (the same one I've always used), I find, right away, that I now have two instances of syslog-ng running. I saw a bug report from back in 2006, mentioning a similar behavior that was related to (I think) syslog-ng restarting itself, but that was supposedly fixed. I'm ass-u-me-ing that something is now causing the daemon to restart itself, but again, when I run my 2.0.9 code on place of the 3.0.2 code, I get just the one copy running. Again, I'm looking for any guidance as to something that might be driving that behavior (and as I said, maybe it's just something stupid on my part, because I'm missing something "new" in the 3.0 behavior). Thanks, in advance, for any help and/or guidance as to what these issues might be. I'm glad to provide further information if it will help. Marvin Nipper syslog-ng.conf: @version:3.0 options { dir_perm(0755); perm(0600); chain_hostnames(no); keep_hostname(yes); log_fifo_size(1000); dns_cache_size(5000); dns_cache_expire(86400); dns_cache_expire_failed(86400); }; source any_udp { udp(); }; source any_tcp { tcp(port(601)); }; destination SEC {pipe("/tmp/sec"); }; destination routers_log { file("/var/adm/log/routers.log" create_dirs(yes)); }; destination ravlin_log { file("/var/adm/log/ravlin.log" create_dirs(yes)); }; destination windows_log { file("/var/adm/log/windows.log" create_dirs(yes)); }; destination workstation_log { file("/var/adm/log/workstation.log" create_dirs(yes)); }; destination catch-all_log { file("/var/adm/log/catch-all.log" create_dirs(yes)); }; destination test { file("/var/adm/log/test.log" create_dirs(yes)); }; destination dev_null {}; filter f_4 { facility(syslog) and level(info..emerg); }; log { source(any_udp); filter(f_4); destination(windows_log); flags(final); }; filter f_1 { facility(local5) and level(debug..emerg); }; log { source(any_udp); filter(f_1); destination(routers_log); flags(final); }; filter SonicWallNoise { match("id=firewall" value("MSG")) and filter(SonicWallMsgs); }; filter SonicWallMsgs { match("m=97" value("MSG")) or match("m=98" value("MSG")) or match("m=537" value("MSG")); }; log { source(any_udp); filter(SonicWallNoise); destination(dev_null); flags(final); }; filter f_3 { facility(local0) and level(debug..emerg); }; log { source(any_udp); filter(f_3); destination(ravlin_log); flags(final); }; log { source(any_tcp); destination(workstation_log); flags(final); }; log { source(any_udp); destination(catch-all_log); flags(final); }; This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

1 0

RFC 5424 compliant syslog library?
by Daniel Mahu 28 May '09

28 May '09

Hello, I'm looking for a linux library that would implement the syslog functions using the new standard format in RFC 5424. Of all the new things in the standard, I'm particularly interested in the timestamp resolution - I need to log timestamps with fractions of a second. Already tried searching google to no avail (only found a NetBSD libc patch, but I need linux). Since syslog-ng can now parse these messages, I thought maybe someone here may have heard about it if such a library existed. thanks, Daniel

2 1

How to name dir or logfile with the time by the message timestamp itself
by Roc Zhou 22 May '09

22 May '09

Hi, I'm now have a syslog-ng to collect the syslog of 2 sites, 1 is in China and another is in US, the host *vip-syslog* is in China, and US send the the messages to *vip-syslog* through a special tunnel, (for reliability, I can build a proxy with disk buffering to relay the syslog messages). Now * vip-syslog* can recevied the messages and works well, but I still suffer a problem. This is the syslog-ng.conf: #cat /etc/syslog-ng/syslog-ng.conf # # configuration file for syslog-ng, customized for remote logging # source s_internal { internal(); }; destination d_syslognglog { file("/var/log/syslog-ng.log"); }; log { source(s_internal); destination(d_syslognglog); }; options { ts_format(rfc3164); chain_hostnames(no); use_dns(yes); dns_cache(yes); dns_cache_hosts(/etc/syslog-ng/hosts); use_fqdn(no); perm(0644); dir_perm(0700); flush_lines(128); flush_timeout(500); log_msg_size(16384); use_time_recvd(no); # recv_time_zone(-07:00); # send_time_zone(-07:00); }; # --------------------------------------------------------------------- # Local sources, filters and destinations are commented out # If you want to replace sysklogd simply uncomment the following # parts and disable sysklogd # # Local sources # source s_local { unix-dgram("/dev/log"); file("/proc/kmsg" log_prefix("kernel:")); }; # # --------------------------------------------------------------------- # Local filters # filter f_messages { level(info..emerg); }; filter f_secure { facility(authpriv); }; filter f_mail { facility(mail); }; filter f_cron { facility(cron); }; filter f_emerg { level(emerg); }; filter f_spooler { level(crit..emerg) and facility(uucp, news); }; filter f_local7 { facility(local7); }; # # --------------------------------------------------------------------- # Local destinations # destination d_messages { file("/var/log/messages"); }; destination d_secure { file("/var/log/secure"); }; destination d_maillog { file("/var/log/maillog"); }; destination d_cron { file("/var/log/cron"); }; destination d_console { usertty("root"); }; destination d_spooler { file("/var/log/spooler"); }; destination d_bootlog { file("/var/log/boot.log"); }; # # --------------------------------------------------------------------- # Local logs - order DOES matter ! # log { source(s_local); filter(f_emerg); destination(d_console); }; log { source(s_local); filter(f_secure); destination(d_secure); flags(final); }; log { source(s_local); filter(f_mail); destination(d_maillog); flags(final); }; log { source(s_local); filter(f_cron); destination(d_cron); flags(final); }; log { source(s_local); filter(f_spooler); destination(d_spooler); }; log { source(s_local); filter(f_local7); destination(d_bootlog); }; log { source(s_local); filter(f_messages); destination(d_messages); }; # --------------------------------------------------------------------- # Remote logging # # Remote sources # source s_remote { tcp(ip(0.0.0.0) port(514)); udp(ip(0.0.0.0) port(514)); }; # --------------------------------------------------------------------- # Remote destinations # template t_message { # template("$STAMP $HOST $PROGRAM[$PID] $MSG\n"); template_escape(no); template("$S_STAMP $HOST $MSG\n"); template_escape(no); }; template t_apache { template("$MSGONLY\n"); template_escape(no); }; # --------------------------------- # Global site with US timezone: destination d_en_alarm { file("/var/log/syslog-ng/$S_YEAR-$S_MONTH-$S_DAY/$HOST/alarm" owner("root") group("root") perm(0640) dir_perm(0750) create_dirs(yes) time_zone(-07:00) template(t_message)); # program("sec.pl") }; destination d_en_necessary { file("/var/log/syslog-ng/$S_YEAR-$S_MONTH-$S_DAY/$HOST/necessary" owner("root") group("root") perm(0640) dir_perm(0750) create_dirs(yes) time_zone(-07:00) template(t_message)); }; destination d_en_cookie_log { file("/var/log/syslog-ng/$S_YEAR-$S_MONTH-$S_DAY/$HOST/cookie_log" owner("root") group("root") perm(0640) dir_perm(0750) create_dirs(yes) time_zone(-07:00) template(t_apache)); }; # --------------------------------- # China site with Beijing timezone: destination d_cn_alarm { file("/var/log/syslog-ng/$S_YEAR-$S_MONTH-$S_DAY/$HOST/alarm" owner("root") group("root") perm(0640) dir_perm(0750) create_dirs(yes) template(t_message)); }; destination d_cn_necessary { file("/var/log/syslog-ng/$S_YEAR-$S_MONTH-$S_DAY/$HOST/necessary" owner("root") group("root") perm(0640) dir_perm(0750) create_dirs(yes) template(t_message)); }; destination d_cn_cookie_log { file("/var/log/syslog-ng/$S_YEAR-$S_MONTH-$S_DAY/$HOST/cookie_log" owner("root") group("root") perm(0640) dir_perm(0750) create_dirs(yes) template(t_apache)); }; destination d_null { file("/dev/null"); }; destination d_sec { program("/etc/syslog-ng/sec.pl -input=\"-\" -conf=/etc/syslog-ng/sec.conf"); }; # --------------------------------------------------------------------- # Remote filters # filter f_en_host { host("(^hz|^us|^hk)_"); }; # # Used by data warehouse(dw): filter f_cn_local1 { facility(local1); }; filter f_en_local2 { facility(local2); }; filter f_en_dw { filter(f_en_host) and filter(f_en_local2); }; filter f_cn_dw { not filter(f_en_host) and filter(f_cn_local1); }; # Used by system admins and security admins: filter f_en_sys { filter(f_en_host) and not filter(f_en_local2); }; filter f_cn_sys { not filter(f_en_host) and not filter(f_cn_local1); }; filter f_necessary { match("necessary") or match("important"); ...... }; filter f_alarm { match("(?i)(error|fail)"); ...... }; ...... # --------------------------------------------------------------------- # Remote log path # log { source(s_remote); filter(f_en_sys); filter(f_necessary); destination(d_en_necessary); }; log { source(s_remote); filter(f_cn_sys); filter(f_necessary); destination(d_cn_necessary); }; log { source(s_remote); filter(f_en_sys); filter(f_alarm); destination(d_en_alarm); }; # log { source(s_remote); filter(f_en_sys); filter(f_necessary); destination(d_en_alarm); }; # --> This will not generate duplicated message! log { source(s_remote); filter(f_cn_sys); filter(f_alarm); destination(d_cn_alarm); }; log { source(s_remote); filter(f_cn_dw); destination(d_cn_cookie_log); }; log { source(s_remote); filter(f_en_dw); destination(d_en_cookie_log); }; log { source(s_remote); destination(d_null); }; NOW as you can see, I want the message to be logged to the dir: /var/log/syslog-ng/$S_YEAR-$S_MONTH-$S_DAY/$HOST/ The logs of China is OK, but the US is not since a timezone translating must be performed. Now the message timestamp in the logfile is right, but the dir name is not, for example: #head /var/log/syslog-ng/2009-05-22/us_search63/alarm May 21 10:50:27 us_search63 kernel: isupdate[7157]: segfault at 0000000000000010 rip 000000000041adad rsp 00007fffaa3cf560 error 4 May 21 18:09:55 us_search63 snmpd[26753]: /etc/snmp/snmpd.conf: line 433: Error: WARNING: This output format is being deprecated - Please use the 'extend' directive instead May 21 18:09:55 us_search63 snmpd[26753]: /etc/snmp/snmpd.conf: line 434: Error: WARNING: This output format is being deprecated - Please use the 'extend' directive instead May 21 18:09:55 us_search63 snmpd[26753]: /etc/snmp/snmpd.conf: line 435: Error: WARNING: This output format is being deprecated - Please use the 'extend' directive instead May 21 18:09:56 us_search63 snmpd[26753]: /etc/snmp/snmpd.conf: line 436: Error: WARNING: This output format is being deprecated - Please use the 'extend' directive instead As you can see, the dir of 2009-05-22 has the records of May 21, which is not what I want. I have tried the MACROS of $DAY, $S_DAY and $R_DAY, seems can not solve this problem. Any suggestions? Thanks.

1 0