January 2006 - syslog-ng - lists.balabit.hu

Escaping problem using pipe to send to databases
by strerror 22 Jan '06

22 Jan '06

heya, I've currently got a problem that I'd have thought other people have run into but all my googling / searching hasn't come up with anything. I'm currently sending my logs to a mysql database (but I believe the problem is generic to any db) with, what seems to be the normal configuration, eg: destination db { pipe("/var/log/mysql.pipe" template("INSERT INTO logs (host, facility, priority, level, tag, datetime, program, msg) VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n") template-escape(yes)); }; The logging all works fine in normal situations. The problem I have is that I have some unusual applications, like OpenCA (www.openca.org) that do weird logging, in the case of OpenCA, dumping xml type logs into syslog. The result of this is that it seems to completely break the pipe as mysql immediately complains that the syntax is not valid sql and the pipe effectively dies. Whats particularly frustrating is that the pipe invoking script is still up and running, so its not like I can easily write a quick script to check for the condition to restart it. I've noticed that this problem can also occur with the odd hostname for my dns server as well. In short, as far as I can see, anyone who is dumping syslogs into a db using pipe would be vulnerable to someone using "logger" to dump a nastily formatted message and then having their pipe collapse. I realise that you can send the syslog to an external program (I rewrote parts of sqlsyslogd.c before using the above syslog-ng.conf) and have it sanitise the input before injecting it into the db, but as documented, there are reasons, mainly performance, as to why you'd want to avoid that mechanism. So my question to the list, is there a nice, elegant way of avoiding badly written applications / malicious users, injecting broken looking entries and causing the pipe to die? ttia, -- strerror http://www.disciplina.net Fingerprint: B6A5 CD59 E6E7 5E71 D387 937A A4EF 4495 5865 0E8F

1 0

duplicate stats counter?
by Paul Krizak 20 Jan '06

20 Jan '06

I've been getting a constant stream of these messages in my syslog-ng log, as much as 70MB worth per day: Jan 19 13:43:39 i-dl385 syslog-ng[25668]: Duplicate stats counter; counter='file(/var/log/logs/$R_YEAR/$R_MONTH/$R_DAY/$HOST-messages.log)' Jan 19 13:43:40 i-dl385 syslog-ng[25668]: Duplicate stats counter; counter='file(/var/log/logs/$R_YEAR/$R_MONTH/$R_DAY/$HOST-messages.log)' Jan 19 13:43:40 i-dl385 syslog-ng[25668]: Duplicate stats counter; counter='file(/var/log/logs/$R_YEAR/$R_MONTH/$R_DAY/$HOST-messages.log)' What does this mean? Are stats getting collected properly? Is there a bug in the stats stuff? Turning on debugging doesn't yield any more useful information... I'm using syslog-ng 1.9.8. The relevant portion of syslog-ng.conf looks like this: source s_external_syslog { udp(port(514)); }; source s_internal_syslog { unix-dgram("/dev/log"); file("/proc/kmsg" log_prefix("kernel:")); }; template t_human_readable { template("$R_FULLDATE $HOST $FACILITY.$PRIORITY $MESSAGE\n"); template-escape(no); }; destination d_host_messages { file( "/var/log/logs/$R_YEAR/$R_MONTH/$R_DAY/$HOST-messages.log" template(t_human_readable) create_dirs(yes) owner(root) group(root) perm(0644) dir_perm(0755) ); }; destination d_big_messages { file ( "/var/log/logs/messages.log" template(t_human_readable) owner(root) group(root) perm(0644) ); }; log { source(s_external_syslog); source(s_internal_syslog); destination(d_big_messages); destination(d_host_messages); }; -- Paul Krizak 5900 E. Ben White Blvd. MS 625 Advanced Micro Devices Austin, TX 78741 Linux/Unix Systems Engineering Phone: (512) 602-8775 Microprocessor Solutions Sector

2 1

Duplicates
by Ed Lucero 19 Jan '06

19 Jan '06

I am getting duplicate log entries for mail.notice and mail.warning. I am also not quite clear on the garbage collecting parameters. Thanks for your help. :) Ed syslog-ng 1.6.9 The initital syslog-ng.conf file was created by the FreeBSD Port. I have modified the entries for mail to isolate mail logging. I am logging for 13 FreeBSD servers and 1 Pix Firewall The daily throughput is 7.3 million log entries per day. This will increase by 5 in the next few months as we migrate from the old mail servers to the new mail servers Below is the current configuration file. Here are the mail entries. destination maillog { file("/var/log/$HOST/maillog"); }; filter f_mail { facility(mail); }; filter f_m_info { level(info,notice,warning,debug); }; log { source(src); filter(f_mail); filter(f_m_info); destination(maillog);}; Hardware configuration ---------------------- DELL Poweredge 2850 Dual 3.0GHZ Xeon Processors 2G Memory OS on Mirrored 73G 10K Ultra320 SCSI SYSLOG on Mirrored 300G 10K Ultra320 SCSI Running FreeBSD 6 using a SMP kernel syslog-ng.conf -------------- # # This sample configuration file is essentially equilivent to the stock # FreeBSD /etc/syslog.conf file. # # # options # options { time_reopen (1); log_fifo_size (999999); use_fqdn(yes); keep_hostname (yes); create_dirs(yes); sync(0); use_dns (yes); dns_cache(yes); dns_cache_size(1000); dns_cache_expire(604800); gc_idle_threshold(600); gc_busy_threshold(500000); }; # # sources # source src { unix-dgram("/var/run/log"); unix-dgram("/var/run/logpriv" perm(0600)); tcp(); udp(); internal(); file("/dev/klog"); }; # # destinations # destination clamav { file("/var/log/$HOST/clamav"); }; destination messages { file("/var/log/$HOST/messages"); }; destination security { file("/var/log/$HOST/security"); }; destination authlog { file("/var/log/$HOST/auth.log"); }; destination maillog { file("/var/log/$HOST/maillog"); }; destination firewall { file("/var/log/firewall/firewall"); }; destination lpd-errs { file("/var/log/$HOST/lpd-errs"); }; destination xferlog { file("/var/log/$HOST/xferlog"); }; destination cron { file("/var/log/$HOST/cron"); }; destination debuglog { file("/var/log/$HOST/debug.log"); }; destination consolelog { file("/var/log/console.log"); }; destination all { file("/var/log/all.log"); }; destination newscrit { file("/var/log/news.crit"); }; destination newserr { file("/var/log/news.err"); }; destination newsnotice { file("/var/log/news.notice"); }; destination slip { file("/var/log/slip.log"); }; destination ppp { file("/var/log/ppp.log"); }; destination console { file("/dev/console"); }; destination allusers { usertty("*"); }; #destination loghost { udp("loghost" port(514)); }; # # log facility filters # filter f_auth { facility(auth); }; filter f_authpriv { facility(authpriv); }; filter f_not_authpriv { not facility(authpriv); }; filter f_console { facility(console); }; filter f_cron { facility(cron); }; filter f_daemon { facility(daemon); }; filter f_ftp { facility(ftp); }; filter f_kern { facility(kern); }; filter f_lpr { facility(lpr); }; filter f_mail { facility(mail); }; filter f_not_mail { not facility(mail); }; filter f_news { facility(news); }; filter f_security { facility(security); }; filter f_user { facility(user); }; filter f_uucp { facility(uucp); }; filter f_local0 { facility(local0); }; filter f_local1 { facility(local1); }; filter f_local2 { facility(local2); }; filter f_local3 { facility(local3); }; filter f_local4 { facility(local4); }; filter f_local5 { facility(local5); }; filter f_local6 { facility(local6); }; filter f_local7 { facility(local7); }; # # log level filters # filter f_emerg { level(emerg); }; filter f_alert { level(alert..emerg); }; filter f_crit { level(crit..emerg); }; filter f_err { level(err..emerg); }; filter f_warning { level(warning..emerg); }; filter f_notice { level(notice..emerg); }; filter f_info { level(info,debug); }; filter f_debug { level(debug..emerg); }; filter f_is_debug { level(debug); }; filter f_m_notice { level(notice); }; filter f_m_info { level(info,notice,warning,debug); }; # # program filters # filter f_ppp { program("ppp"); }; filter f_slip { program("startslip"); }; # # *.err;kern.warning;auth.notice;mail.crit /dev/console # log { source(src); filter(f_err); destination(console); }; log { source(src); filter(f_kern); filter(f_warning); destination(console); }; log { source(src); filter(f_auth); filter(f_notice); destination(console); }; log { source(src); filter(f_mail); filter(f_crit); destination(console); }; # # *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err /var/log/messages # log { source(src); filter(f_notice); filter(f_not_mail); destination(messages); }; log { source(src); filter(f_kern); filter(f_debug); destination(messages); }; log { source(src); filter(f_lpr); filter(f_info); destination(messages); }; log { source(src); filter(f_news); filter(f_err); destination(messages); }; # # clamav # log { source(src); filter(f_local6); destination(clamav); }; # # security.* /var/log/security # log { source(src); filter(f_security); destination(security); }; # # auth.info;authpriv.info /var/log/auth.log log { source(src); filter(f_auth); filter(f_info); destination(authlog); }; log { source(src); filter(f_authpriv); filter(f_info); destination(authlog); }; # # mail.info /var/log/maillog # log { source(src); filter(f_mail); filter(f_m_info); destination(maillog); }; # # lpr.info /var/log/lpd-errs # log { source(src); filter(f_lpr); filter(f_info); destination(lpd-errs); }; # # ftp.info /var/log/xferlog # log { source(src); filter(f_ftp); filter(f_info); destination(xferlog); }; # # cron.* /var/log/cron # log { source(src); filter(f_cron); destination(cron); }; # # *.=debug /var/log/debug.log # log { source(src); filter(f_is_debug); destination(debuglog); }; # # *.emerg * # #log { source(src); filter(f_emerg); destination(allusers); }; # # uncomment this to log all writes to /dev/console to /var/log/console.log # console.info /var/log/console.log # #log { source(src); filter(f_console); filter(f_info); destination(consolelog); }; # # uncomment this to enable logging of all log messages to /var/log/all.log # touch /var/log/all.log and chmod it to mode 600 before it will work # *.* /var/log/all.log # #log { source(src); destination(all); }; # # uncomment this to enable logging to a remote loghost named loghost # *.* @loghost # #log { source(src); destination(loghost); }; # # uncomment these if you're running inn # news.crit /var/log/news/news.crit # news.err /var/log/news/news.err # news.notice /var/log/news/news.notice # #log { source(src); filter(f_news); filter(f_crit); destination(newscrit); }; #log { source(src); filter(f_news); filter(f_err); destination(newserr); }; #log { source(src); filter(f_news); filter(f_notice); destination(newsnotice); }; # # !startslip # *.* /var/log/slip.log # log { source(src); filter(f_slip); destination(slip); }; # # !ppp # *.* /var/log/ppp.log # log { source(src); filter(f_ppp); destination(ppp); };

1 0

error messages in start syslog-ng
by Gustavo Mendes de Carvalho 19 Jan '06

19 Jan '06

Hi Jesse, Tks for the tip, but now, do you have any idea about some good value to define in both parameters ? I defined CONSOLE_LOG_LEVEL=6 (INFO level) and KERNEL_RINGBUF_SIZE=16392 is the default value. I'm concerned about second one. Is it too high or too low ? Tks in advance Gustavo

1 0

error messages in start syslog-ng
by Gustavo Mendes de Carvalho 18 Jan '06

18 Jan '06

Hi All, I made some specific configurations in my syslog-ng.conf file, and when I tried to restart syslog-ng services, I got those messages: CONSOLE_LOG_LEVEL is of unaccepted value. KERNEL_RINGBUF_SIZE is of unaccepted value. Does anybody knows or has some tip about this message ? Tks in advance Gustavo Mendes de Carvalho gmcarvalho(a)gmail.com

2 1

Reference doc for 1.6 ?
by Per Jessen 15 Jan '06

15 Jan '06

http://www.balabit.com/products/syslog_ng/reference-1.6/syslog-ng.html reference-1.6/syslog-ng.html.tar.gz This URL produces a 404 ? /Per Jessen, Zürich -- http://www.spamchek.com/ - managed anti-spam and anti-virus solution. Let us analyse your spam- and virus-threat - up to 2 months for free.

1 1

log to mysql and to logfile simultaneously
by Natxo Asenjo 15 Jan '06

15 Jan '06

hi all, After stfw and not being able to find answer to that question, I decide to ask it here: is it possible to both log to a database (mysql in my case) and to the normal logfiles at the same time? Thanks in advance for your reply, Natxo Asenjo

2 1

mysql + logfiles
by natxete＠asenjo.nl 13 Jan '06

13 Jan '06

hi, Is it possible to log both to mysql and to syslog files? Thanks in advance, -- Groeten, J.I.Asenjo

2 1

defining stats() option causes syntax error when starting syslog-ng
by def 13 Jan '06

13 Jan '06

My syslog is filled with STATS lines every 2 minutes so I decided to delay the stats every 1 hour, or 3600 seconds. I added this to my options line but it causes a syntax error in the conf. The problem also occurred in the previous 1.9 build. Slackware 10.2 # uname -a && syslog-ng -V Linux santo 2.6.14.3 #1 Wed Dec 7 22:59:32 EST 2005 i686 unknown unknown GNU/Linux syslog-ng 1.9.8 # cat /usr/local/etc/syslog-ng.conf | grep options options { stats(3600); long_hostnames(off); sync(0); }; I stop and start syslog-ng. rc.syslog is a script i found but it also happens when just executing syslog-ng by itself. root@santo:/etc/rc.d# ./rc.syslog start syslog-ng service starting. syntax error at 12 root@santo:/etc/rc.d# Now I remove the stats() option # cat /usr/local/etc/syslog-ng.conf | grep options options { long_hostnames(off); sync(0); }; root@santo:/etc/rc.d# ./rc.syslog start syslog-ng service starting. root@santo:/etc/rc.d# syslog-ng starts successfully Thanks for any help you may be able to provide -Sean

3 2

logging of auth-priv
by Peter Koinange 12 Jan '06

12 Jan '06

I have just installed syslog-ng and am logging to a mysql database. I am trying to implement a centralized syslog system. Everything is working well except for auth-priv facility. I cannot seem to get to them sent to the centralized database. When l run test with logger on the remote computer get the following error logger: unknown facility name: auth-priv. however syslog on the remove computer has the entry # The authpriv file has restricted access. authpriv.* /var/log/secure and is logging to the secure file with no problems when I add an entry for the remote compter to send the info authpriv.* @host.me.com the logs do not show up. Can someone advise me on what to I can Peter Koinange

3 5