January 2010 - syslog-ng - lists.balabit.hu

slngctl: syslog-ng debug, trace and statistics utility
by ILLES, Marton 14 Jan '10

14 Jan '10

Hi, PZolee wrote a quite lengthy blog post on troubleshooting and debuging syslog-ng [1] back in December 2009. It popped into my mind that running syslog-ng with debug/trace/verbose option enabled is very handy, but sometimes you do not want to stop syslog-ng just to restart it in a more verbose mode. My idea was to enable syslog-ng to real-time modify these verbosity settings through the control socket which is currently only used for exporting statistical information. I extended the control socket communication to have the ability of modify these logging settings. To make life easier I have created a small utility called slngctl which can be used to communicate with the running syslog-ng process. Using slngctl it also easy to query the statistics collected by syslog-ng. Also on longer term I think other useful small features could be added the slngctl. slngctl commands: marci@octane:$ slngctl Possible commands are: stats Dump syslog-ng statistics verbose Enable/query verbose messages debug Enable/query debug messages trace Enable/query trace messages Querying statistics: marci@octane:$ slngctl stats destination;d_out;;a;processed;0 global;payload_reallocs;;a;processed;0 source;s_tcp;;a;processed;0 global;msg_clones;;a;processed;0 global;sdata_updates;;a;processed;0 center;;received;a;processed;0 center;;queued;a;processed;0 Getting current verbose settings: marci@octane:$ slngctl verbose VERBOSE=1 Enabling trace run-time: marci@octane:$ slngctl trace TRACE=0 marci@octane:$ slngctl trace -s 1 marci@octane:$ slngctl trace TRACE=1 As always you can find the source code in my 3.1 git repository and all feedbacks are very welcome. best, Marton PS: I know slngctl might not be the best name, so I offer a bier to anyone suggesting a better name when visiting us in Budapest, Hungary. :) [1] http://pzolee.blogs.balabit.com/2009/12/troubleshooting-and-debugging-syslo… -- Key fingerprint = F78C 25CA 5F88 6FAF EA21 779D 3279 9F9E 1155 670D

3 3

requirement for @version directive in 3.x config file
by Paul B. Henson 14 Jan '10

14 Jan '10

Our syslog-ng configuration file is generated automatically, from various different bits and pieces based on what roles and services a system provides. Unfortunately, our current implementation would not allow always ensuring that the "@version" directive is the first line in the generated config file. Version 3 of syslog-ng seems very unhappy if that is not the case. This seems somewhat inflexible, would it be possible to have a command line option specifying the configuration file version so it does not have to be on the very first line of the actual file? Any other suggestions (short of modifying our internal processes :) ) to get verison 3 to be happy with a configuration file without this specific directive as the very first line? Thanks much... -- Paul B. Henson | (909) 979-6361 | http://www.csupomona.edu/~henson/ Operating Systems and Network Analyst | henson(a)csupomona.edu California State Polytechnic University | Pomona CA 91768

3 3

Leaking objects in size-4096 cache in slabinfo
by Doug Warner 12 Jan '10

12 Jan '10

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'm running syslog-ng 3.0.4 OSE on Gentoo and have a central syslog host that accepts about 3Mbps of log traffic via a TCP pipe. We've recently been having oom-killer problems and I finally tracked it down to the fact that we seem to be leaking objects in the size-4096 cache. Now, slabinfo is new to me so I could be interpreting this wrong, but restarting syslog-ng definite reclaims this cache. What data can I provide to help track down this problem? I noticed that there seems to be a similar error in 3.1 beta, reported here: http://thread.gmane.org/gmane.comp.syslog-ng/8544 - -Doug -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) iD8DBQFLJ8FqJV36su0A0xIRAizGAKDbVEpHRMhrRgAogUuAh0nCKz9xFwCgtz17 6EN8aU3MxU7bL8Sr4xRx6tI= =1LQg -----END PGP SIGNATURE-----

3 13

Separating Remote Logs
by Nate Hausrath 12 Jan '10

12 Jan '10

Hi everyone, I've set up a central log server using syslog-ng that receives logs from many different locations on the network. I want to break these logs up into different files so I can using another program to parse and interpret each one individually. So for example, I want my Windows logs to go to /var/log/remote/windows.log, which my ASA logs go to /var/log/remote/asa.log. Anything that doesn't fit into a group should go to /var/log/remote/other.log. I'm trying to figure out the best way to do this. I was going to create a custom xml file for the db-parser, but I can't seem to get it to work the way I want. Here is my db-parser XML file so far: <?xml version='1.0' encoding='UTF-8'?> <patterndb version='2' pub_date='2009-12-07'> <ruleset name='capcxml' id='1923-ab2b'> <pattern/> <rules> <rule provider='capc' id='1' class='system'> <description>Detects ASA logs</description> <patterns> <pattern>%ASA</pattern> </patterns> </rule> <rule provider='capc' id='2' class='system'> <description>Detects Windows logs from Snare</description> <patterns> <pattern>MSWinEventLog</pattern> </patterns> </rule> </rules> </ruleset> </patterndb> So my question is, how can I specify a filter in my syslog-ng.conf file that will allow me to separate logs from these two different rules? Along with that, how can I separate everything that doesn't match? For instance, I could have something like this: filter f_class_asa { match("1" value(".classifier_rule_id") type("string") ); }; filter f_class_windows { match("2" value(".classifier_rule_id") type("string") ); }; filter f_class_other { not match("1" value(".classifier_rule_id") type("string")) and not match("2" value(".classifier_rule_id") type("string")); }; log { source(s_remote); parser(p_capc); filter(f_class_asa); destination(df_asa); }; log { source(s_remote); parser(p_capc); filter(f_class_windows); destination(df_windows); }; log { source(s_remote); parser(p_capc); filter(f_class_other); destination(df_other); }; But this doesn't seem to work. Everything gets placed in the df_asa file. Am I approaching this the wrong way? Should I not use db-parser for this task? Thanks for any help! Nate

3 7

Introducing pdbtool patternize
by Peter Gyongyosi 11 Jan '10

11 Jan '10

Hi, As Márton has already written about it [1], lots of guys here at BalaBit spent a lot of time last year creating a pattern database for some 200+ often-used applications. Just like every manual process, this was a tedious task which begged to be automated. Of course it cannot be fully automated as no algorithm can replace an actual person understanding the structure of the logs a piece of software produces (or even looking into the source code to see how they're generated), but still, a tool that can detect similar messages in a log database and generate a pattern database for it would've been real handy. Thus have pdbtool patternize been created. The tool I've written is a part of the pdbtool utility and can be used to generate a pattern database from a bunch of unknown messages. It uses the algorithm developed by Risto Vaarandi for SLCT [2], the main idea of which is using a data clustering technique to find similar log messages and replacing the differing parts with wildcard characters. In our case, the wildcards are @ESTRING:: @ parsers, otherwise, the solution is pretty much the same. It's far from being perfect (the code could be optimized at some places, it needs to load everything into memory so the size of the parseable log file is limited to the RAM in your machine as swapping slows things down really bad, and the log messages are split to words only at spaces, which makes it unable to detect "username=@QSTRING::'@"-type patterns), but it has already produced some impressive results on the test databases I've tried it with. It managed to categorize 95-98% of ~2M lines of logs into 40-50 patterns in a reasonable time and the patterns themselves were pretty readable as well. There're two options that can be set for the patternization. The first one ("-S") is the support value and accepts a floating point number: the percentage of the lines that have to match a pattern candidate to include it in the resulting pattern database. It allows tuning the trade-off between the number of patterns (too much would be hard to maintain) and the coverage these patterns produce. My first tests show that the optimal support value for most log types are around 2.5-5%, but you should check this with your own logs. The other option ("-o") enables a different operation mode. In this case, after a clustering step is completed, the tool does not exit printing out the generated patterns, rather starts yet another clustering on the messages that are not covered with the patterns generated so far. It keeps on doing this while new patterns can be generated. This way, much better coverage can be achieved while still having a low number of patterns -- the larger groups are detected early, but the small groups aren't left out either. Based on my tests, a bit larger support value, around 10-15% is optimal for this operation mode. The code is available in my public syslog-ng repository [3] and I'm more than eager for some feedback. It still needs a larger review from someone more experienced in syslog-ng internals and leaks a little memory here&there but in overall it's mature enough to play around with even with large log databases. greets, Peter [1] http://marci.blogs.balabit.com/2009/12/pattern-database-first-snapshot.html [2] http://ristov.users.sourceforge.net/slct/] [3] http://git.balabit.hu/?p=gyp/syslog-ng-3.1.git;a=summary

1 0

SQL destination function issue
by Christopher O'Brien 07 Jan '10

07 Jan '10

Hi, I am trying to log messages from a Cisco ASA into a MySQL database through syslog-ng 3.0.4. I have the whole thing working except I cannot import data if I want to use a function. Here is an example: Here is the log message that is coming into the syslog server from the ASA device: Dec 04 2009 10:22:23: %ASA-5-305012: Teardown dynamic TCP translation from inside:10.6.42.166/2617 to outside:10.16.19.17/61631 duration 0:01:00 Here is what is logged by syslog-ng: Running SQL query; query='INSERT INTO stop (s_time, e_time, proto, in_ip, in_port, out_ip, out_port) VALUES (\'DATE_SUB(\"2009-12-04 10:22:23\", INTERVAL \"0:01:00\" HOUR_SECOND)\', \'2009-12-04 10:22:23\', \'TCP\', \'10.6.42.166\', \'2617\', \'10.16.19.17\', \'61631\')' Here is what is recorded in the database: +-----+---------------------+---------------------+-------+-------------+---------+---------------+----------+ | idx | s_time | e_time | proto | in_ip | in_port | out_ip | out_port | +-----+---------------------+---------------------+-------+-------------+---------+---------------+----------+ | 166 | 0000-00-00 00:00:00 | 2009-12-04 10:22:23 | TCP | 10.6.42.166 | 2617 | 10.16.19.17 | 61631 | Every field is recorded properly aside from the one that I am trying to use a function in. The purpose of the function is to record the start time of a translation. I get the end time from the syslog timestamp and the duration from the syslog message itself. I am trying to use a builtin MySQL function to record the start time in s_time. I think the problme is syslog-ng is putting single quotes around the whole function so it is being treated like a string literal by MySQL instead of variable data. Is that right? Is there a way to remedy this? Here is my syslog-ng.conf configuration if it helps: @version: 3.0 # $Header: /var/cvsroot/gentoo-x86/app-admin/syslog-ng/files/syslog-ng.conf.gentoo,v 1.7 2007/08/02 04:52:18 mr_bones_ Exp $ # # Syslog-ng default configuration file for Gentoo Linux # contributed by Michael Sterrett options { chain_hostnames(no); # The default action of syslog-ng 1.6.0 is to log a STATS line # to the file every 10 minutes. That's pretty ugly after a while. # Change it to every 12 hours so you get a nice daily update of # how many messages syslog-ng missed (0). stats_freq(43200); }; source src { unix-stream("/dev/log" max-connections(256)); internal(); file("/proc/kmsg"); }; source s_xlate { udp(ip(0.0.0.0) port(514)); }; filter f_xlate_start { host(10.16.19.33) and program("%ASA-5-305011") ; }; filter f_xlate_stop { host(10.16.19.33) and program("%ASA-5-305012") ; }; parser p_xlate { csv-parser(columns("LOG.STATUS", "LOG.TYPE", "LOG.PROTOCOL", "LOG.XLATE", "LOG.FROM","LOG.INSIDE", "LOG.IN_IP", "LOG.IN_PORT", "LOG.TO","LOG.OUTSIDE", "LOG.OUT_IP", "LOG.OUT_PORT", "LOG.DURATION","LOG.HR", "LOG.MIN", "LOG.SEC") delimiters("/: ") flags(escape-none) template("${MSGONLY}")); }; destination messages { file("/var/log/messages"); }; destination d_xlate_start { sql(type(mysql) username("user") password("pass") database("nat") table("start") columns("s_time DATETIME", "proto VARCHAR(4)", "in_ip VARCHAR(15)", "in_port VARCHAR(5)", "out_ip VARCHAR(15)", "out_port VARCHAR(5)") values("$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC", "${LOG.PROTOCOL}", "${LOG.IN_IP}", "${LOG.IN_PORT}", "${LOG.OUT_IP}", "${LOG.OUT_PORT}")); }; destination d_xlate_stop { sql(type(mysql) username("user") password("pass") database("nat") table("stop") columns("s_time DATETIME", "e_time DATETIME", "proto VARCHAR(4)", "in_ip VARCHAR(15)", "in_port VARCHAR(5)", "out_ip VARCHAR(15)", "out_port VARCHAR(5)") values("DATE_SUB(\"$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC\", INTERVAL \"${LOG.HR}:${LOG.MIN}:${LOG.SEC}\" HOUR_SECOND)", "$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC", "${LOG.PROTOCOL}", "${LOG.IN_IP}", "${LOG.IN_PORT}", "${LOG.OUT_IP}", "${LOG.OUT_PORT}")); }; destination console_all { file("/dev/tty12"); }; log { source(s_xlate); filter(f_xlate_start); parser(p_xlate); destination(d_xlate_start); flags(final); }; log { source(s_xlate); filter(f_xlate_stop); parser(p_xlate); destination(d_xlate_stop); flags(final); }; log { source(src); destination(messages); }; log { source(src); destination(console_all); };

2 2

Syslog Blocked
by Tokie 07 Jan '10

07 Jan '10

Hi For All, i have a strange problem. On my server syslog (on Debian Etch) I installed syslog-ng and some times syslog-ng don't works!!! The programs is in execution, but don't receive the log in destination files; Logrotate makes the new file (ex. /var/log/syslog/new) and compress the last with gz; this works!! But the new file remains empty!! If I restart the syslog-ng with /etc/init.d/syslog-ng restart(like below at Jan 5 17:56:34) , it begins to write on new file!! This is my /var/log/messages > Jan 4 06:25:21 s_all(a)sys-server.tokie syslog-ng[2690]: SIGHUP > received, reloading configuration; > Jan 5 06:25:22 s_all(a)sys-server.tokie syslog-ng[2690]: SIGHUP > received, reloading configuration; > Jan 5 17:56:34 s_all(a)sys-server.tokie syslog-ng[2690]: SIGTERM > received, terminating; > Jan 5 17:56:34 s_all(a)sys-server.tokie syslog-ng[2690]: syslog-ng > shutting down; version='2.0.0' > Jan 5 17:56:35 s_all(a)sys-server.tokie syslog-ng[3111]: syslog-ng > starting up; version='2.0.0' Every day in cron should be restart syslog at 06:25 I think the problem is SIGHUP, but I don't know why!! This don't happen every day, but randomly!! Somebody can help me?? Tokie

3 3

AUTO: Shanif Lakhani is out of the office (returning 01/13/2010)
by Shanif Lakhani 05 Jan '10

05 Jan '10

I am out of the office until 01/13/2010. Please contact Robert Holoweckyj and/or Narendra Prasad for any urgent matters. Thanks Note: This is an automated response to your message "syslog-ng Digest, Vol 57, Issue 1" sent on 1/5/10 6:00:02. This is the only notification you will receive while this person is away.

1 0

program_override("kernel") not working in 3.0.4
by Paul B. Henson 05 Jan '10

05 Jan '10

I'm trying to upgrade from 2.1.3 to 3.0.4 under Linux, and am having trouble getting my kernel messages tagged appropriately. My previous configuration had the following: ----- source k_src { pipe("/proc/kmsg" log_prefix("kernel: ")); }; destination iptables { file("/var/log/iptables.log"); }; filter f_iptables { match("iptables:"); }; log { source(k_src); filter(f_iptables); destination(iptables); flags(final); }; destination messages { file("/var/log/messages"); }; log { source(src); source(k_src); destination(messages); }; ----- All of my kernel messages showed up with the "kernel:" prefix, and anything coming from iptables was dropped in a separate log. Based on the documentation, I modified my configuration to the following for 3.0.4: ----- source k_src { file("/proc/kmsg" program_override("kernel")); }; destination iptables { file("/var/log/iptables.log"); }; filter f_iptables { program("^kernel$") and message("^iptables:"); }; log { source(k_src); filter(f_iptables); destination(iptables); flags(final); }; destination messages { file("/var/log/messages"); }; log { source(src); source(k_src); destination(messages); }; ----- However, my kernel messages show up with no prefix. I tried starting up in debug mode: ----- syslog-ng starting up; version='3.0.4' Incoming log entry; line='<6>usb 2-7: USB disconnect, address 8' Filter rule evaluation begins; filter_rule='f_iptables' Filter node evaluation result; filter_result='not-match' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter rule evaluation result; filter_result='not-match', filter_rule='f_iptables' Initializing destination file writer; template='/var/log/messages', filename='/var/log/messages' Incoming log entry; line='<4>iptables: IN=eth0 OUT= MAC=0:05:00:10:97:43:00:0SC147.5.2 S=2... E=8TS00 RC0C T= D363POO2' Filter rule evaluation begins; filter_rule='f_iptables' Filter node evaluation result; filter_result='not-match' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter rule evaluation result; filter_result='not-match', filter_rule='f_iptables' ----- This didn't help me. Am I doing something wrong? I didn't find any complaints of this nature on the mailing list, which leads me to suspect I'm somehow being stupid, any pointers much appreciated. Thanks... -- Paul B. Henson | (909) 979-6361 | http://www.csupomona.edu/~henson/ Operating Systems and Network Analyst | henson(a)csupomona.edu California State Polytechnic University | Pomona CA 91768

2 2