- syslog-ng - lists.balabit.hu

Separating Remote Logs
by Nate Hausrath 12 Jan '10

12 Jan '10

Hi everyone, I've set up a central log server using syslog-ng that receives logs from many different locations on the network. I want to break these logs up into different files so I can using another program to parse and interpret each one individually. So for example, I want my Windows logs to go to /var/log/remote/windows.log, which my ASA logs go to /var/log/remote/asa.log. Anything that doesn't fit into a group should go to /var/log/remote/other.log. I'm trying to figure out the best way to do this. I was going to create a custom xml file for the db-parser, but I can't seem to get it to work the way I want. Here is my db-parser XML file so far: <?xml version='1.0' encoding='UTF-8'?> <patterndb version='2' pub_date='2009-12-07'> <ruleset name='capcxml' id='1923-ab2b'> <pattern/> <rules> <rule provider='capc' id='1' class='system'> <description>Detects ASA logs</description> <patterns> <pattern>%ASA</pattern> </patterns> </rule> <rule provider='capc' id='2' class='system'> <description>Detects Windows logs from Snare</description> <patterns> <pattern>MSWinEventLog</pattern> </patterns> </rule> </rules> </ruleset> </patterndb> So my question is, how can I specify a filter in my syslog-ng.conf file that will allow me to separate logs from these two different rules? Along with that, how can I separate everything that doesn't match? For instance, I could have something like this: filter f_class_asa { match("1" value(".classifier_rule_id") type("string") ); }; filter f_class_windows { match("2" value(".classifier_rule_id") type("string") ); }; filter f_class_other { not match("1" value(".classifier_rule_id") type("string")) and not match("2" value(".classifier_rule_id") type("string")); }; log { source(s_remote); parser(p_capc); filter(f_class_asa); destination(df_asa); }; log { source(s_remote); parser(p_capc); filter(f_class_windows); destination(df_windows); }; log { source(s_remote); parser(p_capc); filter(f_class_other); destination(df_other); }; But this doesn't seem to work. Everything gets placed in the df_asa file. Am I approaching this the wrong way? Should I not use db-parser for this task? Thanks for any help! Nate

3 7

Introducing pdbtool patternize
by Peter Gyongyosi 11 Jan '10

11 Jan '10

Hi, As Márton has already written about it [1], lots of guys here at BalaBit spent a lot of time last year creating a pattern database for some 200+ often-used applications. Just like every manual process, this was a tedious task which begged to be automated. Of course it cannot be fully automated as no algorithm can replace an actual person understanding the structure of the logs a piece of software produces (or even looking into the source code to see how they're generated), but still, a tool that can detect similar messages in a log database and generate a pattern database for it would've been real handy. Thus have pdbtool patternize been created. The tool I've written is a part of the pdbtool utility and can be used to generate a pattern database from a bunch of unknown messages. It uses the algorithm developed by Risto Vaarandi for SLCT [2], the main idea of which is using a data clustering technique to find similar log messages and replacing the differing parts with wildcard characters. In our case, the wildcards are @ESTRING:: @ parsers, otherwise, the solution is pretty much the same. It's far from being perfect (the code could be optimized at some places, it needs to load everything into memory so the size of the parseable log file is limited to the RAM in your machine as swapping slows things down really bad, and the log messages are split to words only at spaces, which makes it unable to detect "username=@QSTRING::'@"-type patterns), but it has already produced some impressive results on the test databases I've tried it with. It managed to categorize 95-98% of ~2M lines of logs into 40-50 patterns in a reasonable time and the patterns themselves were pretty readable as well. There're two options that can be set for the patternization. The first one ("-S") is the support value and accepts a floating point number: the percentage of the lines that have to match a pattern candidate to include it in the resulting pattern database. It allows tuning the trade-off between the number of patterns (too much would be hard to maintain) and the coverage these patterns produce. My first tests show that the optimal support value for most log types are around 2.5-5%, but you should check this with your own logs. The other option ("-o") enables a different operation mode. In this case, after a clustering step is completed, the tool does not exit printing out the generated patterns, rather starts yet another clustering on the messages that are not covered with the patterns generated so far. It keeps on doing this while new patterns can be generated. This way, much better coverage can be achieved while still having a low number of patterns -- the larger groups are detected early, but the small groups aren't left out either. Based on my tests, a bit larger support value, around 10-15% is optimal for this operation mode. The code is available in my public syslog-ng repository [3] and I'm more than eager for some feedback. It still needs a larger review from someone more experienced in syslog-ng internals and leaks a little memory here&there but in overall it's mature enough to play around with even with large log databases. greets, Peter [1] http://marci.blogs.balabit.com/2009/12/pattern-database-first-snapshot.html [2] http://ristov.users.sourceforge.net/slct/] [3] http://git.balabit.hu/?p=gyp/syslog-ng-3.1.git;a=summary

1 0

SQL destination function issue
by Christopher O'Brien 07 Jan '10

07 Jan '10

Hi, I am trying to log messages from a Cisco ASA into a MySQL database through syslog-ng 3.0.4. I have the whole thing working except I cannot import data if I want to use a function. Here is an example: Here is the log message that is coming into the syslog server from the ASA device: Dec 04 2009 10:22:23: %ASA-5-305012: Teardown dynamic TCP translation from inside:10.6.42.166/2617 to outside:10.16.19.17/61631 duration 0:01:00 Here is what is logged by syslog-ng: Running SQL query; query='INSERT INTO stop (s_time, e_time, proto, in_ip, in_port, out_ip, out_port) VALUES (\'DATE_SUB(\"2009-12-04 10:22:23\", INTERVAL \"0:01:00\" HOUR_SECOND)\', \'2009-12-04 10:22:23\', \'TCP\', \'10.6.42.166\', \'2617\', \'10.16.19.17\', \'61631\')' Here is what is recorded in the database: +-----+---------------------+---------------------+-------+-------------+---------+---------------+----------+ | idx | s_time | e_time | proto | in_ip | in_port | out_ip | out_port | +-----+---------------------+---------------------+-------+-------------+---------+---------------+----------+ | 166 | 0000-00-00 00:00:00 | 2009-12-04 10:22:23 | TCP | 10.6.42.166 | 2617 | 10.16.19.17 | 61631 | Every field is recorded properly aside from the one that I am trying to use a function in. The purpose of the function is to record the start time of a translation. I get the end time from the syslog timestamp and the duration from the syslog message itself. I am trying to use a builtin MySQL function to record the start time in s_time. I think the problme is syslog-ng is putting single quotes around the whole function so it is being treated like a string literal by MySQL instead of variable data. Is that right? Is there a way to remedy this? Here is my syslog-ng.conf configuration if it helps: @version: 3.0 # $Header: /var/cvsroot/gentoo-x86/app-admin/syslog-ng/files/syslog-ng.conf.gentoo,v 1.7 2007/08/02 04:52:18 mr_bones_ Exp $ # # Syslog-ng default configuration file for Gentoo Linux # contributed by Michael Sterrett options { chain_hostnames(no); # The default action of syslog-ng 1.6.0 is to log a STATS line # to the file every 10 minutes. That's pretty ugly after a while. # Change it to every 12 hours so you get a nice daily update of # how many messages syslog-ng missed (0). stats_freq(43200); }; source src { unix-stream("/dev/log" max-connections(256)); internal(); file("/proc/kmsg"); }; source s_xlate { udp(ip(0.0.0.0) port(514)); }; filter f_xlate_start { host(10.16.19.33) and program("%ASA-5-305011") ; }; filter f_xlate_stop { host(10.16.19.33) and program("%ASA-5-305012") ; }; parser p_xlate { csv-parser(columns("LOG.STATUS", "LOG.TYPE", "LOG.PROTOCOL", "LOG.XLATE", "LOG.FROM","LOG.INSIDE", "LOG.IN_IP", "LOG.IN_PORT", "LOG.TO","LOG.OUTSIDE", "LOG.OUT_IP", "LOG.OUT_PORT", "LOG.DURATION","LOG.HR", "LOG.MIN", "LOG.SEC") delimiters("/: ") flags(escape-none) template("${MSGONLY}")); }; destination messages { file("/var/log/messages"); }; destination d_xlate_start { sql(type(mysql) username("user") password("pass") database("nat") table("start") columns("s_time DATETIME", "proto VARCHAR(4)", "in_ip VARCHAR(15)", "in_port VARCHAR(5)", "out_ip VARCHAR(15)", "out_port VARCHAR(5)") values("$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC", "${LOG.PROTOCOL}", "${LOG.IN_IP}", "${LOG.IN_PORT}", "${LOG.OUT_IP}", "${LOG.OUT_PORT}")); }; destination d_xlate_stop { sql(type(mysql) username("user") password("pass") database("nat") table("stop") columns("s_time DATETIME", "e_time DATETIME", "proto VARCHAR(4)", "in_ip VARCHAR(15)", "in_port VARCHAR(5)", "out_ip VARCHAR(15)", "out_port VARCHAR(5)") values("DATE_SUB(\"$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC\", INTERVAL \"${LOG.HR}:${LOG.MIN}:${LOG.SEC}\" HOUR_SECOND)", "$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC", "${LOG.PROTOCOL}", "${LOG.IN_IP}", "${LOG.IN_PORT}", "${LOG.OUT_IP}", "${LOG.OUT_PORT}")); }; destination console_all { file("/dev/tty12"); }; log { source(s_xlate); filter(f_xlate_start); parser(p_xlate); destination(d_xlate_start); flags(final); }; log { source(s_xlate); filter(f_xlate_stop); parser(p_xlate); destination(d_xlate_stop); flags(final); }; log { source(src); destination(messages); }; log { source(src); destination(console_all); };

2 2

Syslog Blocked
by Tokie 07 Jan '10

07 Jan '10

Hi For All, i have a strange problem. On my server syslog (on Debian Etch) I installed syslog-ng and some times syslog-ng don't works!!! The programs is in execution, but don't receive the log in destination files; Logrotate makes the new file (ex. /var/log/syslog/new) and compress the last with gz; this works!! But the new file remains empty!! If I restart the syslog-ng with /etc/init.d/syslog-ng restart(like below at Jan 5 17:56:34) , it begins to write on new file!! This is my /var/log/messages > Jan 4 06:25:21 s_all(a)sys-server.tokie syslog-ng[2690]: SIGHUP > received, reloading configuration; > Jan 5 06:25:22 s_all(a)sys-server.tokie syslog-ng[2690]: SIGHUP > received, reloading configuration; > Jan 5 17:56:34 s_all(a)sys-server.tokie syslog-ng[2690]: SIGTERM > received, terminating; > Jan 5 17:56:34 s_all(a)sys-server.tokie syslog-ng[2690]: syslog-ng > shutting down; version='2.0.0' > Jan 5 17:56:35 s_all(a)sys-server.tokie syslog-ng[3111]: syslog-ng > starting up; version='2.0.0' Every day in cron should be restart syslog at 06:25 I think the problem is SIGHUP, but I don't know why!! This don't happen every day, but randomly!! Somebody can help me?? Tokie

3 3

AUTO: Shanif Lakhani is out of the office (returning 01/13/2010)
by Shanif Lakhani 05 Jan '10

05 Jan '10

I am out of the office until 01/13/2010. Please contact Robert Holoweckyj and/or Narendra Prasad for any urgent matters. Thanks Note: This is an automated response to your message "syslog-ng Digest, Vol 57, Issue 1" sent on 1/5/10 6:00:02. This is the only notification you will receive while this person is away.

1 0

program_override("kernel") not working in 3.0.4
by Paul B. Henson 05 Jan '10

05 Jan '10

I'm trying to upgrade from 2.1.3 to 3.0.4 under Linux, and am having trouble getting my kernel messages tagged appropriately. My previous configuration had the following: ----- source k_src { pipe("/proc/kmsg" log_prefix("kernel: ")); }; destination iptables { file("/var/log/iptables.log"); }; filter f_iptables { match("iptables:"); }; log { source(k_src); filter(f_iptables); destination(iptables); flags(final); }; destination messages { file("/var/log/messages"); }; log { source(src); source(k_src); destination(messages); }; ----- All of my kernel messages showed up with the "kernel:" prefix, and anything coming from iptables was dropped in a separate log. Based on the documentation, I modified my configuration to the following for 3.0.4: ----- source k_src { file("/proc/kmsg" program_override("kernel")); }; destination iptables { file("/var/log/iptables.log"); }; filter f_iptables { program("^kernel$") and message("^iptables:"); }; log { source(k_src); filter(f_iptables); destination(iptables); flags(final); }; destination messages { file("/var/log/messages"); }; log { source(src); source(k_src); destination(messages); }; ----- However, my kernel messages show up with no prefix. I tried starting up in debug mode: ----- syslog-ng starting up; version='3.0.4' Incoming log entry; line='<6>usb 2-7: USB disconnect, address 8' Filter rule evaluation begins; filter_rule='f_iptables' Filter node evaluation result; filter_result='not-match' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter rule evaluation result; filter_result='not-match', filter_rule='f_iptables' Initializing destination file writer; template='/var/log/messages', filename='/var/log/messages' Incoming log entry; line='<4>iptables: IN=eth0 OUT= MAC=0:05:00:10:97:43:00:0SC147.5.2 S=2... E=8TS00 RC0C T= D363POO2' Filter rule evaluation begins; filter_rule='f_iptables' Filter node evaluation result; filter_result='not-match' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter rule evaluation result; filter_result='not-match', filter_rule='f_iptables' ----- This didn't help me. Am I doing something wrong? I didn't find any complaints of this nature on the mailing list, which leads me to suspect I'm somehow being stupid, any pointers much appreciated. Thanks... -- Paul B. Henson | (909) 979-6361 | http://www.csupomona.edu/~henson/ Operating Systems and Network Analyst | henson(a)csupomona.edu California State Polytechnic University | Pomona CA 91768

2 2

Chained changing IP addresses to FQDN
by Jason Carr 28 Dec '09

28 Dec '09

Hello, I have a syslog infrastructure using syslog-ng comprising of three tiers: probes that accept messages and forwards them to the broker, a broker that routes and/or duplicates messages to the appropriate storage devices, and storage devices. All forward using the tcp("hostname"); as a destination. Basically it looks like this: machine1 ==udp==> +-------+ +--------+ +-----------+ machine2 ==udp==> | probe | ==tcp==> | broker | ==tcp==> | storage 1 | machine3 ==udp==> +-------+ +--------+ +-----------+ || +-----------+ ++=======tcp==> | storage 2 | +-----------+ The probes rewrite the hostname to the IP address that the packet came from using the chain_hostnames(no) and keep_hostname(no) options. One of the requirements of one specific storage device is to change the IP addresses into hostnames. I'm having difficulty on the storage device switching the IP address in the log message to the FQDN. I have tried all combinations I can think of of chain_hostnames and keep_hostname and cannot seem to get the IP address replaced with the FQDN. Is there a way to replace the IP address with the FQDN while receiving the messages from an intermediary? Thank you, Jason

4 5

troubleshooting and debugging syslog-ng
by Zoltán Pallagi 23 Dec '09

23 Dec '09

Hi, I've sent a new post about it to my blog (http://pzolee.blogs.balabit.com/2009/12/troubleshooting-and-debugging-syslo…) I tried to collect all important things about this issue. I hope this post will make it easier for you to specify the reason of the possible syslog-ng problems and for us to help you. pzolee

1 0

time_sleep in high udp environment?
by Tim Rupp 22 Dec '09

22 Dec '09

Hi, we were debugging message drops on our syslog-ng server (rpm syslog-ng-3.0.5-1.rhel5.x86_64) and found out that if we commented out the time_sleep global option we had set (value of 20) that all the drops went away. Initially we had set this value while reading through the pdf documentation (page 136) figuring that since we had potentially a lot of traffic that this would boost performance. So I guess I'd like to know why we seemed to see the opposite behavior with this option enabled. We don't really do tcp connections, so maybe this option was useless to begin with. Anyways, it made me curious, so if anyone can enlighten me I'd appreciate it. Thanks, Tim

2 1

pdbtool pattern debuging
by ILLES, Marton 22 Dec '09

22 Dec '09

Hi! I have just pushed to my syslog-ng 3.1 git repo some patches to add more detailed debugging/troubleshooting capabilities to patterndb through pdbtool match. Now you can easily see how your patterns matched a given message, which part of the message was matched by literal part of the pattern and which by a parser. All this new feature with a wonderful colorized output if requested, though the colors are still bit ugly... Up to now the easiest way of creating patterns was to do trial & error sequence and try to figure out which part of the pattern was broken. Now you can exactly see where the matching stopped. It is also possible to output the match in parse-able format to be used in scripts or as a backend of some pattern authoring tool. I am still not sure about the colored output nor about the machine parse-able output format, so any feedback, comment, idea or suggestion is very much appreciated. You can see some screen-shots of pdbtool match (with colors) in action on my blog: http://marci.blogs.balabit.com/2009/12/pdbtool-stylists-wanted.html The code is available in my git repo: http://git.balabit.hu/?p=marci/syslog-ng-3.1.git;a=commit;h=c7c3b7b81577507… I did not update the man page so far, but --help works. You should be using the "-D" to enable debuging and "-c" for color output, while "-C" for csv style output. Happy matching! Marton -- Key fingerprint = F78C 25CA 5F88 6FAF EA21 779D 3279 9F9E 1155 670D

1 0