- syslog-ng - lists.balabit.hu

Dear Syslog-Ng, If you can't find in google, try JUSTDIAL.COM
by Vadiraj 18 Jan '10

18 Jan '10

Dear Syslog-Ng,I strongly recommend this website www.justdial.com. It's a world class local search service & I've always found anything I've ever wanted.You can find info on any company, product, or service in over 240 cities in India.You can also call them up 24x7, on phone (69999999), a local call in 240 Indian cities.Ask for anything, you'll get the info on the phone and/or by SMS within 30 secs, and this service is at no cost!For a change, it's an original Indian idea and an Indian company with world class service, and with a vision to spread all over the world.Be a proud Indian and forward this to every Indian you know.Best Wishes,VadirajClick Here to unsubscribe.

1 0

Odd problem after upgrading from 2.1 to 3
by Jerry Bell 18 Jan '10

18 Jan '10

This config worked in syslog-ng 2.1 source NET {udp(port(514)); tcp(port(5140) keep-alive(yes));}; destination A { file("/var/log/A"); }; destination B { file("/var/log/B"); }; filter A { host ("A");}; filter A { host ("A");}; log { source(NET); destination(A); filter(A); }; log { source(NET); destination(B); filter(B); }; worked as expected that is log messages coming from host A went to /var/log/A file and log messages coming from host B went to /var/log/B. The two hostnames are resolved locally in /etc/hosts After upgrading to 3.0.x all the messages duplicated in both files so that messages coming from A are written in /var/log/A AND /var/log/B and vice-versa. Any idea why it does not work in 3? Thank you, Jerry

2 2

Broken TCP connection
by James Pirman 14 Jan '10

14 Jan '10

I am currently having an issue with syslog-ng 3.0.4 where my TCP connection between my client and server is lost throughout the day. By looking at the pcap file from tcpdump I can tell that the TCP connection reset was initiated by the syslog-ng server. The only information that was initially in the log file regarding this disconnection was the following 2 lines: <45>1 2010-01-05T10:29:32.323-06:00 server-db-01 syslog-ng 29213 - [meta sequenceId="2733719"] notice Syslog connection closed; fd='9', client='AF_INET(192.168.27.218:46326)', local='AF_INET(192.168.27.198:20514)' and <46>1 2010-01-05T10:29:32.323-06:00 server-db-01 syslog-ng 29213 - [meta sequenceId="2733720"] info Closing log transport fd; fd='9' In order to get more information, I set the following flags in init.d: "-v -d -t". This did not give me any more information about the TCP disconnect, however I did notice that a lot of my normal messages were preceeded by the following text: <47>1 2010-01-05T10:29:32.323-06:00 server-db-01 syslog-ng 29213 - [meta sequenceId="2733718"] debug Incoming log entry; line= A normal log message then follows the '=' sign. A decent percentage of my messages are preceeded by this throughout the day, but just before the disconnect it appears that all of my messages from server-db-01 are preceeded by the debug line. Any ideas as to what could be going on? I have included my config file below if that helps. Any assistance would be greatly appreciated. -Jim @version: 3.0 #Default configuration file for syslog-ng. # # For a description of syslog-ng configuration file directives, please read # the syslog-ng Administrator's guide at: # # http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html # options { keep_hostname(yes); keep_timestamp(yes); frac_digits(3); }; source all { internal(); syslog(ip("192.168.27.198") port(20514) transport("tcp") log_fetch_limit(100)); }; destination allclientsfile { file("/data/local/Logs/server-$YEAR-$MONTH-$DAY.log" flags(syslog-protocol) flush_timeout(100) create_dirs(yes) dir_owner(jpirman) dir_group(jpirman) owner(jpirman) group(jpirman) template("$PRIORITY $MESSAGE") ); }; destination msgserver { udp("127.0.0.1" port(20515) flush_timeout(100) template("$ISODATE $PROGRAM $PRIORITY $MESSAGE\n")); }; log { source(all); destination(allclientsfile); destination(msgserver);}; _________________________________________________________________ Hotmail: Powerful Free email with security by Microsoft. http://clk.atdmt.com/GBL/go/171222986/direct/01/

3 8

slngctl: syslog-ng debug, trace and statistics utility
by ILLES, Marton 14 Jan '10

14 Jan '10

Hi, PZolee wrote a quite lengthy blog post on troubleshooting and debuging syslog-ng [1] back in December 2009. It popped into my mind that running syslog-ng with debug/trace/verbose option enabled is very handy, but sometimes you do not want to stop syslog-ng just to restart it in a more verbose mode. My idea was to enable syslog-ng to real-time modify these verbosity settings through the control socket which is currently only used for exporting statistical information. I extended the control socket communication to have the ability of modify these logging settings. To make life easier I have created a small utility called slngctl which can be used to communicate with the running syslog-ng process. Using slngctl it also easy to query the statistics collected by syslog-ng. Also on longer term I think other useful small features could be added the slngctl. slngctl commands: marci@octane:$ slngctl Possible commands are: stats Dump syslog-ng statistics verbose Enable/query verbose messages debug Enable/query debug messages trace Enable/query trace messages Querying statistics: marci@octane:$ slngctl stats destination;d_out;;a;processed;0 global;payload_reallocs;;a;processed;0 source;s_tcp;;a;processed;0 global;msg_clones;;a;processed;0 global;sdata_updates;;a;processed;0 center;;received;a;processed;0 center;;queued;a;processed;0 Getting current verbose settings: marci@octane:$ slngctl verbose VERBOSE=1 Enabling trace run-time: marci@octane:$ slngctl trace TRACE=0 marci@octane:$ slngctl trace -s 1 marci@octane:$ slngctl trace TRACE=1 As always you can find the source code in my 3.1 git repository and all feedbacks are very welcome. best, Marton PS: I know slngctl might not be the best name, so I offer a bier to anyone suggesting a better name when visiting us in Budapest, Hungary. :) [1] http://pzolee.blogs.balabit.com/2009/12/troubleshooting-and-debugging-syslo… -- Key fingerprint = F78C 25CA 5F88 6FAF EA21 779D 3279 9F9E 1155 670D

3 3

requirement for @version directive in 3.x config file
by Paul B. Henson 14 Jan '10

14 Jan '10

Our syslog-ng configuration file is generated automatically, from various different bits and pieces based on what roles and services a system provides. Unfortunately, our current implementation would not allow always ensuring that the "@version" directive is the first line in the generated config file. Version 3 of syslog-ng seems very unhappy if that is not the case. This seems somewhat inflexible, would it be possible to have a command line option specifying the configuration file version so it does not have to be on the very first line of the actual file? Any other suggestions (short of modifying our internal processes :) ) to get verison 3 to be happy with a configuration file without this specific directive as the very first line? Thanks much... -- Paul B. Henson | (909) 979-6361 | http://www.csupomona.edu/~henson/ Operating Systems and Network Analyst | henson(a)csupomona.edu California State Polytechnic University | Pomona CA 91768

3 3

Leaking objects in size-4096 cache in slabinfo
by Doug Warner 12 Jan '10

12 Jan '10

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'm running syslog-ng 3.0.4 OSE on Gentoo and have a central syslog host that accepts about 3Mbps of log traffic via a TCP pipe. We've recently been having oom-killer problems and I finally tracked it down to the fact that we seem to be leaking objects in the size-4096 cache. Now, slabinfo is new to me so I could be interpreting this wrong, but restarting syslog-ng definite reclaims this cache. What data can I provide to help track down this problem? I noticed that there seems to be a similar error in 3.1 beta, reported here: http://thread.gmane.org/gmane.comp.syslog-ng/8544 - -Doug -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) iD8DBQFLJ8FqJV36su0A0xIRAizGAKDbVEpHRMhrRgAogUuAh0nCKz9xFwCgtz17 6EN8aU3MxU7bL8Sr4xRx6tI= =1LQg -----END PGP SIGNATURE-----

3 13

Separating Remote Logs
by Nate Hausrath 12 Jan '10

12 Jan '10

Hi everyone, I've set up a central log server using syslog-ng that receives logs from many different locations on the network. I want to break these logs up into different files so I can using another program to parse and interpret each one individually. So for example, I want my Windows logs to go to /var/log/remote/windows.log, which my ASA logs go to /var/log/remote/asa.log. Anything that doesn't fit into a group should go to /var/log/remote/other.log. I'm trying to figure out the best way to do this. I was going to create a custom xml file for the db-parser, but I can't seem to get it to work the way I want. Here is my db-parser XML file so far: <?xml version='1.0' encoding='UTF-8'?> <patterndb version='2' pub_date='2009-12-07'> <ruleset name='capcxml' id='1923-ab2b'> <pattern/> <rules> <rule provider='capc' id='1' class='system'> <description>Detects ASA logs</description> <patterns> <pattern>%ASA</pattern> </patterns> </rule> <rule provider='capc' id='2' class='system'> <description>Detects Windows logs from Snare</description> <patterns> <pattern>MSWinEventLog</pattern> </patterns> </rule> </rules> </ruleset> </patterndb> So my question is, how can I specify a filter in my syslog-ng.conf file that will allow me to separate logs from these two different rules? Along with that, how can I separate everything that doesn't match? For instance, I could have something like this: filter f_class_asa { match("1" value(".classifier_rule_id") type("string") ); }; filter f_class_windows { match("2" value(".classifier_rule_id") type("string") ); }; filter f_class_other { not match("1" value(".classifier_rule_id") type("string")) and not match("2" value(".classifier_rule_id") type("string")); }; log { source(s_remote); parser(p_capc); filter(f_class_asa); destination(df_asa); }; log { source(s_remote); parser(p_capc); filter(f_class_windows); destination(df_windows); }; log { source(s_remote); parser(p_capc); filter(f_class_other); destination(df_other); }; But this doesn't seem to work. Everything gets placed in the df_asa file. Am I approaching this the wrong way? Should I not use db-parser for this task? Thanks for any help! Nate

3 7

Introducing pdbtool patternize
by Peter Gyongyosi 11 Jan '10

11 Jan '10

Hi, As Márton has already written about it [1], lots of guys here at BalaBit spent a lot of time last year creating a pattern database for some 200+ often-used applications. Just like every manual process, this was a tedious task which begged to be automated. Of course it cannot be fully automated as no algorithm can replace an actual person understanding the structure of the logs a piece of software produces (or even looking into the source code to see how they're generated), but still, a tool that can detect similar messages in a log database and generate a pattern database for it would've been real handy. Thus have pdbtool patternize been created. The tool I've written is a part of the pdbtool utility and can be used to generate a pattern database from a bunch of unknown messages. It uses the algorithm developed by Risto Vaarandi for SLCT [2], the main idea of which is using a data clustering technique to find similar log messages and replacing the differing parts with wildcard characters. In our case, the wildcards are @ESTRING:: @ parsers, otherwise, the solution is pretty much the same. It's far from being perfect (the code could be optimized at some places, it needs to load everything into memory so the size of the parseable log file is limited to the RAM in your machine as swapping slows things down really bad, and the log messages are split to words only at spaces, which makes it unable to detect "username=@QSTRING::'@"-type patterns), but it has already produced some impressive results on the test databases I've tried it with. It managed to categorize 95-98% of ~2M lines of logs into 40-50 patterns in a reasonable time and the patterns themselves were pretty readable as well. There're two options that can be set for the patternization. The first one ("-S") is the support value and accepts a floating point number: the percentage of the lines that have to match a pattern candidate to include it in the resulting pattern database. It allows tuning the trade-off between the number of patterns (too much would be hard to maintain) and the coverage these patterns produce. My first tests show that the optimal support value for most log types are around 2.5-5%, but you should check this with your own logs. The other option ("-o") enables a different operation mode. In this case, after a clustering step is completed, the tool does not exit printing out the generated patterns, rather starts yet another clustering on the messages that are not covered with the patterns generated so far. It keeps on doing this while new patterns can be generated. This way, much better coverage can be achieved while still having a low number of patterns -- the larger groups are detected early, but the small groups aren't left out either. Based on my tests, a bit larger support value, around 10-15% is optimal for this operation mode. The code is available in my public syslog-ng repository [3] and I'm more than eager for some feedback. It still needs a larger review from someone more experienced in syslog-ng internals and leaks a little memory here&there but in overall it's mature enough to play around with even with large log databases. greets, Peter [1] http://marci.blogs.balabit.com/2009/12/pattern-database-first-snapshot.html [2] http://ristov.users.sourceforge.net/slct/] [3] http://git.balabit.hu/?p=gyp/syslog-ng-3.1.git;a=summary

1 0

SQL destination function issue
by Christopher O'Brien 07 Jan '10

07 Jan '10

Hi, I am trying to log messages from a Cisco ASA into a MySQL database through syslog-ng 3.0.4. I have the whole thing working except I cannot import data if I want to use a function. Here is an example: Here is the log message that is coming into the syslog server from the ASA device: Dec 04 2009 10:22:23: %ASA-5-305012: Teardown dynamic TCP translation from inside:10.6.42.166/2617 to outside:10.16.19.17/61631 duration 0:01:00 Here is what is logged by syslog-ng: Running SQL query; query='INSERT INTO stop (s_time, e_time, proto, in_ip, in_port, out_ip, out_port) VALUES (\'DATE_SUB(\"2009-12-04 10:22:23\", INTERVAL \"0:01:00\" HOUR_SECOND)\', \'2009-12-04 10:22:23\', \'TCP\', \'10.6.42.166\', \'2617\', \'10.16.19.17\', \'61631\')' Here is what is recorded in the database: +-----+---------------------+---------------------+-------+-------------+---------+---------------+----------+ | idx | s_time | e_time | proto | in_ip | in_port | out_ip | out_port | +-----+---------------------+---------------------+-------+-------------+---------+---------------+----------+ | 166 | 0000-00-00 00:00:00 | 2009-12-04 10:22:23 | TCP | 10.6.42.166 | 2617 | 10.16.19.17 | 61631 | Every field is recorded properly aside from the one that I am trying to use a function in. The purpose of the function is to record the start time of a translation. I get the end time from the syslog timestamp and the duration from the syslog message itself. I am trying to use a builtin MySQL function to record the start time in s_time. I think the problme is syslog-ng is putting single quotes around the whole function so it is being treated like a string literal by MySQL instead of variable data. Is that right? Is there a way to remedy this? Here is my syslog-ng.conf configuration if it helps: @version: 3.0 # $Header: /var/cvsroot/gentoo-x86/app-admin/syslog-ng/files/syslog-ng.conf.gentoo,v 1.7 2007/08/02 04:52:18 mr_bones_ Exp $ # # Syslog-ng default configuration file for Gentoo Linux # contributed by Michael Sterrett options { chain_hostnames(no); # The default action of syslog-ng 1.6.0 is to log a STATS line # to the file every 10 minutes. That's pretty ugly after a while. # Change it to every 12 hours so you get a nice daily update of # how many messages syslog-ng missed (0). stats_freq(43200); }; source src { unix-stream("/dev/log" max-connections(256)); internal(); file("/proc/kmsg"); }; source s_xlate { udp(ip(0.0.0.0) port(514)); }; filter f_xlate_start { host(10.16.19.33) and program("%ASA-5-305011") ; }; filter f_xlate_stop { host(10.16.19.33) and program("%ASA-5-305012") ; }; parser p_xlate { csv-parser(columns("LOG.STATUS", "LOG.TYPE", "LOG.PROTOCOL", "LOG.XLATE", "LOG.FROM","LOG.INSIDE", "LOG.IN_IP", "LOG.IN_PORT", "LOG.TO","LOG.OUTSIDE", "LOG.OUT_IP", "LOG.OUT_PORT", "LOG.DURATION","LOG.HR", "LOG.MIN", "LOG.SEC") delimiters("/: ") flags(escape-none) template("${MSGONLY}")); }; destination messages { file("/var/log/messages"); }; destination d_xlate_start { sql(type(mysql) username("user") password("pass") database("nat") table("start") columns("s_time DATETIME", "proto VARCHAR(4)", "in_ip VARCHAR(15)", "in_port VARCHAR(5)", "out_ip VARCHAR(15)", "out_port VARCHAR(5)") values("$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC", "${LOG.PROTOCOL}", "${LOG.IN_IP}", "${LOG.IN_PORT}", "${LOG.OUT_IP}", "${LOG.OUT_PORT}")); }; destination d_xlate_stop { sql(type(mysql) username("user") password("pass") database("nat") table("stop") columns("s_time DATETIME", "e_time DATETIME", "proto VARCHAR(4)", "in_ip VARCHAR(15)", "in_port VARCHAR(5)", "out_ip VARCHAR(15)", "out_port VARCHAR(5)") values("DATE_SUB(\"$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC\", INTERVAL \"${LOG.HR}:${LOG.MIN}:${LOG.SEC}\" HOUR_SECOND)", "$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC", "${LOG.PROTOCOL}", "${LOG.IN_IP}", "${LOG.IN_PORT}", "${LOG.OUT_IP}", "${LOG.OUT_PORT}")); }; destination console_all { file("/dev/tty12"); }; log { source(s_xlate); filter(f_xlate_start); parser(p_xlate); destination(d_xlate_start); flags(final); }; log { source(s_xlate); filter(f_xlate_stop); parser(p_xlate); destination(d_xlate_stop); flags(final); }; log { source(src); destination(messages); }; log { source(src); destination(console_all); };

2 2

Syslog Blocked
by Tokie 07 Jan '10

07 Jan '10

Hi For All, i have a strange problem. On my server syslog (on Debian Etch) I installed syslog-ng and some times syslog-ng don't works!!! The programs is in execution, but don't receive the log in destination files; Logrotate makes the new file (ex. /var/log/syslog/new) and compress the last with gz; this works!! But the new file remains empty!! If I restart the syslog-ng with /etc/init.d/syslog-ng restart(like below at Jan 5 17:56:34) , it begins to write on new file!! This is my /var/log/messages > Jan 4 06:25:21 s_all(a)sys-server.tokie syslog-ng[2690]: SIGHUP > received, reloading configuration; > Jan 5 06:25:22 s_all(a)sys-server.tokie syslog-ng[2690]: SIGHUP > received, reloading configuration; > Jan 5 17:56:34 s_all(a)sys-server.tokie syslog-ng[2690]: SIGTERM > received, terminating; > Jan 5 17:56:34 s_all(a)sys-server.tokie syslog-ng[2690]: syslog-ng > shutting down; version='2.0.0' > Jan 5 17:56:35 s_all(a)sys-server.tokie syslog-ng[3111]: syslog-ng > starting up; version='2.0.0' Every day in cron should be restart syslog at 06:25 I think the problem is SIGHUP, but I don't know why!! This don't happen every day, but randomly!! Somebody can help me?? Tokie

3 3