- syslog-ng - lists.balabit.hu

Odd problem after upgrading from 2.1 to 3
by Jerry Bell 17 Jan '10

17 Jan '10

This config worked in syslog-ng 2.1 source NET {udp(port(514)); tcp(port(5140) keep-alive(yes));}; destination A { file("/var/log/A"); }; destination B { file("/var/log/B"); }; filter A { host ("A");}; filter A { host ("A");}; log { source(NET); destination(A); filter(A); }; log { source(NET); destination(B); filter(B); }; worked as expected that is log messages coming from host A went to /var/log/A file and log messages coming from host B went to /var/log/B. The two hostnames are resolved locally in /etc/hosts After upgrading to 3.0.x all the messages duplicated in both files so that messages coming from A are written in /var/log/A AND /var/log/B and vice-versa. Any idea why it does not work in 3? Thank you, Jerry

2 2

Broken TCP connection
by James Pirman 14 Jan '10

14 Jan '10

I am currently having an issue with syslog-ng 3.0.4 where my TCP connection between my client and server is lost throughout the day. By looking at the pcap file from tcpdump I can tell that the TCP connection reset was initiated by the syslog-ng server. The only information that was initially in the log file regarding this disconnection was the following 2 lines: <45>1 2010-01-05T10:29:32.323-06:00 server-db-01 syslog-ng 29213 - [meta sequenceId="2733719"] notice Syslog connection closed; fd='9', client='AF_INET(192.168.27.218:46326)', local='AF_INET(192.168.27.198:20514)' and <46>1 2010-01-05T10:29:32.323-06:00 server-db-01 syslog-ng 29213 - [meta sequenceId="2733720"] info Closing log transport fd; fd='9' In order to get more information, I set the following flags in init.d: "-v -d -t". This did not give me any more information about the TCP disconnect, however I did notice that a lot of my normal messages were preceeded by the following text: <47>1 2010-01-05T10:29:32.323-06:00 server-db-01 syslog-ng 29213 - [meta sequenceId="2733718"] debug Incoming log entry; line= A normal log message then follows the '=' sign. A decent percentage of my messages are preceeded by this throughout the day, but just before the disconnect it appears that all of my messages from server-db-01 are preceeded by the debug line. Any ideas as to what could be going on? I have included my config file below if that helps. Any assistance would be greatly appreciated. -Jim @version: 3.0 #Default configuration file for syslog-ng. # # For a description of syslog-ng configuration file directives, please read # the syslog-ng Administrator's guide at: # # http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html # options { keep_hostname(yes); keep_timestamp(yes); frac_digits(3); }; source all { internal(); syslog(ip("192.168.27.198") port(20514) transport("tcp") log_fetch_limit(100)); }; destination allclientsfile { file("/data/local/Logs/server-$YEAR-$MONTH-$DAY.log" flags(syslog-protocol) flush_timeout(100) create_dirs(yes) dir_owner(jpirman) dir_group(jpirman) owner(jpirman) group(jpirman) template("$PRIORITY $MESSAGE") ); }; destination msgserver { udp("127.0.0.1" port(20515) flush_timeout(100) template("$ISODATE $PROGRAM $PRIORITY $MESSAGE\n")); }; log { source(all); destination(allclientsfile); destination(msgserver);}; _________________________________________________________________ Hotmail: Powerful Free email with security by Microsoft. http://clk.atdmt.com/GBL/go/171222986/direct/01/

3 8

slngctl: syslog-ng debug, trace and statistics utility
by ILLES, Marton 14 Jan '10

14 Jan '10

Hi, PZolee wrote a quite lengthy blog post on troubleshooting and debuging syslog-ng [1] back in December 2009. It popped into my mind that running syslog-ng with debug/trace/verbose option enabled is very handy, but sometimes you do not want to stop syslog-ng just to restart it in a more verbose mode. My idea was to enable syslog-ng to real-time modify these verbosity settings through the control socket which is currently only used for exporting statistical information. I extended the control socket communication to have the ability of modify these logging settings. To make life easier I have created a small utility called slngctl which can be used to communicate with the running syslog-ng process. Using slngctl it also easy to query the statistics collected by syslog-ng. Also on longer term I think other useful small features could be added the slngctl. slngctl commands: marci@octane:$ slngctl Possible commands are: stats Dump syslog-ng statistics verbose Enable/query verbose messages debug Enable/query debug messages trace Enable/query trace messages Querying statistics: marci@octane:$ slngctl stats destination;d_out;;a;processed;0 global;payload_reallocs;;a;processed;0 source;s_tcp;;a;processed;0 global;msg_clones;;a;processed;0 global;sdata_updates;;a;processed;0 center;;received;a;processed;0 center;;queued;a;processed;0 Getting current verbose settings: marci@octane:$ slngctl verbose VERBOSE=1 Enabling trace run-time: marci@octane:$ slngctl trace TRACE=0 marci@octane:$ slngctl trace -s 1 marci@octane:$ slngctl trace TRACE=1 As always you can find the source code in my 3.1 git repository and all feedbacks are very welcome. best, Marton PS: I know slngctl might not be the best name, so I offer a bier to anyone suggesting a better name when visiting us in Budapest, Hungary. :) [1] http://pzolee.blogs.balabit.com/2009/12/troubleshooting-and-debugging-syslo… -- Key fingerprint = F78C 25CA 5F88 6FAF EA21 779D 3279 9F9E 1155 670D

3 3

requirement for @version directive in 3.x config file
by Paul B. Henson 14 Jan '10

14 Jan '10

Our syslog-ng configuration file is generated automatically, from various different bits and pieces based on what roles and services a system provides. Unfortunately, our current implementation would not allow always ensuring that the "@version" directive is the first line in the generated config file. Version 3 of syslog-ng seems very unhappy if that is not the case. This seems somewhat inflexible, would it be possible to have a command line option specifying the configuration file version so it does not have to be on the very first line of the actual file? Any other suggestions (short of modifying our internal processes :) ) to get verison 3 to be happy with a configuration file without this specific directive as the very first line? Thanks much... -- Paul B. Henson | (909) 979-6361 | http://www.csupomona.edu/~henson/ Operating Systems and Network Analyst | henson(a)csupomona.edu California State Polytechnic University | Pomona CA 91768

3 3

Leaking objects in size-4096 cache in slabinfo
by Doug Warner 12 Jan '10

12 Jan '10

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'm running syslog-ng 3.0.4 OSE on Gentoo and have a central syslog host that accepts about 3Mbps of log traffic via a TCP pipe. We've recently been having oom-killer problems and I finally tracked it down to the fact that we seem to be leaking objects in the size-4096 cache. Now, slabinfo is new to me so I could be interpreting this wrong, but restarting syslog-ng definite reclaims this cache. What data can I provide to help track down this problem? I noticed that there seems to be a similar error in 3.1 beta, reported here: http://thread.gmane.org/gmane.comp.syslog-ng/8544 - -Doug -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) iD8DBQFLJ8FqJV36su0A0xIRAizGAKDbVEpHRMhrRgAogUuAh0nCKz9xFwCgtz17 6EN8aU3MxU7bL8Sr4xRx6tI= =1LQg -----END PGP SIGNATURE-----

3 13

Separating Remote Logs
by Nate Hausrath 12 Jan '10

12 Jan '10

Hi everyone, I've set up a central log server using syslog-ng that receives logs from many different locations on the network. I want to break these logs up into different files so I can using another program to parse and interpret each one individually. So for example, I want my Windows logs to go to /var/log/remote/windows.log, which my ASA logs go to /var/log/remote/asa.log. Anything that doesn't fit into a group should go to /var/log/remote/other.log. I'm trying to figure out the best way to do this. I was going to create a custom xml file for the db-parser, but I can't seem to get it to work the way I want. Here is my db-parser XML file so far: <?xml version='1.0' encoding='UTF-8'?> <patterndb version='2' pub_date='2009-12-07'> <ruleset name='capcxml' id='1923-ab2b'> <pattern/> <rules> <rule provider='capc' id='1' class='system'> <description>Detects ASA logs</description> <patterns> <pattern>%ASA</pattern> </patterns> </rule> <rule provider='capc' id='2' class='system'> <description>Detects Windows logs from Snare</description> <patterns> <pattern>MSWinEventLog</pattern> </patterns> </rule> </rules> </ruleset> </patterndb> So my question is, how can I specify a filter in my syslog-ng.conf file that will allow me to separate logs from these two different rules? Along with that, how can I separate everything that doesn't match? For instance, I could have something like this: filter f_class_asa { match("1" value(".classifier_rule_id") type("string") ); }; filter f_class_windows { match("2" value(".classifier_rule_id") type("string") ); }; filter f_class_other { not match("1" value(".classifier_rule_id") type("string")) and not match("2" value(".classifier_rule_id") type("string")); }; log { source(s_remote); parser(p_capc); filter(f_class_asa); destination(df_asa); }; log { source(s_remote); parser(p_capc); filter(f_class_windows); destination(df_windows); }; log { source(s_remote); parser(p_capc); filter(f_class_other); destination(df_other); }; But this doesn't seem to work. Everything gets placed in the df_asa file. Am I approaching this the wrong way? Should I not use db-parser for this task? Thanks for any help! Nate

3 7

Introducing pdbtool patternize
by Peter Gyongyosi 11 Jan '10

11 Jan '10

Hi, As Márton has already written about it [1], lots of guys here at BalaBit spent a lot of time last year creating a pattern database for some 200+ often-used applications. Just like every manual process, this was a tedious task which begged to be automated. Of course it cannot be fully automated as no algorithm can replace an actual person understanding the structure of the logs a piece of software produces (or even looking into the source code to see how they're generated), but still, a tool that can detect similar messages in a log database and generate a pattern database for it would've been real handy. Thus have pdbtool patternize been created. The tool I've written is a part of the pdbtool utility and can be used to generate a pattern database from a bunch of unknown messages. It uses the algorithm developed by Risto Vaarandi for SLCT [2], the main idea of which is using a data clustering technique to find similar log messages and replacing the differing parts with wildcard characters. In our case, the wildcards are @ESTRING:: @ parsers, otherwise, the solution is pretty much the same. It's far from being perfect (the code could be optimized at some places, it needs to load everything into memory so the size of the parseable log file is limited to the RAM in your machine as swapping slows things down really bad, and the log messages are split to words only at spaces, which makes it unable to detect "username=@QSTRING::'@"-type patterns), but it has already produced some impressive results on the test databases I've tried it with. It managed to categorize 95-98% of ~2M lines of logs into 40-50 patterns in a reasonable time and the patterns themselves were pretty readable as well. There're two options that can be set for the patternization. The first one ("-S") is the support value and accepts a floating point number: the percentage of the lines that have to match a pattern candidate to include it in the resulting pattern database. It allows tuning the trade-off between the number of patterns (too much would be hard to maintain) and the coverage these patterns produce. My first tests show that the optimal support value for most log types are around 2.5-5%, but you should check this with your own logs. The other option ("-o") enables a different operation mode. In this case, after a clustering step is completed, the tool does not exit printing out the generated patterns, rather starts yet another clustering on the messages that are not covered with the patterns generated so far. It keeps on doing this while new patterns can be generated. This way, much better coverage can be achieved while still having a low number of patterns -- the larger groups are detected early, but the small groups aren't left out either. Based on my tests, a bit larger support value, around 10-15% is optimal for this operation mode. The code is available in my public syslog-ng repository [3] and I'm more than eager for some feedback. It still needs a larger review from someone more experienced in syslog-ng internals and leaks a little memory here&there but in overall it's mature enough to play around with even with large log databases. greets, Peter [1] http://marci.blogs.balabit.com/2009/12/pattern-database-first-snapshot.html [2] http://ristov.users.sourceforge.net/slct/] [3] http://git.balabit.hu/?p=gyp/syslog-ng-3.1.git;a=summary

1 0

SQL destination function issue
by Christopher O'Brien 07 Jan '10

07 Jan '10

Hi, I am trying to log messages from a Cisco ASA into a MySQL database through syslog-ng 3.0.4. I have the whole thing working except I cannot import data if I want to use a function. Here is an example: Here is the log message that is coming into the syslog server from the ASA device: Dec 04 2009 10:22:23: %ASA-5-305012: Teardown dynamic TCP translation from inside:10.6.42.166/2617 to outside:10.16.19.17/61631 duration 0:01:00 Here is what is logged by syslog-ng: Running SQL query; query='INSERT INTO stop (s_time, e_time, proto, in_ip, in_port, out_ip, out_port) VALUES (\'DATE_SUB(\"2009-12-04 10:22:23\", INTERVAL \"0:01:00\" HOUR_SECOND)\', \'2009-12-04 10:22:23\', \'TCP\', \'10.6.42.166\', \'2617\', \'10.16.19.17\', \'61631\')' Here is what is recorded in the database: +-----+---------------------+---------------------+-------+-------------+---------+---------------+----------+ | idx | s_time | e_time | proto | in_ip | in_port | out_ip | out_port | +-----+---------------------+---------------------+-------+-------------+---------+---------------+----------+ | 166 | 0000-00-00 00:00:00 | 2009-12-04 10:22:23 | TCP | 10.6.42.166 | 2617 | 10.16.19.17 | 61631 | Every field is recorded properly aside from the one that I am trying to use a function in. The purpose of the function is to record the start time of a translation. I get the end time from the syslog timestamp and the duration from the syslog message itself. I am trying to use a builtin MySQL function to record the start time in s_time. I think the problme is syslog-ng is putting single quotes around the whole function so it is being treated like a string literal by MySQL instead of variable data. Is that right? Is there a way to remedy this? Here is my syslog-ng.conf configuration if it helps: @version: 3.0 # $Header: /var/cvsroot/gentoo-x86/app-admin/syslog-ng/files/syslog-ng.conf.gentoo,v 1.7 2007/08/02 04:52:18 mr_bones_ Exp $ # # Syslog-ng default configuration file for Gentoo Linux # contributed by Michael Sterrett options { chain_hostnames(no); # The default action of syslog-ng 1.6.0 is to log a STATS line # to the file every 10 minutes. That's pretty ugly after a while. # Change it to every 12 hours so you get a nice daily update of # how many messages syslog-ng missed (0). stats_freq(43200); }; source src { unix-stream("/dev/log" max-connections(256)); internal(); file("/proc/kmsg"); }; source s_xlate { udp(ip(0.0.0.0) port(514)); }; filter f_xlate_start { host(10.16.19.33) and program("%ASA-5-305011") ; }; filter f_xlate_stop { host(10.16.19.33) and program("%ASA-5-305012") ; }; parser p_xlate { csv-parser(columns("LOG.STATUS", "LOG.TYPE", "LOG.PROTOCOL", "LOG.XLATE", "LOG.FROM","LOG.INSIDE", "LOG.IN_IP", "LOG.IN_PORT", "LOG.TO","LOG.OUTSIDE", "LOG.OUT_IP", "LOG.OUT_PORT", "LOG.DURATION","LOG.HR", "LOG.MIN", "LOG.SEC") delimiters("/: ") flags(escape-none) template("${MSGONLY}")); }; destination messages { file("/var/log/messages"); }; destination d_xlate_start { sql(type(mysql) username("user") password("pass") database("nat") table("start") columns("s_time DATETIME", "proto VARCHAR(4)", "in_ip VARCHAR(15)", "in_port VARCHAR(5)", "out_ip VARCHAR(15)", "out_port VARCHAR(5)") values("$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC", "${LOG.PROTOCOL}", "${LOG.IN_IP}", "${LOG.IN_PORT}", "${LOG.OUT_IP}", "${LOG.OUT_PORT}")); }; destination d_xlate_stop { sql(type(mysql) username("user") password("pass") database("nat") table("stop") columns("s_time DATETIME", "e_time DATETIME", "proto VARCHAR(4)", "in_ip VARCHAR(15)", "in_port VARCHAR(5)", "out_ip VARCHAR(15)", "out_port VARCHAR(5)") values("DATE_SUB(\"$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC\", INTERVAL \"${LOG.HR}:${LOG.MIN}:${LOG.SEC}\" HOUR_SECOND)", "$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC", "${LOG.PROTOCOL}", "${LOG.IN_IP}", "${LOG.IN_PORT}", "${LOG.OUT_IP}", "${LOG.OUT_PORT}")); }; destination console_all { file("/dev/tty12"); }; log { source(s_xlate); filter(f_xlate_start); parser(p_xlate); destination(d_xlate_start); flags(final); }; log { source(s_xlate); filter(f_xlate_stop); parser(p_xlate); destination(d_xlate_stop); flags(final); }; log { source(src); destination(messages); }; log { source(src); destination(console_all); };

2 2

Syslog Blocked
by Tokie 07 Jan '10

07 Jan '10

Hi For All, i have a strange problem. On my server syslog (on Debian Etch) I installed syslog-ng and some times syslog-ng don't works!!! The programs is in execution, but don't receive the log in destination files; Logrotate makes the new file (ex. /var/log/syslog/new) and compress the last with gz; this works!! But the new file remains empty!! If I restart the syslog-ng with /etc/init.d/syslog-ng restart(like below at Jan 5 17:56:34) , it begins to write on new file!! This is my /var/log/messages > Jan 4 06:25:21 s_all(a)sys-server.tokie syslog-ng[2690]: SIGHUP > received, reloading configuration; > Jan 5 06:25:22 s_all(a)sys-server.tokie syslog-ng[2690]: SIGHUP > received, reloading configuration; > Jan 5 17:56:34 s_all(a)sys-server.tokie syslog-ng[2690]: SIGTERM > received, terminating; > Jan 5 17:56:34 s_all(a)sys-server.tokie syslog-ng[2690]: syslog-ng > shutting down; version='2.0.0' > Jan 5 17:56:35 s_all(a)sys-server.tokie syslog-ng[3111]: syslog-ng > starting up; version='2.0.0' Every day in cron should be restart syslog at 06:25 I think the problem is SIGHUP, but I don't know why!! This don't happen every day, but randomly!! Somebody can help me?? Tokie

3 3

AUTO: Shanif Lakhani is out of the office (returning 01/13/2010)
by Shanif Lakhani 05 Jan '10

05 Jan '10

I am out of the office until 01/13/2010. Please contact Robert Holoweckyj and/or Narendra Prasad for any urgent matters. Thanks Note: This is an automated response to your message "syslog-ng Digest, Vol 57, Issue 1" sent on 1/5/10 6:00:02. This is the only notification you will receive while this person is away.

1 0

program_override("kernel") not working in 3.0.4
by Paul B. Henson 05 Jan '10

05 Jan '10

I'm trying to upgrade from 2.1.3 to 3.0.4 under Linux, and am having trouble getting my kernel messages tagged appropriately. My previous configuration had the following: ----- source k_src { pipe("/proc/kmsg" log_prefix("kernel: ")); }; destination iptables { file("/var/log/iptables.log"); }; filter f_iptables { match("iptables:"); }; log { source(k_src); filter(f_iptables); destination(iptables); flags(final); }; destination messages { file("/var/log/messages"); }; log { source(src); source(k_src); destination(messages); }; ----- All of my kernel messages showed up with the "kernel:" prefix, and anything coming from iptables was dropped in a separate log. Based on the documentation, I modified my configuration to the following for 3.0.4: ----- source k_src { file("/proc/kmsg" program_override("kernel")); }; destination iptables { file("/var/log/iptables.log"); }; filter f_iptables { program("^kernel$") and message("^iptables:"); }; log { source(k_src); filter(f_iptables); destination(iptables); flags(final); }; destination messages { file("/var/log/messages"); }; log { source(src); source(k_src); destination(messages); }; ----- However, my kernel messages show up with no prefix. I tried starting up in debug mode: ----- syslog-ng starting up; version='3.0.4' Incoming log entry; line='<6>usb 2-7: USB disconnect, address 8' Filter rule evaluation begins; filter_rule='f_iptables' Filter node evaluation result; filter_result='not-match' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter rule evaluation result; filter_result='not-match', filter_rule='f_iptables' Initializing destination file writer; template='/var/log/messages', filename='/var/log/messages' Incoming log entry; line='<4>iptables: IN=eth0 OUT= MAC=0:05:00:10:97:43:00:0SC147.5.2 S=2... E=8TS00 RC0C T= D363POO2' Filter rule evaluation begins; filter_rule='f_iptables' Filter node evaluation result; filter_result='not-match' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter rule evaluation result; filter_result='not-match', filter_rule='f_iptables' ----- This didn't help me. Am I doing something wrong? I didn't find any complaints of this nature on the mailing list, which leads me to suspect I'm somehow being stupid, any pointers much appreciated. Thanks... -- Paul B. Henson | (909) 979-6361 | http://www.csupomona.edu/~henson/ Operating Systems and Network Analyst | henson(a)csupomona.edu California State Polytechnic University | Pomona CA 91768

2 2

Chained changing IP addresses to FQDN
by Jason Carr 28 Dec '09

28 Dec '09

Hello, I have a syslog infrastructure using syslog-ng comprising of three tiers: probes that accept messages and forwards them to the broker, a broker that routes and/or duplicates messages to the appropriate storage devices, and storage devices. All forward using the tcp("hostname"); as a destination. Basically it looks like this: machine1 ==udp==> +-------+ +--------+ +-----------+ machine2 ==udp==> | probe | ==tcp==> | broker | ==tcp==> | storage 1 | machine3 ==udp==> +-------+ +--------+ +-----------+ || +-----------+ ++=======tcp==> | storage 2 | +-----------+ The probes rewrite the hostname to the IP address that the packet came from using the chain_hostnames(no) and keep_hostname(no) options. One of the requirements of one specific storage device is to change the IP addresses into hostnames. I'm having difficulty on the storage device switching the IP address in the log message to the FQDN. I have tried all combinations I can think of of chain_hostnames and keep_hostname and cannot seem to get the IP address replaced with the FQDN. Is there a way to replace the IP address with the FQDN while receiving the messages from an intermediary? Thank you, Jason

4 5

troubleshooting and debugging syslog-ng
by Zoltán Pallagi 23 Dec '09

23 Dec '09

Hi, I've sent a new post about it to my blog (http://pzolee.blogs.balabit.com/2009/12/troubleshooting-and-debugging-syslo…) I tried to collect all important things about this issue. I hope this post will make it easier for you to specify the reason of the possible syslog-ng problems and for us to help you. pzolee

1 0

time_sleep in high udp environment?
by Tim Rupp 22 Dec '09

22 Dec '09

Hi, we were debugging message drops on our syslog-ng server (rpm syslog-ng-3.0.5-1.rhel5.x86_64) and found out that if we commented out the time_sleep global option we had set (value of 20) that all the drops went away. Initially we had set this value while reading through the pdf documentation (page 136) figuring that since we had potentially a lot of traffic that this would boost performance. So I guess I'd like to know why we seemed to see the opposite behavior with this option enabled. We don't really do tcp connections, so maybe this option was useless to begin with. Anyways, it made me curious, so if anyone can enlighten me I'd appreciate it. Thanks, Tim

2 1

pdbtool pattern debuging
by ILLES, Marton 22 Dec '09

22 Dec '09

Hi! I have just pushed to my syslog-ng 3.1 git repo some patches to add more detailed debugging/troubleshooting capabilities to patterndb through pdbtool match. Now you can easily see how your patterns matched a given message, which part of the message was matched by literal part of the pattern and which by a parser. All this new feature with a wonderful colorized output if requested, though the colors are still bit ugly... Up to now the easiest way of creating patterns was to do trial & error sequence and try to figure out which part of the pattern was broken. Now you can exactly see where the matching stopped. It is also possible to output the match in parse-able format to be used in scripts or as a backend of some pattern authoring tool. I am still not sure about the colored output nor about the machine parse-able output format, so any feedback, comment, idea or suggestion is very much appreciated. You can see some screen-shots of pdbtool match (with colors) in action on my blog: http://marci.blogs.balabit.com/2009/12/pdbtool-stylists-wanted.html The code is available in my git repo: http://git.balabit.hu/?p=marci/syslog-ng-3.1.git;a=commit;h=c7c3b7b81577507… I did not update the man page so far, but --help works. You should be using the "-D" to enable debuging and "-c" for color output, while "-C" for csv style output. Happy matching! Marton -- Key fingerprint = F78C 25CA 5F88 6FAF EA21 779D 3279 9F9E 1155 670D

1 0

Memory Exhaust
by blizzards＠libero.it 21 Dec '09

21 Dec '09

Hi all. I'm using syslog-ng 3.0.5. I have a very large configuration, more than 5000 lines, at my site. I write non only to a plain text file, but i redirect all traffic to another syslog-ng server with 10 templetas with udp socket. If i start the service, syslog-ng give me this error: Memory Exhaust at line 3666. If i cut off my configuration at line 3665, the service start correctly. Someone can help me with this issue? Maybe memory leak bug? Gix.

2 1

Sending TCP messages to Syslog-NG server
by James Pirman 18 Dec '09

18 Dec '09

I am having an issue sending TCP messages to a SysLog-NG server running version 3.0.4. For some reason when multiple messages are put into 1 tcp packet, the server queues part of the second packet and then spits it out when it receives the another new message. Has anyone seen this before? Any help would be appreciated. -Jim _________________________________________________________________ Hotmail: Powerful Free email with security by Microsoft. http://clk.atdmt.com/GBL/go/171222986/direct/01/

1 1

Pattern Database first snapshot available
by ILLES, Marton 18 Dec '09

18 Dec '09

Hi, Last week BalaBit made available some 8000 patterns (covering more than 200 applications) for syslog-ng patterndb (or db_parser as you like to call it). The patterns are available under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 (CC by-NC-SA) license. The patterns in their current form are just snapshots of the ongoing effort of providing good quality patterns for various applications. You can download the snapshot of patterns from our website: http://www.balabit.com/downloads/files/patterndb-snapshot/patterndb-2009120… The patterns are partially hand-crafted and also automatically generated from logfiles and from logcheck regexp based database. Some of the patterns also contains example messages which we are using to automatically test the pattern and syslog-ng's db_parser. You can merge the xml files using "pdbtool merge". I would also like to setup a public git repository where anyone interested can follow the patterndb development and can submit patterns or fixes. A patterndb website containing all patterndb related information, links, forums, wikis and other useful documentations is under construction as well. Till than the syslog-ng mailing list a good place for questions, ideas and discussions. As always feedbacks are very welcomed! Happy parsing! Marton -- Key fingerprint = F78C 25CA 5F88 6FAF EA21 779D 3279 9F9E 1155 670D

3 5

syslog-ng OSE 3.1beta2 released
by Balazs Scheidler 18 Dec '09

18 Dec '09

Dear syslog-ng users, A new beta version of the current syslog-ng feature release has been uploaded to our website. https://www.balabit.com/network-security/syslog-ng/opensource-logging-syste… This is a bugfix only release containing blocker issues found in the past three weeks. Please see the NEWS entry below for more details. Any kind of feedback, be it positive or negative is appreciated. 3.1beta2 Fri, 18 Dec 2009 09:19:39 +0100 Bugfixes: * Fixed two major memory leaks, one on input messages, one in patterndb. * Fixed escaped structured data processing. * Fixed expanding SDATA references in templates which have multiple dots in its name (e.g. when an OID is present in the name of the SD param or SD-ID. * pdbtool now correctly zero pads month and day fields in publish date of the merged patterndb file. * pdbtool won't link to unnecessary libraries, which fixes a compilation problem on Solaris with flex installed. -- Bazsi

1 0

new blog about syslog-ng and software testing
by Zoltán Pallagi 17 Dec '09

17 Dec '09

Hi all, I would like you to know that I've created a new blog about syslog-ng, and software testing. http://pzolee.blogs.balabit.com I hope that you will find some useful and interesting informations here -- pzolee

1 0

Error in FreeBSD jail
by Peter Fraser 17 Dec '09

17 Dec '09

Hi All I installed syslog-ng3-3.0.3_3 in a jail, when I start the service, I get this error below Starting syslog_ng. Error binding socket; addr='AF_UNIX(/var/run/log)', error='Permission denied (13)' Error initializing source driver; source='src', id='src#0' Error initializing message pipeline; /usr/local/etc/rc.d/syslog-ng: WARNING: failed to start syslog_ng Does anyone have any ideas on what I could do to troubleshoot or resolve this problem? Thanks for any help.

2 5

supervise/syslog-ng exits
by Kārlis Repsons 17 Dec '09

17 Dec '09

People, I wonder why is there some two messages like this in errors log each day: supervise/syslog-ng[28806]: Daemon exited gracefully, not restarting; exitcode='0' ? What are the possible reasons and what should be done?

6 12

3.03 filtering problem
by Tim Boyer 16 Dec '09

16 Dec '09

OK, I assume that this is something stupid I'm doing - but I've started at the config file for a week, and I'll be darned if I can see it. I've got a fairly vanilla setup, and I'm trying to send dhcpd, snort, etc to separate filters, while at the same time keeping one file with all of the error messages. Like so: # # log sources # source server { file("/proc/kmsg"); unix-stream("/dev/log"); internal(); }; destination dserver_all { file("/var/log/${HOST}.log" owner(root) group(hobbit) perm(0640)); }; destination dserver_cfengine { file("/var/log/cfengine.log"); }; destination dserver_snort { file("/var/log/snort.log"); }; destination dserver_dhcpd { file("/var/log/dhcpd.log"); }; ... filter fserver_snort { program("snort"); }; filter fserver_dhcpd { program("dhcpd"); }; filter fserver_cfengine { program("cfengine"); }; ... log { source(server); destination(dserver_all); }; log { source(server); filter(fserver_snort); destination(dserver_snort); flags(final); }; log { source(server); filter(fserver_cfengine); destination(dserver_cfengine); flags(final); }; log { source(server); filter(fserver_dhcpd); destination(dserver_dhcpd); flags(final); }; The 'log everything' is working fine. Some of the filtered logs are working fine, like the cfengine log: [root@saratoga log]# ls -la /var/log/cfengine.log -rw------- 1 root root 181006 Dec 14 11:15 /var/log/cfengine.log The dhcpd log is empty: [root@saratoga log]# ls -la /var/log/dhcpd.log -rw------- 1 root root 0 Dec 13 02:42 /var/log/dhcpd.log even though there are entries in the combined log that clearly should have hit the dhcpd log: [root@saratoga log]# tail /var/log/saratoga.log Dec 14 11:25:07 saratoga dhcpd: Copyright 2004-2006 Internet Systems Consortium. Dec 14 11:25:07 saratoga dhcpd: All rights reserved. Dec 14 11:25:07 saratoga dhcpd: For info, please visit http://www.isc.org/sw/dhcp/ Dec 14 11:25:07 saratoga dhcpd: Wrote 22 leases to leases file. Dec 14 11:25:07 saratoga dhcpd: Listening on LPF/eth3/00:0e:0c:6d:64:cf/192.168.3/24 What in the world am I missing? Thanks much, -- tim -- -- Tim Boyer Denman Tire Corporation

1 1

3.1 Beta 1 memory leak?
by Martin Holste 16 Dec '09

16 Dec '09

I'm trying the new 3.1 beta 1 build, and even with log_fifo_size(0), syslog-ng consumes all available memory until the kernel out-of-memory manager kills it. At first I thought this had something to do with UDP forwarding to a destination, but even with that disabled, the uncontrolled consumption occurs. I reconfigured with the most basic configuration to just dump everything to /dev/null, and though the leak is slower, the leak continues. Compiled with: Glibc-2.20.3 PCRE 7.9 syslog-ng 3.1beta1 Installer-Version: 3.1beta1 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.1#master#3412dce0235e8d65bc3bebedffe8ff6bed02f45b Compile-Date: Dec 11 2009 11:36:51 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-Sun-Door: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: on Configuration: @version: 3.0 options { ts_format(iso); stats_level(1); stats_freq(30); keep_hostname(no); chain_hostnames(yes); create_dirs(yes); }; source s_tcp { tcp(ip(0.0.0.0) port(514)); }; source s_udp { udp(ip(0.0.0.0) port(514)); }; destination d_null { file("/dev/null"); }; log { source(s_udp); source(s_tcp); destination(d_null); };

2 2

syslog-ng OSE 3.0.4 coredumps when use "internal()" in s_local
by nova club 15 Dec '09

15 Dec '09

Hi there, I am a new user of syslog-ng. Thanks for your time and attention in advance. The problem is syslog-ng coredumps when use "internal()" in "source", however it will work fine without "internal()". The conf file is like: ===>BEGIN @version:3.0 options { dir_perm(0755); perm(0644); chain_hostnames(no); keep_hostname(yes); }; # Default local source. source local { unix-dgram("/var/run/log"); unix-dgram("/var/run/logpriv" perm(0600)); internal(); file("/dev/klog"); }; ... filter f_hecxd { program("hecxd"); }; ... log { source(local); filter(f_hecxd); destination(d_hecxd); }; ==>END The options used for building syslog-ng is: CFLAGS='-Wl,-Bdynamic -liconv -Wl,-Bstatic' --enable-static-linking \ --enable-debug \ --enable-ssl --enable-ipv6=no --enable-tcp-wrapper=no \ --with-pidfile-dir="/var/run/" The platform is: FREEBSD 6.4-RELEASE-p6 i386 I got a very big core dump file (>500M), by backtracing that file, I can see the problem is "g_malloc0" for some reason failed to allocate memory, while syslog-ng did not check the return value and tried to use the "0x0" pointer in a "strlen()" call later. Here is a part of trace information: #0 0x080a2d8d in strlen () No symbol table info available. #1 0x0804e7fe in log_msg_new_internal (prio=43, msg=0x0, flags=6) at logmsg.c:1971 buf = (gchar *) 0x0 self = (LogMessage *) 0x8159000 #2 0x080492b1 in msg_send_internal_message (prio=43, msg=0x0) at messages.c:128 context = (MsgContext *) 0x81893a0 m = (LogMessage *) 0x7 #3 0x0804954c in msg_log_func (log_domain=0x81186ea "GLib", log_flags=6, msg=0x0, user_data=0x0) at messages.c:187 pri = 43 #4 0x08084906 in IA__g_logv (log_domain=0x81186ea "GLib", log_level=G_LOG_LEVEL_ERROR, format=0x811cbac "%s: failed to allocate %u bytes", args1=0xbfbfe6cc "3�\021\bl") at gmessages.c:519 msg = (gchar *) 0x0 was_fatal = 0 was_recursion = 0 i = 2 #5 0x08084ae4 in IA__g_log (log_domain=0x81186ea "GLib", log_level=G_LOG_LEVEL_ERROR, format=0x811cbac "%s: failed to allocate %u bytes") at gmessages.c:569 No locals. #6 0x08084239 in IA__g_malloc0 (n_bytes=108) at gmem.c:155 mem = 0x0 #7 0x0804e5ef in log_msg_new ( msg=0x8a82180 "<47> syslog-ng[3277]: Filter node evaluation result; filter_result='not-match', filter_type='level'\n", length=100, saddr=0x0, flags=6, bad_hostname=0x0, assume_timezone=-1, default_pri=47) at logmsg.c:1873 self = (LogMessage *) 0x810ba1b #8 0x0804e80a in log_msg_new_internal (prio=47, msg=0x817f500 "Filter node evaluation result; filter_result='not-match', filter_type='level'", flags=6) at logmsg.c:1971 buf = (gchar *) 0x8a82180 "<47> syslog-ng[3277]: Filter node evaluation result; filter_result='not-match', filter_type='level'\n" self = (LogMessage *) 0x8159000 #9 0x080492b1 in msg_send_internal_message (prio=47, msg=0x817f500 "Filter node evaluation result; filter_result='not-match', filter_type='level'") at messages.c:128 context = (MsgContext *) 0x81893a0 m = (LogMessage *) 0x8182030 If you need the completed coredump file, just let me know and tell me where to put it :) Any help is much appreciated. Thanks a lot. Shen _________________________________________________________________ Windows Live: Make it easier for your friends to see what you’re up to on Facebook. http://go.microsoft.com/?linkid=9691816

2 4