- syslog-ng - lists.balabit.hu

JSON vs PatternDB for parser performance
by Martin Holste 20 Jan '12

20 Jan '12

Does anyone have a good idea as to which parser is fastest? The pattern matching tree of PatternDB makes me think that it will be faster, but I'd like to see what others think. Testing in my environment would be non-trivial, so I am hoping someone already has. Thanks, Martin

2 3

Error on start of Syslog-NG
by Michael Kolowicz 19 Jan '12

19 Jan '12

I using Syslog-NG on 3.2.6 on Ubuntu 10.04 If I start / reload syslog-ng I see following error: Reloading syslog-ng's config file: WARNING: Configuration file format is too old, please update it to use the 3.2 format as some constructs might operate inefficiently; WARNING: the expected message format is being changed for unix-domain transports to improve syslogd compatibity with syslog-ng 3.2. If you are using custom applications which bypass the syslog() API, you might need the 'expect-hostname' flag to get the old behaviour back; * My Configuration is: @version: 3.2 #ADM_NTINST @ 09:41 13.01.2012 source s_local {internal(); unix-stream("/dev/log");}; source s_udp {udp(port(514));}; destination d_SYSLOG { file("/var/log/syslog");}; destination SUA002254_localhost_SYS_SYSLOG_local {program("/usr/bin/mysql --host=localhost --user=USR_SYS_SYSLOG --password=pw_Syslog SYS_SYSLOG" template("INSERT INTO local (host, source_ip,facility, priority, tag, unixtime,program, msg) VALUES ('$HOST','$SOURCEIP','$FACILITY','$PRIORITY','$TAG','$UNIXTIME','$PROGRAM',' $MSGONLY' );\n") template-escape(yes));}; destination d_udp1514 {udp("192.168.2.108" port (1514); keep-hostname(););}; destination SUA002254_localhost_SYS_SYSLOG_uaagent {sql( type(mysql) host("localhost") username("USR_SYS_SYSLOG") password("pw_Syslog") database("SYS_SYSLOG") table("uaagent") columns("host", "source_ip", "facility", "priority", "tag", "unixtime", "program", "msg") values("$HOST","$SOURCEIP","$FACILITY","$PRIORITY","$TAG", "$UNIXTIME","$PROGRAM","$MSGONLY") indexes("host", "source_ip", "facility", "priority", "tag", "unixtime", "program", "msg"));}; #log_local log { source(s_local); destination(d_SYSLOG); destination(SUA002254_localhost_SYS_SYSLOG_local); }; #log_uaagent log { source(s_udp); destination(d_udp1514); destination(SUA002254_localhost_SYS_SYSLOG_uaagent); destination(d_SYSLOG); }; Freundliche Grüße / Best regards

3 2

Forwarding Syslog-ng Messages
by Michael Kolowicz 19 Jan '12

19 Jan '12

Hello, I want to forward incomming messages to a "LiveMonitor"-Application which is running on a remote machine and listening on UDP-Port 1514. I have create a destination "{udp("192.168.2.122" port (1514));}" but my Monitor-Application didn´t recieve messages (I using Syslog-ng 3.2.5 on ubuntu 10.04) If I sending direct to my application then I see the messages - is my destination configuration wrong? Here my Configuration source s_udp {udp(port(514));}; destination d_udp1514 {udp("192.168.2.108" port (1514); keep-hostname(););}; destination d_SYSLOG { file("/var/log/syslog");}; log { source(s_udp); destination(d_udp1514); destination(d_SYSLOG); }; I have run Whireshark on the System with the application and I see no incoming messages The the log messages are being written to /var/log/syslog? Freundliche Grüße / Best regards

2 1

Syntax error on threading(yes)
by John Morrissey 18 Jan '12

18 Jan '12

syslog-ng 3.3 snapshots as recent as 9 January yield: -- Error parsing config, syntax error, unexpected LL_IDENTIFIER, expecting '}' in /etc/syslog-ng/options.conf at line 6, column 2: included from /etc/syslog-ng/syslog-ng.conf line 5, column 23 threading(yes); ^^^^^^^^^ -- even with an options statement as simple as: options { threading(yes); }; syslog-ng is linked against libpthread and libgthread, so that much seems to be in order. Is there something else I'm missing? john -- John Morrissey _o /\ ---- __o jwm(a)horde.net _-< \_ / \ ---- < \, www.horde.net/ __(_)/_(_)________/ \_______(_) /_(_)__

2 2

Re: [syslog-ng] Error on start of Syslog-NG
by Michael Kolowicz 18 Jan '12

18 Jan '12

> Try changing that unix-stream to unix-dgram, and see if the warning goes > away. No - the "Message" still come up What I not understand: Why Syslog-ng mean, that the format of file is not right. I have a statement with the version-number @version: 3.2

1 0

No logging on Snow Leopard except internal
by Andrew Eberbach 17 Jan '12

17 Jan '12

Hi I'm trying to get syslog-ng working on snow leopard but when it starts all I get is internal messages. Using logger to send a message won't work nor will using another program. Both work with normal syslogd (which I have turned off so as not to conflict). I've also tried loggen -D localhost 514 and loggen -i localhost 514 to no avail. Here's the details: #/usr/local/Cellar/syslog-ng/3.2.4/sbin/syslog-ng -V syslog-ng 3.2.4 Installer-Version: 3.2.4 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#ef7b91e4a1b1f9628c66138b4ae83de7e4c697c6 Compile-Date: Jan 16 2012 05:44:25 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off and here's the config #cat syslog-ng.conf ############################################################################# # Default syslog-ng.conf file which collects all local logs into a # single file called /var/log/messages. # @version: 3.2 @include "scl.conf" source s_local { system(); internal(); }; source s_network { udp(); }; destination d_local { file("/var/log/messages"); }; log { source(s_local); # uncomment this line to open port 514 to receive messages source(s_network); destination(d_local); }; I'm sure there's something stupid that I'm missing but I haven't been able to find it. Any thoughts? Thanks, Andrew

3 5

syslog-ng.ctl socket fd leak in 3.3.3
by Jakub Jankowski 17 Jan '12

17 Jan '12

Hi, can anyone confirm my observations? $ lsof -n -p $(cat /var/run/syslog-ng.pid) | grep -c /var/lib/syslog-ng/syslog-ng.ctl 129 $ syslog-ng-ctl stats >/dev/null; echo $? 0 $ lsof -n -p $(cat /var/run/syslog-ng.pid) | grep -c /var/lib/syslog-ng/syslog-ng.ctl 130 $ I'm running a cronjob that dumps stats every minute, so the number of open fds is growing linearly, soon reaching configured open files limits. I'm using self-compiled 3.3.3: # syslog-ng -V syslog-ng 3.3.3 Installer-Version: 3.3.3 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.3#master#d199a1980be6b23fe24189e86a882812288e292c Compile-Date: Nov 18 2011 00:16:55 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat,afsql Available-Modules: afprog,syslogformat,convertfuncs,afuser,afsocket,afsocket-tls,csvparser,confgen,afsql,afmongodb,dummy,dbparser,affile,basicfuncs Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: off Am I the only one seeing this? Regards, -- Jakub Jankowski|shasta@toxcorp.com|http://toxcorp.com/ GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D

3 5

tuning syslog-ng 3.3.3
by Johnson, Chris (HP TippingPoint Roseville) 17 Jan '12

17 Jan '12

I'm trying to tune syslog-ng 3.3.3. My first attempt is to use the log_fetch_limit parameter, but I'm running into the problem of where to define it! The Admin guide says I can put it in the global options{}, but that generates and error saying it has been taken out of the global area and needs to be specified by the source and I can't seem to find the correct source syntax. My config (pared down): @version: 3.3 @include "scl.conf" options { # log_fetch_limit(100); }; source s_local { system(); internal(); }; ############################################################################# # Service all filter f_all_pgm_01{program("*" type("glob"));}; filter f_all_lvl_01.01{level(info..emerg)}; destination d_all_01{file("/var/log/system.log" suppress(30));}; log { source(s_local); filter(f_all_pgm_01); filter(f_all_lvl_01.01); destination(d_all_01); }; I've tried putting it in: source s_local{ system( log_fetch_limit(100); ); ...}; source s_local{ system(); ... log_fetch_limit(100); }; log{ source(s_local log_fetch_limit(100);); ... }; each generate a 'syslog-ng[1567]: Error parsing configuration;' error. Any ideas on *where* I should put it? Thanks, Chris ---------------------------------------- Christopher Johnson chris.johnson3(a)hp.com<mailto:chris.johnson3@hp.com> HP Software - Security Product Group (916) 785-2817 ----------------------------------------

5 6

Centos 6 Syslog-ng
by Pigri 16 Jan '12

16 Jan '12

Hello! I have problem for Centos 6.Rebuild syslog-ng package for epel repo, because not have SQL support. But i have this problem: Error: The number of columns and values do not match; len_columns='10', len_values='11' Error initializing dest driver; dest='d_apache_mysql', id='d_apache_mysql#0' Error initializing message pipeline; Config: parser p_apache { csv-parser( columns( "APACHE.CLIENT_IP", "APACHE.IDENT_NAME", "APACHE.USER_NAME", "APACHE.TIMESTAMP", "APACHE.REQUEST_URL", "APACHE.REQUEST_STATUS", "APACHE.CONTENT_LENGTH", "APACHE.REFERER", "APACHE.USER_AGENT", "APACHE.VIRTUAL_HOST", "APACHE.SERVER_NAME") flags(escape-none,escape-backslash,escape-double-char,escape-double-char,strip-whitespace) delimiters(" ") quote-pairs('""[]') ); }; destination d_apache_mysql { sql(type(mysql) host("localhost") username("root") password("xxxxx") database("data") table("accesslog") columns("1", "2", "3", "4", "5", "6", "7", "8","9","10") values("$APACHE.CLIENT_IP","$APACHE.IDENT_NAME","$APACHE.USER_NAME","$APACHE.TIMESTAMP","$APACHE.REQUEST_URL","$APACHE.REQUEST_STATUS","$APACHE.CONTENT_LENGTH","$APACHE.REFERER","$APACHE.USER_AGENT","$APACHE.VIRTUAL_HOST","$APACHE.SERVER_NAME") indexes("1", "2", "3", "4", "5", "6", "7", "8","9","10")); }; destination d_apache_me { file("/var/log/test.me.log"); }; filter f_apache_edata { facility(local6) and program(test.me); }; log { source(s_sys); parser(p_apache); filter(f_apache_edata); destination(d_apache_mysql); }; Version: LIBDBI: Version : 0.8.3 Syslog-ng: syslog-ng 3.2.5 Installer-Version: 3.2.5 Revision: ssh+git://bazsi@git.balabit //var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#9d4bea28198bd731df1a61e980a2af5b88d81116 Compile-Date: Jan 8 2012 22:19:13 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: on Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off Strace: http://dl.dropbox.com/u/4683046/debug Thank you for all. Best regards, pigri

3 3

trouble with chain_hostnames
by Matt Van Mater 16 Jan '12

16 Jan '12

Hi all, Many thanks to the developers, I've been a user for many years and have run into a deployment that I can't quite get working the way I want to. Please correct me if I am wrong, but the chain_hostnames global option is intended to append the current syslog-ng hostname to every message processed, no matter how many hosts a message chains through? My messages are being relayed through these hosts: 1) webserver: Ubuntu LTS 10.4.3, syslog-ng 2.0.9 (from standard apt-get sources) 2) proxy: OpenBSD 5.0, syslog-ng 3.1.4 (from prebuilt packages) 3) logserver: Ubuntu LTS 10.4.3, syslog-ng 2.0.9 (from standard apt-get sources) The traffic flows through them this way: webserver --> proxy (tcp 514) --> proxy (tcp 1514) --> ssh tunnel --> logserver (tcp 514) I have chain_hostnames(yes) in the options global variable section all three syslog-ng.conf files, and messages relay through all hosts properly EXCEPT that the proxy server does not seem to append its hostname or IP address to the syslog message when forwarding it on to the logserver. I have experimented with keep_hostnames as both yes and no on all hosts and have not seen what i am expecting... (example: timestamp s_all@webserver_ip/proxy_ip/logserver_ip actual_log_message) An example of what i see today is this: On webserver # echo "webserver sending message to localhost tcp 514" | nc localhsot 514 On loghost# tail -n1 /var/log/messages Jan 4 17:22:16 webserver/webserver/loghost_ip webserver sending message to localhost tcp 514 Note the fact that the message gets transported through all 3 hosts but the proxy hostname isn't being inserted into the chain. So my questions are: 1) Is chain_hostnames supposed to append the hostname to the message, no matter how many hosts a message is relayed through? 1a) Suggestion: can you update the FAQ here: http://www.campin.net/syslog-ng/faq.html#hostname to include an example of a log message relayed through multiple hosts and the expected behavior 2) Is there a known bug in syslog-ng v3.1.4 and / or OpenBSD's example syslog-ng.conf file that is the cause of hostname chaining not working? 2a) Suggestion: can you publish the syslog-ng changelogs in a single file appending changes, or a searchable database? As far as I can tell, I currently have to download the changelog for each individual syslog-ng version released in order to search and see if an issue was released and that is a huge PITA. Thanks! Matt

3 2