January 2012 - syslog-ng - lists.balabit.hu

Re: [syslog-ng] Error on start of Syslog-NG
by Michael Kolowicz 18 Jan '12

18 Jan '12

> Try changing that unix-stream to unix-dgram, and see if the warning goes > away. No - the "Message" still come up What I not understand: Why Syslog-ng mean, that the format of file is not right. I have a statement with the version-number @version: 3.2

1 0

No logging on Snow Leopard except internal
by Andrew Eberbach 17 Jan '12

17 Jan '12

Hi I'm trying to get syslog-ng working on snow leopard but when it starts all I get is internal messages. Using logger to send a message won't work nor will using another program. Both work with normal syslogd (which I have turned off so as not to conflict). I've also tried loggen -D localhost 514 and loggen -i localhost 514 to no avail. Here's the details: #/usr/local/Cellar/syslog-ng/3.2.4/sbin/syslog-ng -V syslog-ng 3.2.4 Installer-Version: 3.2.4 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#ef7b91e4a1b1f9628c66138b4ae83de7e4c697c6 Compile-Date: Jan 16 2012 05:44:25 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off and here's the config #cat syslog-ng.conf ############################################################################# # Default syslog-ng.conf file which collects all local logs into a # single file called /var/log/messages. # @version: 3.2 @include "scl.conf" source s_local { system(); internal(); }; source s_network { udp(); }; destination d_local { file("/var/log/messages"); }; log { source(s_local); # uncomment this line to open port 514 to receive messages source(s_network); destination(d_local); }; I'm sure there's something stupid that I'm missing but I haven't been able to find it. Any thoughts? Thanks, Andrew

3 5

syslog-ng.ctl socket fd leak in 3.3.3
by Jakub Jankowski 17 Jan '12

17 Jan '12

Hi, can anyone confirm my observations? $ lsof -n -p $(cat /var/run/syslog-ng.pid) | grep -c /var/lib/syslog-ng/syslog-ng.ctl 129 $ syslog-ng-ctl stats >/dev/null; echo $? 0 $ lsof -n -p $(cat /var/run/syslog-ng.pid) | grep -c /var/lib/syslog-ng/syslog-ng.ctl 130 $ I'm running a cronjob that dumps stats every minute, so the number of open fds is growing linearly, soon reaching configured open files limits. I'm using self-compiled 3.3.3: # syslog-ng -V syslog-ng 3.3.3 Installer-Version: 3.3.3 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.3#master#d199a1980be6b23fe24189e86a882812288e292c Compile-Date: Nov 18 2011 00:16:55 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat,afsql Available-Modules: afprog,syslogformat,convertfuncs,afuser,afsocket,afsocket-tls,csvparser,confgen,afsql,afmongodb,dummy,dbparser,affile,basicfuncs Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: off Am I the only one seeing this? Regards, -- Jakub Jankowski|shasta@toxcorp.com|http://toxcorp.com/ GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D

3 5

tuning syslog-ng 3.3.3
by Johnson, Chris (HP TippingPoint Roseville) 17 Jan '12

17 Jan '12

I'm trying to tune syslog-ng 3.3.3. My first attempt is to use the log_fetch_limit parameter, but I'm running into the problem of where to define it! The Admin guide says I can put it in the global options{}, but that generates and error saying it has been taken out of the global area and needs to be specified by the source and I can't seem to find the correct source syntax. My config (pared down): @version: 3.3 @include "scl.conf" options { # log_fetch_limit(100); }; source s_local { system(); internal(); }; ############################################################################# # Service all filter f_all_pgm_01{program("*" type("glob"));}; filter f_all_lvl_01.01{level(info..emerg)}; destination d_all_01{file("/var/log/system.log" suppress(30));}; log { source(s_local); filter(f_all_pgm_01); filter(f_all_lvl_01.01); destination(d_all_01); }; I've tried putting it in: source s_local{ system( log_fetch_limit(100); ); ...}; source s_local{ system(); ... log_fetch_limit(100); }; log{ source(s_local log_fetch_limit(100);); ... }; each generate a 'syslog-ng[1567]: Error parsing configuration;' error. Any ideas on *where* I should put it? Thanks, Chris ---------------------------------------- Christopher Johnson chris.johnson3(a)hp.com<mailto:chris.johnson3@hp.com> HP Software - Security Product Group (916) 785-2817 ----------------------------------------

5 6

Centos 6 Syslog-ng
by Pigri 16 Jan '12

16 Jan '12

Hello! I have problem for Centos 6.Rebuild syslog-ng package for epel repo, because not have SQL support. But i have this problem: Error: The number of columns and values do not match; len_columns='10', len_values='11' Error initializing dest driver; dest='d_apache_mysql', id='d_apache_mysql#0' Error initializing message pipeline; Config: parser p_apache { csv-parser( columns( "APACHE.CLIENT_IP", "APACHE.IDENT_NAME", "APACHE.USER_NAME", "APACHE.TIMESTAMP", "APACHE.REQUEST_URL", "APACHE.REQUEST_STATUS", "APACHE.CONTENT_LENGTH", "APACHE.REFERER", "APACHE.USER_AGENT", "APACHE.VIRTUAL_HOST", "APACHE.SERVER_NAME") flags(escape-none,escape-backslash,escape-double-char,escape-double-char,strip-whitespace) delimiters(" ") quote-pairs('""[]') ); }; destination d_apache_mysql { sql(type(mysql) host("localhost") username("root") password("xxxxx") database("data") table("accesslog") columns("1", "2", "3", "4", "5", "6", "7", "8","9","10") values("$APACHE.CLIENT_IP","$APACHE.IDENT_NAME","$APACHE.USER_NAME","$APACHE.TIMESTAMP","$APACHE.REQUEST_URL","$APACHE.REQUEST_STATUS","$APACHE.CONTENT_LENGTH","$APACHE.REFERER","$APACHE.USER_AGENT","$APACHE.VIRTUAL_HOST","$APACHE.SERVER_NAME") indexes("1", "2", "3", "4", "5", "6", "7", "8","9","10")); }; destination d_apache_me { file("/var/log/test.me.log"); }; filter f_apache_edata { facility(local6) and program(test.me); }; log { source(s_sys); parser(p_apache); filter(f_apache_edata); destination(d_apache_mysql); }; Version: LIBDBI: Version : 0.8.3 Syslog-ng: syslog-ng 3.2.5 Installer-Version: 3.2.5 Revision: ssh+git://bazsi@git.balabit //var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#9d4bea28198bd731df1a61e980a2af5b88d81116 Compile-Date: Jan 8 2012 22:19:13 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: on Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off Strace: http://dl.dropbox.com/u/4683046/debug Thank you for all. Best regards, pigri

3 3

trouble with chain_hostnames
by Matt Van Mater 16 Jan '12

16 Jan '12

Hi all, Many thanks to the developers, I've been a user for many years and have run into a deployment that I can't quite get working the way I want to. Please correct me if I am wrong, but the chain_hostnames global option is intended to append the current syslog-ng hostname to every message processed, no matter how many hosts a message chains through? My messages are being relayed through these hosts: 1) webserver: Ubuntu LTS 10.4.3, syslog-ng 2.0.9 (from standard apt-get sources) 2) proxy: OpenBSD 5.0, syslog-ng 3.1.4 (from prebuilt packages) 3) logserver: Ubuntu LTS 10.4.3, syslog-ng 2.0.9 (from standard apt-get sources) The traffic flows through them this way: webserver --> proxy (tcp 514) --> proxy (tcp 1514) --> ssh tunnel --> logserver (tcp 514) I have chain_hostnames(yes) in the options global variable section all three syslog-ng.conf files, and messages relay through all hosts properly EXCEPT that the proxy server does not seem to append its hostname or IP address to the syslog message when forwarding it on to the logserver. I have experimented with keep_hostnames as both yes and no on all hosts and have not seen what i am expecting... (example: timestamp s_all@webserver_ip/proxy_ip/logserver_ip actual_log_message) An example of what i see today is this: On webserver # echo "webserver sending message to localhost tcp 514" | nc localhsot 514 On loghost# tail -n1 /var/log/messages Jan 4 17:22:16 webserver/webserver/loghost_ip webserver sending message to localhost tcp 514 Note the fact that the message gets transported through all 3 hosts but the proxy hostname isn't being inserted into the chain. So my questions are: 1) Is chain_hostnames supposed to append the hostname to the message, no matter how many hosts a message is relayed through? 1a) Suggestion: can you update the FAQ here: http://www.campin.net/syslog-ng/faq.html#hostname to include an example of a log message relayed through multiple hosts and the expected behavior 2) Is there a known bug in syslog-ng v3.1.4 and / or OpenBSD's example syslog-ng.conf file that is the cause of hostname chaining not working? 2a) Suggestion: can you publish the syslog-ng changelogs in a single file appending changes, or a searchable database? As far as I can tell, I currently have to download the changelog for each individual syslog-ng version released in order to search and see if an issue was released and that is a huge PITA. Thanks! Matt

3 2

[PATCH (3.4) 0/4]: json-parser updates
by Gergely Nagy 16 Jan '12

16 Jan '12

Following this mail, a couple of patches will come - they were written in last November, but I don't think I posted them (as I didn't expect to see the json-parser merged so soon). Until now, they were sitting on my feature/3.4/json/parser branch, and that's where they sit now, too. I rebased that branch onto 3.4 master + center.c syntax error fixes (which I posted a week or two ago), and am sending the json parser patches now. They accomplish a few things: it makes the parser handle invalid JSON input properly (by returning an error, instead of crashing); adds support for parsing boolean, array and nested object types. Booleans will get parsed into TRUE or FALSE, nested objects will be turned into dotted notation (which also means that keys with a dot in them won't be handled properly by format-json, once it is updated to support reconstructing nested objects from dotted notation), and arrays will be treated like nested objects, with the array indexes used as a key. As an example, lets look at a JSON input: {"is-example": true, "hats": ["top hat", "baseball cap", "pointy witch-hat"], "auth": { "method": ["looking angry at the security guy", "bribery"], "result": "did not work" } } This would be parsed into the following name-value paris: "is-example"="TRUE" "hats.0"="top hat" "hats.1"="baseball cap" "hats.2"="pointy witch-hat" "auth.method.0"="looking angry at the security guy" "auth.method.1"="bribery" "auth.result"="did not work" The other part of this will be an enhahcement to value-pairs and format-json, that will make syslog-ng able to reconstruct the original JSON (more or less... booleans probably won't be supported, except perhaps optionally with a flag). But that is something I haven't started working on yet.

2 6

Hello, and project for Win32 port
by Ramón García Fernández 16 Jan '12

16 Jan '12

Hello, I am Ramón García Fernández and I work for the IT department of the Comisión Nacional de Mercado de Valores, the Spanish regulator of securities. Would you be interested in an opensource Win32 port of syslog-ng? Best regards, Aviso legal – Comisión Nacional del Mercado de Valores Este mensaje y, en su caso, los ficheros que lleve incorporados, está dirigido exclusivamente a su destinatario y es de carácter confidencial. Si fuere recibido por error o se tuviere conocimiento del mismo sin ser su destinatario, rogamos nos lo comunique por la misma vía o telefónicamente (91 585 15 00) y proceda a su destrucción, debiendo abstenerse de utilizar, transmitir, divulgar o reproducir la información contenida en el mismo. La CNMV se reserva las acciones legales que procedan contra todo tercero que acceda de forma ilegítima al contenido de cualquier mensaje externo procedente de la entidad Para información y consultas visite nuestra web: http://www.cnmv.es Disclaimer - Comisión Nacional del Mercado de Valores This message, its content and any file attached thereto is for the intended recipient only and is confidential. If you have received this e-mail in error or had access to it, you should note that the information in it is private and any use thereof is unauthorised. In such an event please notify us by e-mail or by telephone (+ 34 91 585 15 00). Any reproduction of this e-mail by whatsoever means and any transmission or dissemination thereof to other persons is prohibited. The Comisión Nacional del Mercado de Valores reserves the right to take legal action against any persons unlawfully gaining access to the content of any external message it has emitted For additional information, please visit our website: http://www.cnmv.es

3 4

SQL driver - Table names with dots failing to create
by phil＠mindfury.net 16 Jan '12

16 Jan '12

Hello. I'm running v3.3.3 built from source on a Red Hat EL 5.3 machine. I've been trying to narrow down a problem where hosts sending syslogs to this box aren't showing up in MySQL, so I checked the internal log and found this: Error running SQL query; type='mysql', host='localhost', port='', user='syslog-ng', database='syslog-ng', error='1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near \'.3.4_20120113 (date text, facility text, level text, host text, program text, pi\' at line 1', query='CREATE TABLE messages_172.25.3.4_20120113 (date text, facility text, level text, host text, program text, pid text, message text)' Looks like in MySQL, that table name can be misinterpreted as <database name>.<table name>, the suggestion is to quote the table identifier with backticks. My question is, is there an option for the SQL driver that does this that I'm just not seeing? I know a workaround would be to ensure the IPs for my hosts reverse map to a name, however it may not be feasible in all situations. Any pointers would be appreciated. --Philip

3 2

CSV parser ignoring template()?
by John Morrissey 16 Jan '12

16 Jan '12

I'm upgrading from 3.1.3 to 3.3.3. My configuration has a CSV parser: parser p_find_role { csv-parser( columns("ROLE", "REMAINDER") delimiters("0123456789.") template("${HOST}") ); }; With 3.1.3, this takes the first word of the machine's hostname and stores it in $ROLE, but 3.3.3 seems to ignore template() and operates on the log message instead, storing the first field of the message in $ROLE. $HOST expands correctly in a log template, so that token is still available in 3.3.x. Am I missing something else related to the upgrade? john -- John Morrissey _o /\ ---- __o jwm(a)horde.net _-< \_ / \ ---- < \, www.horde.net/ __(_)/_(_)________/ \_______(_) /_(_)__

3 4