- syslog-ng - lists.balabit.hu

[PATCH (3.4) 0/4]: json-parser updates
by Gergely Nagy 16 Jan '12

16 Jan '12

Following this mail, a couple of patches will come - they were written in last November, but I don't think I posted them (as I didn't expect to see the json-parser merged so soon). Until now, they were sitting on my feature/3.4/json/parser branch, and that's where they sit now, too. I rebased that branch onto 3.4 master + center.c syntax error fixes (which I posted a week or two ago), and am sending the json parser patches now. They accomplish a few things: it makes the parser handle invalid JSON input properly (by returning an error, instead of crashing); adds support for parsing boolean, array and nested object types. Booleans will get parsed into TRUE or FALSE, nested objects will be turned into dotted notation (which also means that keys with a dot in them won't be handled properly by format-json, once it is updated to support reconstructing nested objects from dotted notation), and arrays will be treated like nested objects, with the array indexes used as a key. As an example, lets look at a JSON input: {"is-example": true, "hats": ["top hat", "baseball cap", "pointy witch-hat"], "auth": { "method": ["looking angry at the security guy", "bribery"], "result": "did not work" } } This would be parsed into the following name-value paris: "is-example"="TRUE" "hats.0"="top hat" "hats.1"="baseball cap" "hats.2"="pointy witch-hat" "auth.method.0"="looking angry at the security guy" "auth.method.1"="bribery" "auth.result"="did not work" The other part of this will be an enhahcement to value-pairs and format-json, that will make syslog-ng able to reconstruct the original JSON (more or less... booleans probably won't be supported, except perhaps optionally with a flag). But that is something I haven't started working on yet.

2 6

Hello, and project for Win32 port
by Ramón García Fernández 16 Jan '12

16 Jan '12

Hello, I am Ramón García Fernández and I work for the IT department of the Comisión Nacional de Mercado de Valores, the Spanish regulator of securities. Would you be interested in an opensource Win32 port of syslog-ng? Best regards, Aviso legal – Comisión Nacional del Mercado de Valores Este mensaje y, en su caso, los ficheros que lleve incorporados, está dirigido exclusivamente a su destinatario y es de carácter confidencial. Si fuere recibido por error o se tuviere conocimiento del mismo sin ser su destinatario, rogamos nos lo comunique por la misma vía o telefónicamente (91 585 15 00) y proceda a su destrucción, debiendo abstenerse de utilizar, transmitir, divulgar o reproducir la información contenida en el mismo. La CNMV se reserva las acciones legales que procedan contra todo tercero que acceda de forma ilegítima al contenido de cualquier mensaje externo procedente de la entidad Para información y consultas visite nuestra web: http://www.cnmv.es Disclaimer - Comisión Nacional del Mercado de Valores This message, its content and any file attached thereto is for the intended recipient only and is confidential. If you have received this e-mail in error or had access to it, you should note that the information in it is private and any use thereof is unauthorised. In such an event please notify us by e-mail or by telephone (+ 34 91 585 15 00). Any reproduction of this e-mail by whatsoever means and any transmission or dissemination thereof to other persons is prohibited. The Comisión Nacional del Mercado de Valores reserves the right to take legal action against any persons unlawfully gaining access to the content of any external message it has emitted For additional information, please visit our website: http://www.cnmv.es

3 4

SQL driver - Table names with dots failing to create
by phil＠mindfury.net 16 Jan '12

16 Jan '12

Hello. I'm running v3.3.3 built from source on a Red Hat EL 5.3 machine. I've been trying to narrow down a problem where hosts sending syslogs to this box aren't showing up in MySQL, so I checked the internal log and found this: Error running SQL query; type='mysql', host='localhost', port='', user='syslog-ng', database='syslog-ng', error='1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near \'.3.4_20120113 (date text, facility text, level text, host text, program text, pi\' at line 1', query='CREATE TABLE messages_172.25.3.4_20120113 (date text, facility text, level text, host text, program text, pid text, message text)' Looks like in MySQL, that table name can be misinterpreted as <database name>.<table name>, the suggestion is to quote the table identifier with backticks. My question is, is there an option for the SQL driver that does this that I'm just not seeing? I know a workaround would be to ensure the IPs for my hosts reverse map to a name, however it may not be feasible in all situations. Any pointers would be appreciated. --Philip

3 2

CSV parser ignoring template()?
by John Morrissey 16 Jan '12

16 Jan '12

I'm upgrading from 3.1.3 to 3.3.3. My configuration has a CSV parser: parser p_find_role { csv-parser( columns("ROLE", "REMAINDER") delimiters("0123456789.") template("${HOST}") ); }; With 3.1.3, this takes the first word of the machine's hostname and stores it in $ROLE, but 3.3.3 seems to ignore template() and operates on the log message instead, storing the first field of the message in $ROLE. $HOST expands correctly in a log template, so that token is still available in 3.3.x. Am I missing something else related to the upgrade? john -- John Morrissey _o /\ ---- __o jwm(a)horde.net _-< \_ / \ ---- < \, www.horde.net/ __(_)/_(_)________/ \_______(_) /_(_)__

3 4

patterndb that kills syslong-ng with a segfault.
by Russell Fulton 16 Jan '12

16 Jan '12

I tried to rewrite the program macro with a <value> clause and this caused syslog-ng to die. syslog-ng-3.2.4 I no longer have the example but can reconstruct it if you have problems reproducing the problem. R

2 1

[PATCH (3.4) 0/5] afmongodb cleanups & replica set support
by Gergely Nagy 16 Jan '12

16 Jan '12

Following this email, a few patches will come, various minor touch-ups (such as the cleanup of the default value-pairs config, and the removal of the TODO file) and replica-set support for afmongodb. The replica-set support is an incompatible change compared to 3.3, as it removes the host() and port() options from the driver, replacing both with a servers() option. After the patches, the driver will support replica sets, which means that when connecting to a replica set, and the primary goes down, the driver will reconnect to the new master automatically. Most of this is handled transparently by the underlying libmongo-client library, so the syslog-ng side doesn't actually get much more complicated. These patches have been sitting in my syslog-ng module collection repo since about july, and I kinda forgot about them. It's about time I submit them. (And consequently, the module-collection repo will be abandoned, as it pretty much has been for the past few months anyway. Turns out it's easier for me to maintain module updates within my syslog-ng repo, especially now that I have 3.3 and 3.4 within the same git repo.) All of these are available from the feature/3.4/afmongodb/lmc-sync branch of my git repository.

2 7

Syslog-ng 3.2 connection timeout with firewall
by André Larose 16 Jan '12

16 Jan '12

Hi, I have two syslog-ng 3.2, one client and one server. The two are separated by a firewall and a load balancer. I noticed that after some inactivity I was not able to receive logs from my client. So I started some tcpdump on both servers to check the traffic. From what I see the firewall will close the connection after some time, so when the client sends traffic it gets dropped. I added the keep-alive(yes) and so_keep-alive(yes) and the mark_freq(60) to the configs. But I still do not see keepalive packets with tcpdump. Am I missing other parameters to have "keepalive" traffic sent ? Thank you in advance. ############################################################################# # # Configuration file syslog-ng.conf.client # @version: 3.2 @include "scl.conf" # add global settings options { # Disable DNS usage to gain the process time use_dns(no); flush_lines(1); flush_timeout(500); mark(59); }; source s_TCP { tcp( port(555) flags(no-parse) ); }; source s_syslogng { internal(); }; destination d_syslogng { file("/opt/syslog-ng/logs/$YEAR$MONTH$DAY.syslog-ng.log"); }; destination d_TCP { tcp ( "server1" port(556) keep-alive(yes) so_keepalive(yes) flags(no_multi_line) flush_lines(1) flush_timeout(500) log_fifo_size(4096) template("$MSG\n") template_escape(no) ); }; .... ############################################################################# # # Configuration file syslog-ng.conf.server # @version: 3.2 @include "scl.conf" # add global settings options { # Disable DNS usage to gain the process time use_dns(no); flush_lines(1); flush_timeout(500); mark(59); }; source s_TCP_555 { tcp( port(555) flags(no-parse) max-connections(200) use_dns(no) ); }; source s_TCP_556 { tcp( port(556) flags(no-parse) so_keepalive(yes) use_dns(no) ); }; source s_syslogng { internal(); }; destination d_syslogng { file("/opt/syslog-ng/logs/$YEAR$MONTH$DAY.syslog-ng.log"); }; destination d_local { file("/opt2/syslog-ng/logs/$YEAR$MONTH$DAY.TNT.Messages.log" log_fifo_size(8192) template("$MSG\n") template_escape(no) ); }; ...

2 1

[PATCH (3.4)] center: Fix a few syntax errors.
by Gergely Nagy 16 Jan '12

16 Jan '12

Fix two syntax errors (an unclosed comment, and a double-closing-bracer) that came in with a previous patch. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/center.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/lib/center.c b/lib/center.c index 986ab75..2fc3554 100644 --- a/lib/center.c +++ b/lib/center.c @@ -320,6 +320,7 @@ log_center_instantiate_process_pipe_line(LogCenter *self, LogProcessRule *rule) * * The next member pointer is not holding a reference, but can be * assumed to be kept alive as long as the configuration is running. + */ LogPipe * log_center_init_pipe_line(LogCenter *self, LogConnection *conn, GlobalConfig *cfg, gboolean toplevel, gboolean flow_controlled_parent) { @@ -468,7 +469,6 @@ log_center_init_pipe_line(LogCenter *self, LogConnection *conn, GlobalConfig *cf log_multiplexer_add_next_hop(fork_mpx, sub_pipe); break; } - } default: { g_assert_not_reached(); -- 1.7.7.3

2 1

Re: [syslog-ng] syslog-ng 3.1.2 segmentation fault in Ubuntu 10.10 (Luis Pugoy)
by Luis Pugoy 16 Jan '12

16 Jan '12

> > > Message: 1 > Date: Mon, 5 Dec 2011 19:54:42 +0800 > From: Luis Pugoy <lpugoy(a)insynchq.com> > Subject: Re: [syslog-ng] syslog-ng 3.1.2 segmentation fault in Ubuntu > 10.10 > To: syslog-ng(a)lists.balabit.hu > Message-ID: > <CA+WGxEhKYMQXbYTVUZVScjaXDdjRETGSCk5nZaisW9p-UOQWoA(a)mail.gmail.com > > > Content-Type: text/plain; charset="iso-8859-1" > > > > > Message: 3 > > Date: Mon, 05 Dec 2011 10:10:40 +0100 > > From: Gergely Nagy <algernon(a)balabit.hu> > > Subject: Re: [syslog-ng] syslog-ng 3.1.2 segmentation fault in Ubuntu > > 10.10 > > To: Syslog-ng users' and developers' mailing list > > <syslog-ng(a)lists.balabit.hu> > > Message-ID: <87aa77l6y7.fsf(a)algernon.balabit> > > Content-Type: text/plain > > > > Luis Pugoy <lpugoy(a)insynchq.com> writes: > > > > > Does anyone have any insight into this? The segfault is happening more > > > often now. :( > > > > I'd suggest either upgrading to 3.3 from my repo[1], or if an upgrade is > > not feasible, then if you could compile 3.1 with debug symbols, get it > > to drop a core, and do a backtrace, that would help tremendously. > > > > [1]: http://asylum.madhouse-project.org/projects/debian/ > > > > To do the latter, the following steps should work: > > > > $ apt-get source syslog-ng > > $ sudo apt-get build-dep syslog-ng > > $ cd syslog-ng-* > > $ DEB_BUILD_OPTIONS="debug nostrip" dpkg-buildpackage -us -uc -rfakeroot > > $ sudo dpkg -i ../syslog-ng_*.deb > > > > Then add --enable-core to SYSLOGNG_OPTS in /etc/default/syslog-ng, and > > when it segfaults, it'll make a core in (as far as I remember) /. Once > > that happened, do something like this: > > > > $ gdb -c /core /usr/bin/syslog-ng > > (gdb) thread apply all bt full > > > > This should give us a few hints. > > > > -- > > |8] > > > > I installed a newer version of the PCRE library. That seems to have worked > for now. If it happens again I will be sure to follow your instructions, > thanks. > Well, installing another version of the PCRE library did not solve my problem. syslog-ng is still segfaulting. I tried upgrading to 3.2.2 but that did nothing, so I tried to create the backtrace like you said. However I don't think it was helpful in my case. The following is the output of the backtrace: root@ip-10-34-150-77:/var/lib/syslog-ng# gdb -c core /usr/sbin/syslog-ng GNU gdb (GDB) 7.2-ubuntu Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html > This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/syslog-ng...done. [New Thread 14680] warning: Can't read pathname for load map: Input/output error. Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/librt.so.1 Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libnsl.so.1 Reading symbols from /usr/lib/libgthread-2.0.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgthread-2.0.so.0 Reading symbols from /lib/libglib-2.0.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libglib-2.0.so.0 Reading symbols from /usr/lib/libevtlog.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libevtlog.so.0 Reading symbols from /lib/libssl.so.0.9.8...(no debugging symbols found)...done. Loaded symbols for /lib/libssl.so.0.9.8 Reading symbols from /lib/libcrypto.so.0.9.8...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypto.so.0.9.8 Reading symbols from /lib/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libz.so.1 Reading symbols from /usr/lib/libnet.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libnet.so.1 Reading symbols from /lib/libwrap.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libwrap.so.0 Reading symbols from /usr/lib/libdbi.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libdbi.so.0 Reading symbols from /lib/libcap.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libcap.so.2 Reading symbols from /lib/libpcre.so.3...(no debugging symbols found)...done. Loaded symbols for /lib/libpcre.so.3 Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libpthread.so.0 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libm.so.6 Reading symbols from /lib/libattr.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libattr.so.1 Reading symbols from /lib/libnss_compat.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_compat.so.2 Reading symbols from /lib/libnss_nis.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_nis.so.2 Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_files.so.2 Reading symbols from /lib/libnss_dns.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_dns.so.2 Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libresolv.so.2 Core was generated by `/usr/sbin/syslog-ng -p /var/run/syslog-ng.pid --no-caps --enable-core'. Program terminated with signal 11, Segmentation fault. #0 0x00007fd8440c4e9d in ?? () from /lib/libpcre.so.3 (gdb) thread apply all bt full Thread 1 (Thread 14680): #0 0x00007fd8440c4e9d in ?? () from /lib/libpcre.so.3 No symbol table info available. Cannot access memory at address 0x7fff4a4e3fd8 (gdb) If it is in anyway helpful, I think the only usage of PCRE in my syslog-ng.conf is a rewrite rule to change the Unicode character separator line to a new line: subst("\\p{Zl}", "\n", value("MESSAGE") type("pcre") flags("utf8" "global")); Is it related to my problem in some way?

2 1

[PATCH] afmongodb: Only free the value pairs if there's any.
by Gergely Nagy 16 Jan '12

16 Jan '12

In afmongodb_dd_free() only free self->vp, if it is non-NULL. In case of a syntax error in the configuration file, it may very well be NULL still, and that would lead to us segfaulting shortly after printing the error message. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afmongodb/afmongodb.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/modules/afmongodb/afmongodb.c b/modules/afmongodb/afmongodb.c index cbd0d16..7009b33 100644 --- a/modules/afmongodb/afmongodb.c +++ b/modules/afmongodb/afmongodb.c @@ -495,7 +495,8 @@ afmongodb_dd_free(LogPipe *d) g_free(self->user); g_free(self->password); g_free(self->host); - value_pairs_free(self->vp); + if (self->vp) + value_pairs_free(self->vp); log_dest_driver_free(d); } -- 1.7.7.3

2 1