syslog-ng 3.3.3 and multiple incarnations of the same parser
-----BEGIN PGP SIGNED MESSAGE----- Hi guys, I ran into another problem with 3.3.3. It seems that I can use a cvs-parser only in one and only one log path. If I reuse it in multiple log paths the columns of the cvs parser are empty. I downgraded to 3.0.9 and that resolved my problem. Here is an example config: - --------< SNIP >---------- source s_remote { udp () }; parser p_hostname_site { csv-parser (columns ("REMOTEHOST.SITE", "REMOTEHOST.NAME") delimiters ("-") template ("${HOST}")); }; filter f_local0 { facility (local0); }; filter f_local1 { facility (local1); }; destination d_cacl { file ("/usr/local/var/log/current/${REMOTEHOST.SITE:-other}/cisco-acl.${S_YEAR}${S_MONTH}${S_DAY}${S_HOUR}.log" template ("$S_DATE ${REMOTEHOST.NAME:-none} $MSGHDR$MSG\n") }; destination d_caps { file ("/usr/local/var/log/current/${REMOTEHOST.SITE:-other}/cisco-aps.${S_YEAR}${S_MONTH}${S_DAY}${S_HOUR}.log" template ("$S_DATE ${REMOTEHOST.NAME:-none} $MSGHDR$MSG\n") }; log (source (s_remote); filter (f_local0); parser (p_hostname_site); destination (d_cacl); }; log (source (s_remote); filter (f_local1); parser (p_hostname_site); destination (d_caps); }; - --------< SNIP >---------- The incoming logs for local0 would look like this: Dec 2 16:00:00 florence-VPN-FLORENCE %ASA-6-305011: Built dynamic TCP translation from inside-vlan765 Dec 2 16:00:00 florence-VPN-FLORENCE %ASA-6-302013: Built outbound TCP connection 100466601 for outside-vlan757 Under 3.3.3 it would produce the following logs: /usr/local/var/log/current/other/cisco-acl.2011120216.log: Dec 2 16:00:00 none %ASA-6-305011: Built dynamic TCP translation from inside-vlan765 Dec 2 16:00:00 none %ASA-6-302013: Built outbound TCP connection 100466601 for outside-vlan757 The same configuration with 3.0.9 produces the following: /usr/local/var/log/current/florence/cisco-acl.2011120216.log: Dec 2 16:00:00 VPN-FLORENCE %ASA-6-305011: Built dynamic TCP translation from inside-vlan765 Dec 2 16:00:00 VPN-FLORENCE %ASA-6-302013: Built outbound TCP connection 100466601 for outside-vlan757 If I remove the parser from one of the log paths 3.3.3 would spit out the same output as 3.0.9 but of course, the second log path would need its own parser to make it work. I am unable to check syslog-ng 3.1 or 3.2. - - Michael -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 10.0.3 (Build 1) Charset: us-ascii wsBVAwUBTtjvwJbfnpCg64TVAQFmvwf9EZ3wShhBxIXOGXWiSJguF07HCgw99Wxu amUrvAAeI9nZSj4wvpQCZNLrpPJ2Q7/DCSKTunzopHNJVwMQQvFSwzaAcezNNt4Y ogmTAx+TeyMc4rYuknpG9Khz6WXFTM8rH4imjNziDSxSwkqZrCQDnmYM4B2pxfqo BNZbcRKifbo9mts0FbeD9hpBiYNxt+k9pXPtqUD193ihUWcnNF4NG3hgt/1RXB7E 7v79A6swf30PXSLpdJn8fATptKVrfzzvXlW0ncNolSUyKK7XaY4kgvjsnFba++CS eTil+xkINNnuUz2FDYfEDziTYFeezO/lhSrhE8BiDaRt1Gx10XfbDQ== =2Idy -----END PGP SIGNATURE-----
When parsing a source that has a month and day but no year, the S_YEAR macro does not default to R_YEAR. It seems to be defaulting to R_YEAR+1 What is the intention when there is no year in the source?
Evan Rempel <erempel@uvic.ca> writes:
When parsing a source that has a month and day but no year, the S_YEAR macro does not default to R_YEAR. It seems to be defaulting to R_YEAR+1
What is the intention when there is no year in the source?
This sounds interesting. A quick look at the code didn't reveal anything obviously wrong. I'll see what I can do about it, since reproduction seems easy enough (and then it's just a little bit of gdb-magic away to spot the error). Thanks for the report! -- |8]
On Fri, 2011-12-02 at 23:20 +0100, Gergely Nagy wrote:
Evan Rempel <erempel@uvic.ca> writes:
When parsing a source that has a month and day but no year, the S_YEAR macro does not default to R_YEAR. It seems to be defaulting to R_YEAR+1
What is the intention when there is no year in the source?
This sounds interesting. A quick look at the code didn't reveal anything obviously wrong. I'll see what I can do about it, since reproduction seems easy enough (and then it's just a little bit of gdb-magic away to spot the error).
Thanks for the report!
If there's no year in the incoming timestamp, syslog-ng applies a heuristics to determine the actual year. This heuristics assumes that the incoming message was generated quite close to the current system time. Here's the algorithm (quoting the source): /* detect if the message is coming from last year. If its * month is at least one larger than the current month. This * handles both clocks that are in the future, or in the * past: * in January we receive a message from December (past) => last year * in January we receive a message from February (future) => same year * in December we receive a message from January (future) => next year */ if (tm.tm_mon > nowtm.tm_mon + 1) tm.tm_year--; if (tm.tm_mon < nowtm.tm_mon - 1) tm.tm_year++; -- Bazsi
On Wed, 2011-12-21 at 13:44 +0100, Balazs Scheidler wrote:
On Fri, 2011-12-02 at 23:20 +0100, Gergely Nagy wrote:
Evan Rempel <erempel@uvic.ca> writes:
When parsing a source that has a month and day but no year, the S_YEAR macro does not default to R_YEAR. It seems to be defaulting to R_YEAR+1
What is the intention when there is no year in the source?
This sounds interesting. A quick look at the code didn't reveal anything obviously wrong. I'll see what I can do about it, since reproduction seems easy enough (and then it's just a little bit of gdb-magic away to spot the error).
Thanks for the report!
If there's no year in the incoming timestamp, syslog-ng applies a heuristics to determine the actual year. This heuristics assumes that the incoming message was generated quite close to the current system time.
Here's the algorithm (quoting the source):
/* detect if the message is coming from last year. If its * month is at least one larger than the current month. This * handles both clocks that are in the future, or in the * past: * in January we receive a message from December (past) => last year * in January we receive a message from February (future) => same year * in December we receive a message from January (future) => next year */ if (tm.tm_mon > nowtm.tm_mon + 1) tm.tm_year--; if (tm.tm_mon < nowtm.tm_mon - 1) tm.tm_year++;
BTW: if you want to process historical data, please use a complete timestamp that includes year information. syslog-ng is certainly capable of doing that, but as far as I know rsyslog can do that too. -- Bazsi
On Fri, 2011-12-02 at 10:33 -0500, Michael Hocke wrote:
Hi guys,
I ran into another problem with 3.3.3. It seems that I can use a cvs-parser only in one and only one log path. If I reuse it in multiple log paths the columns of the cvs parser are empty. I downgraded to 3.0.9 and that resolved my problem. Here is an example config:
Thanks for the report and sorry for not replying back any sooner. This patch should fix your issue: Just pushed to 3.3: commit 79fc32703615e045fca4c9aaf85b7ab2915e7ed4 Author: Balazs Scheidler <bazsi@balabit.hu> Date: Wed Dec 21 13:35:28 2011 +0100 csvparser: fixed csv-parser() when applied to multiple log paths Reported-by: Michael Hocke <michael.hocke@nyu.edu> Signed-off-by: Balazs Scheidler <bazsi@balabit.hu> -- Bazsi
participants (4)
-
Balazs Scheidler
-
Evan Rempel
-
Gergely Nagy
-
Michael Hocke