[syslog-ng] Log messages format difference between version 2.0.9 and 3.3.4
Balazs Scheidler
bazsi77 at gmail.com
Wed Dec 5 21:44:46 CET 2012
Hi,
one of the previous syslog-ng versions introduced \ as an escape character. this was a bad idea, because:
* it broke your config
* it plays badly with windows filenames (which tend to contain \ frequently)
I was thinking about reverting that change, however it'll either break your config or break for those who have already adapted to the new format.
I'm thinking about doing it anyway, with big fat warnings if backslashes are used in templates. but it'll not help clarity.
anyway, your issue can be solved by doubling the amount of backslashes you needed before.
----- Original message -----
> Hello all. We are using syslog-ng to transmit messages to a stream and
> an application then reads the messages from the stream and parses them.
> We have been using 2.0.9 normally. But when we updated to 3.3.4 the
> message format coming to the stream has changed.
>
> Btw we are using CentOS 5 with kernerl version 3.5.3
>
> In 2.0.9 we had the following config:
>
> -----------
>
> destination webfilter {
> unix-stream("/var/log/streams/webfilter"
> template("$YEAR-$MONTH-$DAY\\011$HOUR:$MIN:$SEC\\011$HOST\\011$MSG\n")
> template-escape(yes)
> );
> };
>
> filter f_webfilter {
> match("dansguardian\\[.*\\]: .*\t");
> };
>
> log {
> source(s_all);
> filter(f_webfilter);
> destination(webfilter);
> };
>
>
> were receiving these kinds of messages:
>
> 2012-12-05\\01110:40:13\\011ogm42\\011dansguardian[21839]:
> 2012.12.5-10:40:13\\011-\\01110.42.1.208\\011http://i.radikal.com.tr/90x70/2012/12/05/fft68_mf1227270.Jpeg\\011*EXCEPTION*Ayricalikli_kullanici_adina_sahipsiniz.\\011GET\\0112499\\0110\\011\\0111\\011200\\011-\\011\\011\\011-\\011
> 18
>
> You can see that \\011 was a kind of delimeter to us.
>
> -----------
>
> And in the new syslog-ng version we have the following config (with 4
> back slash \\\\, $MSG changed to $MSGHDR$MSGONLY and template-escape
> changed from yes to no):
>
> destination webfilter {
> unix-stream("/var/log/streams/webfilter"
> template("$YEAR-$MONTH-$DAY\\\\011$HOUR:$MIN:$SEC\\\\011$HOST\\\\011$MSGHDR$MSGONLY\n")
> template-escape(no)
> );
> };
>
> filter f_webfilter {
> match("dansguardian\\[.*\\]: .*\t");
> };
>
> log {
> source(s_all);
> filter(f_webfilter);
> destination(webfilter);
> };
>
> We get the following output (The first parts are okay. But the \\011 int
> the $MSGONLY have become \t):
>
> 2012-12-05\\01110:45:24\\011localhost\\011dansguardian[21135]:
> 2012.12.5-10:45:24\t-\t127.0.0.1\thttp://www.ipnedir.com\t*SCANNED*\tGET\t3616\t0\t\t1\t200\ttext/html\t\t\t-\t
> 76
>
> ------------
>
> With this config:
>
> destination webfilter {
> unix-stream("/var/log/streams/webfilter"
> template("$YEAR-$MONTH-$DAY\\\\011$HOUR:$MIN:$SEC\\\\011$HOST\\\\011$MSGHDR$MSGONLY\n")
> template-escape(no)
> );
> };
>
> filter f_webfilter {
> match("dansguardian\\[.*\\]: .*\t");
> };
>
> log {
> source(s_all);
> filter(f_webfilter);
> destination(webfilter);
> };
>
> We get the following output (The first parts are okay. But the \\011 int
> the $MSGONLY are lost)
>
> 2012-12-05\\01110:48:33\\011localhost\\011dansguardian[21135]:
> 2012.12.5-10:48:33-127.0.0.1http://www.ipnedir.comGET361601200text/html-
> 73
>
> -----------------
>
> With this config:
>
> destination webfilter {
> unix-stream("/var/log/streams/webfilter"
> template("$YEAR-$MONTH-$DAY\\011$HOUR:$MIN:$SEC\\011$HOST\\011$MSGHDR$MSGONLY\n")
> template-escape(no)
> );
> };
>
> filter f_webfilter {
> match("dansguardian\\[.*\\]: .*\t");
> };
>
> log {
> source(s_all);
> filter(f_webfilter);
> destination(webfilter);
> };
>
> We get the following output (Now the output has become completely wrong)
>
> 2012-12-0501110:52:06011localhost011dansguardian[21135]:
> 2012.12.5-10:52:06\t-\t127.0.0.1\thttp://www.ipnedir.com\t\tGET\t3616\t0\t\t1\t200\ttext/html\t\t\t-\t
> 89
>
> ---------------------
>
> And finally with this config:
>
> destination webfilter {
> unix-stream("/var/log/streams/webfilter"
> template("$YEAR-$MONTH-$DAY\\011$HOUR:$MIN:$SEC\\011$HOST\\011$MSGHDR$MSGONLY\n")
> template-escape(yes)
> );
> };
>
> filter f_webfilter {
> match("dansguardian\\[.*\\]: .*\t");
> };
>
> log {
> source(s_all);
> filter(f_webfilter);
> destination(webfilter);
> };
>
> We get the following output (Note that this config is the same config
> with the one we used in 2.0.9 except the $MSGHDR$MSGONLY part )
>
> 2012-12-0501110:53:58011localhost011dansguardian[21135]:
> 2012.12.5-10:53:58-127.0.0.1http://www.ipnedir.comGET361601200text/html-
> 184
>
> ---------------------------
>
> So what we want to accomplish is that we want the old output back with
> the new syslog :)
>
> How can we achieve this? Btw when restarting syslog-ng service we get
> the following messages. I don't know if it has anything to do with this.
>
> Restarting syslog-ng: Error opening plugin module; module='dbparser',
> error='/usr/lib/syslog-ng/libdbparser.so: undefined symbol: evt_tag_long'
> Your configuration file uses an obsoleted keyword, please update your
> configuration; keyword='log_prefix', change='program_override'
> WARNING: the match() filter without the use of the value() option is
> deprecated and hinders performance, please update your configuration;
> Your configuration file uses an obsoleted keyword, please update your
> configuration; keyword='sync', change='flush_lines'
>
> Shutting down syslog-ng: [ OK ]
> Starting syslog-ng: Error opening plugin module; module='dbparser',
> error='/usr/lib/syslog-ng/libdbparser.so: undefined symbol: evt_tag_long'
> Your configuration file uses an obsoleted keyword, please update your
> configuration; keyword='log_prefix', change='program_override'
> WARNING: the match() filter without the use of the value() option is
> deprecated and hinders performance, please update your configuration;
> Your configuration file uses an obsoleted keyword, please update your
> configuration; keyword='sync', change='flush_lines'
> WARNING: window sizing for tcp sources were changed in syslog-ng 3.3,
> the configuration value was divided by the value of max-connections().
> The result was too small, clamping to 100 entries. Ensure you have a
> proper log_fifo_size setting to avoid message loss.;
> orig_log_iw_size='10', new_log_iw_size='100', min_log_fifo_size='10000'
> [ OK ]
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng FAQ:
> http://www.balabit.com/wiki/syslog-ng-faq
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20121205/9a25eb31/attachment.htm
More information about the syslog-ng
mailing list