[syslog-ng] Log messages format difference between version 2.0.9 and 3.3.4

[Alptugay Değirmencioğlu]

Hello all. We are using syslog-ng to transmit messages to a stream and 
an application then reads the messages from the stream and parses them. 
We have been using 2.0.9 normally. But when we updated to 3.3.4 the 
message format coming to the stream has changed.

Btw we are using CentOS 5 with kernerl version 3.5.3

In 2.0.9 we had the following config:

-----------

destination webfilter {
         unix-stream("/var/log/streams/webfilter"
template("$YEAR-$MONTH-$DAY\\011$HOUR:$MIN:$SEC\\011$HOST\\011$MSG\n")
                 template-escape(yes)
         );
};

filter f_webfilter {
         match("dansguardian\\[.*\\]: .*\t");
};

log {
         source(s_all);
         filter(f_webfilter);
         destination(webfilter);
};


were receiving these kinds of messages:

2012-12-05\\01110:40:13\\011ogm42\\011dansguardian[21839]: 
2012.12.5-10:40:13\\011-\\01110.42.1.208\\011http://i.radikal.com.tr/90x70/2012/12/05/fft68_mf1227270.Jpeg\\011*EXCEPTION*Ayricalikli_kullanici_adina_sahipsiniz.\\011GET\\0112499\\0110\\011\\0111\\011200\\011-\\011\\011\\011-\\011 
18

You can see that \\011 was a kind of delimeter to us.

-----------

And in the new syslog-ng version we have the following config (with 4 
back slash \\\\, $MSG changed to $MSGHDR$MSGONLY and template-escape 
changed from yes to no):

destination webfilter {
         unix-stream("/var/log/streams/webfilter"
template("$YEAR-$MONTH-$DAY\\\\011$HOUR:$MIN:$SEC\\\\011$HOST\\\\011$MSGHDR$MSGONLY\n")
                 template-escape(no)
         );
};

filter f_webfilter {
     match("dansguardian\\[.*\\]: .*\t");
};

log {
     source(s_all);
     filter(f_webfilter);
     destination(webfilter);
};

We get the following output (The first parts are okay. But the \\011 int 
the $MSGONLY have become \t):

2012-12-05\\01110:45:24\\011localhost\\011dansguardian[21135]: 
2012.12.5-10:45:24\t-\t127.0.0.1\thttp://www.ipnedir.com\t*SCANNED*\tGET\t3616\t0\t\t1\t200\ttext/html\t\t\t-\t 
76

------------

With this config:

destination webfilter {
         unix-stream("/var/log/streams/webfilter"
template("$YEAR-$MONTH-$DAY\\\\011$HOUR:$MIN:$SEC\\\\011$HOST\\\\011$MSGHDR$MSGONLY\n")
                 template-escape(no)
         );
};

filter f_webfilter {
     match("dansguardian\\[.*\\]: .*\t");
};

log {
     source(s_all);
     filter(f_webfilter);
     destination(webfilter);
};

We get the following output (The first parts are okay. But the \\011 int 
the $MSGONLY are lost)

2012-12-05\\01110:48:33\\011localhost\\011dansguardian[21135]: 
2012.12.5-10:48:33-127.0.0.1http://www.ipnedir.comGET361601200text/html- 73

-----------------

With this config:

destination webfilter {
         unix-stream("/var/log/streams/webfilter"
template("$YEAR-$MONTH-$DAY\\011$HOUR:$MIN:$SEC\\011$HOST\\011$MSGHDR$MSGONLY\n")
                 template-escape(no)
         );
};

filter f_webfilter {
     match("dansguardian\\[.*\\]: .*\t");
};

log {
     source(s_all);
     filter(f_webfilter);
     destination(webfilter);
};

We get the following output (Now the output has become completely wrong)

2012-12-0501110:52:06011localhost011dansguardian[21135]: 
2012.12.5-10:52:06\t-\t127.0.0.1\thttp://www.ipnedir.com\t\tGET\t3616\t0\t\t1\t200\ttext/html\t\t\t-\t 
89

---------------------

And finally with this config:

destination webfilter {
         unix-stream("/var/log/streams/webfilter"
template("$YEAR-$MONTH-$DAY\\011$HOUR:$MIN:$SEC\\011$HOST\\011$MSGHDR$MSGONLY\n")
                 template-escape(yes)
         );
};

filter f_webfilter {
     match("dansguardian\\[.*\\]: .*\t");
};

log {
     source(s_all);
     filter(f_webfilter);
     destination(webfilter);
};

We get the following output (Note that this config is the same config 
with the one we used in 2.0.9 except the $MSGHDR$MSGONLY part )

2012-12-0501110:53:58011localhost011dansguardian[21135]: 
2012.12.5-10:53:58-127.0.0.1http://www.ipnedir.comGET361601200text/html- 184

---------------------------

So what we want to accomplish is that we want the old output back with 
the new syslog :)

How can we achieve this? Btw when restarting syslog-ng service we get 
the following messages. I don't know if it has anything to do with this.

Restarting syslog-ng: Error opening plugin module; module='dbparser', 
error='/usr/lib/syslog-ng/libdbparser.so: undefined symbol: evt_tag_long'
Your configuration file uses an obsoleted keyword, please update your 
configuration; keyword='log_prefix', change='program_override'
WARNING: the match() filter without the use of the value() option is 
deprecated and hinders performance, please update your configuration;
Your configuration file uses an obsoleted keyword, please update your 
configuration; keyword='sync', change='flush_lines'

Shutting down syslog-ng:                                   [  OK  ]
Starting syslog-ng: Error opening plugin module; module='dbparser', 
error='/usr/lib/syslog-ng/libdbparser.so: undefined symbol: evt_tag_long'
Your configuration file uses an obsoleted keyword, please update your 
configuration; keyword='log_prefix', change='program_override'
WARNING: the match() filter without the use of the value() option is 
deprecated and hinders performance, please update your configuration;
Your configuration file uses an obsoleted keyword, please update your 
configuration; keyword='sync', change='flush_lines'
WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, 
the configuration value was divided by the value of max-connections(). 
The result was too small, clamping to 100 entries. Ensure you have a 
proper log_fifo_size setting to avoid message loss.; 
orig_log_iw_size='10', new_log_iw_size='100', min_log_fifo_size='10000'
                                                            [  OK  ]