Hello all. We are using syslog-ng to transmit messages to a stream and
an application then reads the messages from the stream and parses them.
We have been using 2.0.9 normally. But when we updated to 3.3.4 the
message format coming to the stream has changed.
Btw we are using CentOS 5 with kernerl version 3.5.3
In 2.0.9 we had the following config:
-----------
destination webfilter {
unix-stream("/var/log/streams/webfilter"
template("$YEAR-$MONTH-$DAY\\011$HOUR:$MIN:$SEC\\011$HOST\\011$MSG\n")
template-escape(yes)
);
};
filter f_webfilter {
match("dansguardian\\[.*\\]: .*\t");
};
log {
source(s_all);
filter(f_webfilter);
destination(webfilter);
};
were receiving these kinds of messages:
2012-12-05\\01110:40:13\\011ogm42\\011dansguardian[21839]:
2012.12.5-10:40:13\\011-\\01110.42.1.208\\011http://i.radikal.com.tr/90x70/2012/12/05/fft68_mf1227270.Jpeg\\011*EXCEPTION*Ayricalikli_kullanici_adina_sahipsiniz.\\011GET\\0112499\\0110\\011\\0111\\011200\\011-\\011\\011\\011-\\011
18
You can see that \\011 was a kind of delimeter to us.
-----------
And in the new syslog-ng version we have the following config (with 4
back slash \\\\, $MSG changed to $MSGHDR$MSGONLY and template-escape
changed from yes to no):
destination webfilter {
unix-stream("/var/log/streams/webfilter"
template("$YEAR-$MONTH-$DAY\\\\011$HOUR:$MIN:$SEC\\\\011$HOST\\\\011$MSGHDR$MSGONLY\n")
template-escape(no)
);
};
filter f_webfilter {
match("dansguardian\\[.*\\]: .*\t");
};
log {
source(s_all);
filter(f_webfilter);
destination(webfilter);
};
We get the following output (The first parts are okay. But the \\011 int
the $MSGONLY have become \t):
2012-12-05\\01110:45:24\\011localhost\\011dansguardian[21135]:
2012.12.5-10:45:24\t-\t127.0.0.1\thttp://www.ipnedir.com\t*SCANNED*\tGET\t3616\t0\t\t1\t200\ttext/html\t\t\t-\t
76
------------
With this config:
destination webfilter {
unix-stream("/var/log/streams/webfilter"
template("$YEAR-$MONTH-$DAY\\\\011$HOUR:$MIN:$SEC\\\\011$HOST\\\\011$MSGHDR$MSGONLY\n")
template-escape(no)
);
};
filter f_webfilter {
match("dansguardian\\[.*\\]: .*\t");
};
log {
source(s_all);
filter(f_webfilter);
destination(webfilter);
};
We get the following output (The first parts are okay. But the \\011 int
the $MSGONLY are lost)
2012-12-05\\01110:48:33\\011localhost\\011dansguardian[21135]:
2012.12.5-10:48:33-127.0.0.1http://www.ipnedir.comGET361601200text/html- 73
-----------------
With this config:
destination webfilter {
unix-stream("/var/log/streams/webfilter"
template("$YEAR-$MONTH-$DAY\\011$HOUR:$MIN:$SEC\\011$HOST\\011$MSGHDR$MSGONLY\n")
template-escape(no)
);
};
filter f_webfilter {
match("dansguardian\\[.*\\]: .*\t");
};
log {
source(s_all);
filter(f_webfilter);
destination(webfilter);
};
We get the following output (Now the output has become completely wrong)
2012-12-0501110:52:06011localhost011dansguardian[21135]:
2012.12.5-10:52:06\t-\t127.0.0.1\thttp://www.ipnedir.com\t\tGET\t3616\t0\t\t1\t200\ttext/html\t\t\t-\t
89
---------------------
And finally with this config:
destination webfilter {
unix-stream("/var/log/streams/webfilter"
template("$YEAR-$MONTH-$DAY\\011$HOUR:$MIN:$SEC\\011$HOST\\011$MSGHDR$MSGONLY\n")
template-escape(yes)
);
};
filter f_webfilter {
match("dansguardian\\[.*\\]: .*\t");
};
log {
source(s_all);
filter(f_webfilter);
destination(webfilter);
};
We get the following output (Note that this config is the same config
with the one we used in 2.0.9 except the $MSGHDR$MSGONLY part )
2012-12-0501110:53:58011localhost011dansguardian[21135]:
2012.12.5-10:53:58-127.0.0.1http://www.ipnedir.comGET361601200text/html- 184
---------------------------
So what we want to accomplish is that we want the old output back with
the new syslog :)
How can we achieve this? Btw when restarting syslog-ng service we get
the following messages. I don't know if it has anything to do with this.
Restarting syslog-ng: Error opening plugin module; module='dbparser',
error='/usr/lib/syslog-ng/libdbparser.so: undefined symbol: evt_tag_long'
Your configuration file uses an obsoleted keyword, please update your
configuration; keyword='log_prefix', change='program_override'
WARNING: the match() filter without the use of the value() option is
deprecated and hinders performance, please update your configuration;
Your configuration file uses an obsoleted keyword, please update your
configuration; keyword='sync', change='flush_lines'
Shutting down syslog-ng: [ OK ]
Starting syslog-ng: Error opening plugin module; module='dbparser',
error='/usr/lib/syslog-ng/libdbparser.so: undefined symbol: evt_tag_long'
Your configuration file uses an obsoleted keyword, please update your
configuration; keyword='log_prefix', change='program_override'
WARNING: the match() filter without the use of the value() option is
deprecated and hinders performance, please update your configuration;
Your configuration file uses an obsoleted keyword, please update your
configuration; keyword='sync', change='flush_lines'
WARNING: window sizing for tcp sources were changed in syslog-ng 3.3,
the configuration value was divided by the value of max-connections().
The result was too small, clamping to 100 entries. Ensure you have a
proper log_fifo_size setting to avoid message loss.;
orig_log_iw_size='10', new_log_iw_size='100', min_log_fifo_size='10000'
[ OK ]