<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="generator" content="Osso Notes">
<title></title></head>
<body>
<p>
<br>Hi,
<br>
<br>one of the previous syslog-ng versions introduced \ as an escape character. this was a bad idea, because:
<br>
<br>* it broke your config
<br>* it plays badly with windows filenames (which tend to contain \ frequently)
<br>
<br>I was thinking about reverting that change, however it'll either break your config or break for those who have already adapted to the new format.
<br>
<br>I'm thinking about doing it anyway, with big fat warnings if backslashes are used in templates. but it'll not help clarity.
<br>
<br>anyway, your issue can be solved by doubling the amount of backslashes you needed before.
<br>
<br>----- Original message -----
<br>> Hello all. We are using syslog-ng to transmit messages to a stream and
<br>> an application then reads the messages from the stream and parses them.
<br>> We have been using 2.0.9 normally. But when we updated to 3.3.4 the
<br>> message format coming to the stream has changed.
<br>>
<br>> Btw we are using CentOS 5 with kernerl version 3.5.3
<br>>
<br>> In 2.0.9 we had the following config:
<br>>
<br>> -----------
<br>>
<br>> destination webfilter {
<br>>  unix-stream("/var/log/streams/webfilter"
<br>> template("$YEAR-$MONTH-$DAY\\011$HOUR:$MIN:$SEC\\011$HOST\\011$MSG\n")
<br>>  template-escape(yes)
<br>>  );
<br>> };
<br>>
<br>> filter f_webfilter {
<br>>  match("dansguardian\\[.*\\]: .*\t");
<br>> };
<br>>
<br>> log {
<br>>  source(s_all);
<br>>  filter(f_webfilter);
<br>>  destination(webfilter);
<br>> };
<br>>
<br>>
<br>> were receiving these kinds of messages:
<br>>
<br>> 2012-12-05\\01110:40:13\\011ogm42\\011dansguardian[21839]:
<br>> 2012.12.5-10:40:13\\011-\\01110.42.1.208\\011<a href="http://i.radikal.com.tr/90x70/2012/12/05/fft68_mf1227270.Jpeg">http://i.radikal.com.tr/90x70/2012/12/05/fft68_mf1227270.Jpeg</a>\\011*EXCEPTION*Ayricalikli_kullanici_adina_sahipsiniz.\\011GET\\0112499\\0110\\011\\0111\\011200\\011-\\011\\011\\011-\\011
<br>> 18
<br>>
<br>> You can see that \\011 was a kind of delimeter to us.
<br>>
<br>> -----------
<br>>
<br>> And in the new syslog-ng version we have the following config (with 4
<br>> back slash \\\\, $MSG changed to $MSGHDR$MSGONLY and template-escape
<br>> changed from yes to no):
<br>>
<br>> destination webfilter {
<br>>  unix-stream("/var/log/streams/webfilter"
<br>> template("$YEAR-$MONTH-$DAY\\\\011$HOUR:$MIN:$SEC\\\\011$HOST\\\\011$MSGHDR$MSGONLY\n")
<br>>  template-escape(no)
<br>>  );
<br>> };
<br>>
<br>> filter f_webfilter {
<br>>  match("dansguardian\\[.*\\]: .*\t");
<br>> };
<br>>
<br>> log {
<br>>  source(s_all);
<br>>  filter(f_webfilter);
<br>>  destination(webfilter);
<br>> };
<br>>
<br>> We get the following output (The first parts are okay. But the \\011 int
<br>> the $MSGONLY have become \t):
<br>>
<br>> 2012-12-05\\01110:45:24\\011localhost\\011dansguardian[21135]:
<br>> 2012.12.5-10:45:24\t-\t127.0.0.1\t<a href="http://www.ipnedir.com">http://www.ipnedir.com</a>\t*SCANNED*\tGET\t3616\t0\t\t1\t200\ttext/html\t\t\t-\t
<br>> 76
<br>>
<br>> ------------
<br>>
<br>> With this config:
<br>>
<br>> destination webfilter {
<br>>  unix-stream("/var/log/streams/webfilter"
<br>> template("$YEAR-$MONTH-$DAY\\\\011$HOUR:$MIN:$SEC\\\\011$HOST\\\\011$MSGHDR$MSGONLY\n")
<br>>  template-escape(no)
<br>>  );
<br>> };
<br>>
<br>> filter f_webfilter {
<br>>  match("dansguardian\\[.*\\]: .*\t");
<br>> };
<br>>
<br>> log {
<br>>  source(s_all);
<br>>  filter(f_webfilter);
<br>>  destination(webfilter);
<br>> };
<br>>
<br>> We get the following output (The first parts are okay. But the \\011 int
<br>> the $MSGONLY are lost)
<br>>
<br>> 2012-12-05\\01110:48:33\\011localhost\\011dansguardian[21135]:
<br>> 2012.12.5-10:48:33-127.0.0.1<a href="http://www.ipnedir.comGET361601200text/html-">http://www.ipnedir.comGET361601200text/html-</a>
<br>> 73
<br>>
<br>> -----------------
<br>>
<br>> With this config:
<br>>
<br>> destination webfilter {
<br>>  unix-stream("/var/log/streams/webfilter"
<br>> template("$YEAR-$MONTH-$DAY\\011$HOUR:$MIN:$SEC\\011$HOST\\011$MSGHDR$MSGONLY\n")
<br>>  template-escape(no)
<br>>  );
<br>> };
<br>>
<br>> filter f_webfilter {
<br>>  match("dansguardian\\[.*\\]: .*\t");
<br>> };
<br>>
<br>> log {
<br>>  source(s_all);
<br>>  filter(f_webfilter);
<br>>  destination(webfilter);
<br>> };
<br>>
<br>> We get the following output (Now the output has become completely wrong)
<br>>
<br>> 2012-12-0501110:52:06011localhost011dansguardian[21135]:
<br>> 2012.12.5-10:52:06\t-\t127.0.0.1\t<a href="http://www.ipnedir.com">http://www.ipnedir.com</a>\t\tGET\t3616\t0\t\t1\t200\ttext/html\t\t\t-\t
<br>> 89
<br>>
<br>> ---------------------
<br>>
<br>> And finally with this config:
<br>>
<br>> destination webfilter {
<br>>  unix-stream("/var/log/streams/webfilter"
<br>> template("$YEAR-$MONTH-$DAY\\011$HOUR:$MIN:$SEC\\011$HOST\\011$MSGHDR$MSGONLY\n")
<br>>  template-escape(yes)
<br>>  );
<br>> };
<br>>
<br>> filter f_webfilter {
<br>>  match("dansguardian\\[.*\\]: .*\t");
<br>> };
<br>>
<br>> log {
<br>>  source(s_all);
<br>>  filter(f_webfilter);
<br>>  destination(webfilter);
<br>> };
<br>>
<br>> We get the following output (Note that this config is the same config
<br>> with the one we used in 2.0.9 except the $MSGHDR$MSGONLY part )
<br>>
<br>> 2012-12-0501110:53:58011localhost011dansguardian[21135]:
<br>> 2012.12.5-10:53:58-127.0.0.1<a href="http://www.ipnedir.comGET361601200text/html-">http://www.ipnedir.comGET361601200text/html-</a>
<br>> 184
<br>>
<br>> ---------------------------
<br>>
<br>> So what we want to accomplish is that we want the old output back with
<br>> the new syslog :)
<br>>
<br>> How can we achieve this? Btw when restarting syslog-ng service we get
<br>> the following messages. I don't know if it has anything to do with this.
<br>>
<br>> Restarting syslog-ng: Error opening plugin module; module='dbparser',
<br>> error='/usr/lib/syslog-ng/libdbparser.so: undefined symbol: evt_tag_long'
<br>> Your configuration file uses an obsoleted keyword, please update your
<br>> configuration; keyword='log_prefix', change='program_override'
<br>> WARNING: the match() filter without the use of the value() option is
<br>> deprecated and hinders performance, please update your configuration;
<br>> Your configuration file uses an obsoleted keyword, please update your
<br>> configuration; keyword='sync', change='flush_lines'
<br>>
<br>> Shutting down syslog-ng:  [  OK  ]
<br>> Starting syslog-ng: Error opening plugin module; module='dbparser',
<br>> error='/usr/lib/syslog-ng/libdbparser.so: undefined symbol: evt_tag_long'
<br>> Your configuration file uses an obsoleted keyword, please update your
<br>> configuration; keyword='log_prefix', change='program_override'
<br>> WARNING: the match() filter without the use of the value() option is
<br>> deprecated and hinders performance, please update your configuration;
<br>> Your configuration file uses an obsoleted keyword, please update your
<br>> configuration; keyword='sync', change='flush_lines'
<br>> WARNING: window sizing for tcp sources were changed in syslog-ng 3.3,
<br>> the configuration value was divided by the value of max-connections().
<br>> The result was too small, clamping to 100 entries. Ensure you have a
<br>> proper log_fifo_size setting to avoid message loss.;
<br>> orig_log_iw_size='10', new_log_iw_size='100', min_log_fifo_size='10000'
<br>>  [  OK  ]
<br>>
<br>>
<br>>
<br>> ______________________________________________________________________________
<br>> Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
<br>> Documentation:
<br>> <a href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a> FAQ:
<br>> <a href="http://www.balabit.com/wiki/syslog-ng-faq">http://www.balabit.com/wiki/syslog-ng-faq</a>
<br>>
<br><br></p>
</body>
</html>