[syslog-ng] How to send tracebacks through syslog-ng?

[Luis Pugoy]

I have determined that the Unicode character is being written into the log
file, so the problem seems to lie with the pcre regular expression used. I
was testing the pcre regular expression format for the rewrite rule and I
found some unusual results. Take for example the rewrite rule

rewrite r_rewrite_subst {  subst("a*", "?", value("MESSAGE") type("pcre")
flags("utf8" "global"));  };

I thought that it would replace 0 or more occurrences of 'a' with a question
mark. However, when the input is 'a', the output becomes 'a?'. It just
appends a question mark at the end. 'aa' also becomes 'aa?'. What is weirder
still is that even if the input does not contain 'a', the question mark is
still appended. So '1' becomes '1?'.

Is there something I'm not understanding with the pcre regular expression
format? What is the correct way to use it?

Thanks.

On Fri, Sep 23, 2011 at 12:29 AM, Luis Pugoy <lpugoy at insynchq.com> wrote:

> Hello,
>
> We have a Python process that logs into a file. Included in the file are
> tracebacks. We would like to send these log entries through syslog-ng. To
> solve the problem of newlines in tracebacks, our plan was to modify the
> Python log formatter to replace all newlines with the Unicode character for
> line separator, U+2028. Syslog-ng then tails the log file and sends the
> entries to the central logging server through tcp, which is setup to replace
> the Unicode character with a newline and record it in a file.
>
> The problem with this approach is that the Unicode character does not seem
> to be persisted in the file. A lot of rewrite rules were tried to replace
> the Unicode character. These are:
>   rewrite r_newlines { subst("\p{Zl}", " test ", value("MESSAGE")
> type("pcre") flags("utf8" "global")); };
>   rewrite r_newlines { subst("\x{2028}", " test ", value("MESSAGE")
> type("pcre") flags("utf8" "global")); };
>   rewrite r_newlines { subst("\u2028", " test ", value("MESSAGE")
> flags("global")); };
>
> but none of them worked. The file source was also set to have UTF-8
> encoding, as well as the tcp source (from the view of the central logging
> server) but these didn't work either. The problem seems to be either the
> rewrite rule or the Unicode character is not persisted in the log file. We
> also tried logging to syslog directly from Python (through /dev/log) but
> that also failed.
>
> Replacing newlines with a string such as " [NEWLINE] " and using syslog-ng
> to rewrite that to newlines worked, so it seems to be a problem with the
> Unicode character. This solution is a last resort though since it could be
> problematic.
>
> Thanks in advance.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20110924/293f7491/attachment.htm