I have determined that the Unicode character is being written into the log file, so the problem seems to lie with the pcre regular expression used. I was testing the pcre regular expression format for the rewrite rule and I found some unusual results. Take for example the rewrite rule<div>
<br></div><div>rewrite r_rewrite_subst { subst("a*", "?", value("MESSAGE") type("pcre") flags("utf8" "global")); };</div><div><br></div><div>I thought that it would replace 0 or more occurrences of 'a' with a question mark. However, when the input is 'a', the output becomes 'a?'. It just appends a question mark at the end. 'aa' also becomes 'aa?'. What is weirder still is that even if the input does not contain 'a', the question mark is still appended. So '1' becomes '1?'.</div>
<div><br></div><div>Is there something I'm not understanding with the pcre regular expression format? What is the correct way to use it?</div><div><br></div><div>Thanks.<br><br><div class="gmail_quote">On Fri, Sep 23, 2011 at 12:29 AM, Luis Pugoy <span dir="ltr"><<a href="mailto:lpugoy@insynchq.com">lpugoy@insynchq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Hello,<div><br></div><div>We have a Python process that logs into a file. Included in the file are tracebacks. We would like to send these log entries through syslog-ng. To solve the problem of newlines in tracebacks, our plan was to modify the Python log formatter to replace all newlines with the Unicode character for line separator, U+2028. Syslog-ng then tails the log file and sends the entries to the central logging server through tcp, which is setup to replace the Unicode character with a newline and record it in a file.</div>
<div><br></div><div>The problem with this approach is that the Unicode character does not seem to be persisted in the file. A lot of rewrite rules were tried to replace the Unicode character. These are:</div><div> rewrite r_newlines { subst("\p{Zl}", " test ", value("MESSAGE") type("pcre") flags("utf8" "global")); };</div>
<div><div> rewrite r_newlines { subst("\x{2028}", " test ", value("MESSAGE") type("pcre") flags("utf8" "global")); };</div></div><div><div><div> rewrite r_newlines { subst("\u2028", " test ", value("MESSAGE") flags("global")); };</div>
</div></div><div><br></div><div>but none of them worked. The file source was also set to have UTF-8 encoding, as well as the tcp source (from the view of the central logging server) but these didn't work either. The problem seems to be either the rewrite rule or the Unicode character is not persisted in the log file. We also tried logging to syslog directly from Python (through /dev/log) but that also failed.</div>
<div><br></div><div>Replacing newlines with a string such as " [NEWLINE] " and using syslog-ng to rewrite that to newlines worked, so it seems to be a problem with the Unicode character. This solution is a last resort though since it could be problematic.</div>
<div><br></div><div>Thanks in advance.</div>
</blockquote></div><br></div>