<html><head></head><body>Hi Bazsi,<br>
<br>
The version of glib is 2.46.2 on FreeBSD 10.3.<br>
<br>
The issue does not occur on the first packet coming through, but when under light load (~100/sec)...<br>
<br>
Tried reducing the number of unprintable chars and now only \0x92 exists in the inbound message it falls over on. It is always a message with \0x92 that causes it to fail.<br>
<br>
Is there a way to have a filter applies before the message is utf8_sanitised using a regular expression or the like?<br>
<br>
What if the assert was removed, what effect would it have?<br>
<br>
Many thanks to all!<br>
<br>
Kr,<br>
<br>
James<br><br><div class="gmail_quote">On 6 January 2017 12:49:28 GMT+00:00, "Scheidler, Balázs" <balazs.scheidler@balabit.com> wrote:<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div dir="ltr"><div>Hi,<br /><br /></div>Attila is right, it would help a lot to see the original log message and your glib version. That code path uses a performance hack that relies on a GLib implementation detail. Either the glib behaviour has changed or another assumption fails, but just looking at the code I don't know what might.<br /></div><div class="gmail_extra"><br clear="all" /><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">-- <br />Bazsi<br /></div></div></div>
<br /><div class="gmail_quote">On Fri, Jan 6, 2017 at 1:41 PM, Szalai, Attila <span dir="ltr"><<a href="mailto:Attila.Szalai@morganstanley.com" target="_blank">Attila.Szalai@morganstanley.com</a>></span> wrote:<br /><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div link="blue" vlink="purple" lang="EN-US">
<p>
</p><div class="m_1970712568857058622WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Hi James,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Checking the source, it means the following:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">The code allocate a buffer 6 times bigger than the original string length to able to hold the escaped form of the utf-8 character.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">The assert means, that the string, after escaping was not fit into this buffer for some reason. Or, to be more precise, the GString implementation decided that
it should reallocate the string, which usually only happen if the string gets too big to fit into its original place. Currently I have no recent glib source to check if I’m right.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">The original string could help a lot to find the root cause.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Ps.: the escaping works as replacing the original byte with \xHH, so theoretically it can only grows from 1 byte to 4, which should fit into a buffer 6 times
bigger than the original size.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> syslog-ng [mailto:<a href="mailto:syslog-ng-bounces@lists.balabit.hu" target="_blank">syslog-ng-bounces@<wbr />lists.balabit.hu</a>]
<b>On Behalf Of </b>James Elstone<br />
<b>Sent:</b> Thursday, January 05, 2017 10:35 PM<br />
<b>To:</b> <a href="mailto:syslog-ng@lists.balabit.hu" target="_blank">syslog-ng@lists.balabit.hu</a><br />
<b>Subject:</b> [syslog-ng] Hitting g_assert when using sanitize-utf8 enabled!<u></u><u></u></span></p>
</div>
</div><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Hi Balabit et al,<br />
<br />
When using the sanitize-utf8 flag I am hitting a g_assert in modules/syslogformat/syslog-<wbr />format.c; what could be causing this?<br />
<br />
Any advice welcome!!<br />
<br />
Kr,<br />
<br />
James<br />
-- <br />
Sent from my Android device with K-9 Mail. Please excuse my brevity.<u></u><u></u></p>
</div></div></div>
</div>
<br /><br />
<hr id="m_1970712568857058622HR1" />
<br /><span style="FONT-SIZE:7.5pt;FONT-FAMILY:Arial;COLOR:#808080">NOTICE:
Morgan Stanley is not acting as a municipal advisor and the opinions or views
contained herein are not intended to be, and do not constitute, advice within
the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer
Protection Act. If you have received this communication in error, please destroy
all electronic and paper copies and notify the sender immediately.
Mistransmission is not intended to waive confidentiality or privilege. Morgan
Stanley reserves the right, to the extent permitted under applicable law, to
monitor electronic communications. This message is subject to terms available at
the following link: <a href="http://www.morganstanley.com/disclaimers" target="_blank">http://www.morganstanley.com/<wbr />disclaimers</a> If you cannot
access these links, please notify us by reply message and we will send the
contents to you. By communicating with Morgan Stanley you consent to the
foregoing and to the voice recording of conversations with personnel of Morgan
Stanley.</span><br />
<p></p>
<p></p>
<p></p>
<p></p><p></p></div>
<br />______________________________<wbr />______________________________<wbr />__________________<br />
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/<wbr />mailman/listinfo/syslog-ng</a><br />
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/<wbr />support/documentation/?<wbr />product=syslog-ng</a><br />
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/<wbr />syslog-ng-faq</a><br />
<br />
<br /></blockquote></div><br /></div>
</blockquote></div><br>
-- <br>
Sent from my Android device with K-9 Mail. Please excuse my brevity.</body></html>