<div dir="auto">Syslog-ng supports breaking up lines based on regényes, but only for files. I could possibly adapt that functionality to network sources of you are willing to try the functionality. But it's best if the source of fixed.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Aug 22, 2022, 14:58 John Law <<a href="mailto:JohnLaw@tfl.gov.uk">JohnLaw@tfl.gov.uk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div style="font-family:Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thank you <span style="font-size:12pt;background-color:rgba(0,0,0,0);display:inline!important">Balázs,</span></div>
<div style="font-family:Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-size:12pt;background-color:rgba(0,0,0,0);display:inline!important"><br>
</span></div>
<div style="font-family:Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-size:12pt;background-color:rgba(0,0,0,0);display:inline!important">I have done some more digging and I don't believe it is syslog that is the issue, I think the traffic is being forwarded to me from another syslog platform and that is
where the issue lies.</span></div>
<div style="font-family:Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-size:12pt;background-color:rgba(0,0,0,0);display:inline!important"><br>
</span></div>
<div style="font-family:Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-size:12pt;background-color:rgba(0,0,0,0);display:inline!important">A packet capture is showing individual events with the correct line breaks, but then it also shows a number of events being sent as one large packet. Unless I can get syslog-ng
to linebreak on regex when the data comes in, I need to go back to the source and get them to address this.</span></div>
<div>
<div style="font-family:Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
PS the template mismatch below was a typo.</div>
<div style="font-family:Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div id="m_-8404207024552464219Signature">
<div>
<div style="font-family:Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
</div>
<div></div>
<div></div>
<div></div>
<div></div>
<div></div>
<div></div>
<div name="divtagdefaultwrapper" style="margin:0px">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif"><span>Kind Regards</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif"><span><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif"><span>John</span></div>
</div>
</div>
</div>
</div>
<div id="m_-8404207024552464219signature_bookmark"></div>
<div id="m_-8404207024552464219appendonsend"></div>
<div style="font-family:Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<hr style="display:inline-block;width:98%">
<div id="m_-8404207024552464219divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> syslog-ng <<a href="mailto:syslog-ng-bounces@lists.balabit.hu" target="_blank" rel="noreferrer">syslog-ng-bounces@lists.balabit.hu</a>> on behalf of Balazs Scheidler <<a href="mailto:bazsi77@gmail.com" target="_blank" rel="noreferrer">bazsi77@gmail.com</a>><br>
<b>Sent:</b> 20 August 2022 06:33<br>
<b>To:</b> Syslog-ng users' and developers' mailing list <<a href="mailto:syslog-ng@lists.balabit.hu" target="_blank" rel="noreferrer">syslog-ng@lists.balabit.hu</a>><br>
<b>Subject:</b> Re: [syslog-ng] Potential issue with line breaking not working correctly when writing logs to disk</font>
<div> </div>
</div>
<div>
<div dir="auto">
<div>Hi,</div>
<div dir="auto"><br>
</div>
<div dir="auto">Sorry I have trouble matching up the config with the log output. Please find some comments inline.</div>
<div dir="auto"><br>
</div>
<div dir="auto">Balázs<br>
<br>
<div dir="auto">
<div dir="ltr">On Fri, Aug 19, 2022, 16:47 John Law <<a href="mailto:JohnLaw@tfl.gov.uk" target="_blank" rel="noreferrer">JohnLaw@tfl.gov.uk</a>> wrote:<br>
</div>
<blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div style="margin:0px 0cm;font-size:11pt;font-family:Calibri,sans-serif">
Hi,
<div><br>
</div>
<div>We have a number of network devices sending syslog traffic to syslog-ng OSE 3.25 installed on RHEL 7.9.</div>
<div><br>
</div>
<div>We are having an issue where multiple events are being written as the same line.</div>
<div><br>
</div>
<div>The config file is </div>
<div><br>
</div>
<div>template-function adm1 "${HOST} message-length=$(length \"${MSG}\") ${MESSAGE}\n";</div>
<div>source s_adm1 {</div>
<div> udp(ip(0.0.0.0) port(5527) );</div>
<div> tcp(ip(0.0.0.0) port(5527) );</div>
<div>};</div>
<div>destination d_adm1 {</div>
<div> file(</div>
<div> "/Data/syslog_data/$LOGHOST/$R_YEAR-$R_MONTH-$R_DAY/adm1/$HOST_FROM/$HOST/$FACILITY.local"</div>
<div> template("$(adm1)\n")</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">You only mention $(adm1) as a template function but this one refers to $(adm1-function)</div>
<div dir="auto"><br>
</div>
<div dir="auto">
<div dir="auto">
<blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div style="margin:0px 0cm;font-size:11pt;font-family:Calibri,sans-serif">
<div> );</div>
<div>};</div>
<div>log { source(s_adm1); destination(d_adm1); flags(final); };</div>
<div><br>
</div>
<div><br>
</div>
<div>A small extract of one of the files is, naturally I have changed the IPs</div>
<div><br>
</div>
<div><br>
</div>
<div><a href="http://webdefence.global.blackspider.com/urlwrap/?q=AXicY3BnUGFmYAgUZWQoyqk0NMjQKy4q08tNzMxJzs8rKcrP0UvOz2UwMvc2jYhMNDU0MDO1MGfIys_Iy0ksdyhJy9FLzy_TK81myCgpKbDS1ze0NNIzBEF9Y0NDC0NdBgYGLR0GBgCskBrr&Z" target="_blank" rel="noreferrer">192.1.1.1/31181-</a>><a href="http://webdefence.global.blackspider.com/urlwrap/?q=AXicY3BnUGFmYAgUZWQoyqk0NMjQKy4q08tNzMxJzs8rKcrP0UvOz2UwMvc2jYhMNDU0MDO1MGfIys_Iy0ksdyhJy9FLzy_TK81myCgpKbDS1ze0NNIz1DPWM9Q3NDZlYGDQ0mFgAABZfBpb&Z" target="_blank" rel="noreferrer">192.1.3.1/135</a>
0x0 source rule r12 N/A N/A 6 ACME-CNB010 ACME CNB 1727847 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN275 <14>Aug 19 11:38:50 sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created
<a href="http://webdefence.global.blackspider.com/urlwrap/?q=AXicBcG9DoIwFAbQz_gMvIGTkduf2Bac3Byc3GBsSLTKhUtqxfj2noMLdlvgVm2Q-Wd0ondeaYpPHmQuWZgGmWDD1XV9dEZ71wS8JM0cv-dyZ3rISp8RqZTlpJRpLXmymow6But0DWB_AP7JSxsl&Z" target="_blank" rel="noreferrer">
192.6.20.1/47250-</a>><a href="http://webdefence.global.blackspider.com/urlwrap/?q=AXicY3BnUGFmYAgUZWQoyqk0NMjQKy4q08tNzMxJzs8rKcrP0UvOz2UwMvc2jYhMNDU0MDO1MGfIys_Iy0ksdyhJy9FLzy_TK81myCgpKbDS1ze0NNIzNDDRMzLQM9Q3MTFmYGDQ0mFgAACs0Brw&Z" target="_blank" rel="noreferrer">192.104.20.1/443</a>
0x0 junos-https <a href="http://webdefence.global.blackspider.com/urlwrap/?q=AXicBcG9DoIwFAbQz_gMvIGTkduf2Bac3Byc3GBsSLTKhUtqxfj2noMLdlvgVm2Q-Wd0ondeaYpPHmQuWZgGmWDD1XV9dEZ71wS8JM0cv-dyZ3rISp8RqZTlpJRpLXmymow6But0DWB_AP7JSxsl&Z" target="_blank" rel="noreferrer">
192.6.20.1/47250-</a>><a href="http://webdefence.global.blackspider.com/urlwrap/?q=AXicY3BnUGFmYAgUZWQoyqk0NMjQKy4q08tNzMxJzs8rKcrP0UvOz2UwMvc2jYhMNDU0MDO1MGfIys_Iy0ksdyhJy9FLzy_TK81myCgpKbDS1ze0NNIzNDDRMzLQM9Q3MTFmYGDQ0mFgAACs0Brw&Z" target="_blank" rel="noreferrer">192.104.20.1/443</a>
0x0 N/A N/A N/A N/A 6 ACME-BWC002 ACME BWC_EDESIX_AWS 917879 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN275 <14>Aug 19 11:38:50 sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created</div>
<div dir="auto"></div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">Can you show me the lines intermixed here? Again this does not seem to have been formatted with $(adm1) at least as defined above.</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">
<div dir="auto">
<blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div style="margin:0px 0cm;font-size:11pt;font-family:Calibri,sans-serif">
<div dir="auto"> <a href="http://webdefence.global.blackspider.com/urlwrap/?q=AXicBcGxDoIwEAbg3_gMvoGT0Wt7sRSZ3ByY3HRsSLTqwZFaML4934cLtmvgulkhy9_ZRN88Ux9f0ulQsgp12oND62_36J2tfB3w1jRI_J3LQ-ipM00fpFLGxhh3YqqILTlzDOz5AGC3BxbJXRsn&Z" target="_blank" rel="noreferrer">192.6.20.1/47252-</a>><a href="http://webdefence.global.blackspider.com/urlwrap/?q=AXicY3BnUGFmYAgUZWQoyqk0NMjQKy4q08tNzMxJzs8rKcrP0UvOz2UwMvc2jYhMNDU0MDO1MGfIys_Iy0ksdyhJy9FLzy_TK81myCgpKbDS1ze0NNIzNDDRMzLQM9Q3MTFmYGDQ0mFgAACs0Brw&Z" target="_blank" rel="noreferrer">192.104.20.1/443</a>
0x0 junos-https <a href="http://webdefence.global.blackspider.com/urlwrap/?q=AXicBcGxDoIwEAbg3_gMvoGT0Wt7sRSZ3ByY3HRsSLTqwZFaML4934cLtmvgulkhy9_ZRN88Ux9f0ulQsgp12oND62_36J2tfB3w1jRI_J3LQ-ipM00fpFLGxhh3YqqILTlzDOz5AGC3BxbJXRsn&Z" target="_blank" rel="noreferrer">
192.6.20.1/47252-</a>><a href="http://webdefence.global.blackspider.com/urlwrap/?q=AXicY3BnUGFmYAgUZWQoyqk0NMjQKy4q08tNzMxJzs8rKcrP0UvOz2UwMvc2jYhMNDU0MDO1MGfIys_Iy0ksdyhJy9FLzy_TK81myCgpKbDS1ze0NNIzNDDRMzLQM9Q3MTFmYGDQ0mFgAACs0Brw&Z" target="_blank" rel="noreferrer">192.104.20.1/443</a>
0x0 N/A N/A N/A N/A 6 ACME-BWC002 ACME BWC_EDESIX_AWS 595069 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN285 <14>Aug 19 11:38:50 sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created
<a href="http://webdefence.global.blackspider.com/urlwrap/?q=AXicBcHBDgExEAbgXzyDN3ASpp3GKHtyc3By49hsQpfZHanuirf3fThhOQcuixmK_thn-pSJ-tRpa0MtptRajxDPcr0lYb-TfcTT8qDpe6x3pYdNNL6Qa303zvEhEJNEYifb4GUDYLUG_sl4Gyg&Z" target="_blank" rel="noreferrer">
192.1.57.1/54205-</a>><a href="http://webdefence.global.blackspider.com/urlwrap/?q=AXicY3BnUGFmYAgUZWQoyqk0NMjQKy4q08tNzMxJzs8rKcrP0UvOz2UwMvc2jYhMNDU0MDO1MGfIys_Iy0ksdyhJy9FLzy_TK81myCgpKbDS1ze0NNIz1DPXM9Q3NDZnYGDQ0mFgAABZxBph&Z" target="_blank" rel="noreferrer">192.1.7.1/137</a>
0x0 junos-nbname <a href="http://webdefence.global.blackspider.com/urlwrap/?q=AXicBcHBDgExEAbgXzyDN3ASpp3GKHtyc3By49hsQpfZHanuirf3fThhOQcuixmK_thn-pSJ-tRpa0MtptRajxDPcr0lYb-TfcTT8qDpe6x3pYdNNL6Qa303zvEhEJNEYifb4GUDYLUG_sl4Gyg&Z" target="_blank" rel="noreferrer">192.1.57.1/54205-</a>><a href="http://webdefence.global.blackspider.com/urlwrap/?q=AXicY3BnUGFmYAgUZWQoyqk0NMjQKy4q08tNzMxJzs8rKcrP0UvOz2UwMvc2jYhMNDU0MDO1MGfIys_Iy0ksdyhJy9FLzy_TK81myCgpKbDS1ze0NNIz1DPXM9Q3NDZnYGDQ0mFgAABZxBph&Z" target="_blank" rel="noreferrer">192.1.7.1/137</a>
0x0 N/A N/A N/A N/A 17 ACME-COMP007 ACME CCA 1735324 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN288 <14>Aug 19 11:38:50 sunny-fwl29 RT_FLOW: RT_FLOW_SESSION_CREATE: session created
<a href="http://webdefence.global.blackspider.com/urlwrap/?q=AXicBcGxDoIwEAbgn_gMvoGT0SvX2Bad3ByY2HRsSLTowZlaMb4934cLNiugW1fI8uc60SfPNMZBep1KVqFeR9jQuustOq69awKemiaJv3O5Cz10pu8LqZT3yRg-WmJygdh4e2j8HsB2ByzJ3Bsy&Z" target="_blank" rel="noreferrer">
192.1.57.1/62486-</a>><a href="http://webdefence.global.blackspider.com/urlwrap/?q=AXicY3BnUGFmYAgUZWQoyqk0NMjQKy4q08tNzMxJzs8rKcrP0UvOz2UwMvc2jYhMNDU0MDO1MGfIys_Iy0ksdyhJy9FLzy_TK81myCgpKbDS1ze0NNIz1DPXM9Q3NDZlYGDQ0mFgAABZtBpf&Z" target="_blank" rel="noreferrer">192.1.7.1/135</a>
0x0 junos-ms-rpc-tcp <a href="http://webdefence.global.blackspider.com/urlwrap/?q=AXicBcGxDoIwEAbgn_gMvoGT0SvX2Bad3ByY2HRsSLTowZlaMb4934cLNiugW1fI8uc60SfPNMZBep1KVqFeR9jQuustOq69awKemiaJv3O5Cz10pu8LqZT3yRg-WmJygdh4e2j8HsB2ByzJ3Bsy&Z" target="_blank" rel="noreferrer">192.1.57.1/62486-</a>><a href="http://webdefence.global.blackspider.com/urlwrap/?q=AXicY3BnUGFmYAgUZWQoyqk0NMjQKy4q08tNzMxJzs8rKcrP0UvOz2UwMvc2jYhMNDU0MDO1MGfIys_Iy0ksdyhJy9FLzy_TK81myCgpKbDS1ze0NNIz1DPXM9Q3NDZlYGDQ0mFgAABZtBpf&Z" target="_blank" rel="noreferrer">192.1.7.1/135</a>
0x0 N/A N/A N/A N/A 6 ACME-COMP007 ACME CCA 1027474 N/A(N/A) reth3.860 UNKNOWN UNKNOWN UNKNOWN249</div>
<div><br>
</div>
I suspect it might be something to do with the amount of logs that are received in very quick succession, but I have no idea how to address this. One thing I have noticed is that it always appears to be the same device that is causing the issue. When I look
at other devices coming in on the same port, they appear to write out correctly.<br>
</div>
<div style="margin:0px 0cm;font-size:11pt;font-family:Calibri,sans-serif">
<br>
</div>
<div style="margin:0px 0cm;font-size:11pt;font-family:Calibri,sans-serif">
Any suggestion would be really welcome.</div>
<div style="margin:0px 0cm;font-size:11pt;font-family:Calibri,sans-serif">
<br>
</div>
<div style="margin:0px 0cm;font-size:11pt;font-family:Calibri,sans-serif">
Thanks</div>
<div style="margin:0px 0cm;font-size:11pt;font-family:Calibri,sans-serif">
<br>
</div>
<div style="margin:0px 0cm;font-size:11pt;font-family:Calibri,sans-serif">
John</div>
</div>
<br>
<br>
<p align="center"><font style="background-color:#ffffff">This message has been scanned for malware by Forcepoint.
</font><a href="http://www.forcepoint.com/" target="_blank" rel="noreferrer"><font color="#000000" style="background-color:#ffffff">www.forcepoint.com</font></a></p>
</div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank" rel="noreferrer">
https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank" rel="noreferrer">
http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank" rel="noreferrer">
http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote>
</div>
</div>
</div>
</div>
</div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div>