<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Hi Diego,<br>
<br>
I have only some experience with python performance in syslog-ng, but I don't think you could significantly improve performance.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
let me think about this:<br>
You have a non-scalable udp source (1 thread), and also a non-scalable file destination (it writes one file in one thread).<br>
If syslog-ng is in threaded mode (by default yes, unless the global option threaded(no) is not set), sources and destinations run in different threads.<br>
Parsers run in the same thread as the sources.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt;">With the python parser, the python code would limit the source thread's performance, while a template function would be invoked in the destination's thread.</span><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt;">Based on this, a template function should be better.<br>
<br>
</span><span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt;">Also, please note syslog-ng always processes one message at a time (except when you do correlation with dbparser() or grouping-by()).</span><span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt;"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
These are just my early thoughts, I'll think about this and write you an update if I found out anything.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Regards,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Gabor</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Diego Billi <diego.billi@labs.it><br>
<b>Sent:</b> Monday, November 2, 2020 19:11<br>
<b>To:</b> syslog-ng@lists.balabit.hu <syslog-ng@lists.balabit.hu><br>
<b>Subject:</b> [syslog-ng] Performance issue with python code in template</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.<br>
<br>
<br>
Hi,<br>
I have a performance problem. This is a skeleton of my syslog-ng configuration.<br>
<br>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>><br>
<br>
python {<br>
<br>
import ...mylibs...<br>
<br>
def t_my_python_function(msg):<br>
...<br>
output_data = .... process msg object ...<br>
...<br>
return output_data<br>
<br>
}<br>
<br>
source s_mysource {<br>
udp( .... );<br>
}<br>
<br>
destination d_mydestination {<br>
file(<br>
"/tmp/mylogs.log"<br>
<br>
template("$(python t_my_python_function)")<br>
);<br>
};<br>
<br>
<br>
log {<br>
source(s_mysource);<br>
<br>
destination(d_mydestination);<br>
<br>
flags(flow-control);<br>
};<br>
<br>
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<br>
<br>
<br>
<br>
I receive syslog messages via UDP with a very very high rate.<br>
<br>
Incoming messages are processed with a template written in python.<br>
<br>
Syslog-ng have performance issues with this flow due to the python code.<br>
<br>
I'm wondering how to speed up this solution.<br>
<br>
Note that the python function can be parallelized (no shared state/data between messages)<br>
<br>
<br>
I'm trying this solution but i don't know if it changes that much.<br>
<br>
------------------------------------------------------------------------<br>
<br>
python {<br>
<br>
import ...mylibs...<br>
<br>
def p_my_python_function(msg):<br>
...<br>
...<br>
output_data = .... process msg object ...<br>
...<br>
...<br>
msg['MY_OUTPUT_DATA'] = output_data<br>
}<br>
<br>
source s_mysource {<br>
...<br>
}<br>
<br>
destination d_mydestination {<br>
file(<br>
"/tmp/mylogs.log"<br>
template("${MY_OUTPUT_DATA}")<br>
);<br>
};<br>
<br>
<br>
log {<br>
source(s_mysource);<br>
<br>
parser(p_my_python_function); <---- moved here (outside destination)<br>
<br>
destination(d_mydestination);<br>
<br>
flags(flow-control);<br>
};<br>
<br>
------------------------------------------------------------------------<br>
<br>
Moving the "processing" outside the "destination" is really useful?<br>
<br>
I'm trying to understand if i can use threading and multi-core supporto of syslog-ng.<br>
<br>
Thank you for your time.<br>
<br>
<br>
Diego.<br>
______________________________________________________________________________<br>
Member info: <a href="https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C4da75df2b2d94833acde08d87f5ab0a1%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637399374802446586%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=6Pcy7KDBUtLd%2Fp1E8nev0jaZ0pmZ1M0NPe%2BmMXfzz4M%3D&reserved=0">
https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C4da75df2b2d94833acde08d87f5ab0a1%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637399374802446586%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=6Pcy7KDBUtLd%2Fp1E8nev0jaZ0pmZ1M0NPe%2BmMXfzz4M%3D&reserved=0</a><br>
Documentation: <a href="https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C4da75df2b2d94833acde08d87f5ab0a1%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637399374802446586%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=lIOC7oIMczH2A2vW%2FTdhEcSusIg5NIRCzzcYcyjew5I%3D&reserved=0">
https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C4da75df2b2d94833acde08d87f5ab0a1%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637399374802446586%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=lIOC7oIMczH2A2vW%2FTdhEcSusIg5NIRCzzcYcyjew5I%3D&reserved=0</a><br>
FAQ: <a href="https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C4da75df2b2d94833acde08d87f5ab0a1%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637399374802446586%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=1lkIPNno0DJLF5irsBtPMi7L07fMYVEcLOb6Fw1LxzM%3D&reserved=0">
https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C4da75df2b2d94833acde08d87f5ab0a1%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637399374802446586%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=1lkIPNno0DJLF5irsBtPMi7L07fMYVEcLOb6Fw1LxzM%3D&reserved=0</a><br>
<br>
</div>
</span></font></div>
</body>
</html>