<div dir="auto">Hi,<div dir="auto"><br></div><div dir="auto">I think the biggest bottleneck would be that Python as a language is using a GIL, a global interpreter lock, which means that only one Python evaluation happens at a time, regardless of how many threads we are using.</div><div dir="auto"><br></div><div dir="auto">With that said, I think it is better to have that single thread on the destination side (e.g. template function) as at the very least you wouldn't affect the source threads with this limitation, but this still means that you'd be using 1 thread for output, and 1 thread for input.</div><div dir="auto"><br></div><div dir="auto">To increase parallelism you could use the so-reuseport() option for the udp source and use multiple of them (this is quite a recent feature), with that you can use multiple threads for reception that each would feed the same destination queue. This will only help if you single thread output can cope with the traffic, and depending on the flow-control setting, this might mean that</div><div dir="auto"><br></div><div dir="auto">1) syslog-ng will actively drop messages (flow control disabled, destination queue full)</div><div dir="auto"><br></div><div dir="auto">2) syslog-ng will suspend the source threads, causing the kernel to drop in the socket queue (flow control enabled)</div><div dir="auto"><br></div><div dir="auto">To improve performance further (or at all), you would need to improve the Python code or you would have to switch to using syslog-ng primitives to achieve the same thing.</div><div dir="auto"><br></div><div dir="auto">You can expect 20-25k messages/sec from Python code on decent hardware, assuming you are not doing much.</div><div dir="auto"><br></div><div dir="auto">As an alternative more hacky solution you could run multiple instances of your Python code using a program () destination and loop the results back to syslog-ng somehow (e.g. a file source or a Unix socket). This way the Python GIL wouldn't be a problem.</div><div dir="auto"><br></div><div dir="auto">Hope this helps,</div><div dir="auto">Bazsi</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Nov 3, 2020, 18:29 Gabor Nagy (gnagy) <<a href="mailto:Gabor.Nagy@oneidentity.com">Gabor.Nagy@oneidentity.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
Hi Diego,<br>
<br>
I have only some experience with python performance in syslog-ng, but I don't think you could significantly improve performance.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
let me think about this:<br>
You have a non-scalable udp source (1 thread), and also a non-scalable file destination (it writes one file in one thread).<br>
If syslog-ng is in threaded mode (by default yes, unless the global option threaded(no) is not set), sources and destinations run in different threads.<br>
Parsers run in the same thread as the sources.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
<span style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:11pt">With the python parser, the python code would limit the source thread's performance, while a template function would be invoked in the destination's thread.</span><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
<span style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:11pt">Based on this, a template function should be better.<br>
<br>
</span><span style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:11pt">Also, please note syslog-ng always processes one message at a time (except when you do correlation with dbparser() or grouping-by()).</span><span style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:11pt"><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
These are just my early thoughts, I'll think about this and write you an update if I found out anything.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
Regards,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
Gabor</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
<br>
</div>
<div id="m_3533719150719378023appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="m_3533719150719378023divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> syslog-ng <<a href="mailto:syslog-ng-bounces@lists.balabit.hu" target="_blank" rel="noreferrer">syslog-ng-bounces@lists.balabit.hu</a>> on behalf of Diego Billi <<a href="mailto:diego.billi@labs.it" target="_blank" rel="noreferrer">diego.billi@labs.it</a>><br>
<b>Sent:</b> Monday, November 2, 2020 19:11<br>
<b>To:</b> <a href="mailto:syslog-ng@lists.balabit.hu" target="_blank" rel="noreferrer">syslog-ng@lists.balabit.hu</a> <<a href="mailto:syslog-ng@lists.balabit.hu" target="_blank" rel="noreferrer">syslog-ng@lists.balabit.hu</a>><br>
<b>Subject:</b> [syslog-ng] Performance issue with python code in template</font>
<div> </div>
</div>
<div><font size="2"><span style="font-size:11pt">
<div>CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.<br>
<br>
<br>
Hi,<br>
I have a performance problem. This is a skeleton of my syslog-ng configuration.<br>
<br>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>><br>
<br>
python {<br>
<br>
import ...mylibs...<br>
<br>
def t_my_python_function(msg):<br>
...<br>
output_data = .... process msg object ...<br>
...<br>
return output_data<br>
<br>
}<br>
<br>
source s_mysource {<br>
udp( .... );<br>
}<br>
<br>
destination d_mydestination {<br>
file(<br>
"/tmp/mylogs.log"<br>
<br>
template("$(python t_my_python_function)")<br>
);<br>
};<br>
<br>
<br>
log {<br>
source(s_mysource);<br>
<br>
destination(d_mydestination);<br>
<br>
flags(flow-control);<br>
};<br>
<br>
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<br>
<br>
<br>
<br>
I receive syslog messages via UDP with a very very high rate.<br>
<br>
Incoming messages are processed with a template written in python.<br>
<br>
Syslog-ng have performance issues with this flow due to the python code.<br>
<br>
I'm wondering how to speed up this solution.<br>
<br>
Note that the python function can be parallelized (no shared state/data between messages)<br>
<br>
<br>
I'm trying this solution but i don't know if it changes that much.<br>
<br>
------------------------------------------------------------------------<br>
<br>
python {<br>
<br>
import ...mylibs...<br>
<br>
def p_my_python_function(msg):<br>
...<br>
...<br>
output_data = .... process msg object ...<br>
...<br>
...<br>
msg['MY_OUTPUT_DATA'] = output_data<br>
}<br>
<br>
source s_mysource {<br>
...<br>
}<br>
<br>
destination d_mydestination {<br>
file(<br>
"/tmp/mylogs.log"<br>
template("${MY_OUTPUT_DATA}")<br>
);<br>
};<br>
<br>
<br>
log {<br>
source(s_mysource);<br>
<br>
parser(p_my_python_function); <---- moved here (outside destination)<br>
<br>
destination(d_mydestination);<br>
<br>
flags(flow-control);<br>
};<br>
<br>
------------------------------------------------------------------------<br>
<br>
Moving the "processing" outside the "destination" is really useful?<br>
<br>
I'm trying to understand if i can use threading and multi-core supporto of syslog-ng.<br>
<br>
Thank you for your time.<br>
<br>
<br>
Diego.<br>
______________________________________________________________________________<br>
Member info: <a href="https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C4da75df2b2d94833acde08d87f5ab0a1%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637399374802446586%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=6Pcy7KDBUtLd%2Fp1E8nev0jaZ0pmZ1M0NPe%2BmMXfzz4M%3D&reserved=0" target="_blank" rel="noreferrer">
https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C4da75df2b2d94833acde08d87f5ab0a1%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637399374802446586%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=6Pcy7KDBUtLd%2Fp1E8nev0jaZ0pmZ1M0NPe%2BmMXfzz4M%3D&reserved=0</a><br>
Documentation: <a href="https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C4da75df2b2d94833acde08d87f5ab0a1%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637399374802446586%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=lIOC7oIMczH2A2vW%2FTdhEcSusIg5NIRCzzcYcyjew5I%3D&reserved=0" target="_blank" rel="noreferrer">
https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C4da75df2b2d94833acde08d87f5ab0a1%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637399374802446586%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=lIOC7oIMczH2A2vW%2FTdhEcSusIg5NIRCzzcYcyjew5I%3D&reserved=0</a><br>
FAQ: <a href="https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C4da75df2b2d94833acde08d87f5ab0a1%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637399374802446586%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=1lkIPNno0DJLF5irsBtPMi7L07fMYVEcLOb6Fw1LxzM%3D&reserved=0" target="_blank" rel="noreferrer">
https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C4da75df2b2d94833acde08d87f5ab0a1%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637399374802446586%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=1lkIPNno0DJLF5irsBtPMi7L07fMYVEcLOb6Fw1LxzM%3D&reserved=0</a><br>
<br>
</div>
</span></font></div>
</div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div>