[syslog-ng] Performance issue with python code in template

[Gabor Nagy (gnagy)]

Hi Diego,

I have only some experience with python performance in syslog-ng, but I don't think you could significantly improve performance.
let me think about this:
You have a non-scalable udp source (1 thread), and also a non-scalable file destination (it writes one file in one thread).
If syslog-ng is in threaded mode (by default yes, unless the global option threaded(no) is not set), sources and destinations run in different threads.
Parsers run in the same thread as the sources.
With the python parser, the python code would limit the source thread's performance, while a template function would be invoked in the destination's thread.
Based on this, a template function should be better.

Also, please note syslog-ng always processes one message at a time (except when you do correlation with dbparser() or grouping-by()).

These are just my early thoughts, I'll think about this and write you an update if I found out anything.

Regards,
Gabor

________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Diego Billi <diego.billi at labs.it>
Sent: Monday, November 2, 2020 19:11
To: syslog-ng at lists.balabit.hu <syslog-ng at lists.balabit.hu>
Subject: [syslog-ng] Performance issue with python code in template

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.


Hi,
I have a performance problem. This is a skeleton of my syslog-ng configuration.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

python {

import ...mylibs...

def t_my_python_function(msg):
    ...
    output_data = .... process msg object ...
    ...
    return output_data

}

source s_mysource {
        udp( ....  );
}

destination d_mydestination {
        file(
            "/tmp/mylogs.log"

            template("$(python t_my_python_function)")
        );
};


log {
        source(s_mysource);

        destination(d_mydestination);

        flags(flow-control);
};

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<



I receive syslog messages via UDP with a very very high rate.

Incoming messages are processed with a template written in python.

Syslog-ng have performance issues with this flow due to the python code.

I'm wondering how to speed up this solution.

Note that the python function can be parallelized (no shared state/data between messages)


I'm trying this solution but i don't know if it changes that much.

------------------------------------------------------------------------

python {

import ...mylibs...

def p_my_python_function(msg):
    ...
    ...
    output_data = .... process msg object ...
    ...
    ...
    msg['MY_OUTPUT_DATA'] =  output_data
}

source s_mysource {
        ...
}

destination d_mydestination {
        file(
            "/tmp/mylogs.log"
            template("${MY_OUTPUT_DATA}")
        );
};


log {
        source(s_mysource);

        parser(p_my_python_function);   <---- moved here (outside destination)

        destination(d_mydestination);

        flags(flow-control);
};

------------------------------------------------------------------------

Moving the "processing" outside the "destination" is really useful?

I'm trying to understand if i can use threading and multi-core supporto of syslog-ng.

Thank you for your time.


Diego.
______________________________________________________________________________
Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C4da75df2b2d94833acde08d87f5ab0a1%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637399374802446586%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=6Pcy7KDBUtLd%2Fp1E8nev0jaZ0pmZ1M0NPe%2BmMXfzz4M%3D&reserved=0
Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C4da75df2b2d94833acde08d87f5ab0a1%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637399374802446586%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=lIOC7oIMczH2A2vW%2FTdhEcSusIg5NIRCzzcYcyjew5I%3D&reserved=0
FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C4da75df2b2d94833acde08d87f5ab0a1%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637399374802446586%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=1lkIPNno0DJLF5irsBtPMi7L07fMYVEcLOb6Fw1LxzM%3D&reserved=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20201103/0d31cab7/attachment.html>