CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.


Hi,
I have a performance problem. This is a skeleton of my syslog-ng configuration.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

python {

import ...mylibs...

def t_my_python_function(msg):
    ...
    output_data = .... process msg object ...
    ...
    return output_data

}

source s_mysource {
        udp( ....  );
}

destination d_mydestination {
        file(
            "/tmp/mylogs.log"

            template("$(python t_my_python_function)")
        );
};


log {
        source(s_mysource);

        destination(d_mydestination);

        flags(flow-control);
};

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<



I receive syslog messages via UDP with a very very high rate.

Incoming messages are processed with a template written in python.

Syslog-ng have performance issues with this flow due to the python code.

I'm wondering how to speed up this solution.

Note that the python function can be parallelized (no shared state/data between messages)


I'm trying this solution but i don't know if it changes that much.

------------------------------------------------------------------------

python {

import ...mylibs...

def p_my_python_function(msg):
    ...
    ...
    output_data = .... process msg object ...
    ...
    ...
    msg['MY_OUTPUT_DATA'] =  output_data
}

source s_mysource {
        ...
}

destination d_mydestination {
        file(
            "/tmp/mylogs.log"
            template("${MY_OUTPUT_DATA}")
        );
};


log {
        source(s_mysource);

        parser(p_my_python_function);   <---- moved here (outside destination)

        destination(d_mydestination);

        flags(flow-control);
};

------------------------------------------------------------------------

Moving the "processing" outside the "destination" is really useful?

I'm trying to understand if i can use threading and multi-core supporto of syslog-ng.

Thank you for your time.


Diego.
______________________________________________________________________________
Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&amp;data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C4da75df2b2d94833acde08d87f5ab0a1%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637399374802446586%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=6Pcy7KDBUtLd%2Fp1E8nev0jaZ0pmZ1M0NPe%2BmMXfzz4M%3D&amp;reserved=0
Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&amp;data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C4da75df2b2d94833acde08d87f5ab0a1%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637399374802446586%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=lIOC7oIMczH2A2vW%2FTdhEcSusIg5NIRCzzcYcyjew5I%3D&amp;reserved=0
FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&amp;data=04%7C01%7Cgabor.nagy%40oneidentity.com%7C4da75df2b2d94833acde08d87f5ab0a1%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637399374802446586%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=1lkIPNno0DJLF5irsBtPMi7L07fMYVEcLOb6Fw1LxzM%3D&amp;reserved=0