[syslog-ng] Performance issue with python code in template

2 Nov 2020

      Hi,
I have a performance problem. This is a skeleton of my syslog-ng configuration.
...
...
...
...
...
...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
python {

import ...mylibs...

def t_my_python_function(msg):
    ...
    output_data = .... process msg object ...
    ...
    return output_data

}

source s_mysource {
        udp( ....  );
}

destination d_mydestination {
        file(
            "/tmp/mylogs.log"

            template("$(python t_my_python_function)")
        );
};

log {
        source(s_mysource); 

        destination(d_mydestination);

        flags(flow-control);
};

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

I receive syslog messages via UDP with a very very high rate.

Incoming messages are processed with a template written in python.

Syslog-ng have performance issues with this flow due to the python code.

I'm wondering how to speed up this solution. 

Note that the python function can be parallelized (no shared state/data between messages) 

I'm trying this solution but i don't know if it changes that much.

------------------------------------------------------------------------

python {

import ...mylibs...

def p_my_python_function(msg):
    ...
    ...
    output_data = .... process msg object ...
    ...
    ...
    msg['MY_OUTPUT_DATA'] =  output_data
}

source s_mysource {
        ...
}

destination d_mydestination {
        file(
            "/tmp/mylogs.log"
            template("${MY_OUTPUT_DATA}")
        );
};

log {
        source(s_mysource); 

        parser(p_my_python_function);   <---- moved here (outside destination)

        destination(d_mydestination);

        flags(flow-control);
};

------------------------------------------------------------------------

Moving the "processing" outside the "destination" is really useful?

I'm trying to understand if i can use threading and multi-core supporto of syslog-ng.

Thank you for your time.

Diego.

[syslog-ng] Performance issue with python code in template

Diego Billi