Hi,
I have a performance problem. This is a skeleton of my syslog-ng configuration.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
python {
import ...mylibs...
def t_my_python_function(msg):
...
output_data = .... process msg object ...
...
return output_data
}
source s_mysource {
udp( .... );
}
destination d_mydestination {
file(
"/tmp/mylogs.log"
template("$(python t_my_python_function)")
);
};
log {
source(s_mysource);
destination(d_mydestination);
flags(flow-control);
};
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
I receive syslog messages via UDP with a very very high rate.
Incoming messages are processed with a template written in python.
Syslog-ng have performance issues with this flow due to the python code.
I'm wondering how to speed up this solution.
Note that the python function can be parallelized (no shared state/data between messages)
I'm trying this solution but i don't know if it changes that much.
------------------------------------------------------------------------
python {
import ...mylibs...
def p_my_python_function(msg):
...
...
output_data = .... process msg object ...
...
...
msg['MY_OUTPUT_DATA'] = output_data
}
source s_mysource {
...
}
destination d_mydestination {
file(
"/tmp/mylogs.log"
template("${MY_OUTPUT_DATA}")
);
};
log {
source(s_mysource);
parser(p_my_python_function); <---- moved here (outside destination)
destination(d_mydestination);
flags(flow-control);
};
------------------------------------------------------------------------
Moving the "processing" outside the "destination" is really useful?
I'm trying to understand if i can use threading and multi-core supporto of syslog-ng.
Thank you for your time.
Diego.