Problem using Python Parser
When I try to load my config with a python parser in it I'm getting this error message: Starting /opt/syslog-ng/sbin/syslog-ng: [2017-12-29T23:00:05.813945] Error loading Python module; module='_syslogng', exception='exceptions.ImportError: No module named _syslogng' [2017-12-29T23:00:05.814066] Error looking Python parser class; parser='p_php_fpm', class='PhpFpmParser', exception='None' [2017-12-29T23:00:05.814116] Error initializing message pipeline; plugin name='python', location='/opt/syslog-ng/etc/syslog-ng.conf:52:20' I build syslog-ng from source with the python options. Here is the -V output syslog-ng 3 (3.11.1) Installer-Version: 3.11.1 Revision: Compile-Date: Dec 29 2017 21:24:13 Module-Directory: /opt/syslog-ng/lib/syslog-ng Module-Path: /opt/syslog-ng/lib/syslog-ng Available-Modules: snmptrapd-parser,affile,cef,afstomp,basicfuncs,pseudofile,tfgetent,afsocket,mod-python,json-plugin,afuser,kvformat,stardate,graphite,dbparser,csvparser,date,afmongodb,system-source,disk-buffer,confgen,linux-kmsg-format,afamqp,map-value-pairs,http,afprog,add-contextual-data,sdjournal,cryptofuncs,syslogformat Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off Enable-Systemd: off Not sure how to fix this as from what I can tell this module is supposed to be compiled in and automatically imported. Ronald Fenner Programmer Game Circus LLC. rfenner@gamecircus.com
The _syslogng module is automatically created from the top level python block in syslog-ng and behaves similarly to the python __main__ module. Do you explicitly import that module using the imports() option? Can you please post your config? On Dec 30, 2017 00:27, "Ronald Fenner" <rfenner@gamecircus.com> wrote:
When I try to load my config with a python parser in it I'm getting this error message: Starting /opt/syslog-ng/sbin/syslog-ng: [2017-12-29T23:00:05.813945] Error loading Python module; module='_syslogng', exception='exceptions.ImportError: No module named _syslogng' [2017-12-29T23:00:05.814066] Error looking Python parser class; parser='p_php_fpm', class='PhpFpmParser', exception='None' [2017-12-29T23:00:05.814116] Error initializing message pipeline; plugin name='python', location='/opt/syslog-ng/etc/syslog-ng.conf:52:20'
I build syslog-ng from source with the python options. Here is the -V output syslog-ng 3 (3.11.1) Installer-Version: 3.11.1 Revision: Compile-Date: Dec 29 2017 21:24:13 Module-Directory: /opt/syslog-ng/lib/syslog-ng Module-Path: /opt/syslog-ng/lib/syslog-ng Available-Modules: snmptrapd-parser,affile,cef, afstomp,basicfuncs,pseudofile,tfgetent,afsocket,mod-python, json-plugin,afuser,kvformat,stardate,graphite,dbparser, csvparser,date,afmongodb,system-source,disk-buffer, confgen,linux-kmsg-format,afamqp,map-value-pairs,http, afprog,add-contextual-data,sdjournal,cryptofuncs,syslogformat Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off Enable-Systemd: off
Not sure how to fix this as from what I can tell this module is supposed to be compiled in and automatically imported.
Ronald Fenner Programmer Game Circus LLC.
rfenner@gamecircus.com
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Here's the config I've redacted the Kafka servers. I've tried adding a @module "mod-python" but it doesn't help. ############################################################################# # Default syslog-ng.conf file which collects all local logs into a # single file called /var/log/messages. # @version: 3.11 @module "mod-java" @include "scl.conf" source s_internal {internal();}; source s_rtl_stream { unix-stream("/var/log/rtl-stream.sock" flags(no-parse)); }; source s_php_fpm { file("/var/log/php-fpm.www.log", flags(no-parse)); }; destination d_kafka_unstructured { kafka ( client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka_2.11-0.11.0.0/libs/") kafka-bootstrap-servers("******") topic("syslog-ng-{{DEPLOYMENT}}") ); }; destination d_kafka_structured { kafka ( client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka_2.11-0.11.0.0/libs/") kafka-bootstrap-servers("*****") topic("${topic}.{{DEPLOYMENT}}") template("$(format-json --scope nv_pairs --exclude MESSAGE)\n") ); }; destination d_syslog_ng { file("/var/log/syslog-ng"); }; destination d_test_log { file("/var/log/test.log"); }; parser p_json { json-parser(); }; parser p_apache { apache-accesslog-parser(prefix("")); }; parser p_php_fpm { python(class("PhpFpmParser")); }; rewrite r_add_access_topic { set("access.log", value("topic")); }; log { source(s_internal); destination(d_syslog_ng); }; log { source(s_rtl_stream); parser(p_json); destination(d_kafka_structured); }; log { source(s_php_fpm); parser(p_php_fpm); destination(d_test_log); }; Here's the actual python parser: python { class PhpFpmParser(object): def parse(self, log_msg): msg = log_msg['MESSAGE'] str_pos = msg.find('] ') if str_pos == -1: return True log_date = msg[1:str_pos] msg = msg[str_pos+2:] str_pos = msg.find(':') if str_pos == -1: return True level = msg[:str_pos] if "Parse" in level: level = "parse" elif "Compile" in level: level = 'compile' elif "Fatal" in level: level = 'fatal' elif "Core" in level: level = 'core' elif "Notice" in level: level = 'notice' elif "Warning" in level: level = 'warning' msg = msg[str_pos+2:].strip() log_msg['err_msg'] = msg log_msg['log_level'] = level log_msg['timestamp_utc'] = log_date return True }; It's stored the the etc/conf.d directory within the syslog path.. Ronald Fenner Programmer Game Circus LLC. rfenner@gamecircus.com
On Dec 29, 2017, at 11:52 PM, Scheidler, Balázs <balazs.scheidler@balabit.com> wrote:
The _syslogng module is automatically created from the top level python block in syslog-ng and behaves similarly to the python __main__ module.
Do you explicitly import that module using the imports() option?
Can you please post your config?
On Dec 30, 2017 00:27, "Ronald Fenner" <rfenner@gamecircus.com> wrote: When I try to load my config with a python parser in it I'm getting this error message: Starting /opt/syslog-ng/sbin/syslog-ng: [2017-12-29T23:00:05.813945] Error loading Python module; module='_syslogng', exception='exceptions.ImportError: No module named _syslogng' [2017-12-29T23:00:05.814066] Error looking Python parser class; parser='p_php_fpm', class='PhpFpmParser', exception='None' [2017-12-29T23:00:05.814116] Error initializing message pipeline; plugin name='python', location='/opt/syslog-ng/etc/syslog-ng.conf:52:20'
I build syslog-ng from source with the python options. Here is the -V output syslog-ng 3 (3.11.1) Installer-Version: 3.11.1 Revision: Compile-Date: Dec 29 2017 21:24:13 Module-Directory: /opt/syslog-ng/lib/syslog-ng Module-Path: /opt/syslog-ng/lib/syslog-ng Available-Modules: snmptrapd-parser,affile,cef,afstomp,basicfuncs,pseudofile,tfgetent,afsocket,mod-python,json-plugin,afuser,kvformat,stardate,graphite,dbparser,csvparser,date,afmongodb,system-source,disk-buffer,confgen,linux-kmsg-format,afamqp,map-value-pairs,http,afprog,add-contextual-data,sdjournal,cryptofuncs,syslogformat Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off Enable-Systemd: off
Not sure how to fix this as from what I can tell this module is supposed to be compiled in and automatically imported.
Ronald Fenner Programmer Game Circus LLC.
rfenner@gamecircus.com
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, The problem is syslog-ng cannot import the user code: [2017-12-29T23:00:05.814066] Error looking Python parser class; parser='p_php_fpm', class='PhpFpmParser', exception='None' Please try either to put the python user code python { class PhpFpmParser(object): ... }; directly into syslog-ng.conf, or you can use @include to include the file that contains the parser code. The _syslogng module is created by syslog-ng config parse: when syslog-ng processes the python keyword with the user code. So the _syslogng import problem should be resolved automatically with the above. Br, Antal On Sat, Dec 30, 2017 at 7:09 AM Ronald Fenner <rfenner@gamecircus.com> wrote:
Here's the config I've redacted the Kafka servers. I've tried adding a @module "mod-python" but it doesn't help.
############################################################################# # Default syslog-ng.conf file which collects all local logs into a # single file called /var/log/messages. #
@version: 3.11 @module "mod-java" @include "scl.conf"
source s_internal {internal();};
source s_rtl_stream { unix-stream("/var/log/rtl-stream.sock" flags(no-parse)); };
source s_php_fpm { file("/var/log/php-fpm.www.log", flags(no-parse)); };
destination d_kafka_unstructured { kafka (
client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka_2.11-0.11.0.0/libs/") kafka-bootstrap-servers("******") topic("syslog-ng-{{DEPLOYMENT}}") ); };
destination d_kafka_structured { kafka (
client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka_2.11-0.11.0.0/libs/") kafka-bootstrap-servers("*****") topic("${topic}.{{DEPLOYMENT}}") template("$(format-json --scope nv_pairs --exclude MESSAGE)\n") ); };
destination d_syslog_ng { file("/var/log/syslog-ng"); };
destination d_test_log { file("/var/log/test.log"); };
parser p_json { json-parser(); };
parser p_apache { apache-accesslog-parser(prefix("")); };
parser p_php_fpm { python(class("PhpFpmParser")); };
rewrite r_add_access_topic { set("access.log", value("topic")); };
log { source(s_internal); destination(d_syslog_ng); };
log { source(s_rtl_stream); parser(p_json); destination(d_kafka_structured); };
log { source(s_php_fpm); parser(p_php_fpm); destination(d_test_log); };
Here's the actual python parser: python { class PhpFpmParser(object): def parse(self, log_msg): msg = log_msg['MESSAGE'] str_pos = msg.find('] ') if str_pos == -1: return True log_date = msg[1:str_pos] msg = msg[str_pos+2:] str_pos = msg.find(':') if str_pos == -1: return True level = msg[:str_pos] if "Parse" in level: level = "parse" elif "Compile" in level: level = 'compile' elif "Fatal" in level: level = 'fatal' elif "Core" in level: level = 'core' elif "Notice" in level: level = 'notice' elif "Warning" in level: level = 'warning' msg = msg[str_pos+2:].strip() log_msg['err_msg'] = msg log_msg['log_level'] = level log_msg['timestamp_utc'] = log_date return True };
It's stored the the etc/conf.d directory within the syslog path..
Ronald Fenner Programmer Game Circus LLC.
rfenner@gamecircus.com
On Dec 29, 2017, at 11:52 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
The _syslogng module is automatically created from the top level python block in syslog-ng and behaves similarly to the python __main__ module.
Do you explicitly import that module using the imports() option?
Can you please post your config?
On Dec 30, 2017 00:27, "Ronald Fenner" <rfenner@gamecircus.com> wrote: When I try to load my config with a python parser in it I'm getting this error message: Starting /opt/syslog-ng/sbin/syslog-ng: [2017-12-29T23:00:05.813945] Error loading Python module; module='_syslogng', exception='exceptions.ImportError: No module named _syslogng' [2017-12-29T23:00:05.814066] Error looking Python parser class; parser='p_php_fpm', class='PhpFpmParser', exception='None' [2017-12-29T23:00:05.814116] Error initializing message pipeline; plugin name='python', location='/opt/syslog-ng/etc/syslog-ng.conf:52:20'
I build syslog-ng from source with the python options. Here is the -V output syslog-ng 3 (3.11.1) Installer-Version: 3.11.1 Revision: Compile-Date: Dec 29 2017 21:24:13 Module-Directory: /opt/syslog-ng/lib/syslog-ng Module-Path: /opt/syslog-ng/lib/syslog-ng Available-Modules: snmptrapd-parser,affile,cef,afstomp,basicfuncs,pseudofile,tfgetent,afsocket,mod-python,json-plugin,afuser,kvformat,stardate,graphite,dbparser,csvparser,date,afmongodb,system-source,disk-buffer,confgen,linux-kmsg-format,afamqp,map-value-pairs,http,afprog,add-contextual-data,sdjournal,cryptofuncs,syslogformat Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off Enable-Systemd: off
Not sure how to fix this as from what I can tell this module is supposed to be compiled in and automatically imported.
Ronald Fenner Programmer Game Circus LLC.
rfenner@gamecircus.com
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Including the file worked. I thought scl.conf was supposed to be including anything in the conf.d directory but looking into the file again it's actually the scl directory. Ronald Fenner Programmer Game Circus LLC. rfenner@gamecircus.com
On Dec 30, 2017, at 1:12 AM, Antal Nemes <antal.nemes@balabit.com> wrote:
Hi,
The problem is syslog-ng cannot import the user code: [2017-12-29T23:00:05.814066] Error looking Python parser class; parser='p_php_fpm', class='PhpFpmParser', exception='None'
Please try either to put the python user code python { class PhpFpmParser(object): ... }; directly into syslog-ng.conf, or you can use @include to include the file that contains the parser code.
The _syslogng module is created by syslog-ng config parse: when syslog-ng processes the python keyword with the user code. So the _syslogng import problem should be resolved automatically with the above.
Br, Antal
On Sat, Dec 30, 2017 at 7:09 AM Ronald Fenner <rfenner@gamecircus.com <mailto:rfenner@gamecircus.com>> wrote: Here's the config I've redacted the Kafka servers. I've tried adding a @module "mod-python" but it doesn't help.
############################################################################# # Default syslog-ng.conf file which collects all local logs into a # single file called /var/log/messages. #
@version: 3.11 @module "mod-java" @include "scl.conf"
source s_internal {internal();};
source s_rtl_stream { unix-stream("/var/log/rtl-stream.sock" flags(no-parse)); };
source s_php_fpm { file("/var/log/php-fpm.www.log", flags(no-parse)); };
destination d_kafka_unstructured { kafka ( client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka_2.11-0.11.0.0/libs/") kafka-bootstrap-servers("******") topic("syslog-ng-{{DEPLOYMENT}}") ); };
destination d_kafka_structured { kafka ( client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka_2.11-0.11.0.0/libs/") kafka-bootstrap-servers("*****") topic("${topic}.{{DEPLOYMENT}}") template("$(format-json --scope nv_pairs --exclude MESSAGE)\n") ); };
destination d_syslog_ng { file("/var/log/syslog-ng"); };
destination d_test_log { file("/var/log/test.log"); };
parser p_json { json-parser(); };
parser p_apache { apache-accesslog-parser(prefix("")); };
parser p_php_fpm { python(class("PhpFpmParser")); };
rewrite r_add_access_topic { set("access.log", value("topic")); };
log { source(s_internal); destination(d_syslog_ng); };
log { source(s_rtl_stream); parser(p_json); destination(d_kafka_structured); };
log { source(s_php_fpm); parser(p_php_fpm); destination(d_test_log); };
Here's the actual python parser: python { class PhpFpmParser(object): def parse(self, log_msg): msg = log_msg['MESSAGE'] str_pos = msg.find('] ') if str_pos == -1: return True log_date = msg[1:str_pos] msg = msg[str_pos+2:] str_pos = msg.find(':') if str_pos == -1: return True level = msg[:str_pos] if "Parse" in level: level = "parse" elif "Compile" in level: level = 'compile' elif "Fatal" in level: level = 'fatal' elif "Core" in level: level = 'core' elif "Notice" in level: level = 'notice' elif "Warning" in level: level = 'warning' msg = msg[str_pos+2:].strip() log_msg['err_msg'] = msg log_msg['log_level'] = level log_msg['timestamp_utc'] = log_date return True };
It's stored the the etc/conf.d directory within the syslog path..
Ronald Fenner Programmer Game Circus LLC.
rfenner@gamecircus.com <mailto:rfenner@gamecircus.com>
On Dec 29, 2017, at 11:52 PM, Scheidler, Balázs <balazs.scheidler@balabit.com <mailto:balazs.scheidler@balabit.com>> wrote:
The _syslogng module is automatically created from the top level python block in syslog-ng and behaves similarly to the python __main__ module.
Do you explicitly import that module using the imports() option?
Can you please post your config?
On Dec 30, 2017 00:27, "Ronald Fenner" <rfenner@gamecircus.com <mailto:rfenner@gamecircus.com>> wrote: When I try to load my config with a python parser in it I'm getting this error message: Starting /opt/syslog-ng/sbin/syslog-ng: [2017-12-29T23:00:05.813945] Error loading Python module; module='_syslogng', exception='exceptions.ImportError: No module named _syslogng' [2017-12-29T23:00:05.814066] Error looking Python parser class; parser='p_php_fpm', class='PhpFpmParser', exception='None' [2017-12-29T23:00:05.814116] Error initializing message pipeline; plugin name='python', location='/opt/syslog-ng/etc/syslog-ng.conf:52:20'
I build syslog-ng from source with the python options. Here is the -V output syslog-ng 3 (3.11.1) Installer-Version: 3.11.1 Revision: Compile-Date: Dec 29 2017 21:24:13 Module-Directory: /opt/syslog-ng/lib/syslog-ng Module-Path: /opt/syslog-ng/lib/syslog-ng Available-Modules: snmptrapd-parser,affile,cef,afstomp,basicfuncs,pseudofile,tfgetent,afsocket,mod-python,json-plugin,afuser,kvformat,stardate,graphite,dbparser,csvparser,date,afmongodb,system-source,disk-buffer,confgen,linux-kmsg-format,afamqp,map-value-pairs,http,afprog,add-contextual-data,sdjournal,cryptofuncs,syslogformat Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off Enable-Systemd: off
Not sure how to fix this as from what I can tell this module is supposed to be compiled in and automatically imported.
Ronald Fenner Programmer Game Circus LLC.
rfenner@gamecircus.com <mailto:rfenner@gamecircus.com>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (3)
-
Antal Nemes
-
Ronald Fenner
-
Scheidler, Balázs