[syslog-ng] Problem using Python Parser

Ronald Fenner rfenner at gamecircus.com
Sat Dec 30 15:51:05 UTC 2017 

Including the file worked. I thought scl.conf was supposed to be including anything in the conf.d directory but looking into the file again it's actually the scl directory.


Ronald Fenner
Programmer
Game Circus LLC.

rfenner at gamecircus.com

> On Dec 30, 2017, at 1:12 AM, Antal Nemes <antal.nemes at balabit.com> wrote:
> 
>    Hi,
> 
> The problem is syslog-ng cannot import the user code: 
> [2017-12-29T23:00:05.814066] Error looking Python parser class; parser='p_php_fpm', class='PhpFpmParser', exception='None'
> 
> Please try either to put the python user code 
> python {
>     class PhpFpmParser(object):
>     ...
> };
> directly into syslog-ng.conf, or you can use @include to include the file that contains the parser code.
> 
> The _syslogng module is created by syslog-ng config parse: when syslog-ng processes the python keyword with the user code. So the _syslogng import problem should be resolved automatically with the above.
> 
> Br,
>   Antal
> 
> On Sat, Dec 30, 2017 at 7:09 AM Ronald Fenner <rfenner at gamecircus.com <mailto:rfenner at gamecircus.com>> wrote:
> Here's the config I've redacted the Kafka servers. I've tried adding a @module "mod-python" but it doesn't help.
> 
> #############################################################################
> # Default syslog-ng.conf file which collects all local logs into a
> # single file called /var/log/messages.
> #
> 
> @version: 3.11
> @module "mod-java"
> @include "scl.conf"
> 
> source s_internal {internal();};
> 
> source s_rtl_stream {
> unix-stream("/var/log/rtl-stream.sock" flags(no-parse));
> };
> 
> source s_php_fpm {
> file("/var/log/php-fpm.www.log", flags(no-parse));
> };
> 
> destination d_kafka_unstructured {
> kafka (
> client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka_2.11-0.11.0.0/libs/")
> kafka-bootstrap-servers("******")
> topic("syslog-ng-{{DEPLOYMENT}}")
> );
> };
> 
> destination d_kafka_structured {
> kafka (
> client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka_2.11-0.11.0.0/libs/")
> kafka-bootstrap-servers("*****")
> topic("${topic}.{{DEPLOYMENT}}")
> template("$(format-json --scope nv_pairs --exclude MESSAGE)\n")
> );
> };
> 
> destination d_syslog_ng {
> file("/var/log/syslog-ng");
> };
> 
> destination d_test_log {
> file("/var/log/test.log");
> };
> 
> parser p_json { json-parser(); };
> 
> parser p_apache { apache-accesslog-parser(prefix("")); };
> 
> parser p_php_fpm { python(class("PhpFpmParser")); };
> 
> rewrite r_add_access_topic {
> set("access.log", value("topic"));
> };
> 
> log {
> source(s_internal);
> destination(d_syslog_ng);
> };
> 
> log {
> source(s_rtl_stream);
> parser(p_json);
> destination(d_kafka_structured);
> };
> 
> log {
> source(s_php_fpm);
> parser(p_php_fpm);
> destination(d_test_log);
> };
> 
> 
> Here's the actual python parser:
> python {
> class PhpFpmParser(object):
>     def parse(self, log_msg):
>         msg = log_msg['MESSAGE']
>         str_pos = msg.find('] ')
>         if str_pos == -1:
>             return True
>         log_date = msg[1:str_pos]
>         msg = msg[str_pos+2:]
>         str_pos = msg.find(':')
>         if str_pos == -1:
>             return True
>         level = msg[:str_pos]
>         if "Parse" in level:
>             level = "parse"
>         elif "Compile" in level:
>             level = 'compile'
>         elif "Fatal" in level:
>             level = 'fatal'
>         elif "Core" in level:
>             level = 'core'
>         elif "Notice" in level:
>             level = 'notice'
>         elif "Warning" in level:
>             level = 'warning'
>         msg = msg[str_pos+2:].strip()
>         log_msg['err_msg'] = msg
>         log_msg['log_level'] = level
>         log_msg['timestamp_utc'] = log_date
>         return True
> };
> 
> It's stored the the etc/conf.d directory within the syslog path..
> 
> Ronald Fenner
> Programmer
> Game Circus LLC.
> 
> rfenner at gamecircus.com <mailto:rfenner at gamecircus.com>
> 
> > On Dec 29, 2017, at 11:52 PM, Scheidler, Balázs <balazs.scheidler at balabit.com <mailto:balazs.scheidler at balabit.com>> wrote:
> >
> > The _syslogng module is automatically created from the top level python block in syslog-ng and behaves similarly to the python __main__ module.
> >
> > Do you explicitly import that module using the imports() option?
> >
> > Can you please post your config?
> >
> > On Dec 30, 2017 00:27, "Ronald Fenner" <rfenner at gamecircus.com <mailto:rfenner at gamecircus.com>> wrote:
> > When I try to load my config with a python parser in it I'm getting this error message:
> > Starting /opt/syslog-ng/sbin/syslog-ng: [2017-12-29T23:00:05.813945] Error loading Python module; module='_syslogng', exception='exceptions.ImportError: No module named _syslogng'
> > [2017-12-29T23:00:05.814066] Error looking Python parser class; parser='p_php_fpm', class='PhpFpmParser', exception='None'
> > [2017-12-29T23:00:05.814116] Error initializing message pipeline; plugin name='python', location='/opt/syslog-ng/etc/syslog-ng.conf:52:20'
> >
> > I build syslog-ng from source with the python options. Here is the -V output
> > syslog-ng 3 (3.11.1)
> > Installer-Version: 3.11.1
> > Revision:
> > Compile-Date: Dec 29 2017 21:24:13
> > Module-Directory: /opt/syslog-ng/lib/syslog-ng
> > Module-Path: /opt/syslog-ng/lib/syslog-ng
> > Available-Modules: snmptrapd-parser,affile,cef,afstomp,basicfuncs,pseudofile,tfgetent,afsocket,mod-python,json-plugin,afuser,kvformat,stardate,graphite,dbparser,csvparser,date,afmongodb,system-source,disk-buffer,confgen,linux-kmsg-format,afamqp,map-value-pairs,http,afprog,add-contextual-data,sdjournal,cryptofuncs,syslogformat
> > Enable-Debug: off
> > Enable-GProf: off
> > Enable-Memtrace: off
> > Enable-IPv6: on
> > Enable-Spoof-Source: off
> > Enable-TCP-Wrapper: off
> > Enable-Linux-Caps: off
> > Enable-Systemd: off
> >
> > Not sure how to fix this as from what I can tell this module is supposed to be compiled in and automatically imported.
> >
> >
> > Ronald Fenner
> > Programmer
> > Game Circus LLC.
> >
> > rfenner at gamecircus.com <mailto:rfenner at gamecircus.com>
> >
> >
> > ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng>
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
> >
> >
> > ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng>
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
> >
> 
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng>
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
> 
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20171230/ab1a189d/attachment.html>

More information about the syslog-ng mailing list