[syslog-ng] Problem using Python Parser

Antal Nemes antal.nemes at balabit.com
Sat Dec 30 07:12:16 UTC 2017 

   Hi,

The problem is syslog-ng cannot import the user code:
[2017-12-29T23:00:05.814066] Error looking Python parser class;
parser='p_php_fpm', class='PhpFpmParser', exception='None'

Please try either to put the python user code
python {
    class PhpFpmParser(object):
    ...
};
directly into syslog-ng.conf, or you can use @include to include the file
that contains the parser code.

The _syslogng module is created by syslog-ng config parse: when syslog-ng
processes the python keyword with the user code. So the _syslogng import
problem should be resolved automatically with the above.

Br,
  Antal

On Sat, Dec 30, 2017 at 7:09 AM Ronald Fenner <rfenner at gamecircus.com>
wrote:

> Here's the config I've redacted the Kafka servers. I've tried adding a
> @module "mod-python" but it doesn't help.
>
>
> #############################################################################
> # Default syslog-ng.conf file which collects all local logs into a
> # single file called /var/log/messages.
> #
>
> @version: 3.11
> @module "mod-java"
> @include "scl.conf"
>
> source s_internal {internal();};
>
> source s_rtl_stream {
> unix-stream("/var/log/rtl-stream.sock" flags(no-parse));
> };
>
> source s_php_fpm {
> file("/var/log/php-fpm.www.log", flags(no-parse));
> };
>
> destination d_kafka_unstructured {
> kafka (
>
> client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka_2.11-0.11.0.0/libs/")
> kafka-bootstrap-servers("******")
> topic("syslog-ng-{{DEPLOYMENT}}")
> );
> };
>
> destination d_kafka_structured {
> kafka (
>
> client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka_2.11-0.11.0.0/libs/")
> kafka-bootstrap-servers("*****")
> topic("${topic}.{{DEPLOYMENT}}")
> template("$(format-json --scope nv_pairs --exclude MESSAGE)\n")
> );
> };
>
> destination d_syslog_ng {
> file("/var/log/syslog-ng");
> };
>
> destination d_test_log {
> file("/var/log/test.log");
> };
>
> parser p_json { json-parser(); };
>
> parser p_apache { apache-accesslog-parser(prefix("")); };
>
> parser p_php_fpm { python(class("PhpFpmParser")); };
>
> rewrite r_add_access_topic {
> set("access.log", value("topic"));
> };
>
> log {
> source(s_internal);
> destination(d_syslog_ng);
> };
>
> log {
> source(s_rtl_stream);
> parser(p_json);
> destination(d_kafka_structured);
> };
>
> log {
> source(s_php_fpm);
> parser(p_php_fpm);
> destination(d_test_log);
> };
>
>
> Here's the actual python parser:
> python {
> class PhpFpmParser(object):
>     def parse(self, log_msg):
>         msg = log_msg['MESSAGE']
>         str_pos = msg.find('] ')
>         if str_pos == -1:
>             return True
>         log_date = msg[1:str_pos]
>         msg = msg[str_pos+2:]
>         str_pos = msg.find(':')
>         if str_pos == -1:
>             return True
>         level = msg[:str_pos]
>         if "Parse" in level:
>             level = "parse"
>         elif "Compile" in level:
>             level = 'compile'
>         elif "Fatal" in level:
>             level = 'fatal'
>         elif "Core" in level:
>             level = 'core'
>         elif "Notice" in level:
>             level = 'notice'
>         elif "Warning" in level:
>             level = 'warning'
>         msg = msg[str_pos+2:].strip()
>         log_msg['err_msg'] = msg
>         log_msg['log_level'] = level
>         log_msg['timestamp_utc'] = log_date
>         return True
> };
>
> It's stored the the etc/conf.d directory within the syslog path..
>
> Ronald Fenner
> Programmer
> Game Circus LLC.
>
> rfenner at gamecircus.com
>
> > On Dec 29, 2017, at 11:52 PM, Scheidler, Balázs <
> balazs.scheidler at balabit.com> wrote:
> >
> > The _syslogng module is automatically created from the top level python
> block in syslog-ng and behaves similarly to the python __main__ module.
> >
> > Do you explicitly import that module using the imports() option?
> >
> > Can you please post your config?
> >
> > On Dec 30, 2017 00:27, "Ronald Fenner" <rfenner at gamecircus.com> wrote:
> > When I try to load my config with a python parser in it I'm getting this
> error message:
> > Starting /opt/syslog-ng/sbin/syslog-ng: [2017-12-29T23:00:05.813945]
> Error loading Python module; module='_syslogng',
> exception='exceptions.ImportError: No module named _syslogng'
> > [2017-12-29T23:00:05.814066] Error looking Python parser class;
> parser='p_php_fpm', class='PhpFpmParser', exception='None'
> > [2017-12-29T23:00:05.814116] Error initializing message pipeline; plugin
> name='python', location='/opt/syslog-ng/etc/syslog-ng.conf:52:20'
> >
> > I build syslog-ng from source with the python options. Here is the -V
> output
> > syslog-ng 3 (3.11.1)
> > Installer-Version: 3.11.1
> > Revision:
> > Compile-Date: Dec 29 2017 21:24:13
> > Module-Directory: /opt/syslog-ng/lib/syslog-ng
> > Module-Path: /opt/syslog-ng/lib/syslog-ng
> > Available-Modules:
> snmptrapd-parser,affile,cef,afstomp,basicfuncs,pseudofile,tfgetent,afsocket,mod-python,json-plugin,afuser,kvformat,stardate,graphite,dbparser,csvparser,date,afmongodb,system-source,disk-buffer,confgen,linux-kmsg-format,afamqp,map-value-pairs,http,afprog,add-contextual-data,sdjournal,cryptofuncs,syslogformat
> > Enable-Debug: off
> > Enable-GProf: off
> > Enable-Memtrace: off
> > Enable-IPv6: on
> > Enable-Spoof-Source: off
> > Enable-TCP-Wrapper: off
> > Enable-Linux-Caps: off
> > Enable-Systemd: off
> >
> > Not sure how to fix this as from what I can tell this module is supposed
> to be compiled in and automatically imported.
> >
> >
> > Ronald Fenner
> > Programmer
> > Game Circus LLC.
> >
> > rfenner at gamecircus.com
> >
> >
> >
> ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >
> >
> >
> ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20171230/123c834c/attachment-0001.html>

More information about the syslog-ng mailing list