Re: [syslog-ng] Problem using Python Parser

30 Dec 2017

      Here's the config I've redacted the Kafka servers. I've tried adding a @module "mod-python" but it doesn't help.

#############################################################################
# Default syslog-ng.conf file which collects all local logs into a
# single file called /var/log/messages.
#

@version: 3.11
@module "mod-java"
@include "scl.conf"

source s_internal {internal();};

source s_rtl_stream {
unix-stream("/var/log/rtl-stream.sock" flags(no-parse));
};

source s_php_fpm {
file("/var/log/php-fpm.www.log", flags(no-parse));
};

destination d_kafka_unstructured {
kafka (
client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka_2.11-0.11.0.0/libs/")
kafka-bootstrap-servers("******")
topic("syslog-ng-{{DEPLOYMENT}}")
);
};

destination d_kafka_structured {
kafka (
client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka_2.11-0.11.0.0/libs/")
kafka-bootstrap-servers("*****")
topic("${topic}.{{DEPLOYMENT}}")
template("$(format-json --scope nv_pairs --exclude MESSAGE)\n")
);
};

destination d_syslog_ng {
file("/var/log/syslog-ng");
};

destination d_test_log {
file("/var/log/test.log");
};

parser p_json { json-parser(); };

parser p_apache { apache-accesslog-parser(prefix("")); };

parser p_php_fpm { python(class("PhpFpmParser")); };

rewrite r_add_access_topic {
set("access.log", value("topic"));
};

log {
source(s_internal);
destination(d_syslog_ng);
};

log {
source(s_rtl_stream);
parser(p_json);
destination(d_kafka_structured);
};

log {
source(s_php_fpm);
parser(p_php_fpm);
destination(d_test_log);
};

Here's the actual python parser:
python {
class PhpFpmParser(object):
    def parse(self, log_msg):
        msg = log_msg['MESSAGE']
        str_pos = msg.find('] ')
        if str_pos == -1:
            return True
        log_date = msg[1:str_pos]
        msg = msg[str_pos+2:]
        str_pos = msg.find(':')
        if str_pos == -1:
            return True
        level = msg[:str_pos]
        if "Parse" in level:
            level = "parse"
        elif "Compile" in level:
            level = 'compile'
        elif "Fatal" in level:
            level = 'fatal'
        elif "Core" in level:
            level = 'core'
        elif "Notice" in level:
            level = 'notice'
        elif "Warning" in level:
            level = 'warning'
        msg = msg[str_pos+2:].strip()
        log_msg['err_msg'] = msg
        log_msg['log_level'] = level
        log_msg['timestamp_utc'] = log_date
        return True
};

It's stored the the etc/conf.d directory within the syslog path..

Ronald Fenner
Programmer
Game Circus LLC.

rfenner@gamecircus.com
...
On Dec 29, 2017, at 11:52 PM, Scheidler, Balázs <balazs.scheidler@balabit.com> wrote:
The _syslogng module is automatically created from the top level python block in syslog-ng and behaves similarly to the python __main__ module.
Do you explicitly import that module using the imports() option?
Can you please post your config?
On Dec 30, 2017 00:27, "Ronald Fenner" <rfenner@gamecircus.com> wrote:
When I try to load my config with a python parser in it I'm getting this error message:
Starting /opt/syslog-ng/sbin/syslog-ng: [2017-12-29T23:00:05.813945] Error loading Python module; module='_syslogng', exception='exceptions.ImportError: No module named _syslogng'
[2017-12-29T23:00:05.814066] Error looking Python parser class; parser='p_php_fpm', class='PhpFpmParser', exception='None'
[2017-12-29T23:00:05.814116] Error initializing message pipeline; plugin name='python', location='/opt/syslog-ng/etc/syslog-ng.conf:52:20'
I build syslog-ng from source with the python options. Here is the -V output
syslog-ng 3 (3.11.1)
Installer-Version: 3.11.1
Revision:
Compile-Date: Dec 29 2017 21:24:13
Module-Directory: /opt/syslog-ng/lib/syslog-ng
Module-Path: /opt/syslog-ng/lib/syslog-ng
Available-Modules: snmptrapd-parser,affile,cef,afstomp,basicfuncs,pseudofile,tfgetent,afsocket,mod-python,json-plugin,afuser,kvformat,stardate,graphite,dbparser,csvparser,date,afmongodb,system-source,disk-buffer,confgen,linux-kmsg-format,afamqp,map-value-pairs,http,afprog,add-contextual-data,sdjournal,cryptofuncs,syslogformat
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: off
Enable-TCP-Wrapper: off
Enable-Linux-Caps: off
Enable-Systemd: off
Not sure how to fix this as from what I can tell this module is supposed to be compiled in and automatically imported.
Ronald Fenner
Programmer
Game Circus LLC.
rfenner@gamecircus.com
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq