[syslog-ng] Problem using Python Parser

[Ronald Fenner]

Here's the config I've redacted the Kafka servers. I've tried adding a @module "mod-python" but it doesn't help.

#############################################################################
# Default syslog-ng.conf file which collects all local logs into a
# single file called /var/log/messages.
#

@version: 3.11
@module "mod-java"
@include "scl.conf"

source s_internal {internal();};

source s_rtl_stream {
unix-stream("/var/log/rtl-stream.sock" flags(no-parse));
};

source s_php_fpm {
file("/var/log/php-fpm.www.log", flags(no-parse));
};

destination d_kafka_unstructured {
kafka (
client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka_2.11-0.11.0.0/libs/")
kafka-bootstrap-servers("******")
topic("syslog-ng-{{DEPLOYMENT}}")
);
};

destination d_kafka_structured {
kafka (
client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka_2.11-0.11.0.0/libs/")
kafka-bootstrap-servers("*****")
topic("${topic}.{{DEPLOYMENT}}")
template("$(format-json --scope nv_pairs --exclude MESSAGE)\n")
);
};

destination d_syslog_ng {
file("/var/log/syslog-ng");
};

destination d_test_log {
file("/var/log/test.log");
};

parser p_json { json-parser(); };

parser p_apache { apache-accesslog-parser(prefix("")); };

parser p_php_fpm { python(class("PhpFpmParser")); };

rewrite r_add_access_topic {
set("access.log", value("topic"));
};

log {
source(s_internal);
destination(d_syslog_ng);
};

log {
source(s_rtl_stream);
parser(p_json);
destination(d_kafka_structured);
};

log {
source(s_php_fpm);
parser(p_php_fpm);
destination(d_test_log);
};


Here's the actual python parser:
python {
class PhpFpmParser(object):
    def parse(self, log_msg):
        msg = log_msg['MESSAGE']
        str_pos = msg.find('] ')
        if str_pos == -1:
            return True
        log_date = msg[1:str_pos]
        msg = msg[str_pos+2:]
        str_pos = msg.find(':')
        if str_pos == -1:
            return True
        level = msg[:str_pos]
        if "Parse" in level:
            level = "parse"
        elif "Compile" in level:
            level = 'compile'
        elif "Fatal" in level:
            level = 'fatal'
        elif "Core" in level:
            level = 'core'
        elif "Notice" in level:
            level = 'notice'
        elif "Warning" in level:
            level = 'warning'
        msg = msg[str_pos+2:].strip()
        log_msg['err_msg'] = msg
        log_msg['log_level'] = level
        log_msg['timestamp_utc'] = log_date
        return True
};

It's stored the the etc/conf.d directory within the syslog path..

Ronald Fenner
Programmer
Game Circus LLC.

rfenner at gamecircus.com

> On Dec 29, 2017, at 11:52 PM, Scheidler, Balázs <balazs.scheidler at balabit.com> wrote:
> 
> The _syslogng module is automatically created from the top level python block in syslog-ng and behaves similarly to the python __main__ module.
> 
> Do you explicitly import that module using the imports() option?
> 
> Can you please post your config?
> 
> On Dec 30, 2017 00:27, "Ronald Fenner" <rfenner at gamecircus.com> wrote:
> When I try to load my config with a python parser in it I'm getting this error message:
> Starting /opt/syslog-ng/sbin/syslog-ng: [2017-12-29T23:00:05.813945] Error loading Python module; module='_syslogng', exception='exceptions.ImportError: No module named _syslogng'
> [2017-12-29T23:00:05.814066] Error looking Python parser class; parser='p_php_fpm', class='PhpFpmParser', exception='None'
> [2017-12-29T23:00:05.814116] Error initializing message pipeline; plugin name='python', location='/opt/syslog-ng/etc/syslog-ng.conf:52:20'
> 
> I build syslog-ng from source with the python options. Here is the -V output
> syslog-ng 3 (3.11.1)
> Installer-Version: 3.11.1
> Revision:
> Compile-Date: Dec 29 2017 21:24:13
> Module-Directory: /opt/syslog-ng/lib/syslog-ng
> Module-Path: /opt/syslog-ng/lib/syslog-ng
> Available-Modules: snmptrapd-parser,affile,cef,afstomp,basicfuncs,pseudofile,tfgetent,afsocket,mod-python,json-plugin,afuser,kvformat,stardate,graphite,dbparser,csvparser,date,afmongodb,system-source,disk-buffer,confgen,linux-kmsg-format,afamqp,map-value-pairs,http,afprog,add-contextual-data,sdjournal,cryptofuncs,syslogformat
> Enable-Debug: off
> Enable-GProf: off
> Enable-Memtrace: off
> Enable-IPv6: on
> Enable-Spoof-Source: off
> Enable-TCP-Wrapper: off
> Enable-Linux-Caps: off
> Enable-Systemd: off
> 
> Not sure how to fix this as from what I can tell this module is supposed to be compiled in and automatically imported.
> 
> 
> Ronald Fenner
> Programmer
> Game Circus LLC.
> 
> rfenner at gamecircus.com
> 
> 
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> 
> 
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>