<div dir="ltr"><div><div><div> Hi,<br><br></div>The problem is syslog-ng cannot import the user code: <br></div><div><div>[2017-12-29T23:00:05.814066] Error looking Python parser class; parser='p_php_fpm', class='PhpFpmParser', exception='None'</div><div><br></div><div>Please try either to put the python user code <br></div><div>python {</div><div> class PhpFpmParser(object):</div><div> ...<br></div><div>};<br></div><div>directly into syslog-ng.conf, or you can use @include to include the file that contains the parser code.</div><div><br></div><div>The _syslogng module is created by syslog-ng config parse: when syslog-ng processes the python keyword with the user code. So the _syslogng import problem should be resolved automatically with the above.<br></div><div><br></div></div><div>Br,</div></div><div> Antal<br></div></div><br><div class="gmail_quote"><div dir="ltr">On Sat, Dec 30, 2017 at 7:09 AM Ronald Fenner <<a href="mailto:rfenner@gamecircus.com">rfenner@gamecircus.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Here's the config I've redacted the Kafka servers. I've tried adding a @module "mod-python" but it doesn't help.<br>
<br>
#############################################################################<br>
# Default syslog-ng.conf file which collects all local logs into a<br>
# single file called /var/log/messages.<br>
#<br>
<br>
@version: 3.11<br>
@module "mod-java"<br>
@include "scl.conf"<br>
<br>
source s_internal {internal();};<br>
<br>
source s_rtl_stream {<br>
unix-stream("/var/log/rtl-stream.sock" flags(no-parse));<br>
};<br>
<br>
source s_php_fpm {<br>
file("/var/log/php-fpm.www.log", flags(no-parse));<br>
};<br>
<br>
destination d_kafka_unstructured {<br>
kafka (<br>
client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka_2.11-0.11.0.0/libs/")<br>
kafka-bootstrap-servers("******")<br>
topic("syslog-ng-{{DEPLOYMENT}}")<br>
);<br>
};<br>
<br>
destination d_kafka_structured {<br>
kafka (<br>
client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka_2.11-0.11.0.0/libs/")<br>
kafka-bootstrap-servers("*****")<br>
topic("${topic}.{{DEPLOYMENT}}")<br>
template("$(format-json --scope nv_pairs --exclude MESSAGE)\n")<br>
);<br>
};<br>
<br>
destination d_syslog_ng {<br>
file("/var/log/syslog-ng");<br>
};<br>
<br>
destination d_test_log {<br>
file("/var/log/test.log");<br>
};<br>
<br>
parser p_json { json-parser(); };<br>
<br>
parser p_apache { apache-accesslog-parser(prefix("")); };<br>
<br>
parser p_php_fpm { python(class("PhpFpmParser")); };<br>
<br>
rewrite r_add_access_topic {<br>
set("access.log", value("topic"));<br>
};<br>
<br>
log {<br>
source(s_internal);<br>
destination(d_syslog_ng);<br>
};<br>
<br>
log {<br>
source(s_rtl_stream);<br>
parser(p_json);<br>
destination(d_kafka_structured);<br>
};<br>
<br>
log {<br>
source(s_php_fpm);<br>
parser(p_php_fpm);<br>
destination(d_test_log);<br>
};<br>
<br>
<br>
Here's the actual python parser:<br>
python {<br>
class PhpFpmParser(object):<br>
def parse(self, log_msg):<br>
msg = log_msg['MESSAGE']<br>
str_pos = msg.find('] ')<br>
if str_pos == -1:<br>
return True<br>
log_date = msg[1:str_pos]<br>
msg = msg[str_pos+2:]<br>
str_pos = msg.find(':')<br>
if str_pos == -1:<br>
return True<br>
level = msg[:str_pos]<br>
if "Parse" in level:<br>
level = "parse"<br>
elif "Compile" in level:<br>
level = 'compile'<br>
elif "Fatal" in level:<br>
level = 'fatal'<br>
elif "Core" in level:<br>
level = 'core'<br>
elif "Notice" in level:<br>
level = 'notice'<br>
elif "Warning" in level:<br>
level = 'warning'<br>
msg = msg[str_pos+2:].strip()<br>
log_msg['err_msg'] = msg<br>
log_msg['log_level'] = level<br>
log_msg['timestamp_utc'] = log_date<br>
return True<br>
};<br>
<br>
It's stored the the etc/conf.d directory within the syslog path..<br>
<br>
Ronald Fenner<br>
Programmer<br>
Game Circus LLC.<br>
<br>
<a href="mailto:rfenner@gamecircus.com" target="_blank">rfenner@gamecircus.com</a><br>
<br>
> On Dec 29, 2017, at 11:52 PM, Scheidler, Balázs <<a href="mailto:balazs.scheidler@balabit.com" target="_blank">balazs.scheidler@balabit.com</a>> wrote:<br>
><br>
> The _syslogng module is automatically created from the top level python block in syslog-ng and behaves similarly to the python __main__ module.<br>
><br>
> Do you explicitly import that module using the imports() option?<br>
><br>
> Can you please post your config?<br>
><br>
> On Dec 30, 2017 00:27, "Ronald Fenner" <<a href="mailto:rfenner@gamecircus.com" target="_blank">rfenner@gamecircus.com</a>> wrote:<br>
> When I try to load my config with a python parser in it I'm getting this error message:<br>
> Starting /opt/syslog-ng/sbin/syslog-ng: [2017-12-29T23:00:05.813945] Error loading Python module; module='_syslogng', exception='exceptions.ImportError: No module named _syslogng'<br>
> [2017-12-29T23:00:05.814066] Error looking Python parser class; parser='p_php_fpm', class='PhpFpmParser', exception='None'<br>
> [2017-12-29T23:00:05.814116] Error initializing message pipeline; plugin name='python', location='/opt/syslog-ng/etc/syslog-ng.conf:52:20'<br>
><br>
> I build syslog-ng from source with the python options. Here is the -V output<br>
> syslog-ng 3 (3.11.1)<br>
> Installer-Version: 3.11.1<br>
> Revision:<br>
> Compile-Date: Dec 29 2017 21:24:13<br>
> Module-Directory: /opt/syslog-ng/lib/syslog-ng<br>
> Module-Path: /opt/syslog-ng/lib/syslog-ng<br>
> Available-Modules: snmptrapd-parser,affile,cef,afstomp,basicfuncs,pseudofile,tfgetent,afsocket,mod-python,json-plugin,afuser,kvformat,stardate,graphite,dbparser,csvparser,date,afmongodb,system-source,disk-buffer,confgen,linux-kmsg-format,afamqp,map-value-pairs,http,afprog,add-contextual-data,sdjournal,cryptofuncs,syslogformat<br>
> Enable-Debug: off<br>
> Enable-GProf: off<br>
> Enable-Memtrace: off<br>
> Enable-IPv6: on<br>
> Enable-Spoof-Source: off<br>
> Enable-TCP-Wrapper: off<br>
> Enable-Linux-Caps: off<br>
> Enable-Systemd: off<br>
><br>
> Not sure how to fix this as from what I can tell this module is supposed to be compiled in and automatically imported.<br>
><br>
><br>
> Ronald Fenner<br>
> Programmer<br>
> Game Circus LLC.<br>
><br>
> <a href="mailto:rfenner@gamecircus.com" target="_blank">rfenner@gamecircus.com</a><br>
><br>
><br>
> ______________________________________________________________________________<br>
> Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
> Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
> FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
><br>
><br>
> ______________________________________________________________________________<br>
> Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
> Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
> FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
><br>
<br>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div>