<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Including the file worked. I thought scl.conf was supposed to be including anything in the conf.d directory but looking into the file again it's actually the scl directory.<div class=""><br class=""></div><div class=""><br class=""><div class="">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class=""><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class=""><div class=""><div class="">Ronald Fenner</div><div class="">Programmer</div><div class="">Game Circus LLC.</div></div><div class=""><br class=""></div><div class=""><a href="mailto:rfenner@gamecircus.com" class="">rfenner@gamecircus.com</a></div></div></div>
</div>
<br class=""><div><blockquote type="cite" class=""><div class="">On Dec 30, 2017, at 1:12 AM, Antal Nemes <<a href="mailto:antal.nemes@balabit.com" class="">antal.nemes@balabit.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class=""><div class=""><div class=""> Hi,<br class=""><br class=""></div>The problem is syslog-ng cannot import the user code: <br class=""></div><div class=""><div class="">[2017-12-29T23:00:05.814066] Error looking Python parser class; parser='p_php_fpm', class='PhpFpmParser', exception='None'</div><div class=""><br class=""></div><div class="">Please try either to put the python user code <br class=""></div><div class="">python {</div><div class=""> class PhpFpmParser(object):</div><div class=""> ...<br class=""></div><div class="">};<br class=""></div><div class="">directly into syslog-ng.conf, or you can use @include to include the file that contains the parser code.</div><div class=""><br class=""></div><div class="">The _syslogng module is created by syslog-ng config parse: when syslog-ng processes the python keyword with the user code. So the _syslogng import problem should be resolved automatically with the above.<br class=""></div><div class=""><br class=""></div></div><div class="">Br,</div></div><div class=""> Antal<br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="">On Sat, Dec 30, 2017 at 7:09 AM Ronald Fenner <<a href="mailto:rfenner@gamecircus.com" class="">rfenner@gamecircus.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Here's the config I've redacted the Kafka servers. I've tried adding a @module "mod-python" but it doesn't help.<br class="">
<br class="">
#############################################################################<br class="">
# Default syslog-ng.conf file which collects all local logs into a<br class="">
# single file called /var/log/messages.<br class="">
#<br class="">
<br class="">
@version: 3.11<br class="">
@module "mod-java"<br class="">
@include "scl.conf"<br class="">
<br class="">
source s_internal {internal();};<br class="">
<br class="">
source s_rtl_stream {<br class="">
unix-stream("/var/log/rtl-stream.sock" flags(no-parse));<br class="">
};<br class="">
<br class="">
source s_php_fpm {<br class="">
file("/var/log/php-fpm.www.log", flags(no-parse));<br class="">
};<br class="">
<br class="">
destination d_kafka_unstructured {<br class="">
kafka (<br class="">
client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka_2.11-0.11.0.0/libs/")<br class="">
kafka-bootstrap-servers("******")<br class="">
topic("syslog-ng-{{DEPLOYMENT}}")<br class="">
);<br class="">
};<br class="">
<br class="">
destination d_kafka_structured {<br class="">
kafka (<br class="">
client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka_2.11-0.11.0.0/libs/")<br class="">
kafka-bootstrap-servers("*****")<br class="">
topic("${topic}.{{DEPLOYMENT}}")<br class="">
template("$(format-json --scope nv_pairs --exclude MESSAGE)\n")<br class="">
);<br class="">
};<br class="">
<br class="">
destination d_syslog_ng {<br class="">
file("/var/log/syslog-ng");<br class="">
};<br class="">
<br class="">
destination d_test_log {<br class="">
file("/var/log/test.log");<br class="">
};<br class="">
<br class="">
parser p_json { json-parser(); };<br class="">
<br class="">
parser p_apache { apache-accesslog-parser(prefix("")); };<br class="">
<br class="">
parser p_php_fpm { python(class("PhpFpmParser")); };<br class="">
<br class="">
rewrite r_add_access_topic {<br class="">
set("access.log", value("topic"));<br class="">
};<br class="">
<br class="">
log {<br class="">
source(s_internal);<br class="">
destination(d_syslog_ng);<br class="">
};<br class="">
<br class="">
log {<br class="">
source(s_rtl_stream);<br class="">
parser(p_json);<br class="">
destination(d_kafka_structured);<br class="">
};<br class="">
<br class="">
log {<br class="">
source(s_php_fpm);<br class="">
parser(p_php_fpm);<br class="">
destination(d_test_log);<br class="">
};<br class="">
<br class="">
<br class="">
Here's the actual python parser:<br class="">
python {<br class="">
class PhpFpmParser(object):<br class="">
def parse(self, log_msg):<br class="">
msg = log_msg['MESSAGE']<br class="">
str_pos = msg.find('] ')<br class="">
if str_pos == -1:<br class="">
return True<br class="">
log_date = msg[1:str_pos]<br class="">
msg = msg[str_pos+2:]<br class="">
str_pos = msg.find(':')<br class="">
if str_pos == -1:<br class="">
return True<br class="">
level = msg[:str_pos]<br class="">
if "Parse" in level:<br class="">
level = "parse"<br class="">
elif "Compile" in level:<br class="">
level = 'compile'<br class="">
elif "Fatal" in level:<br class="">
level = 'fatal'<br class="">
elif "Core" in level:<br class="">
level = 'core'<br class="">
elif "Notice" in level:<br class="">
level = 'notice'<br class="">
elif "Warning" in level:<br class="">
level = 'warning'<br class="">
msg = msg[str_pos+2:].strip()<br class="">
log_msg['err_msg'] = msg<br class="">
log_msg['log_level'] = level<br class="">
log_msg['timestamp_utc'] = log_date<br class="">
return True<br class="">
};<br class="">
<br class="">
It's stored the the etc/conf.d directory within the syslog path..<br class="">
<br class="">
Ronald Fenner<br class="">
Programmer<br class="">
Game Circus LLC.<br class="">
<br class="">
<a href="mailto:rfenner@gamecircus.com" target="_blank" class="">rfenner@gamecircus.com</a><br class="">
<br class="">
> On Dec 29, 2017, at 11:52 PM, Scheidler, Balázs <<a href="mailto:balazs.scheidler@balabit.com" target="_blank" class="">balazs.scheidler@balabit.com</a>> wrote:<br class="">
><br class="">
> The _syslogng module is automatically created from the top level python block in syslog-ng and behaves similarly to the python __main__ module.<br class="">
><br class="">
> Do you explicitly import that module using the imports() option?<br class="">
><br class="">
> Can you please post your config?<br class="">
><br class="">
> On Dec 30, 2017 00:27, "Ronald Fenner" <<a href="mailto:rfenner@gamecircus.com" target="_blank" class="">rfenner@gamecircus.com</a>> wrote:<br class="">
> When I try to load my config with a python parser in it I'm getting this error message:<br class="">
> Starting /opt/syslog-ng/sbin/syslog-ng: [2017-12-29T23:00:05.813945] Error loading Python module; module='_syslogng', exception='exceptions.ImportError: No module named _syslogng'<br class="">
> [2017-12-29T23:00:05.814066] Error looking Python parser class; parser='p_php_fpm', class='PhpFpmParser', exception='None'<br class="">
> [2017-12-29T23:00:05.814116] Error initializing message pipeline; plugin name='python', location='/opt/syslog-ng/etc/syslog-ng.conf:52:20'<br class="">
><br class="">
> I build syslog-ng from source with the python options. Here is the -V output<br class="">
> syslog-ng 3 (3.11.1)<br class="">
> Installer-Version: 3.11.1<br class="">
> Revision:<br class="">
> Compile-Date: Dec 29 2017 21:24:13<br class="">
> Module-Directory: /opt/syslog-ng/lib/syslog-ng<br class="">
> Module-Path: /opt/syslog-ng/lib/syslog-ng<br class="">
> Available-Modules: snmptrapd-parser,affile,cef,afstomp,basicfuncs,pseudofile,tfgetent,afsocket,mod-python,json-plugin,afuser,kvformat,stardate,graphite,dbparser,csvparser,date,afmongodb,system-source,disk-buffer,confgen,linux-kmsg-format,afamqp,map-value-pairs,http,afprog,add-contextual-data,sdjournal,cryptofuncs,syslogformat<br class="">
> Enable-Debug: off<br class="">
> Enable-GProf: off<br class="">
> Enable-Memtrace: off<br class="">
> Enable-IPv6: on<br class="">
> Enable-Spoof-Source: off<br class="">
> Enable-TCP-Wrapper: off<br class="">
> Enable-Linux-Caps: off<br class="">
> Enable-Systemd: off<br class="">
><br class="">
> Not sure how to fix this as from what I can tell this module is supposed to be compiled in and automatically imported.<br class="">
><br class="">
><br class="">
> Ronald Fenner<br class="">
> Programmer<br class="">
> Game Circus LLC.<br class="">
><br class="">
> <a href="mailto:rfenner@gamecircus.com" target="_blank" class="">rfenner@gamecircus.com</a><br class="">
><br class="">
><br class="">
> ______________________________________________________________________________<br class="">
> Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank" class="">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br class="">
> Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank" class="">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br class="">
> FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank" class="">http://www.balabit.com/wiki/syslog-ng-faq</a><br class="">
><br class="">
><br class="">
> ______________________________________________________________________________<br class="">
> Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank" class="">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br class="">
> Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank" class="">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br class="">
> FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank" class="">http://www.balabit.com/wiki/syslog-ng-faq</a><br class="">
><br class="">
<br class="">
______________________________________________________________________________<br class="">
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank" class="">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br class="">
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank" class="">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br class="">
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank" class="">http://www.balabit.com/wiki/syslog-ng-faq</a><br class="">
<br class="">
</blockquote></div>
______________________________________________________________________________<br class="">Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" class="">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br class="">Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" class="">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br class="">FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" class="">http://www.balabit.com/wiki/syslog-ng-faq</a><br class=""><br class=""></div></blockquote></div><br class=""></div></body></html>