<div dir="ltr"><div><div><div><div>Just to finish this thread,<br><br></div>this may not be the proper way but I got it done by extending the regex to start on ": (.*)" <br><br></div>since I cannot remove the default headers, I have to filter match my message that way (I tried to use message() instead of match() but it did not work)<br><br></div>I only found this after scraping this page:<br><br></div>where I found this part of the docs:<a href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/reference-source-syslog-chapter.html">https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/reference-source-syslog-chapter.html</a><br><p><span class="emphasis"><em>no-parse</em></span>: By default, <span class="gmail-phrase">syslog-ng OSE</span> parses incoming messages as syslog messages. The <em class="gmail-parameter"><code>no-parse</code></em> flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. <b>The <span class="gmail-phrase">syslog-ng OSE</span>
application will generate a new syslog header (timestamp, host, and so
on) automatically and put the entire incoming message into the MESSAGE
part of the syslog message (available using the <em class="gmail-parameter"><code>${MESSAGE}</code></em> macro).</b> This flag is useful for parsing messages not complying to the syslog format.</p><p>It would be good to have an expample on the output inside the syslog so the user knows what to filter or the kind of message he's getting after no-parse.</p><p>It would also be wonderful if the docs were linked, so I don't have to search for every term that's not on the same page.</p><p>Anyway, thank you for your attention. <br></p><p><br></p><p><br></p></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 11, 2017 at 12:08 PM, Filipe Cifali <span dir="ltr"><<a href="mailto:cifali.filipe@gmail.com" target="_blank">cifali.filipe@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>Hi Fabien,<br><br></div>just corrected, but that's not enough info for me, I'm using a combined log and it's not separating the virtualhost (not even setting it)<br><br></div>The apache parser is getting me this kind of log:<br><br>{"_apache":{"verb":"HEAD","<wbr>timestamp":"11/Jul/2017:11:48:<wbr>48 -0300","response":"200","<wbr>request":"/h","referrer":"-","<wbr>rawrequest":"HEAD /h HTTP/1.1","ident":"-","<wbr>httpversion":"1.1","clientip":<wbr>"127.0.0.1","bytes":"-","auth"<wbr>:"-","agent":"Monit/5.21.0"}}<br><br></div>coming from this log:<br><br><a href="http://cifa.li" target="_blank">cifa.li</a> 127.0.0.1 - - [11/Jul/2017:11:48:48 -0300] "HEAD /h HTTP/1.1" 200 - "-" "Monit/5.21.0"<br><br><br><div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 11, 2017 at 11:28 AM, Filipe Cifali <span dir="ltr"><<a href="mailto:cifali.filipe@gmail.com" target="_blank">cifali.filipe@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div>Sorry for the double post, just a heads up on this<br><br></div>if I use source { flags(no-parse) } <br><br></div>the first regex match turns into ": $1"<br><br></div>if I comment flags(no-parse)<br><br></div>the first regex match turns into "$1:"<br><br></div>Since the docs states this: <br><br>By default, <span class="m_-7298279881951527347m_8013868931887656693gmail-phrase">syslog-ng OSE</span> parses every message as a syslog message. To disable message parsing, use the <em class="m_-7298279881951527347m_8013868931887656693gmail-parameter"><code>flags(<span class="m_-7298279881951527347m_8013868931887656693gmail-highlight">no-parse</span>)</code></em> option of the source. To explicitly parse a message as a syslog message, use the <em class="m_-7298279881951527347m_8013868931887656693gmail-parameter"><code>syslog</code></em> parser. For details, see <a class="m_-7298279881951527347m_8013868931887656693gmail-xref" href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/parser-syslog.html" title="12.1. Parsing syslog messages" target="_blank">Section 12.1, Parsing syslog messages</a>.<br><br></div>To my understanding, I should use (no-parse) since that message does not come in syslog message format.<br></div><div class="m_-7298279881951527347HOEnZb"><div class="m_-7298279881951527347h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 11, 2017 at 11:13 AM, Filipe Cifali <span dir="ltr"><<a href="mailto:cifali.filipe@gmail.com" target="_blank">cifali.filipe@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>Hi Robert,<br><br></div>I'm on 3.9.1, I have just tried hat example and it returns:<br><br>Error parsing affile, Error compiling template, error=Invalid template function reference, missing function name or inbalanced '(', error_pos='24' in /etc/syslog-ng/conf.d/apache.c<wbr>onf at line 54, column 18:<br> <wbr> <wbr> <wbr> <wbr> included from /etc/syslog-ng/syslog-ng.conf line 68, column 1<br><br> template("$(format-json .apache.*\n"));<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^<br><br></div>Besides the parser itself, the strange part is why the regex returning that extra info at all... <br><div><div><div><br></div></div></div></div><div class="m_-7298279881951527347m_8013868931887656693HOEnZb"><div class="m_-7298279881951527347m_8013868931887656693h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 11, 2017 at 10:59 AM, Fekete, Róbert <span dir="ltr"><<a href="mailto:robert.fekete@balabit.com" target="_blank">robert.fekete@balabit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi, in OSE 3.9 and later there is a dedicated apache parser: <a href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/apache-access-log-parser.html" target="_blank">https://www.balabit.co<wbr>m/documents/syslog-ng-ose-late<wbr>st-guides/en/syslog-ng-ose-gui<wbr>de-admin/html/apache-access-lo<wbr>g-parser.html</a><div><br></div><div>You might want to try it.</div><div><br></div><div>HTH, </div><div><br></div><div>Robert</div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_-7298279881951527347m_8013868931887656693m_-5515862259013602228h5">On Tue, Jul 11, 2017 at 3:11 PM, Filipe Cifali <span dir="ltr"><<a href="mailto:cifali.filipe@gmail.com" target="_blank">cifali.filipe@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="m_-7298279881951527347m_8013868931887656693m_-5515862259013602228h5"><div dir="ltr"><div><div><div>Hi all,<br><br></div>reading the docs I got into this config:<br><br>source s_apache_access_log { <wbr> <wbr> <wbr> <br> file( <wbr> <wbr> <wbr> <wbr> <br> "/var/logs/apache2/access_log"<wbr> <wbr> <wbr> <wbr> <br> follow-freq(1) <wbr> <wbr> <wbr> <wbr> <br> flags(no-parse) <wbr> <wbr> <wbr> <wbr> <br> ); <wbr> <wbr> <wbr> <wbr> <br>}; <wbr> <wbr> <wbr> <wbr> <br> <wbr> <wbr> <wbr> <wbr> <br>filter f_apache_access_log { <wbr> <wbr> <wbr> <br> match( <wbr> <wbr> <wbr> <wbr> <br> '(.*) (.*) - - \[[0-9]{2}\/[A-Z][a-z]{2}\/[0-<wbr>9]{4}:[0-9]{2}:[0-9]{2}:[0-9]{<wbr>2} -0300\] \"(.*) (.*) (.*)\" (.*) (.*) \"-\" (.*)' <br> type("pcre") <wbr> <wbr> <wbr> <wbr> <br> flags("store-matches") <wbr> <wbr> <wbr> <wbr> <br> ); <wbr> <wbr> <wbr> <wbr> <br>}; <wbr> <wbr> <wbr> <wbr> <br> <wbr> <wbr> <wbr> <wbr> <br>rewrite r_apache_access_log { <wbr> <wbr> <wbr> <br> set("$1", value("DOMAIN") condition(filter(f_apache_acce<wbr>ss_log))); <wbr> <wbr> <br> set("$2", value("IP") condition(filter(f_apache_acce<wbr>ss_log))); <wbr> <wbr> <wbr> <br> set("$3", value("HTTP_METHOD") condition(filter(f_apache_acce<wbr>ss_log))); <wbr> <wbr> <br> set("$4", value("URI") condition(filter(f_apache_acce<wbr>ss_log))); <wbr> <wbr> <br> set("$6", value("HTTP_STATUS") condition(filter(f_apache_acce<wbr>ss_log))); <wbr> <wbr> <br> set("$7", value("SIZE") condition(filter(f_apache_acce<wbr>ss_log))); <wbr> <wbr> <br> set("$8", value("USER_AGENT") condition(filter(f_apache_acce<wbr>ss_log))); <wbr> <wbr> <br>}; <wbr> <wbr> <wbr> <wbr> <br> <wbr> <wbr> <wbr> <wbr> <br>destination d_apache_access_log { <wbr> <wbr> <wbr> <br> mongodb( <wbr> <wbr> <wbr> <wbr> <br> # <a href="https://docs.mongodb.com/manual/reference/connection-string/" target="_blank">https://docs.mongodb.com/manua<wbr>l/reference/connection-string/</a><wbr> <wbr> <wbr> <br> persist-name("apache-access-lo<wbr>gs") <wbr> <wbr> <wbr> <br> uri("mongodb://$server_and_por<wbr>t/syslog?wtimeoutMS=60000&sock<wbr>etTimeoutMS=60000&connectTimeo<wbr>utMS=60000") <wbr> <br> collection("logs") <wbr> <wbr> <wbr> <wbr> <br> retries(3600) <wbr> <wbr> <wbr> <wbr> <br> value-pairs( <wbr> <wbr> <wbr> <wbr> <br> pair("HOST", "${HOST}") <wbr> <wbr> <wbr> <wbr> <br> pair("SERVICE", "APACHE") <wbr> <wbr> <wbr> <br> pair("DATE", "${DAY}/${MONTH}/${YEAR}") <wbr> <wbr> <wbr> <wbr> <br> pair("TIME", "${HOUR}:${MIN}") <wbr> <wbr> <wbr> <wbr> <br> pair("MESSAGE", "${MESSAGE}") <wbr> <wbr> <wbr> <br> pair("DOMAIN", "${DOMAIN}") <wbr> <wbr> <wbr> <br> pair("HTTP_STATUS", "${HTTP_STATUS}") <wbr> <wbr> <wbr> <br> pair("HTTP_METHOD", "${HTTP_METHOD}") <wbr> <wbr> <wbr> <br> pair("USER_AGENT", "${USER_AGENT}") <wbr> <wbr> <wbr> <br> pair("SIZE", "${SIZE}") <wbr> <wbr> <wbr> <wbr> <br> pair("URI", "${URI}") <wbr> <wbr> <wbr> <wbr> <br> pair("IP", "${IP}") <wbr> <wbr> <wbr> <wbr> <br> ) <wbr> <wbr> <wbr> <wbr> <br> ); <wbr> <wbr> <wbr> <wbr> <br>}; <wbr> <wbr> <wbr> <wbr> <br> <wbr> <wbr> <wbr> <wbr> <br>log { <wbr> <wbr> <wbr> <wbr> <br> source(s_apache_access_log); <wbr> <wbr> <wbr> <wbr> <br> filter(f_apache_access_log); <wbr> <wbr> <wbr> <wbr> <br> rewrite(r_apache_access_log); <wbr> <wbr> <wbr> <wbr> <br> destination(d_apache_access_lo<wbr>g); <wbr> <wbr> <wbr> <br>}; <wbr> <wbr> <wbr> <wbr> <br><br></div>but I think something is not ok, I'm not sure this is the right way to do it.<br><br></div>This log produces an strange behavior: <br><br><a href="http://www.cifa.li" target="_blank">www.cifa.li</a> 127.0.0.1 - - [11/Jul/2017:09:18:56 -0300] "GET / HTTP/1.1" 200 18652 "<a href="http://cifa.li/" target="_blank">http://cifa.li/</a>" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"<br clear="all"><div><div><div><div><div><div><div><br></div><div>but this one doesn't<br><br><a href="http://cifa.li" target="_blank">cifa.li</a> 127.0.0.1 - - [11/Jul/2017:09:18:56 -0300] "GET / HTTP/1.1"
200 18652 "<a href="http://cifa.li/" target="_blank">http://cifa.li/</a>" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64;
rv:54.0) Gecko/20100101 Firefox/54.0"</div><div><br></div><div>The behavior is (only for subdomains):<br><br></div><div>DOMAIN: ': <a href="http://www.cifa.li" target="_blank">www.cifa.li</a>' <br><br></div><div>corret one<br><br></div><div>DOMAIN: '<a href="http://www.cifa.li" target="_blank">www.cifa.li</a>'<br><br></div><div>The subdomain seems like it's adding stuff that I didn't (or want) to add.<br><br><br></div><div>Am I missing something?<br><br></div><div>Thanks in advance.<span class="m_-7298279881951527347m_8013868931887656693m_-5515862259013602228m_-1243469234872472797HOEnZb"><font color="#888888"><br></font></span></div><span class="m_-7298279881951527347m_8013868931887656693m_-5515862259013602228m_-1243469234872472797HOEnZb"><font color="#888888"><div><br></div><div><br>-- <br><div class="m_-7298279881951527347m_8013868931887656693m_-5515862259013602228m_-1243469234872472797m_-8953235465465869119gmail_signature">[ ]'s<br><br>Filipe Cifali Stangler<br></div>
</div></font></span></div></div></div></div></div></div></div>
<br></div></div>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=syslog<wbr>-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>
<br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=syslog<wbr>-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
<br></blockquote></div><br><br clear="all"><br>-- <br><div class="m_-7298279881951527347m_8013868931887656693m_-5515862259013602228gmail_signature" data-smartmail="gmail_signature">[ ]'s<br><br>Filipe Cifali Stangler<br></div>
</div>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="m_-7298279881951527347m_8013868931887656693gmail_signature" data-smartmail="gmail_signature">[ ]'s<br><br>Filipe Cifali Stangler<br></div>
</div>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="m_-7298279881951527347gmail_signature" data-smartmail="gmail_signature">[ ]'s<br><br>Filipe Cifali Stangler<br></div>
</div>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">[ ]'s<br><br>Filipe Cifali Stangler<br></div>
</div>