<div dir="ltr"><span style="font-size:12.8px">Hi, dunno</span><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">My client config:</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><div>source s_apache_access {</div><div> file("/var/log/apache2/access_<wbr>testsrv_log" follow-freq(1) flags("no-parse") host-override("testsrv_access"<wbr>) program-override("apache-<wbr>access-log"));</div><div>};</div></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><div>destination d_network_apache {.</div><div> syslog("192.168.56.48"</div><div> transport("tcp").</div><div> ip-protocol(4)</div><div> port("16602")</div><div> persist-name("testsrv_apache")<wbr>);</div><div>};</div></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">log { source(s_apache_access); destination(d_network_apache); };<br></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">And my server config for this:</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><div>source s_network_testsrv_apache {</div><div> tcp(ip(192.168.56.48)</div><div> port("16602"));</div><div>};</div></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><div>destination d_apache_access_testsrv {</div><div> file("/var/log/syslog-ng/<wbr>testsrv/apache2/$FULLHOST.<wbr>access"</div><span class="gmail-im"><div> owner("root")</div><div> group("adm")</div><div> perm(0640)</div><div> create_dirs(yes)</div><div> dir_owner("root")</div><div> dir_group("adm")</div></span><div> template ("${MSG}\n")</div><div> persist-name("testsrv_access")<wbr>);</div><div>....</div><div> file("/var/log/archive/$R_<wbr>YEAR/testsrv/apache/$R_MONTH/$<wbr>FULLHOST.access.$R_DAY"</div><span class="gmail-im"><div> owner("root")</div><div> group("adm")</div><div> perm(0640)</div><div> create_dirs(yes)</div><div> dir_owner("root")</div><div> dir_group("adm")</div></span><div> template ("${MSG}\n")</div><div> persist-name("testsrv_access_<wbr>archive"));</div><div>};</div></div><span class="gmail-im" style="font-size:12.8px"><div><br></div><div><div>filter testsrv_apache_access {</div><div> match("apache-access-log")</div><div>};</div></div><div><br></div></span><div style="font-size:12.8px">log { source(s_network_testsrv_<wbr>apache); filter(testsrv_apache_access); destination(d_apache_access_<wbr>testsrv); };</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Thanks, Robert</div></div><div class="gmail_extra"><br><div class="gmail_quote">2017-06-08 17:49 GMT+02:00 Scheidler, Balázs <span dir="ltr"><<a href="mailto:balazs.scheidler@balabit.com" target="_blank">balazs.scheidler@balabit.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><div>Hi,<div><div class="h5"><br><div class="gmail_extra"><br><div class="gmail_quote">On Jun 8, 2017 14:41, "Sandbox" <<a href="mailto:sandboxheh@gmail.com" target="_blank">sandboxheh@gmail.com</a>> wrote:<br type="attribution"><blockquote class="m_-6207913433981906096quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span class="m_-6207913433981906096m_1853049704165417901gmail-im" style="font-size:12.8px"><div><div class="gmail_extra"><div class="gmail_quote">Hi, </div><div class="gmail_quote"><br></div><div class="gmail_quote">Thanks your answer.</div><div class="m_-6207913433981906096quoted-text"><div class="gmail_quote"><br></div><div class="gmail_quote"><br></div><div class="gmail_quote">On Jun 8, 2017 08:38, "Sandbox" <<a href="mailto:sandboxheh@gmail.com" target="_blank">sandboxheh@gmail.com</a>> wrote:<br type="attribution"><blockquote class="m_-6207913433981906096m_1853049704165417901gmail-m_-725075846151490485quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>I just started to test, learn etc syslog-ng, my server configuration is really basic:</div><div><br></div><div>Q: Can i filter (and mark them on client) the incoming logs, so i dont have to open multiple ports for different logs?</div></div></blockquote></div></div></div></div><div dir="auto"><br></div></span><div class="m_-6207913433981906096quoted-text"><div dir="auto" style="font-size:12.8px">| Sure, you can open one port and have it filtered using source ip (netmask() filter), embedded hostname (host() filter) or even message content.</div><div dir="auto" style="font-size:12.8px"><br></div></div><div style="font-size:12.8px">I made some filters, eg:</div><div style="font-size:12.8px"><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">filter testsrv_apache_access {</div><div style="font-size:12.8px"> match("apache-access-log")</div><div style="font-size:12.8px">};</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Q: i tried to use the "program" filter, but for some reason would't work, as you mentioned im using <span style="font-size:12.8px">program-override("apache<wbr>-</span><span style="font-size:12.8px">access-log") on the client and sat up the </span><span style="font-size:12.8px">apache-</span><span style="font-size:12.8px">access-log as filter match on the server side. With this setting it complains about missing value setting.</span></div></div><div class="m_-6207913433981906096quoted-text"><span class="m_-6207913433981906096m_1853049704165417901gmail-im" style="font-size:12.8px"><div dir="auto"><br></div><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="m_-6207913433981906096m_1853049704165417901gmail-m_-725075846151490485quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div><br></div><div>The stored log:</div><div><br></div><div>Q: Why does it store the date 3 times in every logs?</div></div></blockquote></div></div></div><div dir="auto"><br></div></span><div dir="auto" style="font-size:12.8px">| You seem to have received an rfc5424 formatted message, but it was not parsed, maybe because you were using the wrong source driver (syslog() is the one that should handle this format).</div><div dir="auto" style="font-size:12.8px">|</div><div dir="auto" style="font-size:12.8px">| Since it wasnt parsed, syslog-ng assumed the entire line is a $MSG, and prepended its own syslog header. Also, apache itself contains date as well.</div><div dir="auto" style="font-size:12.8px">|</div><div dir="auto" style="font-size:12.8px">| The solution depends on your exact use case. If you want to transport non-syslog data (like apache.log), you'll probably want to dedicate a port to it (so it doesnt mix syslog), or you make sure you | can identify it on the server side.</div><div dir="auto" style="font-size:12.8px"><br></div><div dir="auto" style="font-size:12.8px">| E.g.</div><div dir="auto" style="font-size:12.8px">|</div><div dir="auto" style="font-size:12.8px">| source { file("/var/log/apache/access.l<wbr>og" host-override("hostname") program-override("apache-acces<wbr>s-log") flags(no-parse))); };</div><div dir="auto" style="font-size:12.8px">|</div><div dir="auto" style="font-size:12.8px">| This would read the log file without parsing it, adds $HOST and $PROGRAM fields, which would otherwise be missing.</div><div dir="auto" style="font-size:12.8px">|</div><div dir="auto" style="font-size:12.8px">| Then:</div><div dir="auto" style="font-size:12.8px">|</div><div dir="auto" style="font-size:12.8px">| * send it on to the server using whatever means (tcp and syslog both works), on the wire, the syslog header will be prepended.</div><div dir="auto" style="font-size:12.8px"><br></div></div><div style="font-size:12.8px">I sat up tcp driver and it stopped to send any log to the server. :)</div><div class="m_-6207913433981906096quoted-text"><div dir="auto" style="font-size:12.8px"><br></div><div dir="auto" style="font-size:12.8px">|</div><div dir="auto" style="font-size:12.8px">| * On the server, identify that these are apache logs (based on the $PROGRAM value), then write a file using a custom template, where you only use $MSG:</div><div dir="auto" style="font-size:12.8px">|</div><div dir="auto" style="font-size:12.8px">|file("logfile" template("$MSG\n"));</div><div dir="auto" style="font-size:12.8px"><br></div></div><div style="font-size:12.8px">I am still got this: <13>1 2017-06-08T14:53:54+02:00 testsrv_access apache-access-log - - - 192.168.56.48 - - [08/Jun/2017:14:53:54 +0200] "GET /index.php HTTP/1.1" 304 -</div><div class="m_-6207913433981906096quoted-text"><div style="font-size:12.8px"></div></div></div></blockquote></div></div></div></div></div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto">This means you are still receiving the client messaged using something like tcp(flags(no-parse))</div><div dir="auto"><br></div><div dir="auto">You have to match the destination on the client with the source on the server. How do those look like?</div><div><div class="h5"><div dir="auto"><br></div><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="m_-6207913433981906096quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="m_-6207913433981906096quoted-text"><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><br></div><div dir="auto" style="font-size:12.8px">| This would remove the syslog header in your output file.</div><div dir="auto" style="font-size:12.8px">| Hope this helps</div><div dir="auto" style="font-size:12.8px">| Bazsi</div><div dir="auto" style="font-size:12.8px"><br></div></div><div style="font-size:12.8px">Thanks, Robert</div></div><div class="gmail_extra"><br><div class="gmail_quote"><div class="m_-6207913433981906096elided-text">2017-06-08 10:19 GMT+02:00 Scheidler, Balázs <span dir="ltr"><<a href="mailto:balazs.scheidler@balabit.com" target="_blank">balazs.scheidler@balabit.com</a>></span><wbr>:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="m_-6207913433981906096elided-text"><div dir="auto"><span><div><br><div class="gmail_extra"><br><div class="gmail_quote">On Jun 8, 2017 08:38, "Sandbox" <<a href="mailto:sandboxheh@gmail.com" target="_blank">sandboxheh@gmail.com</a>> wrote:<br type="attribution"><blockquote class="m_-6207913433981906096m_1853049704165417901m_-725075846151490485quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>I just started to test, learn etc syslog-ng, my server configuration is really basic:</div><div><br></div><div>Q: Can i filter (and mark them on client) the incoming logs, so i dont have to open multiple ports for different logs?</div></div></blockquote></div></div></div><div dir="auto"><br></div></span><div dir="auto">Sure, you can open one port and have it filtered using source ip (netmask() filter), embedded hostname (host() filter) or even message content.</div><span><div dir="auto"><br></div><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="m_-6207913433981906096m_1853049704165417901m_-725075846151490485quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div><div><br></div><div>The stored log:</div><div><br></div><div>Q: Why does it store the date 3 times in every logs?</div></div></blockquote></div></div></div><div dir="auto"><br></div></span><div dir="auto">You seem to have received an rfc5424 formatted message, but it was not parsed, maybe because you were using the wrong source driver (syslog() is the one that should handle this format).</div><div dir="auto"><br></div><div dir="auto">Since it wasnt parsed, syslog-ng assumed the entire line is a $MSG, and prepended its own syslog header. Also, apache itself contains date as well.</div><div dir="auto"><br></div><div dir="auto">The solution depends on your exact use case. If you want to transport non-syslog data (like apache.log), you'll probably want to dedicate a port to it (so it doesnt mix syslog), or you make sure you can identify it on the server side.</div><div dir="auto"><br></div><div dir="auto">E.g.</div><div dir="auto"><br></div><div dir="auto">source { file("/var/log/apache/access.l<wbr>og" host-override("hostname") program-override("apache-acces<wbr>s-log") flags(no-parse))); };</div><div dir="auto"><br></div><div dir="auto">This would read the log file without parsing it, adds $HOST and $PROGRAM fields, which would otherwise be missing.</div><div dir="auto"><br></div><div dir="auto">Then:</div><div dir="auto"><br></div><div dir="auto">* send it on to the server using whatever means (tcp and syslog both works), on the wire, the syslog header will be prepended.</div><div dir="auto"><br></div><div dir="auto">* On the server, identify that these are apache logs (based on the $PROGRAM value), then write a file using a custom template, where you only use $MSG:</div><div dir="auto"><br></div><div dir="auto">file("logfile" template("$MSG\n"));</div><div dir="auto"><br></div><div dir="auto">This would remove the syslog header in your output file.</div><div dir="auto">Hope this helps</div><div dir="auto">Bazsi</div><span><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="m_-6207913433981906096m_1853049704165417901m_-725075846151490485quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>Jun 8 08:20:11 192.168.7.30 133 <13>1 2017-06-08T08:20:11+02:00 testweb01 - - - [meta sequenceId="24"] :1 - - [08/Jun/2017:08:20:10 +0200] "GET / HTTP/1.1" 200 3004<br></div><div><br></div><div><br></div><div><br></div></div></blockquote></div></div></div></span></div>
<br></div>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=syslog<wbr>-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>
</blockquote></div><br></div></div></div></div></div>
</blockquote></div><br></div>