<div dir="ltr"><div>The backend server itself would be the syslog-ng. ELBs are not just a simple HTTP proxies, they can act as a TCP forwarder/proxy too. It is a common logging pattern in AWS to use ELBs to load balance log streams between multiple log collection servers. To be able to keep the original source IP and port, ELB uses the Proxy Protocol v1 (<a href="http://www.haproxy.org/download/1.5/doc/proxy-protocol.txt">http://www.haproxy.org/download/1.5/doc/proxy-protocol.txt</a>) which injects a human readable proxy header at the beginning of the TCP stream. So if the log stream is '\n' separated, the first log message would contain the proxy header, but no other log message would contain any reference on the original source ip.<br><br></div><div>So this problem cannot be solved without keeping the state throughout a connection. Because ELB will open a new connection to the backend on every incoming connection from the clients, syslog-ng will see different connections for different clients. The source IP would be the ELB IP, but the port would be varying. <br><br></div><div>With patterndb, you can parse out the proxy header line, and the store the original IP in a log message field. Then you can write a python rewrite/filter/template statement (don't know which would be the best) which can store the original IP of the connection under the key (source_ip,source_port) in a hash table or a Redis DB. With every log message in a connection, you can look up the original IP of the connection by using the key (source_ip, source_port), and add it to the log message.<br><br></div><div>Regards,<br></div><div>Viktor<br></div><div><br><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Oct 18, 2015 at 11:44 AM, Balazs Scheidler <span dir="ltr"><<a href="mailto:bazsi77@gmail.com" target="_blank">bazsi77@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">Hi,</p>
<p dir="ltr">While thinking about the subject (BTW, could you try the parser I sent?), the scenario you were describing doesn't add up.</p>
<p dir="ltr">This line is sent to the backend server, not to a log server, how would you send this format to syslog-ng?</p>
<p dir="ltr">Thanks</p><div class="HOEnZb"><div class="h5">
<div class="gmail_quote">On Oct 15, 2015 19:28, "Nadine Miller" <<a href="mailto:nadine.miller@defpoint.com" target="_blank">nadine.miller@defpoint.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I've searched through the archives and spent some time trying to find<br>
possible answers on the web, but haven't found a definitive answer.<br>
<br>
I'm in a situation where I need to parse syslog streams being<br>
forwarded through an AWS ELB. The normal configuration of the ELB<br>
resets the source IP to be the ELB's IP address. Logs are coming from<br>
multiple AWS VPCs, and we've already discovered duplicate hostnames<br>
across different VPCs, which has mingled logs from different hosts<br>
into one receiving log file.<br>
<br>
The ELB has another mode, referred to as "Proxy Protocol" which adds a<br>
single line to the TCP stream in the form:<br>
<br>
PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP<br>
+ single space + PROXY_IP + single space + CLIENT_PORT + single space<br>
+ PROXY_PORT + "\r\n"<br>
<br>
Example:<br>
<br>
PROXY TCP4 198.51.100.22 203.0.113.7 35646 80\r\n<br>
<br>
Is it possible to use this proxy line in syslog-ng to properly<br>
segregate the log messages? If so, what would be the best method to<br>
use? I've done a lot of filtering/templating with normal UDP syslog<br>
and syslog-ng, but this is the first time I've had to consider<br>
something crazy like this.<br>
<br>
Currently there is no option at this time to change configurations at<br>
endpoints sending the syslog messages, nor can we remove the ELB.<br>
<br>
For reference:<br>
<a href="http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable-proxy-protocol.html" rel="noreferrer" target="_blank">http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable-proxy-protocol.html</a><br>
<br>
Thanks in advance--<br>
=N=<br>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div>
</div></div><br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>