<p dir="ltr">I gave a simpler solution at the end of.the thread.</p>
<div class="gmail_quote">On Oct 20, 2015 16:27, "Nadine Miller" <<a href="mailto:nadine.miller@defpoint.com">nadine.miller@defpoint.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="auto" style="word-wrap:break-word">Thanks Tibor and Laszlo, I will explore both of these options.</div><div dir="auto" style="word-wrap:break-word"><br></div><div dir="auto" style="word-wrap:break-word">=Nadine=<br><div><br>--<br>Nadine Miller<br>Principal Security Engineer<br>email: <a href="mailto:nadine.miller@defpoint.com" target="_blank">nadine.miller@defpoint.com</a><br>phone: <a href="tel:%28408%29%20667-9004" value="+14086679004" target="_blank">(408) 667-9004</a> (Eastern)<br></div><br><blockquote type="cite">On Oct 16, 2015, at 4:04 AM, Tibor Benke <<a href="mailto:ihrwein@gmail.com" target="_blank">ihrwein@gmail.com</a>> wrote:<br><br>Maybe the csv-parser would also work with space as a separator character.<br><br>2015-10-16 9:53 GMT+02:00 Budai, László <<a href="mailto:laszlo.budai@balabit.com" target="_blank">laszlo.budai@balabit.com</a>>:<br>Hi,<br><br>a more specific example that I've created (just a POC):<br><br>@version: 3.7<br>@include "scl.conf"<br><br>#PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP + single space + PROXY_IP + single space + CLIENT_PORT + single space + PROXY_PORT + "\r\n"<br>python {<br>import socket<br>import json<br>def aws_elb_proxy_protocol2json(logmsg, arg):<br> out = {}<br> token_separator = ' '<br> tokens = arg.split(token_separator)<br> if not tokens or len(tokens) < 6:<br> return json.dumps({"aws_elb_proxy_protocol2json.error": "split failure", "message": arg})<br><br> out["proxy"] = tokens[0]<br> out["inet_protocol"] = tokens[1]<br> out["client_ip"] = tokens[2]<br> out["proxy_ip"] = tokens[3]<br> out["client_port"] = tokens[4]<br> out["proxy_port"] = tokens[5]<br><br> return json.dumps(out)<br>};<br><br>block parser aws_elb2json (<br> template("${MSG}")<br> rec-sep(' ')<br> field-sep(' ')<br>) {<br> json-parser(template("$(python aws_elb_proxy_protocol2json `template`)"));<br>};<br> <br>source s_aws_elb {<br> file("/tmp/aws-elb.log" flags(no-parse));<br>};<br> <br>destination d_client_port_odd_json {<br> file("/tmp/aws_elb_client_port_odd_json.log" template("$(format-json -s nv-pairs)\n"));<br>};<br><br>destination d_client_port_even {<br> #file("/tmp/aws_elb_client_port_even.log" template("$(format-json -s nv-pairs)\n"));<br> file("/tmp/aws_elb_client_port_even.log");<br>};<br> <br>filter f_client_port_odd { match("\d*[13579]$" value("client_port")); };<br>filter f_client_port_even { match("\d*[02468]$" value("client_port")); };<br><br>log {<br> source(s_aws_elb);<br> parser { aws_elb2json(); };<br><br> filter(f_client_port_odd);<br> destination(d_client_port_odd_json);<br>};<br><br>log {<br> source(s_aws_elb);<br> parser { aws_elb2json(); };<br><br> filter(f_client_port_even);<br> destination(d_client_port_even);<br>};<br><br><br>regards,<br>L.<br><br>On Thu, Oct 15, 2015 at 7:28 PM, Nadine Miller <<a href="mailto:nadine.miller@defpoint.com" target="_blank">nadine.miller@defpoint.com</a>> wrote:<br>I've searched through the archives and spent some time trying to find<br>possible answers on the web, but haven't found a definitive answer.<br><br>I'm in a situation where I need to parse syslog streams being<br>forwarded through an AWS ELB. The normal configuration of the ELB<br>resets the source IP to be the ELB's IP address. Logs are coming from<br>multiple AWS VPCs, and we've already discovered duplicate hostnames<br>across different VPCs, which has mingled logs from different hosts<br>into one receiving log file.<br><br>The ELB has another mode, referred to as "Proxy Protocol" which adds a<br>single line to the TCP stream in the form:<br><br>PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP<br>+ single space + PROXY_IP + single space + CLIENT_PORT + single space<br>+ PROXY_PORT + "\r\n"<br><br>Example:<br><br>PROXY TCP4 198.51.100.22 203.0.113.7 35646 80\r\n<br><br>Is it possible to use this proxy line in syslog-ng to properly<br>segregate the log messages? If so, what would be the best method to<br>use? I've done a lot of filtering/templating with normal UDP syslog<br>and syslog-ng, but this is the first time I've had to consider<br>something crazy like this.<br><br>Currently there is no option at this time to change configurations at<br>endpoints sending the syslog messages, nor can we remove the ELB.<br><br>For reference:<br><a href="http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable-proxy-protocol.html" target="_blank">http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable-proxy-protocol.html</a><br><br>Thanks in advance--<br>=N=<br>______________________________________________________________________________<br>Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br><br><br><br>______________________________________________________________________________<br>Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br><br><br><br>______________________________________________________________________________<br>Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br><br></blockquote><br></div></div>
<br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></blockquote></div>