[syslog-ng] Parse client IP out of Proxy Protocol Line in TCP syslog->ELB->syslog-ng

[Tibor Benke]

Maybe the csv-parser would also work with space as a separator character.

2015-10-16 9:53 GMT+02:00 Budai, László <laszlo.budai at balabit.com>:

> Hi,
>
> a more specific example that I've created (just a POC):
>
> @version: 3.7
> @include "scl.conf"
>
> #PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP +
> single space + PROXY_IP + single space + CLIENT_PORT + single space +
> PROXY_PORT + "\r\n"
> python {
> import socket
> import json
> def aws_elb_proxy_protocol2json(logmsg, arg):
>   out = {}
>   token_separator = ' '
>   tokens = arg.split(token_separator)
>   if not tokens or len(tokens) < 6:
>     return json.dumps({"aws_elb_proxy_protocol2json.error": "split
> failure", "message": arg})
>
>   out["proxy"] = tokens[0]
>   out["inet_protocol"] = tokens[1]
>   out["client_ip"] = tokens[2]
>   out["proxy_ip"] = tokens[3]
>   out["client_port"] = tokens[4]
>   out["proxy_port"] = tokens[5]
>
>   return json.dumps(out)
> };
>
> block parser aws_elb2json (
>   template("${MSG}")
>   rec-sep(' ')
>   field-sep(' ')
> ) {
>   json-parser(template("$(python aws_elb_proxy_protocol2json
> `template`)"));
> };
>
> source s_aws_elb {
>   file("/tmp/aws-elb.log" flags(no-parse));
> };
>
> destination d_client_port_odd_json {
>     file("/tmp/aws_elb_client_port_odd_json.log" template("$(format-json
> -s nv-pairs)\n"));
> };
>
> destination d_client_port_even {
>     #file("/tmp/aws_elb_client_port_even.log" template("$(format-json -s
> nv-pairs)\n"));
>     file("/tmp/aws_elb_client_port_even.log");
> };
>
> filter f_client_port_odd { match("\d*[13579]$" value("client_port")); };
> filter f_client_port_even { match("\d*[02468]$" value("client_port")); };
>
> log {
>     source(s_aws_elb);
>     parser { aws_elb2json(); };
>
>     filter(f_client_port_odd);
>     destination(d_client_port_odd_json);
> };
>
> log {
>     source(s_aws_elb);
>     parser { aws_elb2json(); };
>
>     filter(f_client_port_even);
>     destination(d_client_port_even);
> };
>
>
> regards,
> L.
>
> On Thu, Oct 15, 2015 at 7:28 PM, Nadine Miller <nadine.miller at defpoint.com
> > wrote:
>
>> I've searched through the archives and spent some time trying to find
>> possible answers on the web, but haven't found a definitive answer.
>>
>> I'm in a situation where I need to parse syslog streams being
>> forwarded through an AWS ELB. The normal configuration of the ELB
>> resets the source IP to be the ELB's IP address. Logs are coming from
>> multiple AWS VPCs, and we've already discovered duplicate hostnames
>> across different VPCs, which has mingled logs from different hosts
>> into one receiving log file.
>>
>> The ELB has another mode, referred to as "Proxy Protocol" which adds a
>> single line to the TCP stream in the form:
>>
>> PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP
>> + single space + PROXY_IP + single space + CLIENT_PORT + single space
>> + PROXY_PORT + "\r\n"
>>
>> Example:
>>
>> PROXY TCP4 198.51.100.22 203.0.113.7 35646 80\r\n
>>
>> Is it possible to use this proxy line in syslog-ng to properly
>> segregate the log messages? If so, what would be the best method to
>> use? I've done a lot of filtering/templating with normal UDP syslog
>> and syslog-ng, but this is the first time I've had to consider
>> something crazy like this.
>>
>> Currently there is no option at this time to change configurations at
>> endpoints sending the  syslog messages, nor can we remove the ELB.
>>
>> For reference:
>>
>> http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable-proxy-protocol.html
>>
>> Thanks in advance--
>> =N=
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20151016/c6d5c7b6/attachment-0001.htm