[syslog-ng] Parse client IP out of Proxy Protocol Line in TCP syslog->ELB->syslog-ng

[Tibor Benke]

I see. But it's also a nice use-case for pattern matching support with the
`match` keyword :)

2015-10-16 10:07 GMT+02:00 Budai, László <laszlo.budai at balabit.com>:

> You are right, but this is something what I wanted to try for some other
> reasons :-)
>
> L.
>
> On Fri, Oct 16, 2015 at 10:04 AM, Tibor Benke <ihrwein at gmail.com> wrote:
>
>> Maybe the csv-parser would also work with space as a separator character.
>>
>> 2015-10-16 9:53 GMT+02:00 Budai, László <laszlo.budai at balabit.com>:
>>
>>> Hi,
>>>
>>> a more specific example that I've created (just a POC):
>>>
>>> @version: 3.7
>>> @include "scl.conf"
>>>
>>> #PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP
>>> + single space + PROXY_IP + single space + CLIENT_PORT + single space +
>>> PROXY_PORT + "\r\n"
>>> python {
>>> import socket
>>> import json
>>> def aws_elb_proxy_protocol2json(logmsg, arg):
>>>   out = {}
>>>   token_separator = ' '
>>>   tokens = arg.split(token_separator)
>>>   if not tokens or len(tokens) < 6:
>>>     return json.dumps({"aws_elb_proxy_protocol2json.error": "split
>>> failure", "message": arg})
>>>
>>>   out["proxy"] = tokens[0]
>>>   out["inet_protocol"] = tokens[1]
>>>   out["client_ip"] = tokens[2]
>>>   out["proxy_ip"] = tokens[3]
>>>   out["client_port"] = tokens[4]
>>>   out["proxy_port"] = tokens[5]
>>>
>>>   return json.dumps(out)
>>> };
>>>
>>> block parser aws_elb2json (
>>>   template("${MSG}")
>>>   rec-sep(' ')
>>>   field-sep(' ')
>>> ) {
>>>   json-parser(template("$(python aws_elb_proxy_protocol2json
>>> `template`)"));
>>> };
>>>
>>> source s_aws_elb {
>>>   file("/tmp/aws-elb.log" flags(no-parse));
>>> };
>>>
>>> destination d_client_port_odd_json {
>>>     file("/tmp/aws_elb_client_port_odd_json.log" template("$(format-json
>>> -s nv-pairs)\n"));
>>> };
>>>
>>> destination d_client_port_even {
>>>     #file("/tmp/aws_elb_client_port_even.log" template("$(format-json -s
>>> nv-pairs)\n"));
>>>     file("/tmp/aws_elb_client_port_even.log");
>>> };
>>>
>>> filter f_client_port_odd { match("\d*[13579]$" value("client_port")); };
>>> filter f_client_port_even { match("\d*[02468]$" value("client_port")); };
>>>
>>> log {
>>>     source(s_aws_elb);
>>>     parser { aws_elb2json(); };
>>>
>>>     filter(f_client_port_odd);
>>>     destination(d_client_port_odd_json);
>>> };
>>>
>>> log {
>>>     source(s_aws_elb);
>>>     parser { aws_elb2json(); };
>>>
>>>     filter(f_client_port_even);
>>>     destination(d_client_port_even);
>>> };
>>>
>>>
>>> regards,
>>> L.
>>>
>>> On Thu, Oct 15, 2015 at 7:28 PM, Nadine Miller <
>>> nadine.miller at defpoint.com> wrote:
>>>
>>>> I've searched through the archives and spent some time trying to find
>>>> possible answers on the web, but haven't found a definitive answer.
>>>>
>>>> I'm in a situation where I need to parse syslog streams being
>>>> forwarded through an AWS ELB. The normal configuration of the ELB
>>>> resets the source IP to be the ELB's IP address. Logs are coming from
>>>> multiple AWS VPCs, and we've already discovered duplicate hostnames
>>>> across different VPCs, which has mingled logs from different hosts
>>>> into one receiving log file.
>>>>
>>>> The ELB has another mode, referred to as "Proxy Protocol" which adds a
>>>> single line to the TCP stream in the form:
>>>>
>>>> PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP
>>>> + single space + PROXY_IP + single space + CLIENT_PORT + single space
>>>> + PROXY_PORT + "\r\n"
>>>>
>>>> Example:
>>>>
>>>> PROXY TCP4 198.51.100.22 203.0.113.7 35646 80\r\n
>>>>
>>>> Is it possible to use this proxy line in syslog-ng to properly
>>>> segregate the log messages? If so, what would be the best method to
>>>> use? I've done a lot of filtering/templating with normal UDP syslog
>>>> and syslog-ng, but this is the first time I've had to consider
>>>> something crazy like this.
>>>>
>>>> Currently there is no option at this time to change configurations at
>>>> endpoints sending the  syslog messages, nor can we remove the ELB.
>>>>
>>>> For reference:
>>>>
>>>> http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable-proxy-protocol.html
>>>>
>>>> Thanks in advance--
>>>> =N=
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20151016/7ae061c2/attachment-0001.htm