[syslog-ng] Parse client IP out of Proxy Protocol Line in TCP syslog->ELB->syslog-ng

[Budai, László]

You are right, but this is something what I wanted to try for some other
reasons :-)

L.

On Fri, Oct 16, 2015 at 10:04 AM, Tibor Benke <ihrwein at gmail.com> wrote:

> Maybe the csv-parser would also work with space as a separator character.
>
> 2015-10-16 9:53 GMT+02:00 Budai, László <laszlo.budai at balabit.com>:
>
>> Hi,
>>
>> a more specific example that I've created (just a POC):
>>
>> @version: 3.7
>> @include "scl.conf"
>>
>> #PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP +
>> single space + PROXY_IP + single space + CLIENT_PORT + single space +
>> PROXY_PORT + "\r\n"
>> python {
>> import socket
>> import json
>> def aws_elb_proxy_protocol2json(logmsg, arg):
>>   out = {}
>>   token_separator = ' '
>>   tokens = arg.split(token_separator)
>>   if not tokens or len(tokens) < 6:
>>     return json.dumps({"aws_elb_proxy_protocol2json.error": "split
>> failure", "message": arg})
>>
>>   out["proxy"] = tokens[0]
>>   out["inet_protocol"] = tokens[1]
>>   out["client_ip"] = tokens[2]
>>   out["proxy_ip"] = tokens[3]
>>   out["client_port"] = tokens[4]
>>   out["proxy_port"] = tokens[5]
>>
>>   return json.dumps(out)
>> };
>>
>> block parser aws_elb2json (
>>   template("${MSG}")
>>   rec-sep(' ')
>>   field-sep(' ')
>> ) {
>>   json-parser(template("$(python aws_elb_proxy_protocol2json
>> `template`)"));
>> };
>>
>> source s_aws_elb {
>>   file("/tmp/aws-elb.log" flags(no-parse));
>> };
>>
>> destination d_client_port_odd_json {
>>     file("/tmp/aws_elb_client_port_odd_json.log" template("$(format-json
>> -s nv-pairs)\n"));
>> };
>>
>> destination d_client_port_even {
>>     #file("/tmp/aws_elb_client_port_even.log" template("$(format-json -s
>> nv-pairs)\n"));
>>     file("/tmp/aws_elb_client_port_even.log");
>> };
>>
>> filter f_client_port_odd { match("\d*[13579]$" value("client_port")); };
>> filter f_client_port_even { match("\d*[02468]$" value("client_port")); };
>>
>> log {
>>     source(s_aws_elb);
>>     parser { aws_elb2json(); };
>>
>>     filter(f_client_port_odd);
>>     destination(d_client_port_odd_json);
>> };
>>
>> log {
>>     source(s_aws_elb);
>>     parser { aws_elb2json(); };
>>
>>     filter(f_client_port_even);
>>     destination(d_client_port_even);
>> };
>>
>>
>> regards,
>> L.
>>
>> On Thu, Oct 15, 2015 at 7:28 PM, Nadine Miller <
>> nadine.miller at defpoint.com> wrote:
>>
>>> I've searched through the archives and spent some time trying to find
>>> possible answers on the web, but haven't found a definitive answer.
>>>
>>> I'm in a situation where I need to parse syslog streams being
>>> forwarded through an AWS ELB. The normal configuration of the ELB
>>> resets the source IP to be the ELB's IP address. Logs are coming from
>>> multiple AWS VPCs, and we've already discovered duplicate hostnames
>>> across different VPCs, which has mingled logs from different hosts
>>> into one receiving log file.
>>>
>>> The ELB has another mode, referred to as "Proxy Protocol" which adds a
>>> single line to the TCP stream in the form:
>>>
>>> PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP
>>> + single space + PROXY_IP + single space + CLIENT_PORT + single space
>>> + PROXY_PORT + "\r\n"
>>>
>>> Example:
>>>
>>> PROXY TCP4 198.51.100.22 203.0.113.7 35646 80\r\n
>>>
>>> Is it possible to use this proxy line in syslog-ng to properly
>>> segregate the log messages? If so, what would be the best method to
>>> use? I've done a lot of filtering/templating with normal UDP syslog
>>> and syslog-ng, but this is the first time I've had to consider
>>> something crazy like this.
>>>
>>> Currently there is no option at this time to change configurations at
>>> endpoints sending the  syslog messages, nor can we remove the ELB.
>>>
>>> For reference:
>>>
>>> http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable-proxy-protocol.html
>>>
>>> Thanks in advance--
>>> =N=
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20151016/f6396aaf/attachment.htm