[syslog-ng] Parse client IP out of Proxy Protocol Line in TCP syslog->ELB->syslog-ng

[Scheidler, Balázs]

Ah, thanks for the explanation. I might want to create support for this
kind of load balancing if someone is willing to test it/give feedback.

Nadine?
On Oct 18, 2015 2:21 PM, "Tusa Viktor" <tusavik at gmail.com> wrote:

> The backend server itself would be the syslog-ng. ELBs are not just a
> simple HTTP proxies, they can act as a TCP forwarder/proxy too. It is a
> common logging pattern in AWS to use ELBs to load balance log streams
> between multiple log collection servers. To be able to keep the original
> source IP and port, ELB uses the Proxy Protocol v1 (
> http://www.haproxy.org/download/1.5/doc/proxy-protocol.txt) which injects
> a human readable proxy header at the beginning of the TCP stream. So if the
> log stream is '\n' separated, the first log message would contain the proxy
> header, but no other log message would contain any reference on the
> original source ip.
>
> So this problem cannot be solved without keeping the state throughout a
> connection. Because ELB will open a new connection to the backend on every
> incoming connection from the clients, syslog-ng will see different
> connections for different clients. The source IP would be the ELB IP, but
> the port would be varying.
>
> With patterndb, you can parse out the proxy header line, and the store the
> original IP in a log message field. Then you can write a python
> rewrite/filter/template statement (don't know which would be the best)
> which can store the original IP of the connection under the key
> (source_ip,source_port) in a hash table or a Redis DB. With every log
> message in a connection, you can look up the original IP of the connection
> by using the key (source_ip, source_port), and add it to the log message.
>
> Regards,
> Viktor
>
>
>
> On Sun, Oct 18, 2015 at 11:44 AM, Balazs Scheidler <bazsi77 at gmail.com>
> wrote:
>
>> Hi,
>>
>> While thinking about the subject (BTW, could you try the parser I sent?),
>> the scenario you were describing doesn't add up.
>>
>> This line is sent to the backend server, not to a log server, how would
>> you send this format to syslog-ng?
>>
>> Thanks
>> On Oct 15, 2015 19:28, "Nadine Miller" <nadine.miller at defpoint.com>
>> wrote:
>>
>>> I've searched through the archives and spent some time trying to find
>>> possible answers on the web, but haven't found a definitive answer.
>>>
>>> I'm in a situation where I need to parse syslog streams being
>>> forwarded through an AWS ELB. The normal configuration of the ELB
>>> resets the source IP to be the ELB's IP address. Logs are coming from
>>> multiple AWS VPCs, and we've already discovered duplicate hostnames
>>> across different VPCs, which has mingled logs from different hosts
>>> into one receiving log file.
>>>
>>> The ELB has another mode, referred to as "Proxy Protocol" which adds a
>>> single line to the TCP stream in the form:
>>>
>>> PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP
>>> + single space + PROXY_IP + single space + CLIENT_PORT + single space
>>> + PROXY_PORT + "\r\n"
>>>
>>> Example:
>>>
>>> PROXY TCP4 198.51.100.22 203.0.113.7 35646 80\r\n
>>>
>>> Is it possible to use this proxy line in syslog-ng to properly
>>> segregate the log messages? If so, what would be the best method to
>>> use? I've done a lot of filtering/templating with normal UDP syslog
>>> and syslog-ng, but this is the first time I've had to consider
>>> something crazy like this.
>>>
>>> Currently there is no option at this time to change configurations at
>>> endpoints sending the  syslog messages, nor can we remove the ELB.
>>>
>>> For reference:
>>>
>>> http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable-proxy-protocol.html
>>>
>>> Thanks in advance--
>>> =N=
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20151018/09539450/attachment.htm