[syslog-ng] Parse client IP out of Proxy Protocol Line in TCP syslog->ELB->syslog-ng

[Budai, László]

Hi,

a more specific example that I've created (just a POC):

@version: 3.7
@include "scl.conf"

#PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP +
single space + PROXY_IP + single space + CLIENT_PORT + single space +
PROXY_PORT + "\r\n"
python {
import socket
import json
def aws_elb_proxy_protocol2json(logmsg, arg):
  out = {}
  token_separator = ' '
  tokens = arg.split(token_separator)
  if not tokens or len(tokens) < 6:
    return json.dumps({"aws_elb_proxy_protocol2json.error": "split
failure", "message": arg})

  out["proxy"] = tokens[0]
  out["inet_protocol"] = tokens[1]
  out["client_ip"] = tokens[2]
  out["proxy_ip"] = tokens[3]
  out["client_port"] = tokens[4]
  out["proxy_port"] = tokens[5]

  return json.dumps(out)
};

block parser aws_elb2json (
  template("${MSG}")
  rec-sep(' ')
  field-sep(' ')
) {
  json-parser(template("$(python aws_elb_proxy_protocol2json `template`)"));
};

source s_aws_elb {
  file("/tmp/aws-elb.log" flags(no-parse));
};

destination d_client_port_odd_json {
    file("/tmp/aws_elb_client_port_odd_json.log" template("$(format-json -s
nv-pairs)\n"));
};

destination d_client_port_even {
    #file("/tmp/aws_elb_client_port_even.log" template("$(format-json -s
nv-pairs)\n"));
    file("/tmp/aws_elb_client_port_even.log");
};

filter f_client_port_odd { match("\d*[13579]$" value("client_port")); };
filter f_client_port_even { match("\d*[02468]$" value("client_port")); };

log {
    source(s_aws_elb);
    parser { aws_elb2json(); };

    filter(f_client_port_odd);
    destination(d_client_port_odd_json);
};

log {
    source(s_aws_elb);
    parser { aws_elb2json(); };

    filter(f_client_port_even);
    destination(d_client_port_even);
};


regards,
L.

On Thu, Oct 15, 2015 at 7:28 PM, Nadine Miller <nadine.miller at defpoint.com>
wrote:

> I've searched through the archives and spent some time trying to find
> possible answers on the web, but haven't found a definitive answer.
>
> I'm in a situation where I need to parse syslog streams being
> forwarded through an AWS ELB. The normal configuration of the ELB
> resets the source IP to be the ELB's IP address. Logs are coming from
> multiple AWS VPCs, and we've already discovered duplicate hostnames
> across different VPCs, which has mingled logs from different hosts
> into one receiving log file.
>
> The ELB has another mode, referred to as "Proxy Protocol" which adds a
> single line to the TCP stream in the form:
>
> PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP
> + single space + PROXY_IP + single space + CLIENT_PORT + single space
> + PROXY_PORT + "\r\n"
>
> Example:
>
> PROXY TCP4 198.51.100.22 203.0.113.7 35646 80\r\n
>
> Is it possible to use this proxy line in syslog-ng to properly
> segregate the log messages? If so, what would be the best method to
> use? I've done a lot of filtering/templating with normal UDP syslog
> and syslog-ng, but this is the first time I've had to consider
> something crazy like this.
>
> Currently there is no option at this time to change configurations at
> endpoints sending the  syslog messages, nor can we remove the ELB.
>
> For reference:
>
> http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable-proxy-protocol.html
>
> Thanks in advance--
> =N=
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20151016/2b57179e/attachment.htm