Thanks Tibor and Laszlo, I will explore both of these options. =Nadine= -- Nadine Miller Principal Security Engineer email: nadine.miller@defpoint.com phone: (408) 667-9004 (Eastern) On Oct 16, 2015, at 4:04 AM, Tibor Benke <ihrwein@gmail.com> wrote: Maybe the csv-parser would also work with space as a separator character. 2015-10-16 9:53 GMT+02:00 Budai, László <laszlo.budai@balabit.com>: Hi, a more specific example that I've created (just a POC): @version: 3.7 @include "scl.conf" #PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP + single space + PROXY_IP + single space + CLIENT_PORT + single space + PROXY_PORT + "\r\n" python { import socket import json def aws_elb_proxy_protocol2json(logmsg, arg): out = {} token_separator = ' ' tokens = arg.split(token_separator) if not tokens or len(tokens) < 6: return json.dumps({"aws_elb_proxy_protocol2json.error": "split failure", "message": arg}) out["proxy"] = tokens[0] out["inet_protocol"] = tokens[1] out["client_ip"] = tokens[2] out["proxy_ip"] = tokens[3] out["client_port"] = tokens[4] out["proxy_port"] = tokens[5] return json.dumps(out) }; block parser aws_elb2json ( template("${MSG}") rec-sep(' ') field-sep(' ') ) { json-parser(template("$(python aws_elb_proxy_protocol2json `template`)")); }; source s_aws_elb { file("/tmp/aws-elb.log" flags(no-parse)); }; destination d_client_port_odd_json { file("/tmp/aws_elb_client_port_odd_json.log" template("$(format-json -s nv-pairs)\n")); }; destination d_client_port_even { #file("/tmp/aws_elb_client_port_even.log" template("$(format-json -s nv-pairs)\n")); file("/tmp/aws_elb_client_port_even.log"); }; filter f_client_port_odd { match("\d*[13579]$" value("client_port")); }; filter f_client_port_even { match("\d*[02468]$" value("client_port")); }; log { source(s_aws_elb); parser { aws_elb2json(); }; filter(f_client_port_odd); destination(d_client_port_odd_json); }; log { source(s_aws_elb); parser { aws_elb2json(); }; filter(f_client_port_even); destination(d_client_port_even); }; regards, L. On Thu, Oct 15, 2015 at 7:28 PM, Nadine Miller <nadine.miller@defpoint.com
wrote: I've searched through the archives and spent some time trying to find possible answers on the web, but haven't found a definitive answer.
I'm in a situation where I need to parse syslog streams being forwarded through an AWS ELB. The normal configuration of the ELB resets the source IP to be the ELB's IP address. Logs are coming from multiple AWS VPCs, and we've already discovered duplicate hostnames across different VPCs, which has mingled logs from different hosts into one receiving log file. The ELB has another mode, referred to as "Proxy Protocol" which adds a single line to the TCP stream in the form: PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP + single space + PROXY_IP + single space + CLIENT_PORT + single space + PROXY_PORT + "\r\n" Example: PROXY TCP4 198.51.100.22 203.0.113.7 35646 80\r\n Is it possible to use this proxy line in syslog-ng to properly segregate the log messages? If so, what would be the best method to use? I've done a lot of filtering/templating with normal UDP syslog and syslog-ng, but this is the first time I've had to consider something crazy like this. Currently there is no option at this time to change configurations at endpoints sending the syslog messages, nor can we remove the ELB. For reference: http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable... Thanks in advance-- =N= ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq