Parse client IP out of Proxy Protocol Line in TCP syslog->ELB->syslog-ng
I've searched through the archives and spent some time trying to find possible answers on the web, but haven't found a definitive answer. I'm in a situation where I need to parse syslog streams being forwarded through an AWS ELB. The normal configuration of the ELB resets the source IP to be the ELB's IP address. Logs are coming from multiple AWS VPCs, and we've already discovered duplicate hostnames across different VPCs, which has mingled logs from different hosts into one receiving log file. The ELB has another mode, referred to as "Proxy Protocol" which adds a single line to the TCP stream in the form: PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP + single space + PROXY_IP + single space + CLIENT_PORT + single space + PROXY_PORT + "\r\n" Example: PROXY TCP4 198.51.100.22 203.0.113.7 35646 80\r\n Is it possible to use this proxy line in syslog-ng to properly segregate the log messages? If so, what would be the best method to use? I've done a lot of filtering/templating with normal UDP syslog and syslog-ng, but this is the first time I've had to consider something crazy like this. Currently there is no option at this time to change configurations at endpoints sending the syslog messages, nor can we remove the ELB. For reference: http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable... Thanks in advance-- =N=
Hi Nadine, maybe this could help you: https://guest.blogs.balabit.com/2015/09/processing-log-messages-with-python-... regards, L. On Thu, Oct 15, 2015 at 7:28 PM, Nadine Miller <nadine.miller@defpoint.com> wrote:
I've searched through the archives and spent some time trying to find possible answers on the web, but haven't found a definitive answer.
I'm in a situation where I need to parse syslog streams being forwarded through an AWS ELB. The normal configuration of the ELB resets the source IP to be the ELB's IP address. Logs are coming from multiple AWS VPCs, and we've already discovered duplicate hostnames across different VPCs, which has mingled logs from different hosts into one receiving log file.
The ELB has another mode, referred to as "Proxy Protocol" which adds a single line to the TCP stream in the form:
PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP + single space + PROXY_IP + single space + CLIENT_PORT + single space + PROXY_PORT + "\r\n"
Example:
PROXY TCP4 198.51.100.22 203.0.113.7 35646 80\r\n
Is it possible to use this proxy line in syslog-ng to properly segregate the log messages? If so, what would be the best method to use? I've done a lot of filtering/templating with normal UDP syslog and syslog-ng, but this is the first time I've had to consider something crazy like this.
Currently there is no option at this time to change configurations at endpoints sending the syslog messages, nor can we remove the ELB.
For reference:
http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable...
Thanks in advance-- =N=
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, a more specific example that I've created (just a POC): @version: 3.7 @include "scl.conf" #PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP + single space + PROXY_IP + single space + CLIENT_PORT + single space + PROXY_PORT + "\r\n" python { import socket import json def aws_elb_proxy_protocol2json(logmsg, arg): out = {} token_separator = ' ' tokens = arg.split(token_separator) if not tokens or len(tokens) < 6: return json.dumps({"aws_elb_proxy_protocol2json.error": "split failure", "message": arg}) out["proxy"] = tokens[0] out["inet_protocol"] = tokens[1] out["client_ip"] = tokens[2] out["proxy_ip"] = tokens[3] out["client_port"] = tokens[4] out["proxy_port"] = tokens[5] return json.dumps(out) }; block parser aws_elb2json ( template("${MSG}") rec-sep(' ') field-sep(' ') ) { json-parser(template("$(python aws_elb_proxy_protocol2json `template`)")); }; source s_aws_elb { file("/tmp/aws-elb.log" flags(no-parse)); }; destination d_client_port_odd_json { file("/tmp/aws_elb_client_port_odd_json.log" template("$(format-json -s nv-pairs)\n")); }; destination d_client_port_even { #file("/tmp/aws_elb_client_port_even.log" template("$(format-json -s nv-pairs)\n")); file("/tmp/aws_elb_client_port_even.log"); }; filter f_client_port_odd { match("\d*[13579]$" value("client_port")); }; filter f_client_port_even { match("\d*[02468]$" value("client_port")); }; log { source(s_aws_elb); parser { aws_elb2json(); }; filter(f_client_port_odd); destination(d_client_port_odd_json); }; log { source(s_aws_elb); parser { aws_elb2json(); }; filter(f_client_port_even); destination(d_client_port_even); }; regards, L. On Thu, Oct 15, 2015 at 7:28 PM, Nadine Miller <nadine.miller@defpoint.com> wrote:
I've searched through the archives and spent some time trying to find possible answers on the web, but haven't found a definitive answer.
I'm in a situation where I need to parse syslog streams being forwarded through an AWS ELB. The normal configuration of the ELB resets the source IP to be the ELB's IP address. Logs are coming from multiple AWS VPCs, and we've already discovered duplicate hostnames across different VPCs, which has mingled logs from different hosts into one receiving log file.
The ELB has another mode, referred to as "Proxy Protocol" which adds a single line to the TCP stream in the form:
PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP + single space + PROXY_IP + single space + CLIENT_PORT + single space + PROXY_PORT + "\r\n"
Example:
PROXY TCP4 198.51.100.22 203.0.113.7 35646 80\r\n
Is it possible to use this proxy line in syslog-ng to properly segregate the log messages? If so, what would be the best method to use? I've done a lot of filtering/templating with normal UDP syslog and syslog-ng, but this is the first time I've had to consider something crazy like this.
Currently there is no option at this time to change configurations at endpoints sending the syslog messages, nor can we remove the ELB.
For reference:
http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable...
Thanks in advance-- =N=
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Maybe the csv-parser would also work with space as a separator character. 2015-10-16 9:53 GMT+02:00 Budai, László <laszlo.budai@balabit.com>:
Hi,
a more specific example that I've created (just a POC):
@version: 3.7 @include "scl.conf"
#PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP + single space + PROXY_IP + single space + CLIENT_PORT + single space + PROXY_PORT + "\r\n" python { import socket import json def aws_elb_proxy_protocol2json(logmsg, arg): out = {} token_separator = ' ' tokens = arg.split(token_separator) if not tokens or len(tokens) < 6: return json.dumps({"aws_elb_proxy_protocol2json.error": "split failure", "message": arg})
out["proxy"] = tokens[0] out["inet_protocol"] = tokens[1] out["client_ip"] = tokens[2] out["proxy_ip"] = tokens[3] out["client_port"] = tokens[4] out["proxy_port"] = tokens[5]
return json.dumps(out) };
block parser aws_elb2json ( template("${MSG}") rec-sep(' ') field-sep(' ') ) { json-parser(template("$(python aws_elb_proxy_protocol2json `template`)")); };
source s_aws_elb { file("/tmp/aws-elb.log" flags(no-parse)); };
destination d_client_port_odd_json { file("/tmp/aws_elb_client_port_odd_json.log" template("$(format-json -s nv-pairs)\n")); };
destination d_client_port_even { #file("/tmp/aws_elb_client_port_even.log" template("$(format-json -s nv-pairs)\n")); file("/tmp/aws_elb_client_port_even.log"); };
filter f_client_port_odd { match("\d*[13579]$" value("client_port")); }; filter f_client_port_even { match("\d*[02468]$" value("client_port")); };
log { source(s_aws_elb); parser { aws_elb2json(); };
filter(f_client_port_odd); destination(d_client_port_odd_json); };
log { source(s_aws_elb); parser { aws_elb2json(); };
filter(f_client_port_even); destination(d_client_port_even); };
regards, L.
On Thu, Oct 15, 2015 at 7:28 PM, Nadine Miller <nadine.miller@defpoint.com
wrote:
I've searched through the archives and spent some time trying to find possible answers on the web, but haven't found a definitive answer.
I'm in a situation where I need to parse syslog streams being forwarded through an AWS ELB. The normal configuration of the ELB resets the source IP to be the ELB's IP address. Logs are coming from multiple AWS VPCs, and we've already discovered duplicate hostnames across different VPCs, which has mingled logs from different hosts into one receiving log file.
The ELB has another mode, referred to as "Proxy Protocol" which adds a single line to the TCP stream in the form:
PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP + single space + PROXY_IP + single space + CLIENT_PORT + single space + PROXY_PORT + "\r\n"
Example:
PROXY TCP4 198.51.100.22 203.0.113.7 35646 80\r\n
Is it possible to use this proxy line in syslog-ng to properly segregate the log messages? If so, what would be the best method to use? I've done a lot of filtering/templating with normal UDP syslog and syslog-ng, but this is the first time I've had to consider something crazy like this.
Currently there is no option at this time to change configurations at endpoints sending the syslog messages, nor can we remove the ELB.
For reference:
http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable...
Thanks in advance-- =N=
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
You are right, but this is something what I wanted to try for some other reasons :-) L. On Fri, Oct 16, 2015 at 10:04 AM, Tibor Benke <ihrwein@gmail.com> wrote:
Maybe the csv-parser would also work with space as a separator character.
2015-10-16 9:53 GMT+02:00 Budai, László <laszlo.budai@balabit.com>:
Hi,
a more specific example that I've created (just a POC):
@version: 3.7 @include "scl.conf"
#PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP + single space + PROXY_IP + single space + CLIENT_PORT + single space + PROXY_PORT + "\r\n" python { import socket import json def aws_elb_proxy_protocol2json(logmsg, arg): out = {} token_separator = ' ' tokens = arg.split(token_separator) if not tokens or len(tokens) < 6: return json.dumps({"aws_elb_proxy_protocol2json.error": "split failure", "message": arg})
out["proxy"] = tokens[0] out["inet_protocol"] = tokens[1] out["client_ip"] = tokens[2] out["proxy_ip"] = tokens[3] out["client_port"] = tokens[4] out["proxy_port"] = tokens[5]
return json.dumps(out) };
block parser aws_elb2json ( template("${MSG}") rec-sep(' ') field-sep(' ') ) { json-parser(template("$(python aws_elb_proxy_protocol2json `template`)")); };
source s_aws_elb { file("/tmp/aws-elb.log" flags(no-parse)); };
destination d_client_port_odd_json { file("/tmp/aws_elb_client_port_odd_json.log" template("$(format-json -s nv-pairs)\n")); };
destination d_client_port_even { #file("/tmp/aws_elb_client_port_even.log" template("$(format-json -s nv-pairs)\n")); file("/tmp/aws_elb_client_port_even.log"); };
filter f_client_port_odd { match("\d*[13579]$" value("client_port")); }; filter f_client_port_even { match("\d*[02468]$" value("client_port")); };
log { source(s_aws_elb); parser { aws_elb2json(); };
filter(f_client_port_odd); destination(d_client_port_odd_json); };
log { source(s_aws_elb); parser { aws_elb2json(); };
filter(f_client_port_even); destination(d_client_port_even); };
regards, L.
On Thu, Oct 15, 2015 at 7:28 PM, Nadine Miller < nadine.miller@defpoint.com> wrote:
I've searched through the archives and spent some time trying to find possible answers on the web, but haven't found a definitive answer.
I'm in a situation where I need to parse syslog streams being forwarded through an AWS ELB. The normal configuration of the ELB resets the source IP to be the ELB's IP address. Logs are coming from multiple AWS VPCs, and we've already discovered duplicate hostnames across different VPCs, which has mingled logs from different hosts into one receiving log file.
The ELB has another mode, referred to as "Proxy Protocol" which adds a single line to the TCP stream in the form:
PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP + single space + PROXY_IP + single space + CLIENT_PORT + single space + PROXY_PORT + "\r\n"
Example:
PROXY TCP4 198.51.100.22 203.0.113.7 35646 80\r\n
Is it possible to use this proxy line in syslog-ng to properly segregate the log messages? If so, what would be the best method to use? I've done a lot of filtering/templating with normal UDP syslog and syslog-ng, but this is the first time I've had to consider something crazy like this.
Currently there is no option at this time to change configurations at endpoints sending the syslog messages, nor can we remove the ELB.
For reference:
http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable...
Thanks in advance-- =N=
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I see. But it's also a nice use-case for pattern matching support with the `match` keyword :) 2015-10-16 10:07 GMT+02:00 Budai, László <laszlo.budai@balabit.com>:
You are right, but this is something what I wanted to try for some other reasons :-)
L.
On Fri, Oct 16, 2015 at 10:04 AM, Tibor Benke <ihrwein@gmail.com> wrote:
Maybe the csv-parser would also work with space as a separator character.
2015-10-16 9:53 GMT+02:00 Budai, László <laszlo.budai@balabit.com>:
Hi,
a more specific example that I've created (just a POC):
@version: 3.7 @include "scl.conf"
#PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP + single space + PROXY_IP + single space + CLIENT_PORT + single space + PROXY_PORT + "\r\n" python { import socket import json def aws_elb_proxy_protocol2json(logmsg, arg): out = {} token_separator = ' ' tokens = arg.split(token_separator) if not tokens or len(tokens) < 6: return json.dumps({"aws_elb_proxy_protocol2json.error": "split failure", "message": arg})
out["proxy"] = tokens[0] out["inet_protocol"] = tokens[1] out["client_ip"] = tokens[2] out["proxy_ip"] = tokens[3] out["client_port"] = tokens[4] out["proxy_port"] = tokens[5]
return json.dumps(out) };
block parser aws_elb2json ( template("${MSG}") rec-sep(' ') field-sep(' ') ) { json-parser(template("$(python aws_elb_proxy_protocol2json `template`)")); };
source s_aws_elb { file("/tmp/aws-elb.log" flags(no-parse)); };
destination d_client_port_odd_json { file("/tmp/aws_elb_client_port_odd_json.log" template("$(format-json -s nv-pairs)\n")); };
destination d_client_port_even { #file("/tmp/aws_elb_client_port_even.log" template("$(format-json -s nv-pairs)\n")); file("/tmp/aws_elb_client_port_even.log"); };
filter f_client_port_odd { match("\d*[13579]$" value("client_port")); }; filter f_client_port_even { match("\d*[02468]$" value("client_port")); };
log { source(s_aws_elb); parser { aws_elb2json(); };
filter(f_client_port_odd); destination(d_client_port_odd_json); };
log { source(s_aws_elb); parser { aws_elb2json(); };
filter(f_client_port_even); destination(d_client_port_even); };
regards, L.
On Thu, Oct 15, 2015 at 7:28 PM, Nadine Miller < nadine.miller@defpoint.com> wrote:
I've searched through the archives and spent some time trying to find possible answers on the web, but haven't found a definitive answer.
I'm in a situation where I need to parse syslog streams being forwarded through an AWS ELB. The normal configuration of the ELB resets the source IP to be the ELB's IP address. Logs are coming from multiple AWS VPCs, and we've already discovered duplicate hostnames across different VPCs, which has mingled logs from different hosts into one receiving log file.
The ELB has another mode, referred to as "Proxy Protocol" which adds a single line to the TCP stream in the form:
PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP + single space + PROXY_IP + single space + CLIENT_PORT + single space + PROXY_PORT + "\r\n"
Example:
PROXY TCP4 198.51.100.22 203.0.113.7 35646 80\r\n
Is it possible to use this proxy line in syslog-ng to properly segregate the log messages? If so, what would be the best method to use? I've done a lot of filtering/templating with normal UDP syslog and syslog-ng, but this is the first time I've had to consider something crazy like this.
Currently there is no option at this time to change configurations at endpoints sending the syslog messages, nor can we remove the ELB.
For reference:
http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable...
Thanks in advance-- =N=
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Thanks Tibor and Laszlo, I will explore both of these options. =Nadine= -- Nadine Miller Principal Security Engineer email: nadine.miller@defpoint.com phone: (408) 667-9004 (Eastern) On Oct 16, 2015, at 4:04 AM, Tibor Benke <ihrwein@gmail.com> wrote: Maybe the csv-parser would also work with space as a separator character. 2015-10-16 9:53 GMT+02:00 Budai, László <laszlo.budai@balabit.com>: Hi, a more specific example that I've created (just a POC): @version: 3.7 @include "scl.conf" #PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP + single space + PROXY_IP + single space + CLIENT_PORT + single space + PROXY_PORT + "\r\n" python { import socket import json def aws_elb_proxy_protocol2json(logmsg, arg): out = {} token_separator = ' ' tokens = arg.split(token_separator) if not tokens or len(tokens) < 6: return json.dumps({"aws_elb_proxy_protocol2json.error": "split failure", "message": arg}) out["proxy"] = tokens[0] out["inet_protocol"] = tokens[1] out["client_ip"] = tokens[2] out["proxy_ip"] = tokens[3] out["client_port"] = tokens[4] out["proxy_port"] = tokens[5] return json.dumps(out) }; block parser aws_elb2json ( template("${MSG}") rec-sep(' ') field-sep(' ') ) { json-parser(template("$(python aws_elb_proxy_protocol2json `template`)")); }; source s_aws_elb { file("/tmp/aws-elb.log" flags(no-parse)); }; destination d_client_port_odd_json { file("/tmp/aws_elb_client_port_odd_json.log" template("$(format-json -s nv-pairs)\n")); }; destination d_client_port_even { #file("/tmp/aws_elb_client_port_even.log" template("$(format-json -s nv-pairs)\n")); file("/tmp/aws_elb_client_port_even.log"); }; filter f_client_port_odd { match("\d*[13579]$" value("client_port")); }; filter f_client_port_even { match("\d*[02468]$" value("client_port")); }; log { source(s_aws_elb); parser { aws_elb2json(); }; filter(f_client_port_odd); destination(d_client_port_odd_json); }; log { source(s_aws_elb); parser { aws_elb2json(); }; filter(f_client_port_even); destination(d_client_port_even); }; regards, L. On Thu, Oct 15, 2015 at 7:28 PM, Nadine Miller <nadine.miller@defpoint.com
wrote: I've searched through the archives and spent some time trying to find possible answers on the web, but haven't found a definitive answer.
I'm in a situation where I need to parse syslog streams being forwarded through an AWS ELB. The normal configuration of the ELB resets the source IP to be the ELB's IP address. Logs are coming from multiple AWS VPCs, and we've already discovered duplicate hostnames across different VPCs, which has mingled logs from different hosts into one receiving log file. The ELB has another mode, referred to as "Proxy Protocol" which adds a single line to the TCP stream in the form: PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP + single space + PROXY_IP + single space + CLIENT_PORT + single space + PROXY_PORT + "\r\n" Example: PROXY TCP4 198.51.100.22 203.0.113.7 35646 80\r\n Is it possible to use this proxy line in syslog-ng to properly segregate the log messages? If so, what would be the best method to use? I've done a lot of filtering/templating with normal UDP syslog and syslog-ng, but this is the first time I've had to consider something crazy like this. Currently there is no option at this time to change configurations at endpoints sending the syslog messages, nor can we remove the ELB. For reference: http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable... Thanks in advance-- =N= ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I gave a simpler solution at the end of.the thread. On Oct 20, 2015 16:27, "Nadine Miller" <nadine.miller@defpoint.com> wrote:
Thanks Tibor and Laszlo, I will explore both of these options.
=Nadine=
-- Nadine Miller Principal Security Engineer email: nadine.miller@defpoint.com phone: (408) 667-9004 (Eastern)
On Oct 16, 2015, at 4:04 AM, Tibor Benke <ihrwein@gmail.com> wrote:
Maybe the csv-parser would also work with space as a separator character.
2015-10-16 9:53 GMT+02:00 Budai, László <laszlo.budai@balabit.com>: Hi,
a more specific example that I've created (just a POC):
@version: 3.7 @include "scl.conf"
#PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP + single space + PROXY_IP + single space + CLIENT_PORT + single space + PROXY_PORT + "\r\n" python { import socket import json def aws_elb_proxy_protocol2json(logmsg, arg): out = {} token_separator = ' ' tokens = arg.split(token_separator) if not tokens or len(tokens) < 6: return json.dumps({"aws_elb_proxy_protocol2json.error": "split failure", "message": arg})
out["proxy"] = tokens[0] out["inet_protocol"] = tokens[1] out["client_ip"] = tokens[2] out["proxy_ip"] = tokens[3] out["client_port"] = tokens[4] out["proxy_port"] = tokens[5]
return json.dumps(out) };
block parser aws_elb2json ( template("${MSG}") rec-sep(' ') field-sep(' ') ) { json-parser(template("$(python aws_elb_proxy_protocol2json `template`)")); };
source s_aws_elb { file("/tmp/aws-elb.log" flags(no-parse)); };
destination d_client_port_odd_json { file("/tmp/aws_elb_client_port_odd_json.log" template("$(format-json -s nv-pairs)\n")); };
destination d_client_port_even { #file("/tmp/aws_elb_client_port_even.log" template("$(format-json -s nv-pairs)\n")); file("/tmp/aws_elb_client_port_even.log"); };
filter f_client_port_odd { match("\d*[13579]$" value("client_port")); }; filter f_client_port_even { match("\d*[02468]$" value("client_port")); };
log { source(s_aws_elb); parser { aws_elb2json(); };
filter(f_client_port_odd); destination(d_client_port_odd_json); };
log { source(s_aws_elb); parser { aws_elb2json(); };
filter(f_client_port_even); destination(d_client_port_even); };
regards, L.
On Thu, Oct 15, 2015 at 7:28 PM, Nadine Miller <nadine.miller@defpoint.com
wrote: I've searched through the archives and spent some time trying to find possible answers on the web, but haven't found a definitive answer.
I'm in a situation where I need to parse syslog streams being forwarded through an AWS ELB. The normal configuration of the ELB resets the source IP to be the ELB's IP address. Logs are coming from multiple AWS VPCs, and we've already discovered duplicate hostnames across different VPCs, which has mingled logs from different hosts into one receiving log file.
The ELB has another mode, referred to as "Proxy Protocol" which adds a single line to the TCP stream in the form:
PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP + single space + PROXY_IP + single space + CLIENT_PORT + single space + PROXY_PORT + "\r\n"
Example:
PROXY TCP4 198.51.100.22 203.0.113.7 35646 80\r\n
Is it possible to use this proxy line in syslog-ng to properly segregate the log messages? If so, what would be the best method to use? I've done a lot of filtering/templating with normal UDP syslog and syslog-ng, but this is the first time I've had to consider something crazy like this.
Currently there is no option at this time to change configurations at endpoints sending the syslog messages, nor can we remove the ELB.
For reference:
http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable...
Thanks in advance-- =N=
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, there's a simpler way to parse that, let me come up with a parser for you. -- Bazsi On Thu, Oct 15, 2015 at 7:28 PM, Nadine Miller <nadine.miller@defpoint.com> wrote:
I've searched through the archives and spent some time trying to find possible answers on the web, but haven't found a definitive answer.
I'm in a situation where I need to parse syslog streams being forwarded through an AWS ELB. The normal configuration of the ELB resets the source IP to be the ELB's IP address. Logs are coming from multiple AWS VPCs, and we've already discovered duplicate hostnames across different VPCs, which has mingled logs from different hosts into one receiving log file.
The ELB has another mode, referred to as "Proxy Protocol" which adds a single line to the TCP stream in the form:
PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP + single space + PROXY_IP + single space + CLIENT_PORT + single space + PROXY_PORT + "\r\n"
Example:
PROXY TCP4 198.51.100.22 203.0.113.7 35646 80\r\n
Is it possible to use this proxy line in syslog-ng to properly segregate the log messages? If so, what would be the best method to use? I've done a lot of filtering/templating with normal UDP syslog and syslog-ng, but this is the first time I've had to consider something crazy like this.
Currently there is no option at this time to change configurations at endpoints sending the syslog messages, nor can we remove the ELB.
For reference:
http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable...
Thanks in advance-- =N=
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, Sorry I was distracted, here's a block definition for this format that you can simply use as a parser: block parser aws-proxy-protocol-header() { csv-parser(delimiters(" ") columns(".aws.proxy", ".aws.inet_protocol", ".aws.client_ip", ".aws.proxy_ip", ".aws.client_port", ".aws.proxy_port")); }; Just put it somewhere in your configuration file, or into a separate file and include it. This is how I tested it. @version: 3.7 block parser aws-proxy-protocol-header() { csv-parser(delimiters(" ") columns(".aws.proxy", ".aws.inet_protocol", ".aws.client_ip", ".aws.proxy_ip", ".aws.client_port", ".aws.proxy_port")); }; log { source { file("aws-sample.log" flags(no-parse)); }; parser { aws-proxy-protocol-header(); }; destination { file("/dev/stdout" template("$(format-json .aws.*)\n")); }; }; -- Bazsi On Thu, Oct 15, 2015 at 7:28 PM, Nadine Miller <nadine.miller@defpoint.com> wrote:
I've searched through the archives and spent some time trying to find possible answers on the web, but haven't found a definitive answer.
I'm in a situation where I need to parse syslog streams being forwarded through an AWS ELB. The normal configuration of the ELB resets the source IP to be the ELB's IP address. Logs are coming from multiple AWS VPCs, and we've already discovered duplicate hostnames across different VPCs, which has mingled logs from different hosts into one receiving log file.
The ELB has another mode, referred to as "Proxy Protocol" which adds a single line to the TCP stream in the form:
PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP + single space + PROXY_IP + single space + CLIENT_PORT + single space + PROXY_PORT + "\r\n"
Example:
PROXY TCP4 198.51.100.22 203.0.113.7 35646 80\r\n
Is it possible to use this proxy line in syslog-ng to properly segregate the log messages? If so, what would be the best method to use? I've done a lot of filtering/templating with normal UDP syslog and syslog-ng, but this is the first time I've had to consider something crazy like this.
Currently there is no option at this time to change configurations at endpoints sending the syslog messages, nor can we remove the ELB.
For reference:
http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable...
Thanks in advance-- =N=
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, While thinking about the subject (BTW, could you try the parser I sent?), the scenario you were describing doesn't add up. This line is sent to the backend server, not to a log server, how would you send this format to syslog-ng? Thanks On Oct 15, 2015 19:28, "Nadine Miller" <nadine.miller@defpoint.com> wrote:
I've searched through the archives and spent some time trying to find possible answers on the web, but haven't found a definitive answer.
I'm in a situation where I need to parse syslog streams being forwarded through an AWS ELB. The normal configuration of the ELB resets the source IP to be the ELB's IP address. Logs are coming from multiple AWS VPCs, and we've already discovered duplicate hostnames across different VPCs, which has mingled logs from different hosts into one receiving log file.
The ELB has another mode, referred to as "Proxy Protocol" which adds a single line to the TCP stream in the form:
PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP + single space + PROXY_IP + single space + CLIENT_PORT + single space + PROXY_PORT + "\r\n"
Example:
PROXY TCP4 198.51.100.22 203.0.113.7 35646 80\r\n
Is it possible to use this proxy line in syslog-ng to properly segregate the log messages? If so, what would be the best method to use? I've done a lot of filtering/templating with normal UDP syslog and syslog-ng, but this is the first time I've had to consider something crazy like this.
Currently there is no option at this time to change configurations at endpoints sending the syslog messages, nor can we remove the ELB.
For reference:
http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable...
Thanks in advance-- =N=
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
The backend server itself would be the syslog-ng. ELBs are not just a simple HTTP proxies, they can act as a TCP forwarder/proxy too. It is a common logging pattern in AWS to use ELBs to load balance log streams between multiple log collection servers. To be able to keep the original source IP and port, ELB uses the Proxy Protocol v1 ( http://www.haproxy.org/download/1.5/doc/proxy-protocol.txt) which injects a human readable proxy header at the beginning of the TCP stream. So if the log stream is '\n' separated, the first log message would contain the proxy header, but no other log message would contain any reference on the original source ip. So this problem cannot be solved without keeping the state throughout a connection. Because ELB will open a new connection to the backend on every incoming connection from the clients, syslog-ng will see different connections for different clients. The source IP would be the ELB IP, but the port would be varying. With patterndb, you can parse out the proxy header line, and the store the original IP in a log message field. Then you can write a python rewrite/filter/template statement (don't know which would be the best) which can store the original IP of the connection under the key (source_ip,source_port) in a hash table or a Redis DB. With every log message in a connection, you can look up the original IP of the connection by using the key (source_ip, source_port), and add it to the log message. Regards, Viktor On Sun, Oct 18, 2015 at 11:44 AM, Balazs Scheidler <bazsi77@gmail.com> wrote:
Hi,
While thinking about the subject (BTW, could you try the parser I sent?), the scenario you were describing doesn't add up.
This line is sent to the backend server, not to a log server, how would you send this format to syslog-ng?
Thanks On Oct 15, 2015 19:28, "Nadine Miller" <nadine.miller@defpoint.com> wrote:
I've searched through the archives and spent some time trying to find possible answers on the web, but haven't found a definitive answer.
I'm in a situation where I need to parse syslog streams being forwarded through an AWS ELB. The normal configuration of the ELB resets the source IP to be the ELB's IP address. Logs are coming from multiple AWS VPCs, and we've already discovered duplicate hostnames across different VPCs, which has mingled logs from different hosts into one receiving log file.
The ELB has another mode, referred to as "Proxy Protocol" which adds a single line to the TCP stream in the form:
PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP + single space + PROXY_IP + single space + CLIENT_PORT + single space + PROXY_PORT + "\r\n"
Example:
PROXY TCP4 198.51.100.22 203.0.113.7 35646 80\r\n
Is it possible to use this proxy line in syslog-ng to properly segregate the log messages? If so, what would be the best method to use? I've done a lot of filtering/templating with normal UDP syslog and syslog-ng, but this is the first time I've had to consider something crazy like this.
Currently there is no option at this time to change configurations at endpoints sending the syslog messages, nor can we remove the ELB.
For reference:
http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable...
Thanks in advance-- =N=
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Ah, thanks for the explanation. I might want to create support for this kind of load balancing if someone is willing to test it/give feedback. Nadine? On Oct 18, 2015 2:21 PM, "Tusa Viktor" <tusavik@gmail.com> wrote:
The backend server itself would be the syslog-ng. ELBs are not just a simple HTTP proxies, they can act as a TCP forwarder/proxy too. It is a common logging pattern in AWS to use ELBs to load balance log streams between multiple log collection servers. To be able to keep the original source IP and port, ELB uses the Proxy Protocol v1 ( http://www.haproxy.org/download/1.5/doc/proxy-protocol.txt) which injects a human readable proxy header at the beginning of the TCP stream. So if the log stream is '\n' separated, the first log message would contain the proxy header, but no other log message would contain any reference on the original source ip.
So this problem cannot be solved without keeping the state throughout a connection. Because ELB will open a new connection to the backend on every incoming connection from the clients, syslog-ng will see different connections for different clients. The source IP would be the ELB IP, but the port would be varying.
With patterndb, you can parse out the proxy header line, and the store the original IP in a log message field. Then you can write a python rewrite/filter/template statement (don't know which would be the best) which can store the original IP of the connection under the key (source_ip,source_port) in a hash table or a Redis DB. With every log message in a connection, you can look up the original IP of the connection by using the key (source_ip, source_port), and add it to the log message.
Regards, Viktor
On Sun, Oct 18, 2015 at 11:44 AM, Balazs Scheidler <bazsi77@gmail.com> wrote:
Hi,
While thinking about the subject (BTW, could you try the parser I sent?), the scenario you were describing doesn't add up.
This line is sent to the backend server, not to a log server, how would you send this format to syslog-ng?
Thanks On Oct 15, 2015 19:28, "Nadine Miller" <nadine.miller@defpoint.com> wrote:
I've searched through the archives and spent some time trying to find possible answers on the web, but haven't found a definitive answer.
I'm in a situation where I need to parse syslog streams being forwarded through an AWS ELB. The normal configuration of the ELB resets the source IP to be the ELB's IP address. Logs are coming from multiple AWS VPCs, and we've already discovered duplicate hostnames across different VPCs, which has mingled logs from different hosts into one receiving log file.
The ELB has another mode, referred to as "Proxy Protocol" which adds a single line to the TCP stream in the form:
PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP + single space + PROXY_IP + single space + CLIENT_PORT + single space + PROXY_PORT + "\r\n"
Example:
PROXY TCP4 198.51.100.22 203.0.113.7 35646 80\r\n
Is it possible to use this proxy line in syslog-ng to properly segregate the log messages? If so, what would be the best method to use? I've done a lot of filtering/templating with normal UDP syslog and syslog-ng, but this is the first time I've had to consider something crazy like this.
Currently there is no option at this time to change configurations at endpoints sending the syslog messages, nor can we remove the ELB.
For reference:
http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable...
Thanks in advance-- =N=
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (6)
-
Balazs Scheidler
-
Budai, László
-
Nadine Miller
-
Scheidler, Balázs
-
Tibor Benke
-
Tusa Viktor