Remote server not keeping message intact
Hi all I'm sending Apache logs to a remote syslog-ng server, but the remote server isn't keeping the message intact. Source: My Apache log format: LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined What the log looks like: 172.27.15.149 - - [12/Nov/2015:08:30:59 +0000] "GET / HTTP/1.1" 200 3594 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36" My syslog-ng configuration: source s_apache { file("/var/log/apache2/access.log" follow_freq(1) flags(no-parse)); }; destination d_apache_tcp { tcp("x.x.x.x" port(514)); }; log { source(s_apache); destination(d_apache_tcp); }; Log server: source s_net { udp(port(514)); tcp(port(514)); }; template apache { template("${MESSAGE}\n"); template-escape(no); }; destination apache { file("/var/log/apachetest" template(apache)); }; What I see in /var/log/apachetest: - - [12/Nov/2015:08:30:59 +0000] "GET / HTTP/1.1" 200 3594 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36" As you can see the IP at the beginning of the log entry is being removed. I've tried using $MSG and $MSGONLY. Any ideas would be greatly appreciated. Gareth
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi, just my 2 cents, but have you tried to drop the use of template in the apache destination? Best regards, Ádám -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBCgAGBQJWRGPrAAoJEISh0dl+cj6LSl4H/0nhB/VqmLi3cQDTV6onLwVx cnrfSbghTv5vj2DUArsPMwR0I5UcS7wDQc9KDNHGAUPuiIvGa2XRPmWolB4F5Ro1 y8OVugHD1jYCAwpZ0BtlFtSlpDFkoXzoSGfbqXaVC4xen/qvixiwQJwasIRjmvSV ObfidKaRjZthh9paYqN8RPTRf+LRlOSz7ZMR5fbSXgms/mf+8w/0lIVgmas+Dr/D ydO706z4zJMNNoTNvO+fTuyGDZdYTPqJ+hZAsG1UPtFm4CNNUPftLozMwX8BbVVp dOf5eMlF7uZ3AWmgRB7Q0HVoEXmnJd3Cn4KT+HEPFnYnWuYdtfKXpgEoSjelce4= =oyLn -----END PGP SIGNATURE-----
On Thu, Nov 12, 2015 at 12:03 PM, MÓZES Ádám István <mozes.adam.istvan@gmail.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
Hi,
just my 2 cents, but have you tried to drop the use of template in the apache destination?
Best regards, Ádám -----BEGIN PGP SIGNATURE----- Version: GnuPG v2
iQEcBAEBCgAGBQJWRGPrAAoJEISh0dl+cj6LSl4H/0nhB/VqmLi3cQDTV6onLwVx cnrfSbghTv5vj2DUArsPMwR0I5UcS7wDQc9KDNHGAUPuiIvGa2XRPmWolB4F5Ro1 y8OVugHD1jYCAwpZ0BtlFtSlpDFkoXzoSGfbqXaVC4xen/qvixiwQJwasIRjmvSV ObfidKaRjZthh9paYqN8RPTRf+LRlOSz7ZMR5fbSXgms/mf+8w/0lIVgmas+Dr/D ydO706z4zJMNNoTNvO+fTuyGDZdYTPqJ+hZAsG1UPtFm4CNNUPftLozMwX8BbVVp dOf5eMlF7uZ3AWmgRB7Q0HVoEXmnJd3Cn4KT+HEPFnYnWuYdtfKXpgEoSjelce4= =oyLn -----END PGP SIGNATURE----- ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
The problem is I'd like to strip out the received timestamp and hostname from the beginning of the log and only have the raw Apache message.
On 11/12/2015 11:14 AM, Gareth Allen wrote:
The problem is I'd like to strip out the received timestamp and hostname from the beginning of the log and only have the raw Apache message.
With the no-parse flag you already disabled syslog-ng's parser so you got the raw apache log on the sending side. However to ensure that the receiver gets the message intact a protocol-compliant on-wire format must get used. You seem to use the old RFC3164 format so try template("<30> $R_ISODATE $HOST $MSG\n") of course you can change the priority, timestamp and hostname fields to whatever suits you best. hth, Sandor
You have two options: 1) use a dedicated port on the server for non-syslog traffic, and also use flags(no-parse) there 2) stick to using a standard transport, but then you should ensure that clients send a complete syslog message there. Although syslog-ng (on the client) does prepend a minimal header to your apache log even without a template, it doesn't prepend a program name explicitly, thus your on-wire apache log will look like this: <pri>$DATE $HOST 172.27.15.149 - - [12/Nov/2015:08:30:59 +0000] "GET / HTTP/1.1" 200... This is standard syslog already and if you don't have a template on the server, this would be reproduced completely. Anyway, the format above is processed by syslog-ng on the server side as as an rfc3164 message, so it will parse the IP address (172.27.15.149) as the program name (bloody heuristics), but then you omitted that with your explicit template(), as you only have ${MESSAGE} there. So you have three options: 2.a) use template ("$MSGHDR$MSG\n") without the $DATE/$HOST macros on the server, this would restore the IP address as part of $MSGHDR 2.b) prepend an application name on the client with an explicit template 2.c) use rfc5424 transport with the syslog() driver, that should keep the $MSG intact as it has less heuristics on the server side to parse all formats. I would use 2.b or 2.c myself, or 1) if you have loads of non-syslog traffic. -- Bazsi On Thu, Nov 12, 2015 at 11:26 AM, Sandor Geller <sandor.geller@ericsson.com> wrote:
On 11/12/2015 11:14 AM, Gareth Allen wrote:
The problem is I'd like to strip out the received timestamp and hostname from the beginning of the log and only have the raw Apache message.
With the no-parse flag you already disabled syslog-ng's parser so you got the raw apache log on the sending side. However to ensure that the receiver gets the message intact a protocol-compliant on-wire format must get used. You seem to use the old RFC3164 format so try
template("<30> $R_ISODATE $HOST $MSG\n")
of course you can change the priority, timestamp and hostname fields to whatever suits you best.
hth,
Sandor
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
The problem is I'd like to strip out the received timestamp and hostname from the beginning of the log and only have the raw Apache message.
The syslog format is quite different from the Common Log Format, therefore parsing the CLF might not be the easiest solution. If you can change the log format in the Apache configuration, that would be much easier. Here is what I found: http://httpd.apache.org/docs/2.2/mod/mod_log_config.html#formats Best Regards, Ádám -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBCgAGBQJWRGt3AAoJEISh0dl+cj6LtN0H/j0kNT6YID0GeC/9r0nrcg5L fOb3hbbsvSD6Odxm83WGBsliZ8jX1jOEUvZb1BhcFU6+njLAybGis/Ali8dQNUbe LGaHUluEUx9vKMaHUbIxTYLG8mU69a/WvDgh+3HA4ZKGmq3JjAWcHos0v1IBCPXU /gekjaWQ9LKobuZbJ1lx6n6Y2FUCqlwQlVxDMl4JxEYNRIZIEN5CmhK3rCRI3l4q w74QcVLcTFHv2Y/Vesfe5uB/5ZI9IHmG2AggH7PznzUH6WJkGJm7w7KE7B+tcuA+ WbqK6JQrcLNWNrk4R3FKHo/fCOUkFAS3grDcKscEk7D+3XQACjLtMMhM3jid98M= =Ztqt -----END PGP SIGNATURE-----
I am not 100% sure of this, but I think that the no-parse flag places the line into the message portion and spoofs the all of the prefix elements such as priority, facility, host, and date, but still processes the PROGRAM and the MSG. If you try using a template of templete apache { template("$PROGRAM $MESSAGE\n"); template-escape(no"); }; I think you might have better success. Evan. On 11/12/2015 12:34 AM, Gareth Allen wrote:
Hi all
I'm sending Apache logs to a remote syslog-ng server, but the remote server isn't keeping the message intact.
Source: My Apache log format: LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
What the log looks like: 172.27.15.149 - - [12/Nov/2015:08:30:59 +0000] "GET / HTTP/1.1" 200 3594 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36"
My syslog-ng configuration: source s_apache { file("/var/log/apache2/access.log" follow_freq(1) flags(no-parse)); };
destination d_apache_tcp { tcp("x.x.x.x" port(514)); };
log { source(s_apache); destination(d_apache_tcp); };
Log server: source s_net { udp(port(514)); tcp(port(514)); };
template apache { template("${MESSAGE}\n"); template-escape(no); };
destination apache { file("/var/log/apachetest" template(apache)); };
What I see in /var/log/apachetest: - - [12/Nov/2015:08:30:59 +0000] "GET / HTTP/1.1" 200 3594 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36"
As you can see the IP at the beginning of the log entry is being removed. I've tried using $MSG and $MSGONLY.
Any ideas would be greatly appreciated. Gareth
participants (5)
-
Evan Rempel
-
Gareth Allen
-
MÓZES Ádám István
-
Sandor Geller
-
Scheidler, Balázs