Using syslog-ng as a relay inject unexpected data
Hello, I'm doing some pretty basic syslog forwarding using syslog-ng 1.6.2. Essentially, I've got the following: destination remoteHost { tcp("1.1.1.1 port 9999"); }; The forwarding is working correctly, however on the remote side all my log lines are prepended with a <number> tag. For example: Some log line turns into: <38>Some log line I've tried creating a custom template, but the <number> is always added to the log lines when the arrive at the remote host. Why is this happening and is there a way to turn it off? Thanks, - Allen Bettilyon
On Wed, 2007-12-19 at 17:41 -0700, Allen Bettilyon wrote:
Hello,
I'm doing some pretty basic syslog forwarding using syslog-ng 1.6.2.
Essentially, I've got the following:
destination remoteHost { tcp("1.1.1.1 port 9999"); };
The forwarding is working correctly, however on the remote side all my log lines are prepended with a <number> tag.
For example: Some log line turns into: <38>Some log line
I've tried creating a custom template, but the <number> is always added to the log lines when the arrive at the remote host.
Why is this happening and is there a way to turn it off?
you are seeing the priority value, but you can indeed turn it off using a template: destination d_dst { tcp("1.1.1.1" template("$DATE $HOST $MSG\n"); }; but as your configuration snippet is obviously not directly copy/pasted from the configuration I cannot tell for sure why your template didn't work. -- Bazsi
Hey Allen, I'd say that if you /are/ seeing '38' (or anything over 23) as a number pre-pended, it's not the facility which was my first guess. Could be reporting PID or other internal identifier of the sender, which some devices I see seem to use. Just speculation.. Does the number change, if so how? To verify that's actually being /sent/ by the syslog-ng forwarder, check the output when logging to a local file as well as the remote forward using the same src:template, and see if it shows up in both, as well look at the packets as they hit the wire and see if it's in the payload. If it IS being sent by your relay, also verify that it isn't actually in the payload sent by your log client. Can you post the template/src/dest stanzas if you find it IS being generated by the syslog-ng relay? There's obvious likelihood that it's not syslog-ng on the sending host in question, but at the receiving end or originating sender adding this. /eli Allen Bettilyon wrote:
Hello,
I'm doing some pretty basic syslog forwarding using syslog-ng 1.6.2.
Essentially, I've got the following:
destination remoteHost { tcp("1.1.1.1 <http://1.1.1.1> port 9999"); };
The forwarding is working correctly, however on the remote side all my log lines are prepended with a <number> tag.
For example: Some log line turns into: <38>Some log line
I've tried creating a custom template, but the <number> is always added to the log lines when the arrive at the remote host.
Why is this happening and is there a way to turn it off?
Thanks,
- Allen Bettilyon
Thanks for the replies. To address a few of the questions: 1) the receiving end is a splunk instance 2) I have verified the existence of the <number> with tcpdump, so its not the receiving end injecting the value. 3) The logs been written locally by syslog-ng do NOT have the number injected 4) The template didn't seem to fix the problem 5) This also happens when using the program() destination Bellow are some details regarding the 2 tests I've ran. The numbers do change but not very quickly. I haven't been able to tell if they increment or decrement or are just random. Quite perplexing. I think my next steps will be to recreate this issue on a totally separate node and installation of syslog-ng. -Allen ----- details regarding the upd forwarder------------- Bellow is the destination clause in its entirety with addresses changed to protect the innocent. I've tried it with and without the NGTOKEN literal just to prove to myself that the number was not part of any of the macros. destination forwardHost { tcp("1.1.1.1" port(1) template("NGTOKEN $ISODATE $FACILITY $LEVEL $MSG\n")); }; Just to sanity check this again, I setup a filter to match local1 traffic and forward it while doing a packet capture from the syslog host using tcpdump in ASCII mode: 13:48:16.736077 IP syslogngHost.47468 > 1.1.1.1.1: P 3847271716:3847271778(62) ack 4053481885 win 5840 <nop,nop,timestamp 11894280 1181945548> E..r4+@.@..) . . . ).l'..P.$..9.....C...... ..~.Fs..*<142>NGTOKEN 2007-12-20T13:48:16-0700 local1 info allen: test* 13:48:16.736572 IP nocbuild01.overstock.com.distinct32 > syslog01.se.overstock.com.47468: . ack 62 win 5792 <nop,nop,timestamp 1181966237 11894280> E..4X{@.8... . ) . .'..l..9..P.b....l...... ------------- details regarding the program() forwarder ----------------- my program consists of: #!/usr/bin/perl while(<STDIN>) { $line = $_; open(F,">>/tmp/loggerOutput") or die "no open: $!"; print F $line . "\n"; close(F); } Running some quick logger tests <142>Dec 20 13:59:38 alshost allen: test <142>Dec 20 13:59:40 alshost allen: test <142>Dec 20 13:59:40 alshost allen: test2 <142>Dec 20 13:59:42 alshost allen: test3 On Dec 20, 2007 11:16 AM, Eli Stair <estair@ilm.com> wrote:
Hey Allen,
I'd say that if you /are/ seeing '38' (or anything over 23) as a number pre-pended, it's not the facility which was my first guess. Could be reporting PID or other internal identifier of the sender, which some devices I see seem to use. Just speculation.. Does the number change, if so how?
To verify that's actually being /sent/ by the syslog-ng forwarder, check the output when logging to a local file as well as the remote forward using the same src:template, and see if it shows up in both, as well look at the packets as they hit the wire and see if it's in the payload. If it IS being sent by your relay, also verify that it isn't actually in the payload sent by your log client. Can you post the template/src/dest stanzas if you find it IS being generated by the syslog-ng relay?
There's obvious likelihood that it's not syslog-ng on the sending host in question, but at the receiving end or originating sender adding this.
/eli
Allen Bettilyon wrote:
Hello,
I'm doing some pretty basic syslog forwarding using syslog-ng 1.6.2.
Essentially, I've got the following:
destination remoteHost { tcp("1.1.1.1 <http://1.1.1.1> port 9999"); };
The forwarding is working correctly, however on the remote side all my log lines are prepended with a <number> tag.
For example: Some log line turns into: <38>Some log line
I've tried creating a custom template, but the <number> is always added to the log lines when the arrive at the remote host.
Why is this happening and is there a way to turn it off?
Thanks,
- Allen Bettilyon
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
Sounds like it may be a bug... I'd try first (cuz it's easier) building 2.0.6 and checking to see if you get the same behaviour (I'm not :). And if it solves the issue but you really want to run 1.x code, turn on debug and start tweaking through the source to find the spot it's prepending that to the outgoing packet (before the template is even used), or gdb step through it. /eli Allen Bettilyon wrote:
Thanks for the replies.
To address a few of the questions:
1) the receiving end is a splunk instance 2) I have verified the existence of the <number> with tcpdump, so its not the receiving end injecting the value. 3) The logs been written locally by syslog-ng do NOT have the number injected 4) The template didn't seem to fix the problem 5) This also happens when using the program() destination
Bellow are some details regarding the 2 tests I've ran. The numbers do change but not very quickly. I haven't been able to tell if they increment or decrement or are just random.
Quite perplexing. I think my next steps will be to recreate this issue on a totally separate node and installation of syslog-ng.
-Allen
----- details regarding the upd forwarder------------- Bellow is the destination clause in its entirety with addresses changed to protect the innocent. I've tried it with and without the NGTOKEN literal just to prove to myself that the number was not part of any of the macros.
destination forwardHost { tcp("1.1.1.1 <http://1.1.1.1>" port(1) template("NGTOKEN $ISODATE $FACILITY $LEVEL $MSG\n")); };
Just to sanity check this again, I setup a filter to match local1 traffic and forward it while doing a packet capture from the syslog host using tcpdump in ASCII mode:
13:48:16.736077 IP syslogngHost.47468 > 1.1.1.1.1: P 3847271716:3847271778(62) ack 4053481885 win 5840 <nop,nop,timestamp 11894280 1181945548> E..r4+@.@..) . . . ).l'..P.$..9.....C...... ..~.Fs..*<142>NGTOKEN 2007-12-20T13:48:16-0700 local1 info allen: test*
13:48:16.736572 IP nocbuild01.overstock.com.distinct32 > syslog01.se.overstock.com.47468: . ack 62 win 5792 <nop,nop,timestamp 1181966237 11894280> E..4X{@.8... . ) . .'..l..9..P.b....l......
------------- details regarding the program() forwarder -----------------
my program consists of: #!/usr/bin/perl while(<STDIN>) { $line = $_; open(F,">>/tmp/loggerOutput") or die "no open: $!"; print F $line . "\n"; close(F); }
Running some quick logger tests
<142>Dec 20 13:59:38 alshost allen: test
<142>Dec 20 13:59:40 alshost allen: test
<142>Dec 20 13:59:40 alshost allen: test2
<142>Dec 20 13:59:42 alshost allen: test3
On Dec 20, 2007 11:16 AM, Eli Stair <estair@ilm.com <mailto:estair@ilm.com>> wrote:
Hey Allen,
I'd say that if you /are/ seeing '38' (or anything over 23) as a number pre-pended, it's not the facility which was my first guess. Could be reporting PID or other internal identifier of the sender, which some devices I see seem to use. Just speculation.. Does the number change, if so how?
To verify that's actually being /sent/ by the syslog-ng forwarder, check the output when logging to a local file as well as the remote forward using the same src:template, and see if it shows up in both, as well look at the packets as they hit the wire and see if it's in the payload. If it IS being sent by your relay, also verify that it isn't actually in the payload sent by your log client. Can you post the template/src/dest stanzas if you find it IS being generated by the syslog-ng relay?
There's obvious likelihood that it's not syslog-ng on the sending host in question, but at the receiving end or originating sender adding this.
/eli
Allen Bettilyon wrote: > Hello, > > I'm doing some pretty basic syslog forwarding using syslog-ng 1.6.2. > > Essentially, I've got the following: > > destination remoteHost { > tcp("1.1.1.1 <http://1.1.1.1> <http://1.1.1.1> port 9999"); > }; > > > The forwarding is working correctly, however on the remote side all my > log lines are prepended with a <number> tag. > > For example: Some log line > turns into: <38>Some log line > > I've tried creating a custom template, but the <number> is always added > to the log lines when the arrive at the remote host. > > Why is this happening and is there a way to turn it off? > > Thanks, > > - Allen Bettilyon > > > > > >
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu <mailto:syslog-ng@lists.balabit.hu> https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
On Thu, 2007-12-20 at 14:05 -0700, Allen Bettilyon wrote:
Thanks for the replies.
To address a few of the questions:
1) the receiving end is a splunk instance 2) I have verified the existence of the <number> with tcpdump, so its not the receiving end injecting the value. 3) The logs been written locally by syslog-ng do NOT have the number injected 4) The template didn't seem to fix the problem 5) This also happens when using the program() destination
Bellow are some details regarding the 2 tests I've ran. The numbers do change but not very quickly. I haven't been able to tell if they increment or decrement or are just random.
Quite perplexing. I think my next steps will be to recreate this issue on a totally separate node and installation of syslog-ng.
-Allen
----- details regarding the upd forwarder------------- Bellow is the destination clause in its entirety with addresses changed to protect the innocent. I've tried it with and without the NGTOKEN literal just to prove to myself that the number was not part of any of the macros.
destination forwardHost { tcp("1.1.1.1" port(1) template("NGTOKEN $ISODATE $FACILITY $LEVEL $MSG\n")); };
Just to sanity check this again, I setup a filter to match local1 traffic and forward it while doing a packet capture from the syslog host using tcpdump in ASCII mode:
13:48:16.736077 IP syslogngHost.47468 > 1.1.1.1.1: P 3847271716:3847271778(62) ack 4053481885 win 5840 <nop,nop,timestamp 11894280 1181945548> E..r4+@.@..) .
Here's a snippet of the NEWS file of syslog-ng 1.6.x: News for the 1.6.3 release Thu, 06 May 2004 11:05:46 +0200 ... * fixed afunix and afinet destination template handling, do not include the PRI value automatically, let the administrator do it explicitly in its template I think you are bitten by this problem which was fixed in 1.6.3, as you wrote, you are using 1.6.2, but if you need to update anyway, I'd recomment to update directly to 2.0.6, the 1.6.x branch is not maintained anymore. -- Bazsi
participants (3)
-
Allen Bettilyon
-
Balazs Scheidler
-
Eli Stair