[syslog-ng] Using syslog-ng as a relay inject unexpected data

Thu Dec 20 22:20:59 CET 2007

Sounds like it may be a bug... I'd try first (cuz it's easier) building 2.0.6 
and checking to see if you get the same behaviour (I'm not :).  And if it 
solves the issue but you really want to run 1.x code, turn on debug and start 
tweaking through the source to find the spot it's prepending that to the 
outgoing packet (before the template is even used), or gdb step through it.

/eli

Allen Bettilyon wrote:
> Thanks for the replies.
> 
> To address a few of the questions:
> 
> 1) the receiving end is a splunk instance
> 2) I have verified the existence of the <number> with tcpdump, so its 
> not the receiving end injecting the value.
> 3) The logs been written locally by syslog-ng do NOT have the number 
> injected
> 4) The template didn't seem to fix the problem
> 5) This also happens when using the program() destination
> 
> Bellow are some details regarding the 2 tests I've ran.  The numbers do 
> change but not very quickly.  I haven't been able to tell if they 
> increment or decrement or are just random.
> 
> Quite perplexing.  I think my next steps will be to recreate this issue 
> on a totally separate node and installation of syslog-ng.
> 
> 
> -Allen
> 
> 
> 
> 
> ----- details regarding the upd forwarder-------------
> Bellow is the destination clause in its entirety with addresses changed 
> to protect the innocent.
> I've tried it with and without the NGTOKEN literal just to prove to 
> myself that the number was not part of any of the macros.
> 
> destination forwardHost {
>         tcp("1.1.1.1 <http://1.1.1.1>" port(1) template("NGTOKEN 
> $ISODATE $FACILITY $LEVEL $MSG\n"));
> };
> 
> Just to sanity check this again, I setup a filter to match local1 
> traffic and forward it while doing a packet capture from the syslog host 
> using tcpdump in ASCII mode:
> 
> 13:48:16.736077 IP syslogngHost.47468 > 1.1.1.1.1: P 
> 3847271716:3847271778(62) ack 4053481885 win 5840 <nop,nop,timestamp 
> 11894280 1181945548>
> E..r4+ at .@..)
> .
> .
> .       ).l'..P.$..9.....C......
> ..~.Fs..*<142>NGTOKEN 2007-12-20T13:48:16-0700 local1 info allen: test*
> 
> 13:48:16.736572 IP nocbuild01.overstock.com.distinct32 > 
> syslog01.se.overstock.com.47468: . ack 62 win 5792 <nop,nop,timestamp 
> 1181966237 11894280>
> E..4X{@.8...
> .       )
> .
> .'..l..9..P.b....l......
> 
> 
> 
> ------------- details regarding the program() forwarder -----------------
> 
> 
> my program consists of:
> #!/usr/bin/perl
> while(<STDIN>)
> {
>    $line = $_;
>    open(F,">>/tmp/loggerOutput") or die "no open: $!";
>    print F $line . "\n";
>    close(F);
> }
> 
> Running some quick logger tests
> 
> <142>Dec 20 13:59:38 alshost allen: test
> 
> <142>Dec 20 13:59:40 alshost allen: test
> 
> <142>Dec 20 13:59:40 alshost allen: test2
> 
> <142>Dec 20 13:59:42 alshost allen: test3
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> On Dec 20, 2007 11:16 AM, Eli Stair <estair at ilm.com 
> <mailto:estair at ilm.com>> wrote:
> 
> 
>     Hey Allen,
> 
>     I'd say that if you /are/ seeing '38' (or anything over 23) as a number
>     pre-pended, it's not the facility which was my first guess.  Could
>     be reporting
>     PID or other internal identifier of the sender, which some devices I
>     see seem
>     to use.  Just speculation.. Does the number change, if so how?
> 
>     To verify that's actually being /sent/ by the syslog-ng forwarder,
>     check the
>     output when logging to a local file as well as the remote forward
>     using the
>     same src:template, and see if it shows up in both, as well look at
>     the packets
>     as they hit the wire and see if it's in the payload.  If it IS being
>     sent by
>     your relay, also verify that it isn't actually in the payload sent
>     by your log
>     client.  Can you post the template/src/dest stanzas if you find it
>     IS being
>     generated by the syslog-ng relay?
> 
>     There's obvious likelihood that it's not syslog-ng on the sending
>     host in
>     question, but at the receiving end or originating sender adding this.
> 
>     /eli
> 
> 
> 
>     Allen Bettilyon wrote:
>      > Hello,
>      >
>      > I'm doing some pretty basic syslog forwarding using syslog-ng 1.6.2.
>      >
>      > Essentially, I've got the following:
>      >
>      > destination remoteHost {
>      >       tcp("1.1.1.1 <http://1.1.1.1> <http://1.1.1.1> port 9999");
>      > };
>      >
>      >
>      > The forwarding is working correctly, however on the remote side
>     all my
>      > log lines are prepended with a <number> tag.
>      >
>      > For example:   Some log line
>      > turns into:  <38>Some log line
>      >
>      > I've tried creating a custom template, but the <number> is always
>     added
>      > to the log lines when the arrive at the remote host.
>      >
>      > Why is this happening and is there a way to turn it off?
>      >
>      > Thanks,
>      >
>      > - Allen Bettilyon
>      >
>      >
>      >
>      >
>      >
>      >
> 
>     _______________________________________________
>     syslog-ng maillist  -   syslog-ng at lists.balabit.hu
>     <mailto:syslog-ng at lists.balabit.hu>
>     https://lists.balabit.hu/mailman/listinfo/syslog-ng
>     Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
> 
> 