Re: [syslog-ng] Syslog-NG 3.0.5 incorrectly parsing messages without program names
In the past I used the MSG macro to get the full syslog message. Now MSG is broken into MSG and MSGHDR.
If a message comes in without a program name such as "exiting on signal 15" or "last message repeated 20 times", syslog-ng incorrectly takes the first word as the program name. Trying to rebuild it with $MSGHDR$MSG causes an erroneous : to be placed on these messages since $MSGHDR is filled in with "exiting" and "last".
I found a previous post and tried adding store-legacy-msghdr. The problem now is that MSGHDR still includes the first word such as "exiting" and "last" but no colon but on messages like "kernel : Kernel logging (proc) stopped.", "kernel: " is put in MSGHDR but then also placed in MSG and MSGONLY.
--- Larry Lo
In the past I used the MSG macro to get the full syslog message. Now MSG is broken into MSG and MSGHDR.
If a message comes in without a program name such as "exiting on signal 15" or "last message repeated 20 times", syslog-ng incorrectly takes the first word as the program name. Trying to rebuild it with $MSGHDR$MSG causes an erroneous : to be placed on these messages since $MSGHDR is filled in with "exiting" and "last".
I found a previous post and tried adding store-legacy-msghdr. The problem now is that MSGHDR still includes the first word such as "exiting" and "last" but no colon but on messages like "kernel : Kernel logging (proc) stopped.", "kernel: " is put in MSGHDR but then also placed in MSG and MSGONLY.
Noticed one more thing, if the PID is included in the syslog message then MSG and MSGONLY do not repeat process and pid when store-legacy-msghdr on.
--- Larry Lo
On Fri, 2009-12-11 at 12:12 -0800, Larry Low wrote:
In the past I used the MSG macro to get the full syslog message. Now MSG is broken into MSG and MSGHDR.
If a message comes in without a program name such as "exiting on signal 15" or "last message repeated 20 times", syslog-ng incorrectly takes the first word as the program name. Trying to rebuild it with $MSGHDR$MSG causes an erroneous : to be placed on these messages since $MSGHDR is filled in with "exiting" and "last".
I found a previous post and tried adding store-legacy-msghdr. The problem now is that MSGHDR still includes the first word such as "exiting" and "last" but no colon but on messages like "kernel : Kernel logging (proc) stopped.", "kernel: " is put in MSGHDR but then also placed in MSG and MSGONLY.
Noticed one more thing, if the PID is included in the syslog message then MSG and MSGONLY do not repeat process and pid when store-legacy-msghdr on.
if you enable 'store-legacy-msghdr' syslog-ng will not try hard to properly parse program/pid information from the incoming message, rather it simply stores all the characters up to the first space/colon character in MSGHDR. But still $MSG will _not_ contain any of that. In order to get the full message you need to use $MSGHDR$MSG in your template (which of course the default contains). This whole stuff was needed to be able to convert old-style (RFC3164) syslog to new style (RFC5424) in which case both the program name and the PID become separate fields in the syslog header. -- Bazsi
On Fri, 2009-12-11 at 12:12 -0800, Larry Low wrote:
In the past I used the MSG macro to get the full syslog message. Now MSG is broken into MSG and MSGHDR.
If a message comes in without a program name such as "exiting on signal 15" or "last message repeated 20 times", syslog-ng incorrectly takes the first word as the program name. Trying to rebuild it with $MSGHDR$MSG causes an erroneous : to be placed on these messages since $MSGHDR is filled in with "exiting" and "last".
I found a previous post and tried adding store-legacy-msghdr. The problem now is that MSGHDR still includes the first word such as "exiting" and "last" but no colon but on messages like "kernel : Kernel logging (proc) stopped.", "kernel: " is put in MSGHDR but then also placed in MSG and MSGONLY.
Noticed one more thing, if the PID is included in the syslog message then MSG and MSGONLY do not repeat process and pid when store-legacy- msghdr on.
if you enable 'store-legacy-msghdr' syslog-ng will not try hard to properly parse program/pid information from the incoming message, rather it simply stores all the characters up to the first space/colon character in MSGHDR.
But still $MSG will _not_ contain any of that. In order to get the full message you need to use
$MSGHDR$MSG
Yes but does not work when parsing "program: ". When this happens both MSGHDR and MSG include "program: ". Both "program " and "program[pid]: ", however, do get placed in MSGHDR and not repeated in MSG.
in your template (which of course the default contains).
This whole stuff was needed to be able to convert old-style (RFC3164) syslog to new style (RFC5424) in which case both the program name and the PID become separate fields in the syslog header.
-- Bazsi
On Mon, 2009-12-14 at 08:21 -0800, Larry Low wrote:
On Fri, 2009-12-11 at 12:12 -0800, Larry Low wrote:
In the past I used the MSG macro to get the full syslog message. Now MSG is broken into MSG and MSGHDR.
If a message comes in without a program name such as "exiting on signal 15" or "last message repeated 20 times", syslog-ng incorrectly takes the first word as the program name. Trying to rebuild it with $MSGHDR$MSG causes an erroneous : to be placed on these messages since $MSGHDR is filled in with "exiting" and "last".
I found a previous post and tried adding store-legacy-msghdr. The problem now is that MSGHDR still includes the first word such as "exiting" and "last" but no colon but on messages like "kernel : Kernel logging (proc) stopped.", "kernel: " is put in MSGHDR but then also placed in MSG and MSGONLY.
Noticed one more thing, if the PID is included in the syslog message then MSG and MSGONLY do not repeat process and pid when store-legacy- msghdr on.
if you enable 'store-legacy-msghdr' syslog-ng will not try hard to properly parse program/pid information from the incoming message, rather it simply stores all the characters up to the first space/colon character in MSGHDR.
But still $MSG will _not_ contain any of that. In order to get the full message you need to use
$MSGHDR$MSG
Yes but does not work when parsing "program: ". When this happens both MSGHDR and MSG include "program: ". Both "program " and "program[pid]: ", however, do get placed in MSGHDR and not repeated in MSG.
I really try to understand, but I can't. Can you please post an example log message, how you feel it should be processed and why syslog-ng doesn't do the right thing? Thanks. -- Bazsi
On Mon, 2009-12-14 at 08:21 -0800, Larry Low wrote:
On Fri, 2009-12-11 at 12:12 -0800, Larry Low wrote:
In the past I used the MSG macro to get the full syslog message. Now MSG is broken into MSG and MSGHDR.
If a message comes in without a program name such as "exiting on signal 15" or "last message repeated 20 times", syslog-ng incorrectly takes the first word as the program name. Trying to rebuild it with $MSGHDR$MSG causes an erroneous : to be placed on these messages since $MSGHDR is filled in with "exiting" and "last".
I found a previous post and tried adding store-legacy-msghdr. The problem now is that MSGHDR still includes the first word such as "exiting" and "last" but no colon but on messages like "kernel : Kernel logging (proc) stopped.", "kernel: " is put in MSGHDR but then also placed in MSG and MSGONLY.
Noticed one more thing, if the PID is included in the syslog message then MSG and MSGONLY do not repeat process and pid when store- legacy- msghdr on.
if you enable 'store-legacy-msghdr' syslog-ng will not try hard to properly parse program/pid information from the incoming message, rather it simply stores all the characters up to the first space/colon character in MSGHDR.
But still $MSG will _not_ contain any of that. In order to get the full message you need to use
$MSGHDR$MSG
Yes but does not work when parsing "program: ". When this happens both MSGHDR and MSG include "program: ". Both "program " and "program[pid]: ", however, do get placed in MSGHDR and not repeated in MSG.
I really try to understand, but I can't. Can you please post an example log message, how you feel it should be processed and why syslog-ng doesn't do the right thing?
Thanks.
With 'store-legacy-msghdr' on. Message = "program: I am logging something." $MSGHDR = "program: " $MSG = "program: I am logging something." $MSGHDR$MSG = "program: program: I am logging something."
-- Bazsi
_______________________________________________________________________ _______ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Tue, 2009-12-15 at 08:06 -0800, Larry Low wrote:
On Mon, 2009-12-14 at 08:21 -0800, Larry Low wrote:
On Fri, 2009-12-11 at 12:12 -0800, Larry Low wrote:
> In the past I used the MSG macro to get the full syslog message. Now > MSG is broken into MSG and MSGHDR. > > If a message comes in without a program name such as "exiting on signal > 15" or "last message repeated 20 times", syslog-ng incorrectly takes > the first word as the program name. Trying to rebuild it with > $MSGHDR$MSG causes an erroneous : to be placed on these messages since > $MSGHDR is filled in with "exiting" and "last".
I found a previous post and tried adding store-legacy-msghdr. The problem now is that MSGHDR still includes the first word such as "exiting" and "last" but no colon but on messages like "kernel : Kernel logging (proc) stopped.", "kernel: " is put in MSGHDR but then also placed in MSG and MSGONLY.
Noticed one more thing, if the PID is included in the syslog message then MSG and MSGONLY do not repeat process and pid when store- legacy- msghdr on.
if you enable 'store-legacy-msghdr' syslog-ng will not try hard to properly parse program/pid information from the incoming message, rather it simply stores all the characters up to the first space/colon character in MSGHDR.
But still $MSG will _not_ contain any of that. In order to get the full message you need to use
$MSGHDR$MSG
Yes but does not work when parsing "program: ". When this happens both MSGHDR and MSG include "program: ". Both "program " and "program[pid]: ", however, do get placed in MSGHDR and not repeated in MSG.
I really try to understand, but I can't. Can you please post an example log message, how you feel it should be processed and why syslog-ng doesn't do the right thing?
Thanks.
With 'store-legacy-msghdr' on.
Message = "program: I am logging something." $MSGHDR = "program: " $MSG = "program: I am logging something." $MSGHDR$MSG = "program: program: I am logging something."
I can't reproduce with either the flag set, or unset. What I did was the following: I've sent "program: I am logging something." to the UDP port opened by syslog-ng (without a full syslog header, but you didn't include that information either), the result was: MSGHDR = [program: ], MSG = [I am logging something.] MSGHDR = [program: ], MSG = [I am logging something.] Then I tried to add a complete syslog header, but it again worked: Dec 15 15:55:55 bzorp program: I am logging something. So, I'm out of ideas what the difference might be. Ah... one idea: did you mark your configuration as '@version: 3.0' properly? -- Bazsi
On Tue, 2009-12-15 at 08:06 -0800, Larry Low wrote:
On Mon, 2009-12-14 at 08:21 -0800, Larry Low wrote:
On Fri, 2009-12-11 at 12:12 -0800, Larry Low wrote:
> > In the past I used the MSG macro to get the full syslog message. Now > > MSG is broken into MSG and MSGHDR. > > > > If a message comes in without a program name such as "exiting on > signal > > 15" or "last message repeated 20 times", syslog-ng incorrectly takes > > the first word as the program name. Trying to rebuild it with > > $MSGHDR$MSG causes an erroneous : to be placed on these messages > since > > $MSGHDR is filled in with "exiting" and "last". > > I found a previous post and tried adding store-legacy- msghdr. The > problem now is that MSGHDR still includes the first word such as > "exiting" and "last" but no colon but on messages like "kernel : Kernel > logging (proc) stopped.", "kernel: " is put in MSGHDR but then also > placed in MSG and MSGONLY.
Noticed one more thing, if the PID is included in the syslog message then MSG and MSGONLY do not repeat process and pid when store- legacy- msghdr on.
if you enable 'store-legacy-msghdr' syslog-ng will not try hard to properly parse program/pid information from the incoming message, rather it simply stores all the characters up to the first space/colon character in MSGHDR.
But still $MSG will _not_ contain any of that. In order to get the full message you need to use
$MSGHDR$MSG
Yes but does not work when parsing "program: ". When this happens both MSGHDR and MSG include "program: ". Both "program " and "program[pid]: ", however, do get placed in MSGHDR and not repeated in MSG.
I really try to understand, but I can't. Can you please post an example log message, how you feel it should be processed and why syslog-ng doesn't do the right thing?
Thanks.
With 'store-legacy-msghdr' on.
Message = "program: I am logging something." $MSGHDR = "program: " $MSG = "program: I am logging something." $MSGHDR$MSG = "program: program: I am logging something."
I can't reproduce with either the flag set, or unset. What I did was the following:
I've sent "program: I am logging something." to the UDP port opened by syslog-ng (without a full syslog header, but you didn't include that information either), the result was:
The exact payload is "<6>kernel: Kernel logging (proc) stopped."
MSGHDR = [program: ], MSG = [I am logging something.] MSGHDR = [program: ], MSG = [I am logging something.]
Then I tried to add a complete syslog header, but it again worked:
Dec 15 15:55:55 bzorp program: I am logging something.
So, I'm out of ideas what the difference might be.
Ah... one idea: did you mark your configuration as '@version: 3.0' properly?
Yes. For now I am sticking with the 2.1.4 but will come back to this later when I have some time.
-- Bazsi
participants (2)
-
Balazs Scheidler
-
Larry Low