[syslog-ng] syslog 3.9 modifying ${MSG}?

Wed Apr 26 14:07:58 UTC 2017

Hi,

I think I've found the reason why it is parsed differently:

- we added support for parsing Cisco style sequence numbers (that's 45 in
your example), so that does not get parsed as program name
- however, we don't support timezones in the timestamps above, so GMT is
being parsed as PROGRAM
- adding support is not as simple as it sounds, as it conflicts with other
log formats, as generally the host or program name is the next field, and
those can be "GMT" as well.

This is what I would do:

- receive the messages using no-parse, then your entire message as received
is available in $MSG
- grab the timestamp with a regexp (e.g. message("<regexp>"
flags(store-matches)))
- use date-parser() to parse the timestamp

-- 
Bazsi

On Wed, Apr 26, 2017 at 3:43 PM, Nik Ambrosch <nik at ambrosch.com> wrote:

> These are Cisco devices, I can reproduce with both old and new versions of
> ios.  Here is another message - i logged it twice, once with no-parse and
> one without that flag.
>
> (using no flags):  %SYS-5-CONFIG_I: Configured from console by xxx on vty0
> (0.0.0.0)
> (no-parse flag):   <189>45: Apr 25 21:36:17.452 GMT: %SYS-5-CONFIG_I:
> Configured from console by xxx on vty0 (0.0.0.0)
>
> Regarding the config & version - we moved from 3.3 to 3.9.  The behavior
> occurs when using sql() and program() destinations, i have not tried others
> but I would assume the same since it’s the value of a macro.
>
> For reference, here is my old and new configuration for my database
> destination - note ${DEVICE_TYPE} is a custom macro.
>
> old:
>
> columns("id int(11) unsigned not null auto_increment primary key", "host
> varchar(100) not null", "facility varchar(10)", "priority varchar(10)",
> "level varchar(10)", "program text", "date date not null", "time time not
> null", "message text not null")
> values("", "$FULLHOST", "$FACILITY", "$PRIORITY", "$LEVEL", "$PROGRAM",
> "$R_YEAR-$R_MONTH-$R_DAY", "$R_HOUR:$R_MIN:$R_SEC", "$MSG")
>
>
> new:
>
> columns("id int(11) unsigned not null auto_increment primary key", "host
> varchar(100) not null", "host_resolved varchar(60)", "facility
> varchar(10)", "priority varchar(10)", "level varchar(10)", "program
> varchar(60)", "date_local datetime not null", "date_remote datetime",
> "device_type varchar(13)", "message mediumblob not null")
> values("0", "${FULLHOST}", "${FULLHOST_FROM}", "${FACILITY}",
> "${PRIORITY}", "${LEVEL}", "${PROGRAM}", "${R_YEAR}-${R_MONTH}-${R_DAY}
> ${R_HOUR}:{$R_MIN}:${R_SEC}", "${S_YEAR}-${S_MONTH}-${S_DAY}
> ${S_HOUR}:${S_MIN}:${S_SEC}", "${DEVICE_TYPE}", "${MSG}")
>
> Thanks.
>
>
>
>
> > On Apr 26, 2017, at 3:45 AM, Sandor Geller <sandor.geller at ericsson.com>
> wrote:
> >
> > Hi,
> >
> > The app producing these logs violates all syslog standards (there are so
> many apps written by people ignoring standards...). In my opinion syslog-ng
> is correct in assuming that the first doublecolon ends the syslog header
> and the string containing the doublecolon is the program name. I  don't
> know how could the older version mis-parse the message to pick up a not
> even existing string from the message as the program name.
> >
> > Handling of $MSG AKA $MESSAGE changed with syslog-ng 3.0 and with recent
> configs (versioned ones having at least 3.0 in the version number) it no
> longer contains the syslog header. Were you using an old (unversioned)
> configfile and relied on 2.x behaviour maybe? syslog-ng outputs warnings
> about such behaviour changes when it starts.
> >
> > The documentation also contains this information so it is a good read.
> >
> > Regards,
> >
> > Sandor
> >
> > On 04/25/2017 09:23 PM, Nik Ambrosch wrote:
> >> After moving from syslog-ng 3.5 to 3.9 i noticed that the contents of
> $PROGRAM and $MSG are being logged differently than before.  Here is how
> they used to be logged:
> >>
> >> # program: 53
> >> # message: Apr 19 09:35:35.713 GMT: %ILPOWER-5-POWER_GRANTED: Interface
> xxx: Power granted                                 |
> >>
> >> That is the full message (as seen on the device) which is optimal
> behavior.  Below is the behavior i’m seeing with syslog-ng 3.9 with similar
> configuration:
> >>
> >> # program: GMT
> >> # message: %ILPOWER-5-POWER_GRANTED: Interface xxx: Power granted
> >>
> >> The value of program and message are altered but everything else is the
> same.  I’ve been investigating the date parser and the flags(no-parse)
> options but haven’t had any luck getting a properly formatted message yet.
> >>
> >> If anyone has any ideas on how to get the old behavior back it would be
> greatly appreciated.
> >>
> >> ____________________________________________________________
> __________________
> >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> >> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >>
> >
> >
> > ____________________________________________________________
> __________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170426/369bf5ac/attachment.html>