[syslog-ng] syslog 3.9 modifying ${MSG}?
Scheidler, Balázs
balazs.scheidler at balabit.com
Thu Apr 27 04:48:39 UTC 2017
this is the patch, it was first released in 3.4
commit 046f90e76137f31e61412d24b9920914ad75edc5
Author: Balazs Scheidler <bazsi at balabit.hu>
Date: Sun May 27 14:24:52 2012 +0200
I prefer not to touch the syslog parsing code, exactly because it is so
fragile. I was thinking about doing a cisco-parser() that would expect a
flags(no-parse) received message and split it in appropriate parts.
hm... This is what I came up with:
# <189>45: Apr 25 21:36:17.452 GMT: %SYS-5-CONFIG_I: Configured from
console by xxx on vty0 (0.0.0.0)
block parser cisco-parser() {
channel {
parser {
# split msg and header right before the '%', Cisco messages may
# have a variable number of ': ' terminated values
csv-parser(delimiters(chars('') strings(': %'))
columns('1', '2') flags(greedy));
csv-parser(delimiters(chars(':')) template("$2") columns('3'));
csv-parser(delimiters(chars('-')) template("$3")
columns('.cisco.facility', '.cisco.severity',
'.cisco.mnemonic'));
};
rewrite {
set('%$2', value("MSG"));
subst("^<[0-9]+>", "", value('1'));
subst("^[0-9]+: ", "", value('1'));
};
parser {
csv-parser(delimiters(chars('.')) columns('1', '2')
template('$1'));
date-parser(format('%b %d %H:%M:%S') template("$1"));
};
rewrite {
unset(value("1"));
unset(value("2"));
unset(value("3"));
};
};
};
This is not perfect yet, for instance date-parser() does not support
fraction of a second parsing. But first we should be trying to fix the
format. Can you check if this indeed parses your message correctly? All you
need to do is receive messages using flags(no-parse), identify cisco
messages somehow (a filter based on IP?), and then apply this parser on the
cisco part. The rest of the messages could be parsed explicitly by
syslog-parser(), that transforms a "raw" message into what it would be like
if it was not received using flags(no-parse).
I'd appreciate feedback, especially with different cisco devices. I haven't
found samples for when device-ids are enabled in logs, which seems to be
ASA specific.
--
Bazsi
On Wed, Apr 26, 2017 at 4:13 PM, Nik Ambrosch <nik at ambrosch.com> wrote:
> Do you recall the version this was added in? I’m guessing 3.8?
>
> There are non-cisco messages that arrive on this interface as well,
> hopefully there are no other conflicts. What do you think the best
> permanent solution is? Our centralized syslog server(s) accept messages
> from many different device types and i’m wondering if this will happen
> again in the future.
>
>
>
> On Apr 26, 2017, at 10:07 AM, Scheidler, Balázs <
> balazs.scheidler at balabit.com> wrote:
>
> Hi,
>
> I think I've found the reason why it is parsed differently:
>
> - we added support for parsing Cisco style sequence numbers (that's 45 in
> your example), so that does not get parsed as program name
> - however, we don't support timezones in the timestamps above, so GMT is
> being parsed as PROGRAM
> - adding support is not as simple as it sounds, as it conflicts with other
> log formats, as generally the host or program name is the next field, and
> those can be "GMT" as well.
>
> This is what I would do:
>
> - receive the messages using no-parse, then your entire message as
> received is available in $MSG
> - grab the timestamp with a regexp (e.g. message("<regexp>"
> flags(store-matches)))
> - use date-parser() to parse the timestamp
>
>
> --
> Bazsi
>
> On Wed, Apr 26, 2017 at 3:43 PM, Nik Ambrosch <nik at ambrosch.com> wrote:
>
>> These are Cisco devices, I can reproduce with both old and new versions
>> of ios. Here is another message - i logged it twice, once with no-parse
>> and one without that flag.
>>
>> (using no flags): %SYS-5-CONFIG_I: Configured from console by xxx on
>> vty0 (0.0.0.0)
>> (no-parse flag): <189>45: Apr 25 21:36:17.452 GMT: %SYS-5-CONFIG_I:
>> Configured from console by xxx on vty0 (0.0.0.0)
>>
>> Regarding the config & version - we moved from 3.3 to 3.9. The behavior
>> occurs when using sql() and program() destinations, i have not tried others
>> but I would assume the same since it’s the value of a macro.
>>
>> For reference, here is my old and new configuration for my database
>> destination - note ${DEVICE_TYPE} is a custom macro.
>>
>> old:
>>
>> columns("id int(11) unsigned not null auto_increment primary key", "host
>> varchar(100) not null", "facility varchar(10)", "priority varchar(10)",
>> "level varchar(10)", "program text", "date date not null", "time time not
>> null", "message text not null")
>> values("", "$FULLHOST", "$FACILITY", "$PRIORITY", "$LEVEL", "$PROGRAM",
>> "$R_YEAR-$R_MONTH-$R_DAY", "$R_HOUR:$R_MIN:$R_SEC", "$MSG")
>>
>>
>> new:
>>
>> columns("id int(11) unsigned not null auto_increment primary key", "host
>> varchar(100) not null", "host_resolved varchar(60)", "facility
>> varchar(10)", "priority varchar(10)", "level varchar(10)", "program
>> varchar(60)", "date_local datetime not null", "date_remote datetime",
>> "device_type varchar(13)", "message mediumblob not null")
>> values("0", "${FULLHOST}", "${FULLHOST_FROM}", "${FACILITY}",
>> "${PRIORITY}", "${LEVEL}", "${PROGRAM}", "${R_YEAR}-${R_MONTH}-${R_DAY}
>> ${R_HOUR}:{$R_MIN}:${R_SEC}", "${S_YEAR}-${S_MONTH}-${S_DAY}
>> ${S_HOUR}:${S_MIN}:${S_SEC}", "${DEVICE_TYPE}", "${MSG}")
>>
>> Thanks.
>>
>>
>>
>>
>> > On Apr 26, 2017, at 3:45 AM, Sandor Geller <sandor.geller at ericsson.com>
>> wrote:
>> >
>> > Hi,
>> >
>> > The app producing these logs violates all syslog standards (there are
>> so many apps written by people ignoring standards...). In my opinion
>> syslog-ng is correct in assuming that the first doublecolon ends the syslog
>> header and the string containing the doublecolon is the program name. I
>> don't know how could the older version mis-parse the message to pick up a
>> not even existing string from the message as the program name.
>> >
>> > Handling of $MSG AKA $MESSAGE changed with syslog-ng 3.0 and with
>> recent configs (versioned ones having at least 3.0 in the version number)
>> it no longer contains the syslog header. Were you using an old
>> (unversioned) configfile and relied on 2.x behaviour maybe? syslog-ng
>> outputs warnings about such behaviour changes when it starts.
>> >
>> > The documentation also contains this information so it is a good read.
>> >
>> > Regards,
>> >
>> > Sandor
>> >
>> > On 04/25/2017 09:23 PM, Nik Ambrosch wrote:
>> >> After moving from syslog-ng 3.5 to 3.9 i noticed that the contents of
>> $PROGRAM and $MSG are being logged differently than before. Here is how
>> they used to be logged:
>> >>
>> >> # program: 53
>> >> # message: Apr 19 09:35:35.713 GMT: %ILPOWER-5-POWER_GRANTED:
>> Interface xxx: Power granted |
>> >>
>> >> That is the full message (as seen on the device) which is optimal
>> behavior. Below is the behavior i’m seeing with syslog-ng 3.9 with similar
>> configuration:
>> >>
>> >> # program: GMT
>> >> # message: %ILPOWER-5-POWER_GRANTED: Interface xxx: Power granted
>> >>
>> >> The value of program and message are altered but everything else is
>> the same. I’ve been investigating the date parser and the flags(no-parse)
>> options but haven’t had any luck getting a properly formatted message yet.
>> >>
>> >> If anyone has any ideas on how to get the old behavior back it would
>> be greatly appreciated.
>> >>
>> >> ____________________________________________________________
>> __________________
>> >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> >> Documentation: http://www.balabit.com/support/documentation/?product=
>> syslog-ng
>> >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>> >>
>> >
>> >
>> > ____________________________________________________________
>> __________________
>> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> > Documentation: http://www.balabit.com/support/documentation/?product=
>> syslog-ng
>> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>> >
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=
>> syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170427/f9070a8f/attachment.html>
More information about the syslog-ng
mailing list