[syslog-ng] Problems parsing Cisco syslogs

[N. Max Pierson]

Balázs,

I modified the plugin.conf XML file (https://pastebin.com/9BruYy0S - added
lines 62-65) for the cisco-parser and I was able to capture NX-OS syslogs.
I also noticed that the default parser is not matching syslogs that do not
include the Cisco mnemonic (both for IOS and NX-OS). While I was able to
fix the NX-OS issues (which was a date parsing problem), i'm not sure what
needs to be changed/added in this file so that even logs that do not
contain the mnemonic are matched. Can anyone shed some light on what part
of this parser needs to be changed to resolve the issue? I can post some
examples of a log that doesn't contain the mnemonic should you need them.

Regards,
Max

On Tue, Feb 26, 2019 at 11:14 AM Scheidler, Balázs <
balazs.scheidler at oneidentity.com> wrote:

> The cisco-parser() should take care about messages from cisco routers
> assuming they were received using flags(no-parse). It will automatically
> detect timestamp formats, but its not perfect, so if you encounter
> something that it doesn't properly parse, do let us know, so we can add it.
>
> Also, default-network-drivers() makes it possible to receive both cisco
> and non-cisco logs on the same port, automatically recognizing the
> appropriate format. This driver relies on the app-parser() framework, which
> can be extended by application specific parsers. With that you can
> construct your specific source driver configuration if you find
> default-network-drivers () too complicated.
>
> You can find all of these in the documentation, and their source in the
> syslog-ng configuration library (scl for short, usually in
> /usr/share/syslog-ng/include/scl).
>
> On Tue, Feb 26, 2019, 18:08 Sandor Geller <sandor.geller at ericsson.com
> wrote:
>
>> Hello,
>>
>> When the no-parse flag is used then the macros referencing various parts
>> of the message aren't filled in. HOST could get looked up using a reverse
>> DNS lookup unless the keep_hostname option is set. The syslog priority is
>> set to user.notice when parsing is disabled.
>>
>> Did you take a look at the existing cisco parser? Using it or adopting it
>> should ease your job. If you could configure the Ciscos to use other port
>> than anything else (which speaks syslog, Cisco devices usually aren't
>> such...) would be even better.
>>
>> Regards,
>> Sandor
>>
>> On 02/26/2019 05:36 PM, N. Max Pierson wrote:
>>
>> Hi List,
>>
>> I have been trying to get something in place that can parse syslogs from
>> various Cisco devices. The message format is almost the same with a few
>> exceptions. Here is what I have tried and it works but now it has created
>> another problem I do not know how to troubleshoot.
>>
>> So that I could see exactly what was being parsed, I disabled the default
>> parsing using the below.
>>
>> source s_network { udp(ip(0.0.0.0) port(514) flags(no-parse)); };
>>
>> rewrite r_cisco{
>> subst('^<\d+>(\d+:|:)\s+(\.\w+|\w+)\s+\d+\s+\d+\s\d+:\d+:\d+:\s|^<\d+>:\s+\d+\s+\w+\s+\d+\s+\d+:\d+:\d+\s\w+:\s|^<\d+>(\d+:|:)\s',
>> "", value("MESSAGE"), type("pcre"), flags("ignore-case")); };
>>
>> destination d_mysql {
>>     sql(type(mysql)
>>     host("127.0.0.1")
>>     username("syslog-ng")
>>     password("password")
>>     database("syslog")
>>     table("messages_${HOST}")
>>     columns("datetime datetime", "host varchar(50)", "level varchar(10)",
>> "message text")
>>     values("${R_YEAR}-${R_MONTH}-${R_DAY} ${R_HOUR}:${R_MIN}:${R_SEC}",
>> "${HOST}", "${LEVEL}", "${MESSAGE}")
>>     indexes("datetime", "level")
>>     );
>> };
>>
>> log { source(s_network); rewrite(r_cisco); destination(d_mysql); };
>>
>> This works perfectly as it formats the message as I want and covers IOS
>> and NX-OS devices. The problem is when I turned off the default parser, now
>> all of my logs show "notice" in the $LEVEL macro and doesn't reflect the
>> real message header level. The $HOST macro still works fine however.
>>
>> Is this the expected behavior that the message header fields are not
>> parsed as well as the $MESSAGE itself not being parsed? How can map the
>> header level field properly to the $LEVEL marco if I disable the default
>> parser?
>>
>> Regards,
>> Max
>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20190228/62eed9ac/attachment.html>