[syslog-ng] Problems parsing Cisco syslogs

Thu Feb 28 23:44:53 UTC 2019

Log samples would be appreciated.

On Thu, Feb 28, 2019, 18:11 N. Max Pierson <nmaxpierson at gmail.com wrote:

> Balázs,
>
> I modified the plugin.conf XML file (https://pastebin.com/9BruYy0S -
> added lines 62-65) for the cisco-parser and I was able to capture NX-OS
> syslogs. I also noticed that the default parser is not matching syslogs
> that do not include the Cisco mnemonic (both for IOS and NX-OS). While I
> was able to fix the NX-OS issues (which was a date parsing problem), i'm
> not sure what needs to be changed/added in this file so that even logs that
> do not contain the mnemonic are matched. Can anyone shed some light on what
> part of this parser needs to be changed to resolve the issue? I can post
> some examples of a log that doesn't contain the mnemonic should you need
> them.
>
> Regards,
> Max
>
> On Tue, Feb 26, 2019 at 11:14 AM Scheidler, Balázs <
> balazs.scheidler at oneidentity.com> wrote:
>
>> The cisco-parser() should take care about messages from cisco routers
>> assuming they were received using flags(no-parse). It will automatically
>> detect timestamp formats, but its not perfect, so if you encounter
>> something that it doesn't properly parse, do let us know, so we can add it.
>>
>> Also, default-network-drivers() makes it possible to receive both cisco
>> and non-cisco logs on the same port, automatically recognizing the
>> appropriate format. This driver relies on the app-parser() framework, which
>> can be extended by application specific parsers. With that you can
>> construct your specific source driver configuration if you find
>> default-network-drivers () too complicated.
>>
>> You can find all of these in the documentation, and their source in the
>> syslog-ng configuration library (scl for short, usually in
>> /usr/share/syslog-ng/include/scl).
>>
>> On Tue, Feb 26, 2019, 18:08 Sandor Geller <sandor.geller at ericsson.com
>> wrote:
>>
>>> Hello,
>>>
>>> When the no-parse flag is used then the macros referencing various parts
>>> of the message aren't filled in. HOST could get looked up using a reverse
>>> DNS lookup unless the keep_hostname option is set. The syslog priority is
>>> set to user.notice when parsing is disabled.
>>>
>>> Did you take a look at the existing cisco parser? Using it or adopting
>>> it should ease your job. If you could configure the Ciscos to use other
>>> port than anything else (which speaks syslog, Cisco devices usually aren't
>>> such...) would be even better.
>>>
>>> Regards,
>>> Sandor
>>>
>>> On 02/26/2019 05:36 PM, N. Max Pierson wrote:
>>>
>>> Hi List,
>>>
>>> I have been trying to get something in place that can parse syslogs from
>>> various Cisco devices. The message format is almost the same with a few
>>> exceptions. Here is what I have tried and it works but now it has created
>>> another problem I do not know how to troubleshoot.
>>>
>>> So that I could see exactly what was being parsed, I disabled the
>>> default parsing using the below.
>>>
>>> source s_network { udp(ip(0.0.0.0) port(514) flags(no-parse)); };
>>>
>>> rewrite r_cisco{
>>> subst('^<\d+>(\d+:|:)\s+(\.\w+|\w+)\s+\d+\s+\d+\s\d+:\d+:\d+:\s|^<\d+>:\s+\d+\s+\w+\s+\d+\s+\d+:\d+:\d+\s\w+:\s|^<\d+>(\d+:|:)\s',
>>> "", value("MESSAGE"), type("pcre"), flags("ignore-case")); };
>>>
>>> destination d_mysql {
>>>     sql(type(mysql)
>>>     host("127.0.0.1")
>>>     username("syslog-ng")
>>>     password("password")
>>>     database("syslog")
>>>     table("messages_${HOST}")
>>>     columns("datetime datetime", "host varchar(50)", "level
>>> varchar(10)", "message text")
>>>     values("${R_YEAR}-${R_MONTH}-${R_DAY} ${R_HOUR}:${R_MIN}:${R_SEC}",
>>> "${HOST}", "${LEVEL}", "${MESSAGE}")
>>>     indexes("datetime", "level")
>>>     );
>>> };
>>>
>>> log { source(s_network); rewrite(r_cisco); destination(d_mysql); };
>>>
>>> This works perfectly as it formats the message as I want and covers IOS
>>> and NX-OS devices. The problem is when I turned off the default parser, now
>>> all of my logs show "notice" in the $LEVEL macro and doesn't reflect the
>>> real message header level. The $HOST macro still works fine however.
>>>
>>> Is this the expected behavior that the message header fields are not
>>> parsed as well as the $MESSAGE itself not being parsed? How can map the
>>> header level field properly to the $LEVEL marco if I disable the default
>>> parser?
>>>
>>> Regards,
>>> Max
>>>
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20190301/73843b1d/attachment-0001.html>