[syslog-ng] Problems parsing Cisco syslogs

Fri Mar 1 22:15:47 UTC 2019

Here are 2 different samples of logs without the mnemonic, one is IOS and
the other is NX-OS

IOS, no-parse log to file
--
Mar  1 16:06:07 hostname1 <185>78919861: -Traceback= 151644C 151763C
17F0094 1B8BDE8 1B82858

NX-OS, no-parse log to file
--
Mar  1 16:10:41 hostname1 <189>: 2019 Mar  1 16:10:41 CST: last message
repeated 3 times

Regards,
Max

On Thu, Feb 28, 2019 at 5:45 PM Scheidler, Balázs <
balazs.scheidler at oneidentity.com> wrote:

> Log samples would be appreciated.
>
> On Thu, Feb 28, 2019, 18:11 N. Max Pierson <nmaxpierson at gmail.com wrote:
>
>> Balázs,
>>
>> I modified the plugin.conf XML file (https://pastebin.com/9BruYy0S -
>> added lines 62-65) for the cisco-parser and I was able to capture NX-OS
>> syslogs. I also noticed that the default parser is not matching syslogs
>> that do not include the Cisco mnemonic (both for IOS and NX-OS). While I
>> was able to fix the NX-OS issues (which was a date parsing problem), i'm
>> not sure what needs to be changed/added in this file so that even logs that
>> do not contain the mnemonic are matched. Can anyone shed some light on what
>> part of this parser needs to be changed to resolve the issue? I can post
>> some examples of a log that doesn't contain the mnemonic should you need
>> them.
>>
>> Regards,
>> Max
>>
>> On Tue, Feb 26, 2019 at 11:14 AM Scheidler, Balázs <
>> balazs.scheidler at oneidentity.com> wrote:
>>
>>> The cisco-parser() should take care about messages from cisco routers
>>> assuming they were received using flags(no-parse). It will automatically
>>> detect timestamp formats, but its not perfect, so if you encounter
>>> something that it doesn't properly parse, do let us know, so we can add it.
>>>
>>> Also, default-network-drivers() makes it possible to receive both cisco
>>> and non-cisco logs on the same port, automatically recognizing the
>>> appropriate format. This driver relies on the app-parser() framework, which
>>> can be extended by application specific parsers. With that you can
>>> construct your specific source driver configuration if you find
>>> default-network-drivers () too complicated.
>>>
>>> You can find all of these in the documentation, and their source in the
>>> syslog-ng configuration library (scl for short, usually in
>>> /usr/share/syslog-ng/include/scl).
>>>
>>> On Tue, Feb 26, 2019, 18:08 Sandor Geller <sandor.geller at ericsson.com
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> When the no-parse flag is used then the macros referencing various
>>>> parts of the message aren't filled in. HOST could get looked up using a
>>>> reverse DNS lookup unless the keep_hostname option is set. The syslog
>>>> priority is set to user.notice when parsing is disabled.
>>>>
>>>> Did you take a look at the existing cisco parser? Using it or adopting
>>>> it should ease your job. If you could configure the Ciscos to use other
>>>> port than anything else (which speaks syslog, Cisco devices usually aren't
>>>> such...) would be even better.
>>>>
>>>> Regards,
>>>> Sandor
>>>>
>>>> On 02/26/2019 05:36 PM, N. Max Pierson wrote:
>>>>
>>>> Hi List,
>>>>
>>>> I have been trying to get something in place that can parse syslogs
>>>> from various Cisco devices. The message format is almost the same with a
>>>> few exceptions. Here is what I have tried and it works but now it has
>>>> created another problem I do not know how to troubleshoot.
>>>>
>>>> So that I could see exactly what was being parsed, I disabled the
>>>> default parsing using the below.
>>>>
>>>> source s_network { udp(ip(0.0.0.0) port(514) flags(no-parse)); };
>>>>
>>>> rewrite r_cisco{
>>>> subst('^<\d+>(\d+:|:)\s+(\.\w+|\w+)\s+\d+\s+\d+\s\d+:\d+:\d+:\s|^<\d+>:\s+\d+\s+\w+\s+\d+\s+\d+:\d+:\d+\s\w+:\s|^<\d+>(\d+:|:)\s',
>>>> "", value("MESSAGE"), type("pcre"), flags("ignore-case")); };
>>>>
>>>> destination d_mysql {
>>>>     sql(type(mysql)
>>>>     host("127.0.0.1")
>>>>     username("syslog-ng")
>>>>     password("password")
>>>>     database("syslog")
>>>>     table("messages_${HOST}")
>>>>     columns("datetime datetime", "host varchar(50)", "level
>>>> varchar(10)", "message text")
>>>>     values("${R_YEAR}-${R_MONTH}-${R_DAY} ${R_HOUR}:${R_MIN}:${R_SEC}",
>>>> "${HOST}", "${LEVEL}", "${MESSAGE}")
>>>>     indexes("datetime", "level")
>>>>     );
>>>> };
>>>>
>>>> log { source(s_network); rewrite(r_cisco); destination(d_mysql); };
>>>>
>>>> This works perfectly as it formats the message as I want and covers IOS
>>>> and NX-OS devices. The problem is when I turned off the default parser, now
>>>> all of my logs show "notice" in the $LEVEL macro and doesn't reflect the
>>>> real message header level. The $HOST macro still works fine however.
>>>>
>>>> Is this the expected behavior that the message header fields are not
>>>> parsed as well as the $MESSAGE itself not being parsed? How can map the
>>>> header level field properly to the $LEVEL marco if I disable the default
>>>> parser?
>>>>
>>>> Regards,
>>>> Max
>>>>
>>>>
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20190301/d22a21d3/attachment.html>