[syslog-ng] Problems parsing Cisco syslogs

Sat Mar 2 20:37:06 UTC 2019

Can you please include the log sample as it was received by syslog-ng? This
seems to have been mangled already.

On Fri, Mar 1, 2019, 23:16 N. Max Pierson <nmaxpierson at gmail.com wrote:

> Here are 2 different samples of logs without the mnemonic, one is IOS and
> the other is NX-OS
>
> IOS, no-parse log to file
> --
> Mar  1 16:06:07 hostname1 <185>78919861: -Traceback= 151644C 151763C
> 17F0094 1B8BDE8 1B82858
>
>
> NX-OS, no-parse log to file
> --
> Mar  1 16:10:41 hostname1 <189>: 2019 Mar  1 16:10:41 CST: last message
> repeated 3 times
>
> Regards,
> Max
>
> On Thu, Feb 28, 2019 at 5:45 PM Scheidler, Balázs <
> balazs.scheidler at oneidentity.com> wrote:
>
>> Log samples would be appreciated.
>>
>> On Thu, Feb 28, 2019, 18:11 N. Max Pierson <nmaxpierson at gmail.com wrote:
>>
>>> Balázs,
>>>
>>> I modified the plugin.conf XML file (https://pastebin.com/9BruYy0S -
>>> added lines 62-65) for the cisco-parser and I was able to capture NX-OS
>>> syslogs. I also noticed that the default parser is not matching syslogs
>>> that do not include the Cisco mnemonic (both for IOS and NX-OS). While I
>>> was able to fix the NX-OS issues (which was a date parsing problem), i'm
>>> not sure what needs to be changed/added in this file so that even logs that
>>> do not contain the mnemonic are matched. Can anyone shed some light on what
>>> part of this parser needs to be changed to resolve the issue? I can post
>>> some examples of a log that doesn't contain the mnemonic should you need
>>> them.
>>>
>>> Regards,
>>> Max
>>>
>>> On Tue, Feb 26, 2019 at 11:14 AM Scheidler, Balázs <
>>> balazs.scheidler at oneidentity.com> wrote:
>>>
>>>> The cisco-parser() should take care about messages from cisco routers
>>>> assuming they were received using flags(no-parse). It will automatically
>>>> detect timestamp formats, but its not perfect, so if you encounter
>>>> something that it doesn't properly parse, do let us know, so we can add it.
>>>>
>>>> Also, default-network-drivers() makes it possible to receive both cisco
>>>> and non-cisco logs on the same port, automatically recognizing the
>>>> appropriate format. This driver relies on the app-parser() framework, which
>>>> can be extended by application specific parsers. With that you can
>>>> construct your specific source driver configuration if you find
>>>> default-network-drivers () too complicated.
>>>>
>>>> You can find all of these in the documentation, and their source in the
>>>> syslog-ng configuration library (scl for short, usually in
>>>> /usr/share/syslog-ng/include/scl).
>>>>
>>>> On Tue, Feb 26, 2019, 18:08 Sandor Geller <sandor.geller at ericsson.com
>>>> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> When the no-parse flag is used then the macros referencing various
>>>>> parts of the message aren't filled in. HOST could get looked up using a
>>>>> reverse DNS lookup unless the keep_hostname option is set. The syslog
>>>>> priority is set to user.notice when parsing is disabled.
>>>>>
>>>>> Did you take a look at the existing cisco parser? Using it or adopting
>>>>> it should ease your job. If you could configure the Ciscos to use other
>>>>> port than anything else (which speaks syslog, Cisco devices usually aren't
>>>>> such...) would be even better.
>>>>>
>>>>> Regards,
>>>>> Sandor
>>>>>
>>>>> On 02/26/2019 05:36 PM, N. Max Pierson wrote:
>>>>>
>>>>> Hi List,
>>>>>
>>>>> I have been trying to get something in place that can parse syslogs
>>>>> from various Cisco devices. The message format is almost the same with a
>>>>> few exceptions. Here is what I have tried and it works but now it has
>>>>> created another problem I do not know how to troubleshoot.
>>>>>
>>>>> So that I could see exactly what was being parsed, I disabled the
>>>>> default parsing using the below.
>>>>>
>>>>> source s_network { udp(ip(0.0.0.0) port(514) flags(no-parse)); };
>>>>>
>>>>> rewrite r_cisco{
>>>>> subst('^<\d+>(\d+:|:)\s+(\.\w+|\w+)\s+\d+\s+\d+\s\d+:\d+:\d+:\s|^<\d+>:\s+\d+\s+\w+\s+\d+\s+\d+:\d+:\d+\s\w+:\s|^<\d+>(\d+:|:)\s',
>>>>> "", value("MESSAGE"), type("pcre"), flags("ignore-case")); };
>>>>>
>>>>> destination d_mysql {
>>>>>     sql(type(mysql)
>>>>>     host("127.0.0.1")
>>>>>     username("syslog-ng")
>>>>>     password("password")
>>>>>     database("syslog")
>>>>>     table("messages_${HOST}")
>>>>>     columns("datetime datetime", "host varchar(50)", "level
>>>>> varchar(10)", "message text")
>>>>>     values("${R_YEAR}-${R_MONTH}-${R_DAY}
>>>>> ${R_HOUR}:${R_MIN}:${R_SEC}", "${HOST}", "${LEVEL}", "${MESSAGE}")
>>>>>     indexes("datetime", "level")
>>>>>     );
>>>>> };
>>>>>
>>>>> log { source(s_network); rewrite(r_cisco); destination(d_mysql); };
>>>>>
>>>>> This works perfectly as it formats the message as I want and covers
>>>>> IOS and NX-OS devices. The problem is when I turned off the default parser,
>>>>> now all of my logs show "notice" in the $LEVEL macro and doesn't reflect
>>>>> the real message header level. The $HOST macro still works fine however.
>>>>>
>>>>> Is this the expected behavior that the message header fields are not
>>>>> parsed as well as the $MESSAGE itself not being parsed? How can map the
>>>>> header level field properly to the $LEVEL marco if I disable the default
>>>>> parser?
>>>>>
>>>>> Regards,
>>>>> Max
>>>>>
>>>>>
>>>>>
>>>>> ______________________________________________________________________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>>>>
>>>>> ______________________________________________________________________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation:
>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20190302/90fa771d/attachment.html>