[syslog-ng] Problems parsing Cisco syslogs

Mon Mar 4 15:27:24 UTC 2019

The 2 samples that I sent were recorded with the no-parse option on the
network source and sent to a file destination with no other options
specified. Is this what you need or do you need the raw message itself? If
you need the raw message itself, how do I get that (configuration wise)? I
thought the no parse option gave you the raw message minus the headers.

Regards,
Max

On Sat, Mar 2, 2019 at 2:37 PM Scheidler, Balázs <
balazs.scheidler at oneidentity.com> wrote:

> Can you please include the log sample as it was received by syslog-ng?
> This seems to have been mangled already.
>
> On Fri, Mar 1, 2019, 23:16 N. Max Pierson <nmaxpierson at gmail.com wrote:
>
>> Here are 2 different samples of logs without the mnemonic, one is IOS and
>> the other is NX-OS
>>
>> IOS, no-parse log to file
>> --
>> Mar  1 16:06:07 hostname1 <185>78919861: -Traceback= 151644C 151763C
>> 17F0094 1B8BDE8 1B82858
>>
>>
>> NX-OS, no-parse log to file
>> --
>> Mar  1 16:10:41 hostname1 <189>: 2019 Mar  1 16:10:41 CST: last message
>> repeated 3 times
>>
>> Regards,
>> Max
>>
>> On Thu, Feb 28, 2019 at 5:45 PM Scheidler, Balázs <
>> balazs.scheidler at oneidentity.com> wrote:
>>
>>> Log samples would be appreciated.
>>>
>>> On Thu, Feb 28, 2019, 18:11 N. Max Pierson <nmaxpierson at gmail.com wrote:
>>>
>>>> Balázs,
>>>>
>>>> I modified the plugin.conf XML file (https://pastebin.com/9BruYy0S -
>>>> added lines 62-65) for the cisco-parser and I was able to capture NX-OS
>>>> syslogs. I also noticed that the default parser is not matching syslogs
>>>> that do not include the Cisco mnemonic (both for IOS and NX-OS). While I
>>>> was able to fix the NX-OS issues (which was a date parsing problem), i'm
>>>> not sure what needs to be changed/added in this file so that even logs that
>>>> do not contain the mnemonic are matched. Can anyone shed some light on what
>>>> part of this parser needs to be changed to resolve the issue? I can post
>>>> some examples of a log that doesn't contain the mnemonic should you need
>>>> them.
>>>>
>>>> Regards,
>>>> Max
>>>>
>>>> On Tue, Feb 26, 2019 at 11:14 AM Scheidler, Balázs <
>>>> balazs.scheidler at oneidentity.com> wrote:
>>>>
>>>>> The cisco-parser() should take care about messages from cisco routers
>>>>> assuming they were received using flags(no-parse). It will automatically
>>>>> detect timestamp formats, but its not perfect, so if you encounter
>>>>> something that it doesn't properly parse, do let us know, so we can add it.
>>>>>
>>>>> Also, default-network-drivers() makes it possible to receive both
>>>>> cisco and non-cisco logs on the same port, automatically recognizing the
>>>>> appropriate format. This driver relies on the app-parser() framework, which
>>>>> can be extended by application specific parsers. With that you can
>>>>> construct your specific source driver configuration if you find
>>>>> default-network-drivers () too complicated.
>>>>>
>>>>> You can find all of these in the documentation, and their source in
>>>>> the syslog-ng configuration library (scl for short, usually in
>>>>> /usr/share/syslog-ng/include/scl).
>>>>>
>>>>> On Tue, Feb 26, 2019, 18:08 Sandor Geller <sandor.geller at ericsson.com
>>>>> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> When the no-parse flag is used then the macros referencing various
>>>>>> parts of the message aren't filled in. HOST could get looked up using a
>>>>>> reverse DNS lookup unless the keep_hostname option is set. The syslog
>>>>>> priority is set to user.notice when parsing is disabled.
>>>>>>
>>>>>> Did you take a look at the existing cisco parser? Using it or
>>>>>> adopting it should ease your job. If you could configure the Ciscos to use
>>>>>> other port than anything else (which speaks syslog, Cisco devices usually
>>>>>> aren't such...) would be even better.
>>>>>>
>>>>>> Regards,
>>>>>> Sandor
>>>>>>
>>>>>> On 02/26/2019 05:36 PM, N. Max Pierson wrote:
>>>>>>
>>>>>> Hi List,
>>>>>>
>>>>>> I have been trying to get something in place that can parse syslogs
>>>>>> from various Cisco devices. The message format is almost the same with a
>>>>>> few exceptions. Here is what I have tried and it works but now it has
>>>>>> created another problem I do not know how to troubleshoot.
>>>>>>
>>>>>> So that I could see exactly what was being parsed, I disabled the
>>>>>> default parsing using the below.
>>>>>>
>>>>>> source s_network { udp(ip(0.0.0.0) port(514) flags(no-parse)); };
>>>>>>
>>>>>> rewrite r_cisco{
>>>>>> subst('^<\d+>(\d+:|:)\s+(\.\w+|\w+)\s+\d+\s+\d+\s\d+:\d+:\d+:\s|^<\d+>:\s+\d+\s+\w+\s+\d+\s+\d+:\d+:\d+\s\w+:\s|^<\d+>(\d+:|:)\s',
>>>>>> "", value("MESSAGE"), type("pcre"), flags("ignore-case")); };
>>>>>>
>>>>>> destination d_mysql {
>>>>>>     sql(type(mysql)
>>>>>>     host("127.0.0.1")
>>>>>>     username("syslog-ng")
>>>>>>     password("password")
>>>>>>     database("syslog")
>>>>>>     table("messages_${HOST}")
>>>>>>     columns("datetime datetime", "host varchar(50)", "level
>>>>>> varchar(10)", "message text")
>>>>>>     values("${R_YEAR}-${R_MONTH}-${R_DAY}
>>>>>> ${R_HOUR}:${R_MIN}:${R_SEC}", "${HOST}", "${LEVEL}", "${MESSAGE}")
>>>>>>     indexes("datetime", "level")
>>>>>>     );
>>>>>> };
>>>>>>
>>>>>> log { source(s_network); rewrite(r_cisco); destination(d_mysql); };
>>>>>>
>>>>>> This works perfectly as it formats the message as I want and covers
>>>>>> IOS and NX-OS devices. The problem is when I turned off the default parser,
>>>>>> now all of my logs show "notice" in the $LEVEL macro and doesn't reflect
>>>>>> the real message header level. The $HOST macro still works fine however.
>>>>>>
>>>>>> Is this the expected behavior that the message header fields are not
>>>>>> parsed as well as the $MESSAGE itself not being parsed? How can map the
>>>>>> header level field properly to the $LEVEL marco if I disable the default
>>>>>> parser?
>>>>>>
>>>>>> Regards,
>>>>>> Max
>>>>>>
>>>>>>
>>>>>>
>>>>>> ______________________________________________________________________________
>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>
>>>>>>
>>>>>>
>>>>>> ______________________________________________________________________________
>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>> Documentation:
>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>
>>>>>>
>>>>> ______________________________________________________________________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation:
>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20190304/70a50a33/attachment-0001.html>