[syslog-ng] Problems parsing Cisco syslogs

N. Max Pierson nmaxpierson at gmail.com
Wed Mar 6 20:45:35 UTC 2019 

Balázs,,

I have decided to not worry about anything without the cisco mnemonic and
use the cisco parser until I can come up with a better solution. I made
that modification to the plugin.conf file and now I am able to receive
NX-OS messages but I have discovered an issue that seems to be with the
cisco parser itself. When I enable it, all of my messages are coming in
with a "notice" level so it seems as though the cisco parser isn't properly
parsing the header correctly. I even moved back to the default plugin.conf
file to make sure my modifications didn't break it in the process. Here is
my config if that helps.....

source s_network { udp(ip(0.0.0.0) port(514) flags(no-parse)); };

template t_cisco { template("${R_YEAR}-${R_MONTH}-${R_DAY}
${R_HOUR}:${R_MIN}:${R_SEC} ${HOST} ${LEVEL} ${MESSAGE}\n"); };

destination d_test { file("/var/log/network-test" template(t_cisco)); };

parser p_cisco { cisco-parser(); };

log { source(s_network); parser(p_cisco); destination(d_test); };

Regards,
Max

On Mon, Mar 4, 2019 at 2:00 PM Scheidler, Balázs <
balazs.scheidler at oneidentity.com> wrote:

> It was a bit strange that nxos starts with a colon and that a cisco
> message didn't have a mnemonic. Cisco messages by my experience always
> contain a triplet and this one isn't, and doesn't seem to have a timestamp
> either. If you say that this was recorded the way you mentioned then i
> believe you :), they just looked a bit odd.
>
> Do all messages from the same device look the same? Or these are just the
> exceptions?
>
> Also what is nxos?
>
> On Sat, Mar 2, 2019, 21:37 Scheidler, Balázs <
> balazs.scheidler at oneidentity.com wrote:
>
>> Can you please include the log sample as it was received by syslog-ng?
>> This seems to have been mangled already.
>>
>> On Fri, Mar 1, 2019, 23:16 N. Max Pierson <nmaxpierson at gmail.com wrote:
>>
>>> Here are 2 different samples of logs without the mnemonic, one is IOS
>>> and the other is NX-OS
>>>
>>> IOS, no-parse log to file
>>> --
>>> Mar  1 16:06:07 hostname1 <185>78919861: -Traceback= 151644C 151763C
>>> 17F0094 1B8BDE8 1B82858
>>>
>>>
>>> NX-OS, no-parse log to file
>>> --
>>> Mar  1 16:10:41 hostname1 <189>: 2019 Mar  1 16:10:41 CST: last message
>>> repeated 3 times
>>>
>>> Regards,
>>> Max
>>>
>>> On Thu, Feb 28, 2019 at 5:45 PM Scheidler, Balázs <
>>> balazs.scheidler at oneidentity.com> wrote:
>>>
>>>> Log samples would be appreciated.
>>>>
>>>> On Thu, Feb 28, 2019, 18:11 N. Max Pierson <nmaxpierson at gmail.com
>>>> wrote:
>>>>
>>>>> Balázs,
>>>>>
>>>>> I modified the plugin.conf XML file (https://pastebin.com/9BruYy0S -
>>>>> added lines 62-65) for the cisco-parser and I was able to capture NX-OS
>>>>> syslogs. I also noticed that the default parser is not matching syslogs
>>>>> that do not include the Cisco mnemonic (both for IOS and NX-OS). While I
>>>>> was able to fix the NX-OS issues (which was a date parsing problem), i'm
>>>>> not sure what needs to be changed/added in this file so that even logs that
>>>>> do not contain the mnemonic are matched. Can anyone shed some light on what
>>>>> part of this parser needs to be changed to resolve the issue? I can post
>>>>> some examples of a log that doesn't contain the mnemonic should you need
>>>>> them.
>>>>>
>>>>> Regards,
>>>>> Max
>>>>>
>>>>> On Tue, Feb 26, 2019 at 11:14 AM Scheidler, Balázs <
>>>>> balazs.scheidler at oneidentity.com> wrote:
>>>>>
>>>>>> The cisco-parser() should take care about messages from cisco routers
>>>>>> assuming they were received using flags(no-parse). It will automatically
>>>>>> detect timestamp formats, but its not perfect, so if you encounter
>>>>>> something that it doesn't properly parse, do let us know, so we can add it.
>>>>>>
>>>>>> Also, default-network-drivers() makes it possible to receive both
>>>>>> cisco and non-cisco logs on the same port, automatically recognizing the
>>>>>> appropriate format. This driver relies on the app-parser() framework, which
>>>>>> can be extended by application specific parsers. With that you can
>>>>>> construct your specific source driver configuration if you find
>>>>>> default-network-drivers () too complicated.
>>>>>>
>>>>>> You can find all of these in the documentation, and their source in
>>>>>> the syslog-ng configuration library (scl for short, usually in
>>>>>> /usr/share/syslog-ng/include/scl).
>>>>>>
>>>>>> On Tue, Feb 26, 2019, 18:08 Sandor Geller <sandor.geller at ericsson.com
>>>>>> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> When the no-parse flag is used then the macros referencing various
>>>>>>> parts of the message aren't filled in. HOST could get looked up using a
>>>>>>> reverse DNS lookup unless the keep_hostname option is set. The syslog
>>>>>>> priority is set to user.notice when parsing is disabled.
>>>>>>>
>>>>>>> Did you take a look at the existing cisco parser? Using it or
>>>>>>> adopting it should ease your job. If you could configure the Ciscos to use
>>>>>>> other port than anything else (which speaks syslog, Cisco devices usually
>>>>>>> aren't such...) would be even better.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Sandor
>>>>>>>
>>>>>>> On 02/26/2019 05:36 PM, N. Max Pierson wrote:
>>>>>>>
>>>>>>> Hi List,
>>>>>>>
>>>>>>> I have been trying to get something in place that can parse syslogs
>>>>>>> from various Cisco devices. The message format is almost the same with a
>>>>>>> few exceptions. Here is what I have tried and it works but now it has
>>>>>>> created another problem I do not know how to troubleshoot.
>>>>>>>
>>>>>>> So that I could see exactly what was being parsed, I disabled the
>>>>>>> default parsing using the below.
>>>>>>>
>>>>>>> source s_network { udp(ip(0.0.0.0) port(514) flags(no-parse)); };
>>>>>>>
>>>>>>> rewrite r_cisco{
>>>>>>> subst('^<\d+>(\d+:|:)\s+(\.\w+|\w+)\s+\d+\s+\d+\s\d+:\d+:\d+:\s|^<\d+>:\s+\d+\s+\w+\s+\d+\s+\d+:\d+:\d+\s\w+:\s|^<\d+>(\d+:|:)\s',
>>>>>>> "", value("MESSAGE"), type("pcre"), flags("ignore-case")); };
>>>>>>>
>>>>>>> destination d_mysql {
>>>>>>>     sql(type(mysql)
>>>>>>>     host("127.0.0.1")
>>>>>>>     username("syslog-ng")
>>>>>>>     password("password")
>>>>>>>     database("syslog")
>>>>>>>     table("messages_${HOST}")
>>>>>>>     columns("datetime datetime", "host varchar(50)", "level
>>>>>>> varchar(10)", "message text")
>>>>>>>     values("${R_YEAR}-${R_MONTH}-${R_DAY}
>>>>>>> ${R_HOUR}:${R_MIN}:${R_SEC}", "${HOST}", "${LEVEL}", "${MESSAGE}")
>>>>>>>     indexes("datetime", "level")
>>>>>>>     );
>>>>>>> };
>>>>>>>
>>>>>>> log { source(s_network); rewrite(r_cisco); destination(d_mysql); };
>>>>>>>
>>>>>>> This works perfectly as it formats the message as I want and covers
>>>>>>> IOS and NX-OS devices. The problem is when I turned off the default parser,
>>>>>>> now all of my logs show "notice" in the $LEVEL macro and doesn't reflect
>>>>>>> the real message header level. The $HOST macro still works fine however.
>>>>>>>
>>>>>>> Is this the expected behavior that the message header fields are not
>>>>>>> parsed as well as the $MESSAGE itself not being parsed? How can map the
>>>>>>> header level field properly to the $LEVEL marco if I disable the default
>>>>>>> parser?
>>>>>>>
>>>>>>> Regards,
>>>>>>> Max
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ______________________________________________________________________________
>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ______________________________________________________________________________
>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>> Documentation:
>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>
>>>>>>>
>>>>>> ______________________________________________________________________________
>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>> Documentation:
>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>
>>>>>>
>>>>> ______________________________________________________________________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation:
>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20190306/a4314109/attachment-0001.html>

More information about the syslog-ng mailing list