[syslog-ng] Problems parsing Cisco syslogs

Péter, Kókai peter.kokai at oneidentity.com
Fri Mar 8 07:49:47 UTC 2019 

Hello,

There are a few ways you could achieve that, but neither of them is built
directly to handle *severity*; in a sense that they are general text
handling options.

Option1: (the slowest one) use *if* - *elif* - *else* chain match/set (You
can see a few example in scl)
Option2: (I would recommend this option) add-contextual-data , this
provides a way to enrich the logs based on a csv
Option3: (This is really risky) I've created a PR (
https://github.com/balabit/syslog-ng/pull/2602) "fixing" the level value of
level macro with *cisco-parser*, with this patch it should set the
*.cisco.severity* so you could use the old macros

Please not that for the last option you have to compile syslog-ng yourself,
and there is no guarantee that it is going to be upstreamed at all.

Note: there are possible other ways with template as well.

--
Kokan

On Thu, Mar 7, 2019 at 8:22 PM N. Max Pierson <nmaxpierson at gmail.com> wrote:

> Hi Kokan,
>
> I didn't catch that previously, so thanks for pointing that out for me. I
> made the changes to reflect the ${.cisco.severity} macro but now I am not
> getting anything in my database table. It seems to work if I log it to a
> file, but for some reason when I log to mysql, it doesn't put anything in
> the column I had for level. My config is below. Also, is there any way to
> show the text version of the severity instead of the number?
>
> source s_network { udp(ip(0.0.0.0) port(514) flags(no-parse)); };
>
> template t_cisco { template("${R_YEAR}-${R_MONTH}-${R_DAY}
> ${R_HOUR}:${R_MIN}:${R_SEC} ${HOST} ${.cisco.severity} ${MESSAGE}\n"); };
>
> destination d_test { file("/var/log/network-test" template(t_cisco)); };
>
> destination d_mysql {
>     sql(type(mysql)
>     host("127.0.0.1")
>     username("syslog-ng")
>     password("password")
>     database("syslog")
>     table("messages_${HOST}")
>     columns("datetime datetime", "host varchar(50)", "level varchar(10)",
> "message text")
>     values("${R_YEAR}-${R_MONTH}-${R_DAY} ${R_HOUR}:${R_MIN}:${R_SEC}",
> "${HOST}", "${.cisco.severity}", "${MESSAGE}")
>     indexes("datetime", "level")
>     );
> };
>
> log { source(s_network); parser(p_cisco); destination(d_test); };
>
> log { source(s_network); parser(p_cisco); destination(d_mysql); };
>
> Regards,
> Max
>
>
> On Thu, Mar 7, 2019 at 1:50 AM Péter, Kókai <peter.kokai at oneidentity.com>
> wrote:
>
>> Hello,
>>
>> The *cisco-parser* as of now cannot set the LEVEL macro, as that is a
>> *hard* macro, those cannot be set directly from configuration setters (for
>> example: rewrite { set("lvl" value("LEVEL"); };).
>> The header is parsed into the ".prefix.*" macros, the default *prefix* is
>> ".cisco.".
>>
>> I think with shifting *parser*s implemented as syslog-ng configuration
>> rather then C, it might be a good idea to lift the soft vs hard macro via a
>> proper setters.
>>
>> Note1: if you are using *flags(no-parse)* it is going to set a default
>> level, changing at least that default level is possible per source
>> "network( default_level(...) flags(no-parse) ...);"
>>
>> Note2: if you really want to set LEVEL macro, well you can do it but not
>> by design, and it might break at any point. It is possible to shadow hard
>> macros setting patterns, instead of LEVEL you could set LEVLE[L] pattern,
>> well which is kinda only matches LEVEL anyway. The way it works it creates
>> a LEVEL soft macro. IMPORTANT: this leaves the message in a semi-state, as
>> some code is going to use ${LEVEL} and some code is going to use the
>> internal level representation which still stays the same old level.
>>
>> --
>> Kokan
>>
>> On Wed, Mar 6, 2019 at 9:45 PM N. Max Pierson <nmaxpierson at gmail.com>
>> wrote:
>>
>>> Balázs,,
>>>
>>> I have decided to not worry about anything without the cisco mnemonic
>>> and use the cisco parser until I can come up with a better solution. I made
>>> that modification to the plugin.conf file and now I am able to receive
>>> NX-OS messages but I have discovered an issue that seems to be with the
>>> cisco parser itself. When I enable it, all of my messages are coming in
>>> with a "notice" level so it seems as though the cisco parser isn't properly
>>> parsing the header correctly. I even moved back to the default plugin.conf
>>> file to make sure my modifications didn't break it in the process. Here is
>>> my config if that helps.....
>>>
>>> source s_network { udp(ip(0.0.0.0) port(514) flags(no-parse)); };
>>>
>>> template t_cisco { template("${R_YEAR}-${R_MONTH}-${R_DAY}
>>> ${R_HOUR}:${R_MIN}:${R_SEC} ${HOST} ${LEVEL} ${MESSAGE}\n"); };
>>>
>>> destination d_test { file("/var/log/network-test" template(t_cisco)); };
>>>
>>> parser p_cisco { cisco-parser(); };
>>>
>>> log { source(s_network); parser(p_cisco); destination(d_test); };
>>>
>>> Regards,
>>> Max
>>>
>>> On Mon, Mar 4, 2019 at 2:00 PM Scheidler, Balázs <
>>> balazs.scheidler at oneidentity.com> wrote:
>>>
>>>> It was a bit strange that nxos starts with a colon and that a cisco
>>>> message didn't have a mnemonic. Cisco messages by my experience always
>>>> contain a triplet and this one isn't, and doesn't seem to have a timestamp
>>>> either. If you say that this was recorded the way you mentioned then i
>>>> believe you :), they just looked a bit odd.
>>>>
>>>> Do all messages from the same device look the same? Or these are just
>>>> the exceptions?
>>>>
>>>> Also what is nxos?
>>>>
>>>> On Sat, Mar 2, 2019, 21:37 Scheidler, Balázs <
>>>> balazs.scheidler at oneidentity.com wrote:
>>>>
>>>>> Can you please include the log sample as it was received by syslog-ng?
>>>>> This seems to have been mangled already.
>>>>>
>>>>> On Fri, Mar 1, 2019, 23:16 N. Max Pierson <nmaxpierson at gmail.com
>>>>> wrote:
>>>>>
>>>>>> Here are 2 different samples of logs without the mnemonic, one is IOS
>>>>>> and the other is NX-OS
>>>>>>
>>>>>> IOS, no-parse log to file
>>>>>> --
>>>>>> Mar  1 16:06:07 hostname1 <185>78919861: -Traceback= 151644C 151763C
>>>>>> 17F0094 1B8BDE8 1B82858
>>>>>>
>>>>>>
>>>>>> NX-OS, no-parse log to file
>>>>>> --
>>>>>> Mar  1 16:10:41 hostname1 <189>: 2019 Mar  1 16:10:41 CST: last
>>>>>> message repeated 3 times
>>>>>>
>>>>>> Regards,
>>>>>> Max
>>>>>>
>>>>>> On Thu, Feb 28, 2019 at 5:45 PM Scheidler, Balázs <
>>>>>> balazs.scheidler at oneidentity.com> wrote:
>>>>>>
>>>>>>> Log samples would be appreciated.
>>>>>>>
>>>>>>> On Thu, Feb 28, 2019, 18:11 N. Max Pierson <nmaxpierson at gmail.com
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Balázs,
>>>>>>>>
>>>>>>>> I modified the plugin.conf XML file (https://pastebin.com/9BruYy0S
>>>>>>>> - added lines 62-65) for the cisco-parser and I was able to capture NX-OS
>>>>>>>> syslogs. I also noticed that the default parser is not matching syslogs
>>>>>>>> that do not include the Cisco mnemonic (both for IOS and NX-OS). While I
>>>>>>>> was able to fix the NX-OS issues (which was a date parsing problem), i'm
>>>>>>>> not sure what needs to be changed/added in this file so that even logs that
>>>>>>>> do not contain the mnemonic are matched. Can anyone shed some light on what
>>>>>>>> part of this parser needs to be changed to resolve the issue? I can post
>>>>>>>> some examples of a log that doesn't contain the mnemonic should you need
>>>>>>>> them.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Max
>>>>>>>>
>>>>>>>> On Tue, Feb 26, 2019 at 11:14 AM Scheidler, Balázs <
>>>>>>>> balazs.scheidler at oneidentity.com> wrote:
>>>>>>>>
>>>>>>>>> The cisco-parser() should take care about messages from cisco
>>>>>>>>> routers assuming they were received using flags(no-parse). It will
>>>>>>>>> automatically detect timestamp formats, but its not perfect, so if you
>>>>>>>>> encounter something that it doesn't properly parse, do let us know, so we
>>>>>>>>> can add it.
>>>>>>>>>
>>>>>>>>> Also, default-network-drivers() makes it possible to receive both
>>>>>>>>> cisco and non-cisco logs on the same port, automatically recognizing the
>>>>>>>>> appropriate format. This driver relies on the app-parser() framework, which
>>>>>>>>> can be extended by application specific parsers. With that you can
>>>>>>>>> construct your specific source driver configuration if you find
>>>>>>>>> default-network-drivers () too complicated.
>>>>>>>>>
>>>>>>>>> You can find all of these in the documentation, and their source
>>>>>>>>> in the syslog-ng configuration library (scl for short, usually in
>>>>>>>>> /usr/share/syslog-ng/include/scl).
>>>>>>>>>
>>>>>>>>> On Tue, Feb 26, 2019, 18:08 Sandor Geller <
>>>>>>>>> sandor.geller at ericsson.com wrote:
>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> When the no-parse flag is used then the macros referencing
>>>>>>>>>> various parts of the message aren't filled in. HOST could get looked up
>>>>>>>>>> using a reverse DNS lookup unless the keep_hostname option is set. The
>>>>>>>>>> syslog priority is set to user.notice when parsing is disabled.
>>>>>>>>>>
>>>>>>>>>> Did you take a look at the existing cisco parser? Using it or
>>>>>>>>>> adopting it should ease your job. If you could configure the Ciscos to use
>>>>>>>>>> other port than anything else (which speaks syslog, Cisco devices usually
>>>>>>>>>> aren't such...) would be even better.
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Sandor
>>>>>>>>>>
>>>>>>>>>> On 02/26/2019 05:36 PM, N. Max Pierson wrote:
>>>>>>>>>>
>>>>>>>>>> Hi List,
>>>>>>>>>>
>>>>>>>>>> I have been trying to get something in place that can parse
>>>>>>>>>> syslogs from various Cisco devices. The message format is almost the same
>>>>>>>>>> with a few exceptions. Here is what I have tried and it works but now it
>>>>>>>>>> has created another problem I do not know how to troubleshoot.
>>>>>>>>>>
>>>>>>>>>> So that I could see exactly what was being parsed, I disabled the
>>>>>>>>>> default parsing using the below.
>>>>>>>>>>
>>>>>>>>>> source s_network { udp(ip(0.0.0.0) port(514) flags(no-parse)); };
>>>>>>>>>>
>>>>>>>>>> rewrite r_cisco{
>>>>>>>>>> subst('^<\d+>(\d+:|:)\s+(\.\w+|\w+)\s+\d+\s+\d+\s\d+:\d+:\d+:\s|^<\d+>:\s+\d+\s+\w+\s+\d+\s+\d+:\d+:\d+\s\w+:\s|^<\d+>(\d+:|:)\s',
>>>>>>>>>> "", value("MESSAGE"), type("pcre"), flags("ignore-case")); };
>>>>>>>>>>
>>>>>>>>>> destination d_mysql {
>>>>>>>>>>     sql(type(mysql)
>>>>>>>>>>     host("127.0.0.1")
>>>>>>>>>>     username("syslog-ng")
>>>>>>>>>>     password("password")
>>>>>>>>>>     database("syslog")
>>>>>>>>>>     table("messages_${HOST}")
>>>>>>>>>>     columns("datetime datetime", "host varchar(50)", "level
>>>>>>>>>> varchar(10)", "message text")
>>>>>>>>>>     values("${R_YEAR}-${R_MONTH}-${R_DAY}
>>>>>>>>>> ${R_HOUR}:${R_MIN}:${R_SEC}", "${HOST}", "${LEVEL}", "${MESSAGE}")
>>>>>>>>>>     indexes("datetime", "level")
>>>>>>>>>>     );
>>>>>>>>>> };
>>>>>>>>>>
>>>>>>>>>> log { source(s_network); rewrite(r_cisco); destination(d_mysql);
>>>>>>>>>> };
>>>>>>>>>>
>>>>>>>>>> This works perfectly as it formats the message as I want and
>>>>>>>>>> covers IOS and NX-OS devices. The problem is when I turned off the default
>>>>>>>>>> parser, now all of my logs show "notice" in the $LEVEL macro and doesn't
>>>>>>>>>> reflect the real message header level. The $HOST macro still works fine
>>>>>>>>>> however.
>>>>>>>>>>
>>>>>>>>>> Is this the expected behavior that the message header fields are
>>>>>>>>>> not parsed as well as the $MESSAGE itself not being parsed? How can map the
>>>>>>>>>> header level field properly to the $LEVEL marco if I disable the default
>>>>>>>>>> parser?
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Max
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ______________________________________________________________________________
>>>>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>>>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ______________________________________________________________________________
>>>>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>>>>> Documentation:
>>>>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> ______________________________________________________________________________
>>>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>>>> Documentation:
>>>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>>>
>>>>>>>>>
>>>>>>>> ______________________________________________________________________________
>>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>>> Documentation:
>>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>>
>>>>>>>>
>>>>>>> ______________________________________________________________________________
>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>> Documentation:
>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>
>>>>>>>
>>>>>> ______________________________________________________________________________
>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>> Documentation:
>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>
>>>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20190308/9b4e6dfc/attachment-0001.html>

More information about the syslog-ng mailing list