[syslog-ng] Problems parsing Cisco syslogs
Péter, Kókai
peter.kokai at oneidentity.com
Thu Mar 7 07:50:10 UTC 2019
Hello,
The *cisco-parser* as of now cannot set the LEVEL macro, as that is a
*hard* macro, those cannot be set directly from configuration setters (for
example: rewrite { set("lvl" value("LEVEL"); };).
The header is parsed into the ".prefix.*" macros, the default *prefix* is
".cisco.".
I think with shifting *parser*s implemented as syslog-ng configuration
rather then C, it might be a good idea to lift the soft vs hard macro via a
proper setters.
Note1: if you are using *flags(no-parse)* it is going to set a default
level, changing at least that default level is possible per source
"network( default_level(...) flags(no-parse) ...);"
Note2: if you really want to set LEVEL macro, well you can do it but not by
design, and it might break at any point. It is possible to shadow hard
macros setting patterns, instead of LEVEL you could set LEVLE[L] pattern,
well which is kinda only matches LEVEL anyway. The way it works it creates
a LEVEL soft macro. IMPORTANT: this leaves the message in a semi-state, as
some code is going to use ${LEVEL} and some code is going to use the
internal level representation which still stays the same old level.
--
Kokan
On Wed, Mar 6, 2019 at 9:45 PM N. Max Pierson <nmaxpierson at gmail.com> wrote:
> Balázs,,
>
> I have decided to not worry about anything without the cisco mnemonic and
> use the cisco parser until I can come up with a better solution. I made
> that modification to the plugin.conf file and now I am able to receive
> NX-OS messages but I have discovered an issue that seems to be with the
> cisco parser itself. When I enable it, all of my messages are coming in
> with a "notice" level so it seems as though the cisco parser isn't properly
> parsing the header correctly. I even moved back to the default plugin.conf
> file to make sure my modifications didn't break it in the process. Here is
> my config if that helps.....
>
> source s_network { udp(ip(0.0.0.0) port(514) flags(no-parse)); };
>
> template t_cisco { template("${R_YEAR}-${R_MONTH}-${R_DAY}
> ${R_HOUR}:${R_MIN}:${R_SEC} ${HOST} ${LEVEL} ${MESSAGE}\n"); };
>
> destination d_test { file("/var/log/network-test" template(t_cisco)); };
>
> parser p_cisco { cisco-parser(); };
>
> log { source(s_network); parser(p_cisco); destination(d_test); };
>
> Regards,
> Max
>
> On Mon, Mar 4, 2019 at 2:00 PM Scheidler, Balázs <
> balazs.scheidler at oneidentity.com> wrote:
>
>> It was a bit strange that nxos starts with a colon and that a cisco
>> message didn't have a mnemonic. Cisco messages by my experience always
>> contain a triplet and this one isn't, and doesn't seem to have a timestamp
>> either. If you say that this was recorded the way you mentioned then i
>> believe you :), they just looked a bit odd.
>>
>> Do all messages from the same device look the same? Or these are just the
>> exceptions?
>>
>> Also what is nxos?
>>
>> On Sat, Mar 2, 2019, 21:37 Scheidler, Balázs <
>> balazs.scheidler at oneidentity.com wrote:
>>
>>> Can you please include the log sample as it was received by syslog-ng?
>>> This seems to have been mangled already.
>>>
>>> On Fri, Mar 1, 2019, 23:16 N. Max Pierson <nmaxpierson at gmail.com wrote:
>>>
>>>> Here are 2 different samples of logs without the mnemonic, one is IOS
>>>> and the other is NX-OS
>>>>
>>>> IOS, no-parse log to file
>>>> --
>>>> Mar 1 16:06:07 hostname1 <185>78919861: -Traceback= 151644C 151763C
>>>> 17F0094 1B8BDE8 1B82858
>>>>
>>>>
>>>> NX-OS, no-parse log to file
>>>> --
>>>> Mar 1 16:10:41 hostname1 <189>: 2019 Mar 1 16:10:41 CST: last message
>>>> repeated 3 times
>>>>
>>>> Regards,
>>>> Max
>>>>
>>>> On Thu, Feb 28, 2019 at 5:45 PM Scheidler, Balázs <
>>>> balazs.scheidler at oneidentity.com> wrote:
>>>>
>>>>> Log samples would be appreciated.
>>>>>
>>>>> On Thu, Feb 28, 2019, 18:11 N. Max Pierson <nmaxpierson at gmail.com
>>>>> wrote:
>>>>>
>>>>>> Balázs,
>>>>>>
>>>>>> I modified the plugin.conf XML file (https://pastebin.com/9BruYy0S -
>>>>>> added lines 62-65) for the cisco-parser and I was able to capture NX-OS
>>>>>> syslogs. I also noticed that the default parser is not matching syslogs
>>>>>> that do not include the Cisco mnemonic (both for IOS and NX-OS). While I
>>>>>> was able to fix the NX-OS issues (which was a date parsing problem), i'm
>>>>>> not sure what needs to be changed/added in this file so that even logs that
>>>>>> do not contain the mnemonic are matched. Can anyone shed some light on what
>>>>>> part of this parser needs to be changed to resolve the issue? I can post
>>>>>> some examples of a log that doesn't contain the mnemonic should you need
>>>>>> them.
>>>>>>
>>>>>> Regards,
>>>>>> Max
>>>>>>
>>>>>> On Tue, Feb 26, 2019 at 11:14 AM Scheidler, Balázs <
>>>>>> balazs.scheidler at oneidentity.com> wrote:
>>>>>>
>>>>>>> The cisco-parser() should take care about messages from cisco
>>>>>>> routers assuming they were received using flags(no-parse). It will
>>>>>>> automatically detect timestamp formats, but its not perfect, so if you
>>>>>>> encounter something that it doesn't properly parse, do let us know, so we
>>>>>>> can add it.
>>>>>>>
>>>>>>> Also, default-network-drivers() makes it possible to receive both
>>>>>>> cisco and non-cisco logs on the same port, automatically recognizing the
>>>>>>> appropriate format. This driver relies on the app-parser() framework, which
>>>>>>> can be extended by application specific parsers. With that you can
>>>>>>> construct your specific source driver configuration if you find
>>>>>>> default-network-drivers () too complicated.
>>>>>>>
>>>>>>> You can find all of these in the documentation, and their source in
>>>>>>> the syslog-ng configuration library (scl for short, usually in
>>>>>>> /usr/share/syslog-ng/include/scl).
>>>>>>>
>>>>>>> On Tue, Feb 26, 2019, 18:08 Sandor Geller <
>>>>>>> sandor.geller at ericsson.com wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> When the no-parse flag is used then the macros referencing various
>>>>>>>> parts of the message aren't filled in. HOST could get looked up using a
>>>>>>>> reverse DNS lookup unless the keep_hostname option is set. The syslog
>>>>>>>> priority is set to user.notice when parsing is disabled.
>>>>>>>>
>>>>>>>> Did you take a look at the existing cisco parser? Using it or
>>>>>>>> adopting it should ease your job. If you could configure the Ciscos to use
>>>>>>>> other port than anything else (which speaks syslog, Cisco devices usually
>>>>>>>> aren't such...) would be even better.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Sandor
>>>>>>>>
>>>>>>>> On 02/26/2019 05:36 PM, N. Max Pierson wrote:
>>>>>>>>
>>>>>>>> Hi List,
>>>>>>>>
>>>>>>>> I have been trying to get something in place that can parse syslogs
>>>>>>>> from various Cisco devices. The message format is almost the same with a
>>>>>>>> few exceptions. Here is what I have tried and it works but now it has
>>>>>>>> created another problem I do not know how to troubleshoot.
>>>>>>>>
>>>>>>>> So that I could see exactly what was being parsed, I disabled the
>>>>>>>> default parsing using the below.
>>>>>>>>
>>>>>>>> source s_network { udp(ip(0.0.0.0) port(514) flags(no-parse)); };
>>>>>>>>
>>>>>>>> rewrite r_cisco{
>>>>>>>> subst('^<\d+>(\d+:|:)\s+(\.\w+|\w+)\s+\d+\s+\d+\s\d+:\d+:\d+:\s|^<\d+>:\s+\d+\s+\w+\s+\d+\s+\d+:\d+:\d+\s\w+:\s|^<\d+>(\d+:|:)\s',
>>>>>>>> "", value("MESSAGE"), type("pcre"), flags("ignore-case")); };
>>>>>>>>
>>>>>>>> destination d_mysql {
>>>>>>>> sql(type(mysql)
>>>>>>>> host("127.0.0.1")
>>>>>>>> username("syslog-ng")
>>>>>>>> password("password")
>>>>>>>> database("syslog")
>>>>>>>> table("messages_${HOST}")
>>>>>>>> columns("datetime datetime", "host varchar(50)", "level
>>>>>>>> varchar(10)", "message text")
>>>>>>>> values("${R_YEAR}-${R_MONTH}-${R_DAY}
>>>>>>>> ${R_HOUR}:${R_MIN}:${R_SEC}", "${HOST}", "${LEVEL}", "${MESSAGE}")
>>>>>>>> indexes("datetime", "level")
>>>>>>>> );
>>>>>>>> };
>>>>>>>>
>>>>>>>> log { source(s_network); rewrite(r_cisco); destination(d_mysql); };
>>>>>>>>
>>>>>>>> This works perfectly as it formats the message as I want and covers
>>>>>>>> IOS and NX-OS devices. The problem is when I turned off the default parser,
>>>>>>>> now all of my logs show "notice" in the $LEVEL macro and doesn't reflect
>>>>>>>> the real message header level. The $HOST macro still works fine however.
>>>>>>>>
>>>>>>>> Is this the expected behavior that the message header fields are
>>>>>>>> not parsed as well as the $MESSAGE itself not being parsed? How can map the
>>>>>>>> header level field properly to the $LEVEL marco if I disable the default
>>>>>>>> parser?
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Max
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ______________________________________________________________________________
>>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ______________________________________________________________________________
>>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>>> Documentation:
>>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>>
>>>>>>>>
>>>>>>> ______________________________________________________________________________
>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>> Documentation:
>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>
>>>>>>>
>>>>>> ______________________________________________________________________________
>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>> Documentation:
>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>
>>>>>>
>>>>> ______________________________________________________________________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation:
>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20190307/14a757c9/attachment-0001.html>
More information about the syslog-ng
mailing list