[syslog-ng] Problems parsing Cisco syslogs
N. Max Pierson
nmaxpierson at gmail.com
Thu Mar 7 19:21:58 UTC 2019
Hi Kokan,
I didn't catch that previously, so thanks for pointing that out for me. I
made the changes to reflect the ${.cisco.severity} macro but now I am not
getting anything in my database table. It seems to work if I log it to a
file, but for some reason when I log to mysql, it doesn't put anything in
the column I had for level. My config is below. Also, is there any way to
show the text version of the severity instead of the number?
source s_network { udp(ip(0.0.0.0) port(514) flags(no-parse)); };
template t_cisco { template("${R_YEAR}-${R_MONTH}-${R_DAY}
${R_HOUR}:${R_MIN}:${R_SEC} ${HOST} ${.cisco.severity} ${MESSAGE}\n"); };
destination d_test { file("/var/log/network-test" template(t_cisco)); };
destination d_mysql {
sql(type(mysql)
host("127.0.0.1")
username("syslog-ng")
password("password")
database("syslog")
table("messages_${HOST}")
columns("datetime datetime", "host varchar(50)", "level varchar(10)",
"message text")
values("${R_YEAR}-${R_MONTH}-${R_DAY} ${R_HOUR}:${R_MIN}:${R_SEC}",
"${HOST}", "${.cisco.severity}", "${MESSAGE}")
indexes("datetime", "level")
);
};
log { source(s_network); parser(p_cisco); destination(d_test); };
log { source(s_network); parser(p_cisco); destination(d_mysql); };
Regards,
Max
On Thu, Mar 7, 2019 at 1:50 AM Péter, Kókai <peter.kokai at oneidentity.com>
wrote:
> Hello,
>
> The *cisco-parser* as of now cannot set the LEVEL macro, as that is a
> *hard* macro, those cannot be set directly from configuration setters (for
> example: rewrite { set("lvl" value("LEVEL"); };).
> The header is parsed into the ".prefix.*" macros, the default *prefix* is
> ".cisco.".
>
> I think with shifting *parser*s implemented as syslog-ng configuration
> rather then C, it might be a good idea to lift the soft vs hard macro via a
> proper setters.
>
> Note1: if you are using *flags(no-parse)* it is going to set a default
> level, changing at least that default level is possible per source
> "network( default_level(...) flags(no-parse) ...);"
>
> Note2: if you really want to set LEVEL macro, well you can do it but not
> by design, and it might break at any point. It is possible to shadow hard
> macros setting patterns, instead of LEVEL you could set LEVLE[L] pattern,
> well which is kinda only matches LEVEL anyway. The way it works it creates
> a LEVEL soft macro. IMPORTANT: this leaves the message in a semi-state, as
> some code is going to use ${LEVEL} and some code is going to use the
> internal level representation which still stays the same old level.
>
> --
> Kokan
>
> On Wed, Mar 6, 2019 at 9:45 PM N. Max Pierson <nmaxpierson at gmail.com>
> wrote:
>
>> Balázs,,
>>
>> I have decided to not worry about anything without the cisco mnemonic and
>> use the cisco parser until I can come up with a better solution. I made
>> that modification to the plugin.conf file and now I am able to receive
>> NX-OS messages but I have discovered an issue that seems to be with the
>> cisco parser itself. When I enable it, all of my messages are coming in
>> with a "notice" level so it seems as though the cisco parser isn't properly
>> parsing the header correctly. I even moved back to the default plugin.conf
>> file to make sure my modifications didn't break it in the process. Here is
>> my config if that helps.....
>>
>> source s_network { udp(ip(0.0.0.0) port(514) flags(no-parse)); };
>>
>> template t_cisco { template("${R_YEAR}-${R_MONTH}-${R_DAY}
>> ${R_HOUR}:${R_MIN}:${R_SEC} ${HOST} ${LEVEL} ${MESSAGE}\n"); };
>>
>> destination d_test { file("/var/log/network-test" template(t_cisco)); };
>>
>> parser p_cisco { cisco-parser(); };
>>
>> log { source(s_network); parser(p_cisco); destination(d_test); };
>>
>> Regards,
>> Max
>>
>> On Mon, Mar 4, 2019 at 2:00 PM Scheidler, Balázs <
>> balazs.scheidler at oneidentity.com> wrote:
>>
>>> It was a bit strange that nxos starts with a colon and that a cisco
>>> message didn't have a mnemonic. Cisco messages by my experience always
>>> contain a triplet and this one isn't, and doesn't seem to have a timestamp
>>> either. If you say that this was recorded the way you mentioned then i
>>> believe you :), they just looked a bit odd.
>>>
>>> Do all messages from the same device look the same? Or these are just
>>> the exceptions?
>>>
>>> Also what is nxos?
>>>
>>> On Sat, Mar 2, 2019, 21:37 Scheidler, Balázs <
>>> balazs.scheidler at oneidentity.com wrote:
>>>
>>>> Can you please include the log sample as it was received by syslog-ng?
>>>> This seems to have been mangled already.
>>>>
>>>> On Fri, Mar 1, 2019, 23:16 N. Max Pierson <nmaxpierson at gmail.com wrote:
>>>>
>>>>> Here are 2 different samples of logs without the mnemonic, one is IOS
>>>>> and the other is NX-OS
>>>>>
>>>>> IOS, no-parse log to file
>>>>> --
>>>>> Mar 1 16:06:07 hostname1 <185>78919861: -Traceback= 151644C 151763C
>>>>> 17F0094 1B8BDE8 1B82858
>>>>>
>>>>>
>>>>> NX-OS, no-parse log to file
>>>>> --
>>>>> Mar 1 16:10:41 hostname1 <189>: 2019 Mar 1 16:10:41 CST: last
>>>>> message repeated 3 times
>>>>>
>>>>> Regards,
>>>>> Max
>>>>>
>>>>> On Thu, Feb 28, 2019 at 5:45 PM Scheidler, Balázs <
>>>>> balazs.scheidler at oneidentity.com> wrote:
>>>>>
>>>>>> Log samples would be appreciated.
>>>>>>
>>>>>> On Thu, Feb 28, 2019, 18:11 N. Max Pierson <nmaxpierson at gmail.com
>>>>>> wrote:
>>>>>>
>>>>>>> Balázs,
>>>>>>>
>>>>>>> I modified the plugin.conf XML file (https://pastebin.com/9BruYy0S
>>>>>>> - added lines 62-65) for the cisco-parser and I was able to capture NX-OS
>>>>>>> syslogs. I also noticed that the default parser is not matching syslogs
>>>>>>> that do not include the Cisco mnemonic (both for IOS and NX-OS). While I
>>>>>>> was able to fix the NX-OS issues (which was a date parsing problem), i'm
>>>>>>> not sure what needs to be changed/added in this file so that even logs that
>>>>>>> do not contain the mnemonic are matched. Can anyone shed some light on what
>>>>>>> part of this parser needs to be changed to resolve the issue? I can post
>>>>>>> some examples of a log that doesn't contain the mnemonic should you need
>>>>>>> them.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Max
>>>>>>>
>>>>>>> On Tue, Feb 26, 2019 at 11:14 AM Scheidler, Balázs <
>>>>>>> balazs.scheidler at oneidentity.com> wrote:
>>>>>>>
>>>>>>>> The cisco-parser() should take care about messages from cisco
>>>>>>>> routers assuming they were received using flags(no-parse). It will
>>>>>>>> automatically detect timestamp formats, but its not perfect, so if you
>>>>>>>> encounter something that it doesn't properly parse, do let us know, so we
>>>>>>>> can add it.
>>>>>>>>
>>>>>>>> Also, default-network-drivers() makes it possible to receive both
>>>>>>>> cisco and non-cisco logs on the same port, automatically recognizing the
>>>>>>>> appropriate format. This driver relies on the app-parser() framework, which
>>>>>>>> can be extended by application specific parsers. With that you can
>>>>>>>> construct your specific source driver configuration if you find
>>>>>>>> default-network-drivers () too complicated.
>>>>>>>>
>>>>>>>> You can find all of these in the documentation, and their source in
>>>>>>>> the syslog-ng configuration library (scl for short, usually in
>>>>>>>> /usr/share/syslog-ng/include/scl).
>>>>>>>>
>>>>>>>> On Tue, Feb 26, 2019, 18:08 Sandor Geller <
>>>>>>>> sandor.geller at ericsson.com wrote:
>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> When the no-parse flag is used then the macros referencing various
>>>>>>>>> parts of the message aren't filled in. HOST could get looked up using a
>>>>>>>>> reverse DNS lookup unless the keep_hostname option is set. The syslog
>>>>>>>>> priority is set to user.notice when parsing is disabled.
>>>>>>>>>
>>>>>>>>> Did you take a look at the existing cisco parser? Using it or
>>>>>>>>> adopting it should ease your job. If you could configure the Ciscos to use
>>>>>>>>> other port than anything else (which speaks syslog, Cisco devices usually
>>>>>>>>> aren't such...) would be even better.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Sandor
>>>>>>>>>
>>>>>>>>> On 02/26/2019 05:36 PM, N. Max Pierson wrote:
>>>>>>>>>
>>>>>>>>> Hi List,
>>>>>>>>>
>>>>>>>>> I have been trying to get something in place that can parse
>>>>>>>>> syslogs from various Cisco devices. The message format is almost the same
>>>>>>>>> with a few exceptions. Here is what I have tried and it works but now it
>>>>>>>>> has created another problem I do not know how to troubleshoot.
>>>>>>>>>
>>>>>>>>> So that I could see exactly what was being parsed, I disabled the
>>>>>>>>> default parsing using the below.
>>>>>>>>>
>>>>>>>>> source s_network { udp(ip(0.0.0.0) port(514) flags(no-parse)); };
>>>>>>>>>
>>>>>>>>> rewrite r_cisco{
>>>>>>>>> subst('^<\d+>(\d+:|:)\s+(\.\w+|\w+)\s+\d+\s+\d+\s\d+:\d+:\d+:\s|^<\d+>:\s+\d+\s+\w+\s+\d+\s+\d+:\d+:\d+\s\w+:\s|^<\d+>(\d+:|:)\s',
>>>>>>>>> "", value("MESSAGE"), type("pcre"), flags("ignore-case")); };
>>>>>>>>>
>>>>>>>>> destination d_mysql {
>>>>>>>>> sql(type(mysql)
>>>>>>>>> host("127.0.0.1")
>>>>>>>>> username("syslog-ng")
>>>>>>>>> password("password")
>>>>>>>>> database("syslog")
>>>>>>>>> table("messages_${HOST}")
>>>>>>>>> columns("datetime datetime", "host varchar(50)", "level
>>>>>>>>> varchar(10)", "message text")
>>>>>>>>> values("${R_YEAR}-${R_MONTH}-${R_DAY}
>>>>>>>>> ${R_HOUR}:${R_MIN}:${R_SEC}", "${HOST}", "${LEVEL}", "${MESSAGE}")
>>>>>>>>> indexes("datetime", "level")
>>>>>>>>> );
>>>>>>>>> };
>>>>>>>>>
>>>>>>>>> log { source(s_network); rewrite(r_cisco); destination(d_mysql); };
>>>>>>>>>
>>>>>>>>> This works perfectly as it formats the message as I want and
>>>>>>>>> covers IOS and NX-OS devices. The problem is when I turned off the default
>>>>>>>>> parser, now all of my logs show "notice" in the $LEVEL macro and doesn't
>>>>>>>>> reflect the real message header level. The $HOST macro still works fine
>>>>>>>>> however.
>>>>>>>>>
>>>>>>>>> Is this the expected behavior that the message header fields are
>>>>>>>>> not parsed as well as the $MESSAGE itself not being parsed? How can map the
>>>>>>>>> header level field properly to the $LEVEL marco if I disable the default
>>>>>>>>> parser?
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Max
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ______________________________________________________________________________
>>>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ______________________________________________________________________________
>>>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>>>> Documentation:
>>>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>>>
>>>>>>>>>
>>>>>>>> ______________________________________________________________________________
>>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>>> Documentation:
>>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>>
>>>>>>>>
>>>>>>> ______________________________________________________________________________
>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>> Documentation:
>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>
>>>>>>>
>>>>>> ______________________________________________________________________________
>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>> Documentation:
>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>
>>>>>>
>>>>> ______________________________________________________________________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation:
>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20190307/a9ee7d9c/attachment-0001.html>
More information about the syslog-ng
mailing list