[syslog-ng] Rewriting Cisco messages

Tue Feb 7 23:15:40 CET 2012

Ah, didn't know/forgot you had seen ELSA, so I guess my last email
looks a little dumb ;)  I'd be interested in any ELSA-Logstash
comparisons or feedback off-list.

So, you've pre-filtered for Cisco messages, and modified my original
pattern match from an anchored "^%" to just "%" so that any message
will be trimmed.  Since I can filter for just Cisco, that won't work
for me, but I am interested in what the CPU performance is like when
comparing the unanchored regex with the anchored regex.  Adding the
other two filter patterns bumped my CPU up 10-15% on 7-10k logs/sec.

On Tue, Feb 7, 2012 at 4:01 PM, Thomas Wollner <tw at wollner-net.de> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi Martin,
>
> mine is a simplified one of yours (and yes stolen / inspired from elsa;-))
>
> filter f_rewrite_cisco_program {
>  match('%([^:]+):\s+([^\n]+)' value("MESSAGE") type("pcre")
> flags("store-matches" "nobackref"));
> };
>
> rewrite r_cisco_program {
>  set("$1", value("PROGRAM") condition(filter(f_rewrite_cisco_program)));
>  set("$2", value("MESSAGE") condition(filter(f_rewrite_cisco_program)));
> };
>
> just one step and in the rewrite everything before the % gets whiped out.
>
> I`m pre-filtering for cisco based logs (which have the mnemonic in the
> message) and then rewrite only those. I`have logs from different
> network vendors and some servers too. The combined pre-filter together
> with the rewrite I newer catched a "false" one. Currently I have
> around 6-8 k msg/sec and I do not notice an impact on cpu utilization
> compared to the setup without the rewriting.
>
>
> best regards,
>
> Tom
>
>
>
>
> On 07.02.2012 22:45, Martin Holste wrote:
>> Yep, my regexes should account for all of those weird formats.
>> So, your method is a two-step method in which you first clear out
>> everything before the percent sign, then you match from the percent
>> to the colon?  Are you only receiving Cisco syslog?  Do you have a
>> config example?
>>
>> On Tue, Feb 7, 2012 at 2:16 PM, Thomas Wollner <tw at wollner-net.de>
>> wrote:
>>
>>
>> Martin,
>>
>> the configs on cisco routers are very different. One can use too
>> many options how timestamps are built and that crap makes it to the
>> message part of the log.
>>
>> Just an example: #service timestamps log myrouter(config)#service
>> timestamps log ? datetime  Timestamp with date and time uptime
>> Timestamp with system uptime <cr>
>>
>> myrouter(config)#service timestamps log date
>> myrouter(config)#service timestamps log datetime ? localtime
>> Use local time zone for timestamps msec           Include
>> milliseconds in timestamp show-timezone  Add time zone information
>> to timestamp year           Include year in timestamp <cr>
>>
>> Because of the too many options I used to rewrite all that crap
>> with nothing until I find a %FACILITY-PRIORITY-PROGRAM - a so
>> called mnemonic - and set that as the $PROGRAM without leading %.
>> That way I can use a simple regex "everything between % and :" is
>> program and everything until end of line is message.
>>
>> The above options are just for ios based devices. ios-xe, ios-xr,
>> nx-os are out in the wild now and I`m quite shure they will
>> operate different...
>>
>> just my 2 cents,
>>
>> best regards,
>>
>> Tom
>>
>>
>>
>> On 07.02.2012 16:26, Martin Holste wrote:
>>>>> I spent some time yesterday working out the proper regexes
>>>>> to handle the many ways Cisco sends its timestamps, depending
>>>>> on how the device is configured.  However, I feel like my
>>>>> solution can be improved upon, so I'd like to see if there's
>>>>> a better way.  Here's what I've got so far that seems to be
>>>>> working, though there seems to be a 10-15% CPU penalty at the
>>>>> moment:
>>>>>
>>>>> #4w4d: %LINEPROTO-5-UPDOWN: Line protocol on Interface
>>>>> FastEthernet0/15, changed state to down #1y46w:
>>>>> %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered
>>>>> on GigabitEthernet0/1 (1), with S-COR-02 GigabitEthernet2/15
>>>>> (40). filter f_rewrite_cisco_program_3 {
>>>>> match('^\d+[ywdh]\d+[ywdh]: (%[^:]+): ([^\n]+)'
>>>>> value("MSGONLY") type("pcre") flags("store-matches"
>>>>> "nobackref")); };
>>>>>
>>>>> #Feb  6 16:43:32.219: %LINK-3-UPDOWN: Interface
>>>>> FastEthernet2/0/42, changed state to up filter
>>>>> f_rewrite_cisco_program_2 {
>>>>> match('^[\*\.]?(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s+\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}(?:\.\d+)?(?:
>>>>>
>>>>>
>>
>>>>>
> [A-Z]{3})?: (%[^:]+): ([^\n]+)'
>>>>> value("MSGONLY") type("pcre") flags("store-matches"
>>>>> "nobackref")); };
>>>>>
>>>>> # Others where MSGONLY starts with PROGRAM filter
>>>>> f_rewrite_cisco_program { match('^(%[A-Z]+\-\d\-[0-9A-Z]+):
>>>>> ([^\n]+)' value("MSGONLY") type("pcre")
>>>>> flags("store-matches" "nobackref")); };
>>>>>
>>>>> rewrite r_cisco_program { set("$1", value("PROGRAM")
>>>>> condition(filter(f_rewrite_cisco_program) or
>>>>> filter(f_rewrite_cisco_program_2) or
>>>>> filter(f_rewrite_cisco_program_3))); set("$2",
>>>>> value("MESSAGE") condition(filter(f_rewrite_cisco_program)
>>>>> or filter(f_rewrite_cisco_program_2) or
>>>>> filter(f_rewrite_cisco_program_3))); };
>>>>>
>>>>> This works, but is there a better way?
>>>>> ______________________________________________________________________________
>>>>>
>>>>>
>>
>>>>>
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation:
>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>
>>>>>
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>> ______________________________________________________________________________
>>>
>>>
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>> ______________________________________________________________________________
>>
>>
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.2 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iD8DBQFPMZ88TCCRT+dccOYRAkD4AJ9RZS5LT44x9oYsNvVEvQpQNgZfQwCePCdP
> TVucxhRlGcgD14mRqYMhzD8=
> =v6vK
> -----END PGP SIGNATURE-----
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>