[syslog-ng] Windows Agent default install template

Evan Rempel erempel at uvic.ca
Mon Sep 30 17:28:07 CEST 2013


Yes, I know that this can all be done with RFC5424 message transport, but
the Syslog-ng Agent for Windows has a message template for RFC3164 message transport
and that is what I am referring to.

When using RFC3164 it is even more important that the message have a format/template
that can be dealt with by a central log collection destination. For syslog-ng that
means that the message should be easy to parse with patterndb tools.
In my opinion the default message template makes parsing more difficult than it should
because it takes the structured data (MESSAGE_ID, EVENT_NAME, EVENT_MESSAGE etc) and
concatenates them together without any structure. Starting with structured data
and changing it to unstructured data is not helpful. I realize that having only one
message body in RFC3164 is difficult, but using a template like the one I have
proposed minimizes the difficulty.

Thanks for your thorough response.

Evan.

On 09/30/2013 01:05 AM, Fekete Robert wrote:
> Hi Evan,
>
> To make this work as it really should, I think we should take a slightly
> different angle at the problem:
>
> - AFAIK, when using the RFC5424 message format, the agent adds every message
> field to the SDATA as name-value pairs (except for the EVENT_MESSAGE). So all
> this data is already available in a parsed and structured format. This means
> that only the EVENT_MESSAGE part needs to be identified and parsed.
>
> - Since the Event ID is already available as a macro, it is easy to filter only
> the messages of the specific Event ID. Now we would only need two things:
> patterns for the EVENT_MESSAGE part of the messages (if needed), and a way to
> apply these patterns to these messages. Something like:
> " if $EVENT_ID == x , then parse EVENT_MESSAGE with EVENT_ID_x.xml"
> I _think_ that extending the patternb in syslog-ng to take templates so that it
> can be used only on the EVENT_MESSAGE part should be relatively simple, but I am
> not a programmer, so I might err here. Then only the problem of creating the
> appropriate configuration remains (adding tons of filters and patterndb parsers
> does not sound too convenient), but I guess this can be solved as well.
>
> - There are two other things that syslog-ng could do: syslog-ng should recognize
> if it received a message that contains an eventlog message (this should be
> fairly simple, for example, by checking if the EVENT_ID sdata field is set), and
> act accordingly (for example, parse the message with a specific patterndb, or
> when writing the message to a file, use an eventlog-specific template to make it
> more readable, for example, the template that you suggested).
>
> - Another useful thing could be to simplify all the above. Starting with
> syslog-ng Premium Edition 5 LTS (due to be released in a few weeks), the
> syslog-n Windows agent and the Premium Edition will use the same code-base,
> meaning that if there are patterns for the eventlog messages, then technically,
> it could be possible in the future add the pattern database to the Windows
> agent, and do the parsing on the clients. Then the agent could transfer the
> already parsed messages to the server, which then just had to use the parsed
> data to whatever you wanted to.
>
> Kind Regards,
>
> Robert
>
> On 09/27/2013 05:21 PM, Evan Rempel wrote:
>>
>> We are in the process of a hug log filtering/profiling project where we are using syslog-ng to
>> profile/classify *EVERY* message that comes through our syslog server.
>> This means that we will have 1,000's of patterns in our patterndb, but we accept that.
>>
>> We have purchased the Syslog-ng Agent for Windows and are not profiling all of our Windows messages as well.
>>
>> Now to the challenge. The default message template is very difficult to parse using the pattern database.
>> The problem is that many of the fields from a windows computer can contain spaces in them, and there
>> are no delimiters in the default template. Additionally, the $EVENT_ID that windows places on
>> each message, which is unique for each message type within the scope of each application that is logging, is placed
>> at the end of the log line after the free form dext description. making a patterndb pattern to parse through the
>> free form text just to get to the Event ID is very difficult.
>>
>>
>> I wanted to open the discussion of using a different, and IMHO much more usefull, default template
>> for MS Windows event logs. The pattern below has the following properties:
>>
>> - wraps each field in a [] pair
>> - places the most definitive value, the Event ID, at the beginning of the template
>> - places the least definitive and unstructured data, EVENT_MESSAGE, at the end of the template
>>
>>
>> EventID[${EVENT_ID}] Log[${EVENT_NAME}] Type[${EVENT_TYPE}] User[${EVENT_USERNAME}] ${EVENT_MESSAGE}
>>
>>
>> Let the discussion begin!
>>
>> Thanks for listening.
>>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>



More information about the syslog-ng mailing list