[syslog-ng] Windows Agent default install template

[Peter Gyongyosi]

Hi Evan,

thank you for sharing your thoughts -- and I am really interested in 
what others have to say on this. As Robert pointed out, when we 
ourselves want to deal with structured event log information coming from 
the Windows Agent we advise everyone to use the RFC5424 format and use 
the SDATA fields for filtering or do statistics based on the various 
fields in the event log messages. The RFC3164-formatting of such 
messages are, as a result, more of a fallback solution and we haven't 
thought much about its parseability (at least I haven't, maybe Bazsi or 
the others will correct me :)).

Nevertheless, you are right that transforming structured information 
into a format that is hard to parse is never a good idea, and also that 
formatting event log messages better in the Agent would allow us to use 
a pattern database on the other side. It is true that it'd be "nicer" 
and probably more error-safe to transfer the messages in a structured 
way (via RFC5424, json-destination&source or some other more efficient 
protocol we'd add) and match for name-value pairs in this structured 
data within patterns (eg. "if EVENTID==X and EVENTTYPE==Y than match()") 
-- and that'd happen over time. However, that will require an overhaul 
of the current pattern database model which is heavily focused on the 
$MSG part of the logs, while simply agreeing on a more sensible default 
format for that $MSG part would allow us to start writing and using 
patterns to parse and classify messages from event logs instantly.

So I am all for coming up with a more sensible default and if there's an 
agreement I'm more than happy to include it (even as a default) in the 
next version of the Agent.

By the way, while writing your patterns, you might want to take a look 
at this, if you haven't already:

<https://github.com/balabit/syslog-ng-patterndb/tree/master/os/windows2008>https://github.com/balabit/syslog-ng-patterndb/blob/master/os/windows2008/win2008.xml

This uses the current format, but tries to classify the most important 
messages that appear on a Windows 2008 Server and extract much more 
information than just the event ID or the username -- you could use it 
as a basis. And of course we'd be happy to include your patterns here as 
well if you are willing to share them.

Thanks,

Peter
(who is, for the record, responsible for the commercial side of the 
syslog-ng product line at BalaBit)

On 09/30/2013 05:28 PM, Evan Rempel wrote:
> Yes, I know that this can all be done with RFC5424 message transport, but
> the Syslog-ng Agent for Windows has a message template for RFC3164 message transport
> and that is what I am referring to.
>
> When using RFC3164 it is even more important that the message have a format/template
> that can be dealt with by a central log collection destination. For syslog-ng that
> means that the message should be easy to parse with patterndb tools.
> In my opinion the default message template makes parsing more difficult than it should
> because it takes the structured data (MESSAGE_ID, EVENT_NAME, EVENT_MESSAGE etc) and
> concatenates them together without any structure. Starting with structured data
> and changing it to unstructured data is not helpful. I realize that having only one
> message body in RFC3164 is difficult, but using a template like the one I have
> proposed minimizes the difficulty.
>
> Thanks for your thorough response.
>
> Evan.
>
> On 09/30/2013 01:05 AM, Fekete Robert wrote:
>> Hi Evan,
>>
>> To make this work as it really should, I think we should take a slightly
>> different angle at the problem:
>>
>> - AFAIK, when using the RFC5424 message format, the agent adds every message
>> field to the SDATA as name-value pairs (except for the EVENT_MESSAGE). So all
>> this data is already available in a parsed and structured format. This means
>> that only the EVENT_MESSAGE part needs to be identified and parsed.
>>
>> - Since the Event ID is already available as a macro, it is easy to filter only
>> the messages of the specific Event ID. Now we would only need two things:
>> patterns for the EVENT_MESSAGE part of the messages (if needed), and a way to
>> apply these patterns to these messages. Something like:
>> " if $EVENT_ID == x , then parse EVENT_MESSAGE with EVENT_ID_x.xml"
>> I _think_ that extending the patternb in syslog-ng to take templates so that it
>> can be used only on the EVENT_MESSAGE part should be relatively simple, but I am
>> not a programmer, so I might err here. Then only the problem of creating the
>> appropriate configuration remains (adding tons of filters and patterndb parsers
>> does not sound too convenient), but I guess this can be solved as well.
>>
>> - There are two other things that syslog-ng could do: syslog-ng should recognize
>> if it received a message that contains an eventlog message (this should be
>> fairly simple, for example, by checking if the EVENT_ID sdata field is set), and
>> act accordingly (for example, parse the message with a specific patterndb, or
>> when writing the message to a file, use an eventlog-specific template to make it
>> more readable, for example, the template that you suggested).
>>
>> - Another useful thing could be to simplify all the above. Starting with
>> syslog-ng Premium Edition 5 LTS (due to be released in a few weeks), the
>> syslog-n Windows agent and the Premium Edition will use the same code-base,
>> meaning that if there are patterns for the eventlog messages, then technically,
>> it could be possible in the future add the pattern database to the Windows
>> agent, and do the parsing on the clients. Then the agent could transfer the
>> already parsed messages to the server, which then just had to use the parsed
>> data to whatever you wanted to.
>>
>> Kind Regards,
>>
>> Robert
>>
>> On 09/27/2013 05:21 PM, Evan Rempel wrote:
>>> We are in the process of a hug log filtering/profiling project where we are using syslog-ng to
>>> profile/classify *EVERY* message that comes through our syslog server.
>>> This means that we will have 1,000's of patterns in our patterndb, but we accept that.
>>>
>>> We have purchased the Syslog-ng Agent for Windows and are not profiling all of our Windows messages as well.
>>>
>>> Now to the challenge. The default message template is very difficult to parse using the pattern database.
>>> The problem is that many of the fields from a windows computer can contain spaces in them, and there
>>> are no delimiters in the default template. Additionally, the $EVENT_ID that windows places on
>>> each message, which is unique for each message type within the scope of each application that is logging, is placed
>>> at the end of the log line after the free form dext description. making a patterndb pattern to parse through the
>>> free form text just to get to the Event ID is very difficult.
>>>
>>>
>>> I wanted to open the discussion of using a different, and IMHO much more usefull, default template
>>> for MS Windows event logs. The pattern below has the following properties:
>>>
>>> - wraps each field in a [] pair
>>> - places the most definitive value, the Event ID, at the beginning of the template
>>> - places the least definitive and unstructured data, EVENT_MESSAGE, at the end of the template
>>>
>>>
>>> EventID[${EVENT_ID}] Log[${EVENT_NAME}] Type[${EVENT_TYPE}] User[${EVENT_USERNAME}] ${EVENT_MESSAGE}
>>>
>>>
>>> Let the discussion begin!
>>>
>>> Thanks for listening.
>>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20131002/660e7040/attachment.htm