Hi Evan, To make this work as it really should, I think we should take a slightly different angle at the problem: - AFAIK, when using the RFC5424 message format, the agent adds every message field to the SDATA as name-value pairs (except for the EVENT_MESSAGE). So all this data is already available in a parsed and structured format. This means that only the EVENT_MESSAGE part needs to be identified and parsed. - Since the Event ID is already available as a macro, it is easy to filter only the messages of the specific Event ID. Now we would only need two things: patterns for the EVENT_MESSAGE part of the messages (if needed), and a way to apply these patterns to these messages. Something like: " if $EVENT_ID == x , then parse EVENT_MESSAGE with EVENT_ID_x.xml " I _think_ that extending the patternb in syslog-ng to take templates so that it can be used only on the EVENT_MESSAGE part should be relatively simple, but I am not a programmer, so I might err here. Then only the problem of creating the appropriate configuration remains (adding tons of filters and patterndb parsers does not sound too convenient), but I guess this can be solved as well. - There are two other things that syslog-ng could do: syslog-ng should recognize if it received a message that contains an eventlog message (this should be fairly simple, for example, by checking if the EVENT_ID sdata field is set), and act accordingly (for example, parse the message with a specific patterndb, or when writing the message to a file, use an eventlog-specific template to make it more readable, for example, the template that you suggested). - Another useful thing could be to simplify all the above. Starting with syslog-ng Premium Edition 5 LTS (due to be released in a few weeks), the syslog-n Windows agent and the Premium Edition will use the same code-base, meaning that if there are patterns for the eventlog messages, then technically, it could be possible in the future add the pattern database to the Windows agent, and do the parsing on the clients. Then the agent could transfer the already parsed messages to the server, which then just had to use the parsed data to whatever you wanted to. Kind Regards, Robert On 09/27/2013 05:21 PM, Evan Rempel wrote:

We are in the process of a hug log filtering/profiling project where we are using syslog-ng to profile/classify *EVERY* message that comes through our syslog server. This means that we will have 1,000's of patterns in our patterndb, but we accept that.

We have purchased the Syslog-ng Agent for Windows and are not profiling all of our Windows messages as well.

Now to the challenge. The default message template is very difficult to parse using the pattern database. The problem is that many of the fields from a windows computer can contain spaces in them, and there are no delimiters in the default template. Additionally, the $EVENT_ID that windows places on each message, which is unique for each message type within the scope of each application that is logging, is placed at the end of the log line after the free form dext description. making a patterndb pattern to parse through the free form text just to get to the Event ID is very difficult.

I wanted to open the discussion of using a different, and IMHO much more usefull, default template for MS Windows event logs. The pattern below has the following properties:

- wraps each field in a [] pair - places the most definitive value, the Event ID, at the beginning of the template - places the least definitive and unstructured data, EVENT_MESSAGE, at the end of the template

EventID[${EVENT_ID}] Log[${EVENT_NAME}] Type[${EVENT_TYPE}] User[${EVENT_USERNAME}] ${EVENT_MESSAGE}

Let the discussion begin!

Thanks for listening.

Hi Evan, thank you for sharing your thoughts -- and I am really interested in what others have to say on this. As Robert pointed out, when we ourselves want to deal with structured event log information coming from the Windows Agent we advise everyone to use the RFC5424 format and use the SDATA fields for filtering or do statistics based on the various fields in the event log messages. The RFC3164-formatting of such messages are, as a result, more of a fallback solution and we haven't thought much about its parseability (at least I haven't, maybe Bazsi or the others will correct me :)). Nevertheless, you are right that transforming structured information into a format that is hard to parse is never a good idea, and also that formatting event log messages better in the Agent would allow us to use a pattern database on the other side. It is true that it'd be "nicer" and probably more error-safe to transfer the messages in a structured way (via RFC5424, json-destination&source or some other more efficient protocol we'd add) and match for name-value pairs in this structured data within patterns (eg. "if EVENTID==X and EVENTTYPE==Y than match()") -- and that'd happen over time. However, that will require an overhaul of the current pattern database model which is heavily focused on the $MSG part of the logs, while simply agreeing on a more sensible default format for that $MSG part would allow us to start writing and using patterns to parse and classify messages from event logs instantly. So I am all for coming up with a more sensible default and if there's an agreement I'm more than happy to include it (even as a default) in the next version of the Agent. By the way, while writing your patterns, you might want to take a look at this, if you haven't already: <https://github.com/balabit/syslog-ng-patterndb/tree/master/os/windows2008>https://github.com/balabit/syslog-ng-patterndb/blob/master/os/windows2008/win2008.xml This uses the current format, but tries to classify the most important messages that appear on a Windows 2008 Server and extract much more information than just the event ID or the username -- you could use it as a basis. And of course we'd be happy to include your patterns here as well if you are willing to share them. Thanks, Peter (who is, for the record, responsible for the commercial side of the syslog-ng product line at BalaBit) On 09/30/2013 05:28 PM, Evan Rempel wrote:

Yes, I know that this can all be done with RFC5424 message transport, but the Syslog-ng Agent for Windows has a message template for RFC3164 message transport and that is what I am referring to.

When using RFC3164 it is even more important that the message have a format/template that can be dealt with by a central log collection destination. For syslog-ng that means that the message should be easy to parse with patterndb tools. In my opinion the default message template makes parsing more difficult than it should because it takes the structured data (MESSAGE_ID, EVENT_NAME, EVENT_MESSAGE etc) and concatenates them together without any structure. Starting with structured data and changing it to unstructured data is not helpful. I realize that having only one message body in RFC3164 is difficult, but using a template like the one I have proposed minimizes the difficulty.

Thanks for your thorough response.

Evan.

On 09/30/2013 01:05 AM, Fekete Robert wrote:

...

Hi Evan,

To make this work as it really should, I think we should take a slightly different angle at the problem:

- AFAIK, when using the RFC5424 message format, the agent adds every message field to the SDATA as name-value pairs (except for the EVENT_MESSAGE). So all this data is already available in a parsed and structured format. This means that only the EVENT_MESSAGE part needs to be identified and parsed.

- Since the Event ID is already available as a macro, it is easy to filter only the messages of the specific Event ID. Now we would only need two things: patterns for the EVENT_MESSAGE part of the messages (if needed), and a way to apply these patterns to these messages. Something like: " if $EVENT_ID == x , then parse EVENT_MESSAGE with EVENT_ID_x.xml" I _think_ that extending the patternb in syslog-ng to take templates so that it can be used only on the EVENT_MESSAGE part should be relatively simple, but I am not a programmer, so I might err here. Then only the problem of creating the appropriate configuration remains (adding tons of filters and patterndb parsers does not sound too convenient), but I guess this can be solved as well.

- There are two other things that syslog-ng could do: syslog-ng should recognize if it received a message that contains an eventlog message (this should be fairly simple, for example, by checking if the EVENT_ID sdata field is set), and act accordingly (for example, parse the message with a specific patterndb, or when writing the message to a file, use an eventlog-specific template to make it more readable, for example, the template that you suggested).

- Another useful thing could be to simplify all the above. Starting with syslog-ng Premium Edition 5 LTS (due to be released in a few weeks), the syslog-n Windows agent and the Premium Edition will use the same code-base, meaning that if there are patterns for the eventlog messages, then technically, it could be possible in the future add the pattern database to the Windows agent, and do the parsing on the clients. Then the agent could transfer the already parsed messages to the server, which then just had to use the parsed data to whatever you wanted to.

Kind Regards,

Robert

On 09/27/2013 05:21 PM, Evan Rempel wrote:

...

We are in the process of a hug log filtering/profiling project where we are using syslog-ng to profile/classify *EVERY* message that comes through our syslog server. This means that we will have 1,000's of patterns in our patterndb, but we accept that.

We have purchased the Syslog-ng Agent for Windows and are not profiling all of our Windows messages as well.

Now to the challenge. The default message template is very difficult to parse using the pattern database. The problem is that many of the fields from a windows computer can contain spaces in them, and there are no delimiters in the default template. Additionally, the $EVENT_ID that windows places on each message, which is unique for each message type within the scope of each application that is logging, is placed at the end of the log line after the free form dext description. making a patterndb pattern to parse through the free form text just to get to the Event ID is very difficult.

I wanted to open the discussion of using a different, and IMHO much more usefull, default template for MS Windows event logs. The pattern below has the following properties:

- wraps each field in a [] pair - places the most definitive value, the Event ID, at the beginning of the template - places the least definitive and unstructured data, EVENT_MESSAGE, at the end of the template

EventID[${EVENT_ID}] Log[${EVENT_NAME}] Type[${EVENT_TYPE}] User[${EVENT_USERNAME}] ${EVENT_MESSAGE}

Let the discussion begin!

Thanks for listening.

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq