Re: [syslog-ng] Parsing message in unstructured rows

7 Sep 2015

      Would patterndb be an option?Thinking one to match each log line type each parsing out the date and whatever else you want. 
Jim

Sent from my Verizon Wireless 4G LTE smartphone

-------- Original message --------
From: Giovanni Mancuso <giovanni.mancuso@par-tec.it> 
Date: 09/07/2015  3:26 PM  (GMT-05:00) 
To: "Scheidler, Balázs" <balazs.scheidler@balabit.com>, Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> 
Subject: Re: [syslog-ng] Parsing message in unstructured rows 

Il 07 settembre 2015 21:10:59
"Scheidler, Balázs" <balazs.scheidler@balabit.com> ha scritto:
...
On Sep 7, 2015 6:11 PM,
"Giovanni Mancuso" <giovanni.mancuso@par-tec.it>
...
wrote:
...
...
...
...
Hi,
...
...
...
...
I have an application that log date in every rows. The problem is
that
...
the string isn't in specific part of MESSAGE, but it could be the
first
...
element or the last element, or in the middle :-) :-)
...
...
...
...
For example (only MESSAGE):
...
...
User: user1@example.com Date: 12/12/2014 Status: OK
...
...
User: user2@example.com ID: 1234 Status: DEL .......  Date:
03/05/2014
...
...
Date: 05/08/2015 User: user3@example.com ....... Stauts: OK
...
...
.........
...
...
...
...
I want split the log in more files arranged in different path
builded by
...
the date information, for example /LOGS/YYYY/MM/DD/mylog.log.
...
...
...
...
I do:
...
...
rewrite r_rewrite_set {
...
...
    set("$(python get_data)", value("APP.DATE"));
...
...
};
...
...
...
...
python{
...
...
import re
...
...
def get_data(logmsg):
...
...
   out=None
...
...
   vars(logmsg)
...
...
   out = re.findall(" Date: (\d\d/\d\d/\d\d\d\d) ",
logmsg.MESSAGE)
...
...
   if len(out) == 1:
...
...
       return out[0]
...
...
   else:
...
...
       raise
Exception("Invalid match")
...
...
};
...
...
...
...
In this way i have in APP.DATE the date.
...
...
Now i have a some questions:
...
...
1) Is there another way to do this without python?
...

...
Well, this should be possible with a simple regexp filter.
I try to use a filter regexp,
but i don't find the solution. Have you an example?

...
...
2) In this way for every message, syslog-ng forks and exec a
python
...
interpreter?
...

...
No, it embeds a Python interpreter.
...

...
...
3) Is there a way to add custom SDATA field from python? Or is
there a
...
way to create APP.DATE from python without rewrite rule?
...

...
Not right now.
...

...
...
4) Is there a documentation about python{}? I only found a post
in a blog.
...

...
It's being prepared by the tech writer team in BalaBit
...

...
...
...
...
Thanks
...
...
...
...
...
...
...
______________________________________________________________________________
...
...
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
...
...
Documentation:
...
http://www.balabit.com/support/documentation/?product=syslog-ng
...
...
FAQ: http://www.balabit.com/wiki/syslog-ng-faq
...
...
...
...
Inviato con AquaMail per
Android

http://www.aqua-mail.com

Re: [syslog-ng] Parsing message in unstructured rows

Jim Hendrick