Parsing message in unstructured rows
Hi, I have an application that log date in every rows. The problem is that the string isn't in specific part of MESSAGE, but it could be the first element or the last element, or in the middle :-) :-) For example (only MESSAGE): User: user1@example.com *Date: 12/12/2014* Status: OK User: user2@example.com ID: 1234 Status: DEL ....... *Date: 03/05/2014 Date: 05/08/2015 *User:**user3@example.com ....... Stauts: OK* *......... I want split the log in more files arranged in different path builded by the date information, for example /LOGS/YYYY/MM/DD/mylog.log. I do: Konsole output rewrite r_rewrite_set { set("$(python get_data)", value("APP.DATE")); }; python{ import re def get_data(logmsg): out=None vars(logmsg) out = re.findall(" Date: (\d\d/\d\d/\d\d\d\d) ", logmsg.MESSAGE) if len(out) == 1: return out[0] else: raise Exception("Invalid match") }; In this way i have in APP.DATE the date. Now i have a some questions: 1) Is there another way to do this without python? 2) In this way for every message, syslog-ng forks and exec a python interpreter? 3) Is there a way to add custom SDATA field from python? Or is there a way to create APP.DATE from python without rewrite rule? 4) Is there a documentation about python{}? I only found a post in a blog. Thanks
On Sep 7, 2015 6:11 PM, "Giovanni Mancuso" <giovanni.mancuso@par-tec.it> wrote:
Hi,
I have an application that log date in every rows. The problem is that
the string isn't in specific part of MESSAGE, but it could be the first element or the last element, or in the middle :-) :-)
For example (only MESSAGE): User: user1@example.com Date: 12/12/2014 Status: OK User: user2@example.com ID: 1234 Status: DEL ....... Date: 03/05/2014 Date: 05/08/2015 User: user3@example.com ....... Stauts: OK .........
I want split the log in more files arranged in different path builded by
the date information, for example /LOGS/YYYY/MM/DD/mylog.log.
I do: rewrite r_rewrite_set { set("$(python get_data)", value("APP.DATE")); };
python{ import re def get_data(logmsg): out=None vars(logmsg) out = re.findall(" Date: (\d\d/\d\d/\d\d\d\d) ", logmsg.MESSAGE) if len(out) == 1: return out[0] else: raise Exception("Invalid match") };
In this way i have in APP.DATE the date. Now i have a some questions: 1) Is there another way to do this without python?
Well, this should be possible with a simple regexp filter.
2) In this way for every message, syslog-ng forks and exec a python interpreter?
No, it embeds a Python interpreter.
3) Is there a way to add custom SDATA field from python? Or is there a way to create APP.DATE from python without rewrite rule?
Not right now.
4) Is there a documentation about python{}? I only found a post in a blog.
It's being prepared by the tech writer team in BalaBit
Thanks
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Il 07 settembre 2015 21:10:59 "Scheidler, Balázs" <balazs.scheidler@balabit.com> ha scritto:
On Sep 7, 2015 6:11 PM, "Giovanni Mancuso" <giovanni.mancuso@par-tec.it> wrote:
Hi,
I have an application that log date in every rows. The problem is that
the string isn't in specific part of MESSAGE, but it could be the first element or the last element, or in the middle :-) :-)
For example (only MESSAGE): User: user1@example.com Date: 12/12/2014 Status: OK User: user2@example.com ID: 1234 Status: DEL ....... Date: 03/05/2014 Date: 05/08/2015 User: user3@example.com ....... Stauts: OK .........
I want split the log in more files arranged in different path builded by
the date information, for example /LOGS/YYYY/MM/DD/mylog.log.
I do: rewrite r_rewrite_set { set("$(python get_data)", value("APP.DATE")); };
python{ import re def get_data(logmsg): out=None vars(logmsg) out = re.findall(" Date: (\d\d/\d\d/\d\d\d\d) ", logmsg.MESSAGE) if len(out) == 1: return out[0] else: raise Exception("Invalid match") };
In this way i have in APP.DATE the date. Now i have a some questions: 1) Is there another way to do this without python?
Well, this should be possible with a simple regexp filter.
I try to use a filter regexp, but i don't find the solution. Have you an example?
2) In this way for every message, syslog-ng forks and exec a python interpreter?
No, it embeds a Python interpreter.
3) Is there a way to add custom SDATA field from python? Or is there a way to create APP.DATE from python without rewrite rule?
Not right now.
4) Is there a documentation about python{}? I only found a post in a blog.
It's being prepared by the tech writer team in BalaBit
Thanks
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Inviato con AquaMail per Android http://www.aqua-mail.com
filter extract_date { message("Date: ([0-9]+)" flags(store-matches)); The important part is store-matches because by default syslog-ng doesn't store the values of the groups. By default syslog-ng stores the value as $1, but that can be changed using the pcre syntax. (?P<name>group) http://regular-expressions.mobi/named.html On Sep 7, 2015 9:25 PM, "Giovanni Mancuso" <giovanni.mancuso@par-tec.it> wrote:
Il 07 settembre 2015 21:10:59 "Scheidler, Balázs" < balazs.scheidler@balabit.com> ha scritto:
On Sep 7, 2015 6:11 PM, "Giovanni Mancuso" <giovanni.mancuso@par-tec.it> wrote:
Hi,
I have an application that log date in every rows. The problem is that
the string isn't in specific part of MESSAGE, but it could be the first element or the last element, or in the middle :-) :-)
For example (only MESSAGE): User: user1@example.com Date: 12/12/2014 Status: OK User: user2@example.com ID: 1234 Status: DEL ....... Date: 03/05/2014 Date: 05/08/2015 User: user3@example.com ....... Stauts: OK .........
I want split the log in more files arranged in different path builded
by the date information, for example /LOGS/YYYY/MM/DD/mylog.log.
I do: rewrite r_rewrite_set { set("$(python get_data)", value("APP.DATE")); };
python{ import re def get_data(logmsg): out=None vars(logmsg) out = re.findall(" Date: (\d\d/\d\d/\d\d\d\d) ", logmsg.MESSAGE) if len(out) == 1: return out[0] else: raise Exception("Invalid match") };
In this way i have in APP.DATE the date. Now i have a some questions: 1) Is there another way to do this without python?
Well, this should be possible with a simple regexp filter.
I try to use a filter regexp, but i don't find the solution. Have you an example?
2) In this way for every message, syslog-ng forks and exec a python interpreter?
No, it embeds a Python interpreter.
3) Is there a way to add custom SDATA field from python? Or is there a way to create APP.DATE from python without rewrite rule?
Not right now.
4) Is there a documentation about python{}? I only found a post in a
blog.
It's being prepared by the tech writer team in BalaBit
Thanks
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Inviato con AquaMail per Android http://www.aqua-mail.com
participants (2)
-
Giovanni Mancuso
-
Scheidler, Balázs