PCRE in patterndb with back substitution
I have a case where an application logs something like {date} {host} {program}: my first line... ...my second line... ...and my third line. I want to make a correlation and unwrap these lines into {date} {host} {program}: my first line my second line and my third line. I started writing the patterndb to do this, but matching the ... at the end of the line is difficult, so I used @PCRE:line:(.*)\.\.\.$@ but I then need to only use the $1 to set a value <values> <value name="mymessage">$1</value> </values> Would this be the correct syntax to do this? Is there an easier way that would perform well? Thanks,
The $1 is not set in this case, you can however use template functions in the value part. E.g. set line based on the @PCRE@ matcher and overwrite its value using an expression $(substr $line 0 -3) Would that work for you? On Mar 15, 2018 02:08, "Evan Rempel" <erempel@uvic.ca> wrote:
I have a case where an application logs something like
{date} {host} {program}: my first line... ...my second line... ...and my third line.
I want to make a correlation and unwrap these lines into
{date} {host} {program}: my first line my second line and my third line.
I started writing the patterndb to do this, but matching the ... at the end
of the line is difficult, so I used @PCRE:line:(.*)\.\.\.$@
but I then need to only use the $1 to set a value
<values> <value name="mymessage">$1</value> </values>
Would this be the correct syntax to do this?
Is there an easier way that would perform well?
Thanks,
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Elen! Does your application log into a file? Because then you could use multi-line file source with a well-defined prefix as the "{date} {host} {program}:". Regards, Gabor On Thu, Mar 15, 2018 at 7:10 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
The $1 is not set in this case, you can however use template functions in the value part. E.g. set line based on the @PCRE@ matcher and overwrite its value using an expression $(substr $line 0 -3)
Would that work for you?
On Mar 15, 2018 02:08, "Evan Rempel" <erempel@uvic.ca> wrote:
I have a case where an application logs something like
{date} {host} {program}: my first line... ...my second line... ...and my third line.
I want to make a correlation and unwrap these lines into
{date} {host} {program}: my first line my second line and my third line.
I started writing the patterndb to do this, but matching the ... at the end
of the line is difficult, so I used @PCRE:line:(.*)\.\.\.$@
but I then need to only use the $1 to set a value
<values> <value name="mymessage">$1</value> </values>
Would this be the correct syntax to do this?
Is there an easier way that would perform well?
Thanks,
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Sorry Evan for mistyping your name. :) On Tue, Mar 20, 2018 at 2:06 PM, Nagy, Gábor <gabor.nagy@balabit.com> wrote:
Hi Elen!
Does your application log into a file? Because then you could use multi-line file source with a well-defined prefix as the "{date} {host} {program}:".
Regards, Gabor
On Thu, Mar 15, 2018 at 7:10 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
The $1 is not set in this case, you can however use template functions in the value part. E.g. set line based on the @PCRE@ matcher and overwrite its value using an expression $(substr $line 0 -3)
Would that work for you?
On Mar 15, 2018 02:08, "Evan Rempel" <erempel@uvic.ca> wrote:
I have a case where an application logs something like
{date} {host} {program}: my first line... ...my second line... ...and my third line.
I want to make a correlation and unwrap these lines into
{date} {host} {program}: my first line my second line and my third line.
I started writing the patterndb to do this, but matching the ... at the end
of the line is difficult, so I used @PCRE:line:(.*)\.\.\.$@
but I then need to only use the $1 to set a value
<values> <value name="mymessage">$1</value> </values>
Would this be the correct syntax to do this?
Is there an easier way that would perform well?
Thanks,
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
No problem about my name. My fast fingers make tonnes of errors. The application does not log into a file, so that isn't a really good option. I have the patterndb working for this, however, I came across another line that is ... 20 more and has a continuation line preceding it that does NOT end in ... so I have filter that one out. Does anyone handle java stack dumps gracefully :-) Evan On 03/20/2018 06:07 AM, Nagy, Gábor wrote:
Sorry Evan for mistyping your name. :)
On Tue, Mar 20, 2018 at 2:06 PM, Nagy, Gábor <gabor.nagy@balabit.com <mailto:gabor.nagy@balabit.com>> wrote:
Hi Elen!
Does your application log into a file? Because then you could use multi-line file source with a well-defined prefix as the "{date} {host} {program}:".
Regards, Gabor
On Thu, Mar 15, 2018 at 7:10 AM, Scheidler, Balázs <balazs.scheidler@balabit.com <mailto:balazs.scheidler@balabit.com>> wrote:
The $1 is not set in this case, you can however use template functions in the value part. E.g. set line based on the @PCRE@ matcher and overwrite its value using an expression $(substr $line 0 -3)
Would that work for you?
On Mar 15, 2018 02:08, "Evan Rempel" <erempel@uvic.ca <mailto:erempel@uvic.ca>> wrote:
I have a case where an application logs something like
{date} {host} {program}: my first line... ...my second line... ...and my third line.
I want to make a correlation and unwrap these lines into
{date} {host} {program}: my first line my second line and my third line.
I started writing the patterndb to do this, but matching the ... at the end
of the line is difficult, so I used @PCRE:line:(.*)\.\.\.$@
but I then need to only use the $1 to set a value
<values> <value name="mymessage">$1</value> </values>
Would this be the correct syntax to do this?
Is there an easier way that would perform well?
Thanks,
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
N�n�r����)em�h�yhiם�w^��
I see that the complexity of that regex expression would increase hugely if you want to solve. I'm still thinking about other possibilities before focusing on a patterndb solution. What kind of source do you use for that application? Where is it logging to? Gabor On Tue, Mar 20, 2018 at 2:19 PM, Evan Rempel <erempel@uvic.ca> wrote:
No problem about my name. My fast fingers make tonnes of errors.
The application does not log into a file, so that isn't a really good option. I have the patterndb working for this, however, I came across another line that is
... 20 more
and has a continuation line preceding it that does NOT end in ... so I have filter that one out.
Does anyone handle java stack dumps gracefully :-)
Evan
On 03/20/2018 06:07 AM, Nagy, Gábor wrote:
Sorry Evan for mistyping your name. :)
On Tue, Mar 20, 2018 at 2:06 PM, Nagy, Gábor <gabor.nagy@balabit.com> wrote:
Hi Elen!
Does your application log into a file? Because then you could use multi-line file source with a well-defined prefix as the "{date} {host} {program}:".
Regards, Gabor
On Thu, Mar 15, 2018 at 7:10 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
The $1 is not set in this case, you can however use template functions in the value part. E.g. set line based on the @PCRE@ matcher and overwrite its value using an expression $(substr $line 0 -3)
Would that work for you?
On Mar 15, 2018 02:08, "Evan Rempel" <erempel@uvic.ca> wrote:
I have a case where an application logs something like
{date} {host} {program}: my first line... ...my second line... ...and my third line.
I want to make a correlation and unwrap these lines into
{date} {host} {program}: my first line my second line and my third line.
I started writing the patterndb to do this, but matching the ... at the end
of the line is difficult, so I used @PCRE:line:(.*)\.\.\.$@
but I then need to only use the $1 to set a value
<values> <value name="mymessage">$1</value> </values>
Would this be the correct syntax to do this?
Is there an easier way that would perform well?
Thanks,
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
N �n�r����)em�h�yhiם�w^��
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
The source in this case is a fava application logging with log4j2. They log to a syslog tcp socket on the local host. What I have is a java stack trace that looks like. 2018-03-20T00:05:00 briard daemon.err iiq1r: ERROR api.Aggregator - Exception during aggregation. Reason: java.lang.RuntimeException: sailpoint.tools.GeneralException: Errors returned from IQService. The changeToken refers to a time before the start of the current change log. 2018-03-20T00:05:00 briard daemon.err java.lang.RuntimeException: sailpoint.tools.GeneralException: Errors returned from IQService. The changeToken refers to a time before the start of the current change log. 2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.SharePointRWConnector$SharePointIterator.hasNext(SharePointRWConnector.java:700) 2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.ConnectorProxy$CustomizingIterator.peek(ConnectorProxy.java:829) 2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.ConnectorProxy$CustomizingIterator.hasNext(ConnectorProxy.java:856) 2018-03-20T00:05:00 briard daemon.err at sailpoint.api.Aggregator.aggregateAccounts(Aggregator.java:2799) 2018-03-20T00:05:00 briard daemon.err at sailpoint.api.Aggregator.primaryAccountAggregation(Aggregator.java:2498) 2018-03-20T00:05:00 briard daemon.err at sailpoint.api.Aggregator.aggregateApplication(Aggregator.java:2348) 2018-03-20T00:05:00 briard daemon.err at sailpoint.api.Aggregator.phaseAggregate(Aggregator.java:2250) 2018-03-20T00:05:00 briard daemon.err at sailpoint.api.Aggregator.execute(Aggregator.java:1868) 2018-03-20T00:05:00 briard daemon.err at sailpoint.task.ResourceIdentityScan.doUnpartitioned(ResourceIdentityScan.java:219) 2018-03-20T00:05:00 briard daemon.err at sailpoint.task.ResourceIdentityScan.execute(ResourceIdentityScan.java:199) 2018-03-20T00:05:00 briard daemon.err at sailpoint.api.TaskManager.runSync(TaskManager.java:796) 2018-03-20T00:05:00 briard daemon.err at sailpoint.scheduler.JobAdapter.execute(JobAdapter.java:123) 2018-03-20T00:05:00 briard daemon.err at org.quartz.core.JobRunShell.run(JobRunShell.java:202) 2018-03-20T00:05:00 briard daemon.err at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) 2018-03-20T00:05:00 briard daemon.err Caused by: sailpoint.tools.GeneralException: Errors returned from IQService. The changeToken refers to a time before the start of the current change log. 2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.RPCService.checkForErrors(RPCService.java:518) 2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.RPCService.parseResponse(RPCService.java:445) 2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.RPCService.execute(RPCService.java:394) 2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.SharePointRWConnector$SharePointIterator.getNextBlock(SharePointRWConnector.java:608) 2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.SharePointRWConnector$SharePointIterator.hasNext(SharePointRWConnector.java:663) 2018-03-20T00:05:00 briard daemon.err ... 13 more The first line has the application name, and then all of the others are really just part of the multi-linem message. Unfortunately this is arriving on a tcp socket, which does not support multi-line messages. Does log4j2 support syslog protocol? Does log4j2 support json format? That's won't solve my first issue in that the application actually breaks the messages. 2018-03-20T00:00:15 briard daemon.debug iiq1r: DEBUG idam.SyslogStats - syslogEvents,env=preprod,server=boerboel,className=sailpoint.provisioning.PlanEvaluator,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.api.Workflower,eventLevel=WARN count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.provisioning.PlanEvaluator,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.connector.LDAPConnector,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel.comp.uv... 2018-03-20T00:00:15 briard daemon.debug ...ic.ca,className=sailpoint.scheduler.JobAdapter,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.provisioning.PlanEvaluator,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.scheduler.JobAdapter,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.task.Housekeeper$WorkflowerThread,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.provisioning.PlanEvaluator,eventLevel=ERR... 2018-03-20T00:00:15 briard daemon.debug ...OR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.connector.LDAPConnector,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.api.Workflower,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.request.RequestHandler,eventLevel=WARN count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.provisioning.PlanEvaluator,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,classN... 2018-03-20T00:00:15 briard daemon.debug ...ame=sailpoint.provisioning.PlanEvaluator,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.provisioning.PlanEvaluator,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.scheduler.JobAdapter,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.scheduler.JobAdapter,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.provisioning.PlanEvaluator,eventLevel=ERROR count=1 1521529200 I will follow up with out java group to see what options are available to us.. On 03/20/2018 06:56 AM, Nagy, Gábor wrote:
I see that the complexity of that regex expression would increase hugely if you want to solve.
I'm still thinking about other possibilities before focusing on a patterndb solution. What kind of source do you use for that application? Where is it logging to?
Gabor
On Tue, Mar 20, 2018 at 2:19 PM, Evan Rempel <erempel@uvic.ca <mailto:erempel@uvic.ca>> wrote:
No problem about my name. My fast fingers make tonnes of errors.
The application does not log into a file, so that isn't a really good option. I have the patterndb working for this, however, I came across another line that is
... 20 more
and has a continuation line preceding it that does NOT end in ... so I have filter that one out.
Does anyone handle java stack dumps gracefully :-)
Evan
On 03/20/2018 06:07 AM, Nagy, Gábor wrote:
Sorry Evan for mistyping your name. :)
On Tue, Mar 20, 2018 at 2:06 PM, Nagy, Gábor <gabor.nagy@balabit.com <mailto:gabor.nagy@balabit.com>> wrote:
Hi Elen!
Does your application log into a file? Because then you could use multi-line file source with a well-defined prefix as the "{date} {host} {program}:".
Regards, Gabor
On Thu, Mar 15, 2018 at 7:10 AM, Scheidler, Balázs <balazs.scheidler@balabit.com <mailto:balazs.scheidler@balabit.com>> wrote:
The $1 is not set in this case, you can however use template functions in the value part. E.g. set line based on the @PCRE@ matcher and overwrite its value using an expression $(substr $line 0 -3)
Would that work for you?
On Mar 15, 2018 02:08, "Evan Rempel" <erempel@uvic.ca <mailto:erempel@uvic.ca>> wrote:
I have a case where an application logs something like
{date} {host} {program}: my first line... ...my second line... ...and my third line.
I want to make a correlation and unwrap these lines into
{date} {host} {program}: my first line my second line and my third line.
I started writing the patterndb to do this, but matching the ... at the end
of the line is difficult, so I used @PCRE:line:(.*)\.\.\.$@
but I then need to only use the $1 to set a value
<values> <value name="mymessage">$1</value> </values>
Would this be the correct syntax to do this?
Is there an easier way that would perform well?
-- Evan
On Tue, Mar 20, 2018 at 06:19:46AM -0700, Evan Rempel wrote:
Does anyone handle java stack dumps gracefully :-)
The best option for java apps is to configure those to log to JSON. Of course, it all depends on what logging lib they use (logback, SLF4J, log4j2, or any combination of those)
On 03/20/2018 07:19 AM, Fabien Wernli wrote:
On Tue, Mar 20, 2018 at 06:19:46AM -0700, Evan Rempel wrote:
Does anyone handle java stack dumps gracefully :-) The best option for java apps is to configure those to log to JSON. Of course, it all depends on what logging lib they use (logback, SLF4J, log4j2, or any combination of those)
Mostly we use log4j2 Do you have a good config for that library?
On Tue, Mar 20, 2018 at 07:58:54AM -0700, Evan Rempel wrote:
Mostly we use log4j2 Do you have a good config for that library?
Then you can use either JSON: https://stackoverflow.com/questions/41362658/log4j2-how-to-configure-json-la... Or RFC5424! : https://stackoverflow.com/questions/26856447/log4j2-using-syslog-appender-an...
participants (4)
-
Evan Rempel
-
Fabien Wernli
-
Nagy, Gábor
-
Scheidler, Balázs