The source in this case is a fava application logging with log4j2. They log to a syslog tcp socket on the local host. What I have is a java stack trace that looks like. 2018-03-20T00:05:00 briard daemon.err iiq1r: ERROR api.Aggregator - Exception during aggregation. Reason: java.lang.RuntimeException: sailpoint.tools.GeneralException: Errors returned from IQService. The changeToken refers to a time before the start of the current change log. 2018-03-20T00:05:00 briard daemon.err java.lang.RuntimeException: sailpoint.tools.GeneralException: Errors returned from IQService. The changeToken refers to a time before the start of the current change log. 2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.SharePointRWConnector$SharePointIterator.hasNext(SharePointRWConnector.java:700) 2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.ConnectorProxy$CustomizingIterator.peek(ConnectorProxy.java:829) 2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.ConnectorProxy$CustomizingIterator.hasNext(ConnectorProxy.java:856) 2018-03-20T00:05:00 briard daemon.err at sailpoint.api.Aggregator.aggregateAccounts(Aggregator.java:2799) 2018-03-20T00:05:00 briard daemon.err at sailpoint.api.Aggregator.primaryAccountAggregation(Aggregator.java:2498) 2018-03-20T00:05:00 briard daemon.err at sailpoint.api.Aggregator.aggregateApplication(Aggregator.java:2348) 2018-03-20T00:05:00 briard daemon.err at sailpoint.api.Aggregator.phaseAggregate(Aggregator.java:2250) 2018-03-20T00:05:00 briard daemon.err at sailpoint.api.Aggregator.execute(Aggregator.java:1868) 2018-03-20T00:05:00 briard daemon.err at sailpoint.task.ResourceIdentityScan.doUnpartitioned(ResourceIdentityScan.java:219) 2018-03-20T00:05:00 briard daemon.err at sailpoint.task.ResourceIdentityScan.execute(ResourceIdentityScan.java:199) 2018-03-20T00:05:00 briard daemon.err at sailpoint.api.TaskManager.runSync(TaskManager.java:796) 2018-03-20T00:05:00 briard daemon.err at sailpoint.scheduler.JobAdapter.execute(JobAdapter.java:123) 2018-03-20T00:05:00 briard daemon.err at org.quartz.core.JobRunShell.run(JobRunShell.java:202) 2018-03-20T00:05:00 briard daemon.err at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) 2018-03-20T00:05:00 briard daemon.err Caused by: sailpoint.tools.GeneralException: Errors returned from IQService. The changeToken refers to a time before the start of the current change log. 2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.RPCService.checkForErrors(RPCService.java:518) 2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.RPCService.parseResponse(RPCService.java:445) 2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.RPCService.execute(RPCService.java:394) 2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.SharePointRWConnector$SharePointIterator.getNextBlock(SharePointRWConnector.java:608) 2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.SharePointRWConnector$SharePointIterator.hasNext(SharePointRWConnector.java:663) 2018-03-20T00:05:00 briard daemon.err ... 13 more The first line has the application name, and then all of the others are really just part of the multi-linem message. Unfortunately this is arriving on a tcp socket, which does not support multi-line messages. Does log4j2 support syslog protocol? Does log4j2 support json format? That's won't solve my first issue in that the application actually breaks the messages. 2018-03-20T00:00:15 briard daemon.debug iiq1r: DEBUG idam.SyslogStats - syslogEvents,env=preprod,server=boerboel,className=sailpoint.provisioning.PlanEvaluator,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.api.Workflower,eventLevel=WARN count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.provisioning.PlanEvaluator,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.connector.LDAPConnector,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel.comp.uv... 2018-03-20T00:00:15 briard daemon.debug ...ic.ca,className=sailpoint.scheduler.JobAdapter,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.provisioning.PlanEvaluator,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.scheduler.JobAdapter,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.task.Housekeeper$WorkflowerThread,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.provisioning.PlanEvaluator,eventLevel=ERR... 2018-03-20T00:00:15 briard daemon.debug ...OR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.connector.LDAPConnector,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.api.Workflower,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.request.RequestHandler,eventLevel=WARN count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.provisioning.PlanEvaluator,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,classN... 2018-03-20T00:00:15 briard daemon.debug ...ame=sailpoint.provisioning.PlanEvaluator,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.provisioning.PlanEvaluator,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.scheduler.JobAdapter,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.scheduler.JobAdapter,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.provisioning.PlanEvaluator,eventLevel=ERROR count=1 1521529200 I will follow up with out java group to see what options are available to us.. On 03/20/2018 06:56 AM, Nagy, Gábor wrote:
I see that the complexity of that regex expression would increase hugely if you want to solve.
I'm still thinking about other possibilities before focusing on a patterndb solution. What kind of source do you use for that application? Where is it logging to?
Gabor
On Tue, Mar 20, 2018 at 2:19 PM, Evan Rempel <erempel@uvic.ca <mailto:erempel@uvic.ca>> wrote:
No problem about my name. My fast fingers make tonnes of errors.
The application does not log into a file, so that isn't a really good option. I have the patterndb working for this, however, I came across another line that is
... 20 more
and has a continuation line preceding it that does NOT end in ... so I have filter that one out.
Does anyone handle java stack dumps gracefully :-)
Evan
On 03/20/2018 06:07 AM, Nagy, Gábor wrote:
Sorry Evan for mistyping your name. :)
On Tue, Mar 20, 2018 at 2:06 PM, Nagy, Gábor <gabor.nagy@balabit.com <mailto:gabor.nagy@balabit.com>> wrote:
Hi Elen!
Does your application log into a file? Because then you could use multi-line file source with a well-defined prefix as the "{date} {host} {program}:".
Regards, Gabor
On Thu, Mar 15, 2018 at 7:10 AM, Scheidler, Balázs <balazs.scheidler@balabit.com <mailto:balazs.scheidler@balabit.com>> wrote:
The $1 is not set in this case, you can however use template functions in the value part. E.g. set line based on the @PCRE@ matcher and overwrite its value using an expression $(substr $line 0 -3)
Would that work for you?
On Mar 15, 2018 02:08, "Evan Rempel" <erempel@uvic.ca <mailto:erempel@uvic.ca>> wrote:
I have a case where an application logs something like
{date} {host} {program}: my first line... ...my second line... ...and my third line.
I want to make a correlation and unwrap these lines into
{date} {host} {program}: my first line my second line and my third line.
I started writing the patterndb to do this, but matching the ... at the end
of the line is difficult, so I used @PCRE:line:(.*)\.\.\.$@
but I then need to only use the $1 to set a value
<values> <value name="mymessage">$1</value> </values>
Would this be the correct syntax to do this?
Is there an easier way that would perform well?
-- Evan