The source in this case is a fava application logging with log4j2.
They log to a syslog tcp socket on the local host.

What I have is a java stack trace that looks like.

2018-03-20T00:05:00 briard daemon.err iiq1r: ERROR api.Aggregator  - Exception during aggregation. Reason: java.lang.RuntimeException: sailpoint.tools.GeneralException: Errors returned from IQService. The changeToken refers to a time before the start of the current change log.
2018-03-20T00:05:00 briard daemon.err java.lang.RuntimeException: sailpoint.tools.GeneralException: Errors returned from IQService. The changeToken refers to a time before the start of the current change log.
2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.SharePointRWConnector$SharePointIterator.hasNext(SharePointRWConnector.java:700)
2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.ConnectorProxy$CustomizingIterator.peek(ConnectorProxy.java:829)
2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.ConnectorProxy$CustomizingIterator.hasNext(ConnectorProxy.java:856)
2018-03-20T00:05:00 briard daemon.err at sailpoint.api.Aggregator.aggregateAccounts(Aggregator.java:2799)
2018-03-20T00:05:00 briard daemon.err at sailpoint.api.Aggregator.primaryAccountAggregation(Aggregator.java:2498)
2018-03-20T00:05:00 briard daemon.err at sailpoint.api.Aggregator.aggregateApplication(Aggregator.java:2348)
2018-03-20T00:05:00 briard daemon.err at sailpoint.api.Aggregator.phaseAggregate(Aggregator.java:2250)
2018-03-20T00:05:00 briard daemon.err at sailpoint.api.Aggregator.execute(Aggregator.java:1868)
2018-03-20T00:05:00 briard daemon.err at sailpoint.task.ResourceIdentityScan.doUnpartitioned(ResourceIdentityScan.java:219)
2018-03-20T00:05:00 briard daemon.err at sailpoint.task.ResourceIdentityScan.execute(ResourceIdentityScan.java:199)
2018-03-20T00:05:00 briard daemon.err at sailpoint.api.TaskManager.runSync(TaskManager.java:796)
2018-03-20T00:05:00 briard daemon.err at sailpoint.scheduler.JobAdapter.execute(JobAdapter.java:123)
2018-03-20T00:05:00 briard daemon.err at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
2018-03-20T00:05:00 briard daemon.err at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
2018-03-20T00:05:00 briard daemon.err Caused by: sailpoint.tools.GeneralException: Errors returned from IQService. The changeToken refers to a time before the start of the current change log.
2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.RPCService.checkForErrors(RPCService.java:518)
2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.RPCService.parseResponse(RPCService.java:445)
2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.RPCService.execute(RPCService.java:394)
2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.SharePointRWConnector$SharePointIterator.getNextBlock(SharePointRWConnector.java:608)
2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.SharePointRWConnector$SharePointIterator.hasNext(SharePointRWConnector.java:663)
2018-03-20T00:05:00 briard daemon.err ... 13 more


The first line has the application name, and then all of the others are really just part of the multi-linem message. Unfortunately this is arriving on a tcp
socket, which does not support multi-line messages.

Does log4j2 support syslog protocol?
Does log4j2 support json format?

That's won't solve my first issue in that the application actually breaks the messages.

2018-03-20T00:00:15 briard daemon.debug iiq1r: DEBUG idam.SyslogStats  - syslogEvents,env=preprod,server=boerboel,className=sailpoint.provisioning.PlanEvaluator,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.api.Workflower,eventLevel=WARN count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.provisioning.PlanEvaluator,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.connector.LDAPConnector,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel.comp.uv...

2018-03-20T00:00:15 briard daemon.debug ...ic.ca,className=sailpoint.scheduler.JobAdapter,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.provisioning.PlanEvaluator,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.scheduler.JobAdapter,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.task.Housekeeper$WorkflowerThread,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.provisioning.PlanEvaluator,eventLevel=ERR...

2018-03-20T00:00:15 briard daemon.debug ...OR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.connector.LDAPConnector,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.api.Workflower,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.request.RequestHandler,eventLevel=WARN count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.provisioning.PlanEvaluator,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,classN...

2018-03-20T00:00:15 briard daemon.debug ...ame=sailpoint.provisioning.PlanEvaluator,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.provisioning.PlanEvaluator,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.scheduler.JobAdapter,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.scheduler.JobAdapter,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.provisioning.PlanEvaluator,eventLevel=ERROR count=1 1521529200


I will follow up with out java group to see what options are available to us..


On 03/20/2018 06:56 AM, Nagy, Gábor wrote:
I see that the complexity of that regex expression would increase hugely if you want to solve.

I'm still thinking about other possibilities before focusing on a patterndb solution.
What kind of source do you use for that application? Where is it logging to?

Gabor


On Tue, Mar 20, 2018 at 2:19 PM, Evan Rempel <erempel@uvic.ca> wrote:
No problem about my name. My fast fingers make tonnes of errors.

The application does not log into a file, so that isn't a really good option.
I have the patterndb working for this, however, I came across another line that is

... 20 more

and has a continuation line preceding it that does NOT end in ... so I have filter
that one out.

Does anyone handle java stack dumps gracefully :-)

Evan


On 03/20/2018 06:07 AM, Nagy, Gábor wrote:
Sorry Evan for mistyping your name. :)

On Tue, Mar 20, 2018 at 2:06 PM, Nagy, Gábor <gabor.nagy@balabit.com> wrote:
Hi Elen!

Does your application log into a file? Because then you could use multi-line file source with a well-defined prefix as the "{date} {host} {program}:".

Regards,
Gabor

On Thu, Mar 15, 2018 at 7:10 AM, Scheidler, Balázs <balazs.scheidler@balabit.com> wrote:
The $1 is not set in this case, you can however use template functions in the value part. E.g. set line based on the @PCRE@ matcher and overwrite its value using an expression $(substr $line 0 -3)

Would that work for you?


On Mar 15, 2018 02:08, "Evan Rempel" <erempel@uvic.ca> wrote:
I have a case where an application logs something like

{date} {host} {program}: my first line...
...my second line...
...and my third line.


I want to make a correlation and unwrap these lines into

{date} {host} {program}: my first line my second line and my third line.


I started writing the patterndb to do this, but matching the ... at the end

of the line is difficult, so I used @PCRE:line:(.*)\.\.\.$@

but I then need to only use the $1 to set a value

<values>
 <value name="mymessage">$1</value>
</values>


Would this be the correct syntax to do this?

Is there an easier way that would perform well?

-- 
Evan