[syslog-ng] PCRE in patterndb with back substitution

Tue Mar 20 15:10:54 UTC 2018

The source in this case is a fava application logging with log4j2.
They log to a syslog tcp socket on the local host.

What I have is a java stack trace that looks like.

2018-03-20T00:05:00 briard daemon.err iiq1r: ERROR api.Aggregator - Exception during aggregation. Reason: java.lang.RuntimeException: sailpoint.tools.GeneralException: Errors returned from IQService. The changeToken refers to a time before the start of the current change log.
2018-03-20T00:05:00 briard daemon.err java.lang.RuntimeException: sailpoint.tools.GeneralException: Errors returned from IQService. The changeToken refers to a time before the start of the current change log.
2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.SharePointRWConnector$SharePointIterator.hasNext(SharePointRWConnector.java:700)
2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.ConnectorProxy$CustomizingIterator.peek(ConnectorProxy.java:829)
2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.ConnectorProxy$CustomizingIterator.hasNext(ConnectorProxy.java:856)
2018-03-20T00:05:00 briard daemon.err at sailpoint.api.Aggregator.aggregateAccounts(Aggregator.java:2799)
2018-03-20T00:05:00 briard daemon.err at sailpoint.api.Aggregator.primaryAccountAggregation(Aggregator.java:2498)
2018-03-20T00:05:00 briard daemon.err at sailpoint.api.Aggregator.aggregateApplication(Aggregator.java:2348)
2018-03-20T00:05:00 briard daemon.err at sailpoint.api.Aggregator.phaseAggregate(Aggregator.java:2250)
2018-03-20T00:05:00 briard daemon.err at sailpoint.api.Aggregator.execute(Aggregator.java:1868)
2018-03-20T00:05:00 briard daemon.err at sailpoint.task.ResourceIdentityScan.doUnpartitioned(ResourceIdentityScan.java:219)
2018-03-20T00:05:00 briard daemon.err at sailpoint.task.ResourceIdentityScan.execute(ResourceIdentityScan.java:199)
2018-03-20T00:05:00 briard daemon.err at sailpoint.api.TaskManager.runSync(TaskManager.java:796)
2018-03-20T00:05:00 briard daemon.err at sailpoint.scheduler.JobAdapter.execute(JobAdapter.java:123)
2018-03-20T00:05:00 briard daemon.err at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
2018-03-20T00:05:00 briard daemon.err at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
2018-03-20T00:05:00 briard daemon.err Caused by: sailpoint.tools.GeneralException: Errors returned from IQService. The changeToken refers to a time before the start of the current change log.
2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.RPCService.checkForErrors(RPCService.java:518)
2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.RPCService.parseResponse(RPCService.java:445)
2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.RPCService.execute(RPCService.java:394)
2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.SharePointRWConnector$SharePointIterator.getNextBlock(SharePointRWConnector.java:608)
2018-03-20T00:05:00 briard daemon.err at sailpoint.connector.SharePointRWConnector$SharePointIterator.hasNext(SharePointRWConnector.java:663)
2018-03-20T00:05:00 briard daemon.err ... 13 more

The first line has the application name, and then all of the others are really just part of the multi-linem message. Unfortunately this is arriving on a tcp
socket, which does not support multi-line messages.

Does log4j2 support syslog protocol?
Does log4j2 support json format?

That's won't solve my first issue in that the application actually breaks the messages.

2018-03-20T00:00:15 briard daemon.debug iiq1r: DEBUG idam.SyslogStats  - syslogEvents,env=preprod,server=boerboel,className=sailpoint.provisioning.PlanEvaluator,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.api.Workflower,eventLevel=WARN count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.provisioning.PlanEvaluator,eventLevel=ERROR count=1 1521529200 
syslogEvents,env=preprod,server=boerboel,className=sailpoint.connector.LDAPConnector,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel.comp.uv...

2018-03-20T00:00:15 briard daemon.debug ...ic.ca,className=sailpoint.scheduler.JobAdapter,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.provisioning.PlanEvaluator,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.scheduler.JobAdapter,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.task.Housekeeper$WorkflowerThread,eventLevel=ERROR count=1 1521529200 
syslogEvents,env=preprod,server=boerboel,className=sailpoint.provisioning.PlanEvaluator,eventLevel=ERR...

2018-03-20T00:00:15 briard daemon.debug ...OR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.connector.LDAPConnector,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.api.Workflower,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.request.RequestHandler,eventLevel=WARN count=1 1521529200 
syslogEvents,env=preprod,server=boerboel,className=sailpoint.provisioning.PlanEvaluator,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,classN...

2018-03-20T00:00:15 briard daemon.debug ...ame=sailpoint.provisioning.PlanEvaluator,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.provisioning.PlanEvaluator,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.scheduler.JobAdapter,eventLevel=ERROR count=1 1521529200 syslogEvents,env=preprod,server=boerboel,className=sailpoint.scheduler.JobAdapter,eventLevel=ERROR count=1 1521529200 
syslogEvents,env=preprod,server=boerboel,className=sailpoint.provisioning.PlanEvaluator,eventLevel=ERROR count=1 1521529200

I will follow up with out java group to see what options are available to us..

On 03/20/2018 06:56 AM, Nagy, Gábor wrote:
> I see that the complexity of that regex expression would increase hugely if you want to solve.
>
> I'm still thinking about other possibilities before focusing on a patterndb solution.
> What kind of source do you use for that application? Where is it logging to?
>
> Gabor
>
>
> On Tue, Mar 20, 2018 at 2:19 PM, Evan Rempel <erempel at uvic.ca <mailto:erempel at uvic.ca>> wrote:
>
>     No problem about my name. My fast fingers make tonnes of errors.
>
>     The application does not log into a file, so that isn't a really good option.
>     I have the patterndb working for this, however, I came across another line that is
>
>     ... 20 more
>
>     and has a continuation line preceding it that does NOT end in ... so I have filter
>     that one out.
>
>     Does anyone handle java stack dumps gracefully :-)
>
>     Evan
>
>
>     On 03/20/2018 06:07 AM, Nagy, Gábor wrote:
>>     Sorry Evan for mistyping your name. :)
>>
>>     On Tue, Mar 20, 2018 at 2:06 PM, Nagy, Gábor <gabor.nagy at balabit.com <mailto:gabor.nagy at balabit.com>> wrote:
>>
>>         Hi Elen!
>>
>>         Does your application log into a file? Because then you could use multi-line file source with a well-defined prefix as the "{date} {host} {program}:".
>>
>>         Regards,
>>         Gabor
>>
>>         On Thu, Mar 15, 2018 at 7:10 AM, Scheidler, Balázs <balazs.scheidler at balabit.com <mailto:balazs.scheidler at balabit.com>> wrote:
>>
>>             The $1 is not set in this case, you can however use template functions in the value part. E.g. set line based on the @PCRE@ matcher and overwrite its value using an expression $(substr $line 0 -3)
>>
>>             Would that work for you?
>>
>>
>>             On Mar 15, 2018 02:08, "Evan Rempel" <erempel at uvic.ca <mailto:erempel at uvic.ca>> wrote:
>>
>>                 I have a case where an application logs something like
>>
>>                 {date} {host} {program}: my first line...
>>                 ...my second line...
>>                 ...and my third line.
>>
>>
>>                 I want to make a correlation and unwrap these lines into
>>
>>                 {date} {host} {program}: my first line my second line and my third line.
>>
>>
>>                 I started writing the patterndb to do this, but matching the ... at the end
>>
>>                 of the line is difficult, so I used @PCRE:line:(.*)\.\.\.$@
>>
>>                 but I then need to only use the $1 to set a value
>>
>>                 <values>
>>                  <value name="mymessage">$1</value>
>>                 </values>
>>
>>
>>                 Would this be the correct syntax to do this?
>>
>>                 Is there an easier way that would perform well?
>>

-- 
Evan

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180320/63b35e88/attachment-0001.html>