[syslog-ng] Issue with Juniper SRX structured stream logs getting parsed by syslog-ng

Wed Jun 13 18:26:17 UTC 2018

Dear Brad,

You should try adding something like this

template("${ISODATE} ${HOST} ${SDATA} ${MESSAGE}\n")

to the file destination's definition.

Thus it would become:

destination syslog_log {
file("/var/log/syslog/$HOST/$FACILITY/$YEAR$MONTH/$YEAR$MONTH$DAY-$FACILITY.log"

template("${ISODATE} ${HOST} ${SDATA} ${MESSAGE}\n")

owner(root)
group(root)
perm(0600)
dir_perm(0700)
create_dirs(yes)
);

Regards,
János

--
Janos SZIGETVARI
RHCE, License no. 150-053-692
<https://www.redhat.com/rhtapps/verify/?certId=150-053-692>

__ at __˚V˚
Make the switch to open (source) applications, protocols, formats now:
- windows -> Linux, iexplore -> Firefox, msoffice -> LibreOffice
- msn -> jabber protocol (Pidgin, Google Talk)
- mp3 -> ogg, wmv -> ogg, jpg -> png, doc/xls/ppt -> odt/ods/odp

Brad Riemann <briemann at nmi.com> ezt írta (időpont: 2018. jún. 13., Sze,
16:59):

> János,
>
> I did, it should be in the gist i sent over:
> https://gist.github.com/robotman321/217a69d456e543831936b1209b8bf501
>
>
> Fabian,
>
> I tried adding  flags(syslog-protocol) to the syslog() object however
> there were no changes in output.
>
> Thanks guys.
> -Brad
>
> On Wed, Jun 13, 2018 at 3:01 AM SZIGETVÁRI János <jszigetvari at gmail.com>
> wrote:
> >
> > Hi Brad,
> >
> > Could you please share with us the destination configuration?
> > I have a hunch that the problem may be that the SRX sends the log data
> totally as SDATA (with the message part being empty), and by default SDATA
> is not written into file destinations, so you only get an empty message
> part written to your file destination.
> > So in order to write the data contained in the SDATA fields to a file,
> you will have to edit the destination's template to (explicitly) include
> those fields as well.
> >
> > Please let me know if my assumptions were correct.
> > Thanks!
> >
> > Regards,
> > János
> > --
> > Janos SZIGETVARI
> > RHCE, License no. 150-053-692
> >
> > LinkedIn: linkedin.com/in/janosszigetvari
> > E-mail: janos at szigetvari.com, jszigetvari at gmail.com
> > Phone: +36209440412 (Hungary)
> >
> > __ at __˚V˚
> > Make the switch to open (source) applications, protocols, formats now:
> > - windows -> Linux, iexplore -> Firefox, msoffice -> LibreOffice
> > - msn -> jabber protocol (Pidgin, Google Talk)
> > - mp3 -> ogg, wmv -> ogg, jpg -> png, doc/xls/ppt -> odt/ods/odp
> >
> >
> > Brad Riemann <briemann at nmi.com> ezt írta (időpont: 2018. jún. 13., Sze,
> 0:10):
> >>
> >> Hello,
> >>
> >> I've been banging my head against the wall for a while with this
> >> issue, I have a Juniper SRX that sends traffic logs to syslog-ng
> >> (centos 6, 6, version 3.13.2-2), the key being they are structured so
> >> the look like the following.
> >>
> >> [2018-06-12T19:03:54.384014] Incoming log entry; line='<14>1
> >> 2018-06-12T19:03:54.125Z corefw-site-0 RT_FLOW - RT_FLOW_SESSION_CLOSE
> >> [junos at 2636.1.1.1.2.40 reason="idle Timeout" source-address="10.1.1.1"
> >> source-port="56998" destination-address="0.0.0.0"
> >> destination-port="443" service-name="junos-https"
> >> nat-source-address="10.100.1.1" nat-source-port="39155"
> >> nat-destination-address="0.0.0.0" nat-destination-port="443"
> >> src-nat-rule-type="source rule" src-nat-rule-name="test-servers"
> >> dst-nat-rule-type="N/A" dst-nat-rule-name="N/A" protocol-id="6"
> >> policy-name="allow-outbound" source-zone-name="test-zone"
> >> destination-zone-name="untrust" session-id-32="499236"
> >> packets-from-client="3" bytes-from-client="180"
> >> packets-from-server="0" bytes-from-server="0" elapsed-time="19"
> >> application="UNKNOWN" nested-application="UNKNOWN" username="N/A"
> >> roles="N/A" packet-incoming-interface="reth0.1" encrypted="UNKNOWN"]'
> >>
> >> I can confirm that syslog-ng is seeing the whole message as part of
> >> the incoming log entry, however it's just not showing up.
> >>
> >> I've dug through RFC 5424 and these logs DO conform but my confusion
> >> still remains. When I check the logs that were written the line looks
> >> like the following, which seems to just trunk everything after the
> >> RT_FLOW of the message header.
> >>
> >> 2018-06-12T19:03:54.125+00:00 corefw-site-0 RT_FLOW:
> >>
> >> I've run syslog-ng in debug mode, which was how I was able to confirm
> >> the outgoing message matches the prior line as well as the incoming
> >> message matching the expected data from my srx.
> >>
> >> [2018-06-12T19:03:54.385505] Outgoing message;
> >> message='2018-06-12T19:03:54.125+00:00 corefw-site-0 RT_FLOW: \x0a'
> >>
> >> Here is a snippet of my syslog config, it isn't that complex (at least
> >> I think so..) and i'm running out of hair to pull, any input would be
> >> GREATLY appreciated.
> >>
> >> https://gist.github.com/robotman321/217a69d456e543831936b1209b8bf501
> >>
> >> -Brad
> >>
> ______________________________________________________________________________
> >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> >> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >>
> >
> ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180613/7ad32eb6/attachment.html>