[syslog-ng] An External Application For Parsing Existing Syslog-ng Log File

[Laszlo Szemere (lszemere)]

Dear Sezer,

 I am in a little bit of trouble with your email. I can not clearly understand your goals, and don't know anything about your current environment. I will add some notes to your email, hoping that it will trigger some new ideas for you.

Please imagine an application which is capable of parsing and displaying syslog-ng log messages which are written in the order given as template below:
${ISODATE} ${HOST} ${FACILITY} ${LEVEL} ${LEVEL_NUM} ${TAGS} ${MESSAGE}
Syslog-ng is parsing the log messages in the first place. So rather than parsing them again, it is more convenient to use some structured format as an output (like json) if your intention is to forward those log messages for further processing.

2) Assume that you have a syslog-ng log file and you want to parse this file manually, is there any known tool/ibrary for this job?
If you CAN'T change the output template: I found Python very good at text processing. But if you are more familiar with other programming languages, I would recommend to use those instead of learning a new one.
If you CAN change the output format: see my notes under question 3.

3) Do you suggest using Database functionality of syslog-ng to save logs in DB to make parsing easier later on?
Parsing means, that you want to extract some information from the raw data you have.
A) There are common applications for this job, I recommend the "SIEM" search term as a starting point. Syslog-ng supports many of those out of the box.
B) If you want to do this processing by yourself: Using a DB is just an another "structured data format", like json or csv, etc.. I recommend to approach this problem from the processing end, not from the syslog-ng output. Try to define what do you want to extract from the raw data, select the most convenient programming language / application for the task, it will determine the data format automatically.

Fun note: If you select approach B, and you are familiar with the Python language, you don't have to write those log messages into files at all. Syslog-ng has a "Python destination" (Péter wrote a blog post about it: https://www.syslog-ng.com/community/b/blog/posts/python-destination-getting-started), where your own code can process the log messages on the fly.


Best regards,
Laci


Laszlo Szemere
Software Engineer

Quest | One Identity | Balabit
<mailto:laszlo.szemere at oneidentity.com>
<mailto:laszlo.szemere at oneidentity.com>laszlo.szemere at oneidentity.com<mailto:laszlo.szemere at oneidentity.com>

Address: 1117 Budapest, Alíz utca 2. (Balabit-Europe Kft.)


________________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Sezer BAGLAN <sezerbaglan at gmail.com>
Sent: Wednesday, June 12, 2019 10:59
To: syslog-ng at lists.balabit.hu
Subject: [syslog-ng] An External Application For Parsing Existing Syslog-ng     Log File

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

Dear all,

Please imagine an application which is capable of parsing and displaying syslog-ng log messages which are written in the order given as template below:
${ISODATE} ${HOST} ${FACILITY} ${LEVEL} ${LEVEL_NUM} ${TAGS} ${MESSAGE}

1) What is your opinion about the best practices to handle this issue? Please note that I'm not talking about logging messages with filters and parsers. I'm talking about parsing an existing syslog-ng log file.

2) Assume that you have a syslog-ng log file and you want to parse this file manually, is there any known tool/ibrary for this job?

3) Do you suggest using Database functionality of syslog-ng to save logs in DB to make parsing easier later on?

Thanks in advance.

Kind regards,
Sezer




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20190618/1a262366/attachment.html>