[syslog-ng] Convert logs back into syslog-ng logs

Tue Jul 7 06:39:48 UTC 2020

Hello Mark,
 your setup is really good. If those files are written with the "file" destination without using any templates, than the default format is BSD style syslog and "file" source will also read it without any problem.

 I can only see two possible pitfalls with this setup:

 1. BSD style syslog format (RFC3164) do not handle SDATA (beside other things). So if your logs uses any enrichment, than those will be lost as you stated.
     You can prevent this by changing to RFC5424, by using the "flags(syslog-protocol)" on both the source and destination side.

 2. The file source/destination do not use message framing. So if your logs uses any new line (\n) character (i.e. in case of embedded debug, stack trace messages) it can cause problems with message parsing while reading it back from the files.
    - The easiest way to prevent this problem is to deal with the new line characters before writing those logs into the file. - "flags(no-multi-line)"
    - However simply removing them, might be not good for you. A less fortunate solution will depends on your message context, and uses the "multi-line-*" options (https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.25/administration-guide/19#TOPIC-1349359) of the file source.

Best regards,
Laci

________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Faine, Mark R. (MSFC-IS40)[NICS] <mark.faine at nasa.gov>
Sent: Monday, July 6, 2020 21:04
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: Re: [syslog-ng] Convert logs back into syslog-ng logs

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

syslog-ng is writing the logs originally.   They are gzipped older files.  If we find a gap in the logs we use these backed up logs on a secondary server to fill the gap.  I was hoping to just read them from the file and forward them to the syslog-ng server with the gap.

They come from many different sources from VPNs, firewalls, switches, etc. but I think the format is syslog format.

They all look like this but of course the IP and MSG varies.

Jul  1 01:09:44 IP MSG

These would be gzipped files that are on another server and not being written to any longer.  My concern was that there were originally headers and such that are not necessarily printed in the log file as it is written and they may be important in the processing of the messages.

Thanks,

-Mark

From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> On Behalf Of Laszlo Szemere (lszemere)
Sent: Monday, July 6, 2020 13:39
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: [EXTERNAL] Re: [syslog-ng] Convert logs back into syslog-ng logs

Hello Mark,

 there are many (too many?) options to do this. To avoid "solving a problem that doesn't exist", we should start from your specification. Some helping question:

 - What is writing those files in the first place? (If the soul purpose of those files is to store messages temporary, Syslog-ng has a built in file buffer solution.)

 - What is the format of the individual messages in those files?

 - What is the life cycle of those files. (It is always hazardous if two application is writing and reading the same file at the same time without any synchronization.)

To speed things up:

IF !!! If your file is in syslog format, than a simple "file" source will do the job for you. For further information please read our administration guide: https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.21/administration-guide/18#TOPIC-1180429<https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__www.syslog-2Dng.com_technical-2Ddocuments_doc_syslog-2Dng-2Dopen-2Dsource-2Dedition_3.21_administration-2Dguide_18-23TOPIC-2D1180429%26d%3DDwMFAg%26c%3DApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiMk5Csz9Zk%26r%3DzMyZvtxRXMBKZZYKVMke9zplWK320p3d51BzuU4jwWo%26m%3DsWPK1NMGXp0akTJ1ru-sUVBli50Z1R1VWi2zEt5lWcY%26s%3DhiWjVhnphhXKrqoh5qL182DxTfqNCwEgltKNnA18bqs%26e%3D&data=02%7C01%7CLaszlo.Szemere%40oneidentity.com%7Ca0f64e3c240b446549ff08d821df6d82%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637296590810369694&sdata=lreO0YsUAVXdusZUxDqaYMzGq8lGRn9mn%2FpehxUqUu8%3D&reserved=0>

Best regards,

Laci

________________________________

From: syslog-ng <syslog-ng-bounces at lists.balabit.hu<mailto:syslog-ng-bounces at lists.balabit.hu>> on behalf of Faine, Mark R. (MSFC-IS40)[NICS] <mark.faine at nasa.gov<mailto:mark.faine at nasa.gov>>
Sent: Monday, July 6, 2020 20:23
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
Subject: [syslog-ng] Convert logs back into syslog-ng logs

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

I would like to read lines from a file back into log data that syslog-ng can parse.  What would be involved in doing this?

Thanks,

-Mark

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20200707/4b645347/attachment.html>