[syslog-ng] Convert logs back into syslog-ng logs

[Faine, Mark R. (MSFC-IS40)[NICS]]

syslog-ng is writing the logs originally.   They are gzipped older files.  If we find a gap in the logs we use these backed up logs on a secondary server to fill the gap.  I was hoping to just read them from the file and forward them to the syslog-ng server with the gap.

They come from many different sources from VPNs, firewalls, switches, etc. but I think the format is syslog format.

They all look like this but of course the IP and MSG varies.
Jul  1 01:09:44 IP MSG

These would be gzipped files that are on another server and not being written to any longer.  My concern was that there were originally headers and such that are not necessarily printed in the log file as it is written and they may be important in the processing of the messages.

Thanks,
-Mark

From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> On Behalf Of Laszlo Szemere (lszemere)
Sent: Monday, July 6, 2020 13:39
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: [EXTERNAL] Re: [syslog-ng] Convert logs back into syslog-ng logs

Hello Mark,
 there are many (too many?) options to do this. To avoid "solving a problem that doesn't exist", we should start from your specification. Some helping question:

 - What is writing those files in the first place? (If the soul purpose of those files is to store messages temporary, Syslog-ng has a built in file buffer solution.)
 - What is the format of the individual messages in those files?
 - What is the life cycle of those files. (It is always hazardous if two application is writing and reading the same file at the same time without any synchronization.)


To speed things up:
IF !!! If your file is in syslog format, than a simple "file" source will do the job for you. For further information please read our administration guide: https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.21/administration-guide/18#TOPIC-1180429<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.syslog-2Dng.com_technical-2Ddocuments_doc_syslog-2Dng-2Dopen-2Dsource-2Dedition_3.21_administration-2Dguide_18-23TOPIC-2D1180429&d=DwMFAg&c=ApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiMk5Csz9Zk&r=zMyZvtxRXMBKZZYKVMke9zplWK320p3d51BzuU4jwWo&m=sWPK1NMGXp0akTJ1ru-sUVBli50Z1R1VWi2zEt5lWcY&s=hiWjVhnphhXKrqoh5qL182DxTfqNCwEgltKNnA18bqs&e=>


Best regards,
Laci


________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu<mailto:syslog-ng-bounces at lists.balabit.hu>> on behalf of Faine, Mark R. (MSFC-IS40)[NICS] <mark.faine at nasa.gov<mailto:mark.faine at nasa.gov>>
Sent: Monday, July 6, 2020 20:23
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
Subject: [syslog-ng] Convert logs back into syslog-ng logs

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.


I would like to read lines from a file back into log data that syslog-ng can parse.  What would be involved in doing this?



Thanks,

-Mark


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20200706/315df1cc/attachment-0001.html>