[syslog-ng] Forwarding locally stored RAW messages

[Evan Rempel]

It really depends on the format of your stored message files. If any 
information is missing, then you can't expect to recreate the original 
source messages.

We store the messages in a format defined as

template("<$PRI>$ISODATE $FULLHOST $FACILITY.$LEVEL $MSGHDR$MESSAGE\n");

which looks like

2023-04-13T06:15:01.613-07:00 my.host.name cron.info CROND[662460]: 
(root) CMD (/usr/local/sbin/fscheck)


These messages can be read back with a no-parse option from a named pipe

source s_replay { pipe("/var/log/syslog.pipes/replay" 
log_iw_size(100000) log_fetch_limit(5000) flags(no-parse) ); };

and parsed with a pattern database (attached) with

parser p_rawsyslog {
         db_parser(
file("/usr/local/etc/syslog-ng/patterndb.d/reprocess.xml")
                 inject_mode(internal)
         );
};


log {
         source(s_replay);
         parser(p_rawsyslog);
...
};

Then sent to any destination with templated values from the parsing.

template t_replay     { template("<$pri>$parsedate $parsehost 
$parsemessage\n"); template_escape(no); };

message files can be fed into this by the command

cat log.file.name >> /var/log/syslog.pipes/replay

Hope that helps as a starting point.

Evan

On 2023-04-13 04:45, Dragan Zecevic wrote:
>
> Hi,
> we have syslog-ng Open Source Edition 3.33 and we are storing syslog 
> messages from some systems into log files locally on a partition on 
> syslog-ng server.
> Each day those log files are compressed.
> In order to better analyze some logs if needed we would like to 
> extract some log files and ingest them to SIEM.
>
> Is it possible to make some forwarder that will read these RAW syslog 
> messages from a log file and send them via syslog to SIEM?
>
> Thank you.
>
> Br,
> Dragan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20230413/984dad6b/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: reprocess.xml
Type: text/xml
Size: 47909 bytes
Desc: not available
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20230413/984dad6b/attachment-0001.xml>