<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Thanks Evan I will try your advice and log directive. <div class=""><br class=""></div><div class=""><div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Sep 8, 2016, at 10:31 PM, Evan Rempel <<a href="mailto:erempel@uvic.ca" class="">erempel@uvic.ca</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta content="text/html; charset=utf-8" http-equiv="Content-Type" class="">
<div bgcolor="#FFFFFF" text="#000000" class="">
<div class="moz-cite-prefix">Your original question was "Is there a
better way to archive my syslog data", but attempting to answer
your question anyway.<br class="">
<br class="">
If the source is not syslog data, but you are using syslog-ng to
read the data and place it into some storage platform (splunk,
database or elasticsearch), then by definition it is syslog data.
There will be default program, facility, level on each event.
Writing this data to a file with my suggested format will allow
for replaying of the log data. For the non-syslog sources, the
replay of the syslog stream is functionally equivalent to
re-reading the source data. It is just an intermediate format of
the data.<br class="">
<br class="">
This is not limited to syslog data. It can be applied to any time
series event style data. This covers all logging data and also
audit data (audit "log" really is "log" data).<br class="">
<br class="">
The challenge with using blocks of raw data is that you require
the SAME storage engine when replaying your logs. Even different
versions of the storage engine may have different raw formats.
using a text file for archiving would allow replaying the logs
even years later when a different storage and presentation
framework is used. We have had to reply logs as far back as 7
years for audit reporting.<br class="">
<br class="">
I hope this provides some deeper rationale for the text file
format.<br class="">
<br class="">
Evan.<br class="">
<br class="">
On 09/08/2016 06:56 PM, Scot Needy wrote:<br class="">
</div>
<blockquote cite="mid:3E22C1C9-9275-41F4-A583-F140CAF611A6@gmail.com" type="cite" class="">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" class="">
<div class=""><br class="">
</div>
<div class="">That looks like it would work but if the source
isn’t syslog data ? </div>
<div class=""><br class="">
</div>
<div class="">I pondered this question when looking at the disk
buffer settings in 3.8. </div>
<div class="">If my data is in ES, MySQL or some other parsed
presentation platform then all I really need to write out blocks
of raw data to disk and archive for SOX compliance. </div>
<div class=""><br class="">
</div>
<div class="">Let the storage archive, compress, encrypt and
de-dup. </div>
<div class=""><br class="">
</div>
<div class="">Just </div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">
<blockquote type="cite" class="">
<div class="">On Sep 8, 2016, at 5:48 PM, Evan Rempel <<a moz-do-not-send="true" href="mailto:erempel@uvic.ca" class=""></a><a class="moz-txt-link-abbreviated" href="mailto:erempel@uvic.ca">erempel@uvic.ca</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div bgcolor="#FFFFFF" text="#000000" class="">
<div class="moz-cite-prefix">You would have to use a
syslog-ng template defined as<br class="">
<br class="">
"<$PRI>$ISO_DATE $FULLHOST $MSGHDR$MESSAGE\n"<br class="">
<br class="">
and that would be the "on the wire" format. The you can
netcat this to the syslog port which would replay the
log.<br class="">
<br class="">
Evan.<br class="">
<br class="">
On 09/08/2016 02:39 PM, Scot Needy wrote:<br class="">
</div>
<blockquote cite="mid:B148DE0A-1A67-4D21-AD1B-36C78F23C013@gmail.com" type="cite" class="">
<pre class="" wrap="">Thanks Even,
Maybe I’m doing something wrong.
I see some of the headers but not all.
2016-09-08T17:31:17-04:00 <IPADDR> <HOST> %ASA-3-305006: regular translation creation failed for imp src ….
I guess what I was looking for is something that I could write blocks of data to and read from irregardless of the type of data.
Something almost like tcpdump or buffer that could rotate every hour or so and then be used to re-play directly back into syslog-ng.
Log > YYYY.MM.DD.HH_compressed_data.
</pre>
<blockquote type="cite" class="">
<pre class="" wrap="">On Sep 8, 2016, at 5:13 PM, Evan Rempel <a moz-do-not-send="true" class="moz-txt-link-rfc2396E" href="mailto:erempel@uvic.ca"><erempel@uvic.ca></a> wrote:
If you write the syslog data to a file with all of the data, then it can be ingested again.
We save the data to the files like
2016-09-08T00:00:01-07:00 host.name.here <a moz-do-not-send="true" href="http://mail.info/" class="">mail.info</a> sm-mta[9521]: u887019I009521: milter=mimedefang, action=rcpt, continue
and then we have a syslog-ng pattern database that will turn this back into an "on the wiire" format that can be sent to any syslog port.
Alternatively, you could save it in the file as the original format, and then it can just be sent back to syslog when you need to. Human readablility is not quite as good, but if you have the data in splunk or whatever, then humans don't need to read the raw files.
Evan.
On 09/08/2016 01:32 PM, Scot Needy wrote:
</pre>
<blockquote type="cite" class="">
<pre class="" wrap="">Hi All,
Is there a better way to archive my syslog data once it’s been received and injected into elk,spunk or whatever ?
I see how I can log my data to flat files using date macros then compress and archive as normal data but is there a better way to write out the data in the same format it was received to facilitate re-injesting ?
For example: My syslog data on file does not include the original syslog header data .
Syslog is just one example.
</pre>
</blockquote>
</blockquote>
</blockquote>
</div>
</div>
</blockquote>
</div>
</blockquote>
<br class="">
</div>
______________________________________________________________________________<br class="">Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" class="">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br class="">Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" class="">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br class="">FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" class="">http://www.balabit.com/wiki/syslog-ng-faq</a><br class=""><br class=""></div></blockquote></div><br class=""></div></div></body></html>