[syslog-ng] syslog-ng Digest, Vol 137, Issue 11

Szabó, István istvan.szabo at balabit.com
Fri Sep 9 09:43:10 CEST 2016


Hi Scot,

syslog-ng in its premium edition offers logstore:
https://www.balabit.com/documents/syslog-ng-pe-6.0-guides/en/syslog-ng-pe-guide-admin/html-single/index.html#configuring-destinations-logstore

Disclaimer: I'm employed by Balabit (being the product manager of syslog-ng
there)  I don't want to push anything here, the only reason I wrote this is
because I felt there is an exact match with what you required and by
checking the above documentation you can at least see whether this is
indeed the functionality what you are after.

/Istvan


Date: Thu, 8 Sep 2016 21:56:55 -0400
> From: Scot Needy <scotrn at gmail.com>
> Subject: Re: [syslog-ng] Archive module ?
> To: Syslog-ng users' and developers' mailing list
>         <syslog-ng at lists.balabit.hu>
> Message-ID: <3E22C1C9-9275-41F4-A583-F140CAF611A6 at gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
>
> That looks like it would work but if the source isn?t syslog data ?
>
> I pondered this question when looking at the disk buffer settings in 3.8.
> If my data is in ES, MySQL or some other parsed presentation platform then
> all I really need to write out blocks of raw data to disk and archive for
> SOX compliance.
>
> Let the storage archive, compress, encrypt and de-dup.
>
> Just
>
>
> > On Sep 8, 2016, at 5:48 PM, Evan Rempel <erempel at uvic.ca> wrote:
> >
> > You would have to use a syslog-ng template defined as
> >
> > "<$PRI>$ISO_DATE $FULLHOST $MSGHDR$MESSAGE\n"
> >
> > and that would be the "on the wire" format. The you can netcat this to
> the syslog port which would replay the log.
> >
> > Evan.
> >
> > On 09/08/2016 02:39 PM, Scot Needy wrote:
> >> Thanks Even,
> >>
> >>  Maybe I?m doing something wrong.
> >> I see some of the headers but not all.
> >>
> >> 2016-09-08T17:31:17-04:00 <IPADDR> <HOST> %ASA-3-305006:  regular
> translation creation failed for imp src ?.
> >>
> >> I guess what I was looking for is something that I could write blocks
> of data to and read from irregardless of the type of data.
> >> Something almost like tcpdump or buffer that could rotate every hour or
> so and then be used to re-play directly back into syslog-ng.
> >>
> >> Log > YYYY.MM.DD.HH_compressed_data.
> >>
> >>
> >>
> >>
> >>
> >>> On Sep 8, 2016, at 5:13 PM, Evan Rempel <erempel at uvic.ca> <mailto:
> erempel at uvic.ca> wrote:
> >>>
> >>> If you write the syslog data to a file with all of the data, then it
> can be ingested again.
> >>>
> >>> We save the data to the files like
> >>>
> >>> 2016-09-08T00:00:01-07:00 host.name.here mail.info sm-mta[9521]:
> u887019I009521: milter=mimedefang, action=rcpt, continue
> >>>
> >>> and then we have a syslog-ng pattern database that will turn this back
> into an "on the wiire" format that can be sent to any syslog port.
> >>>
> >>> Alternatively, you could save it in the file as the original format,
> and then it can just be sent back to syslog when you need to. Human
> readablility is not quite as good, but if you have the data in splunk or
> whatever, then humans don't need to read the raw files.
> >>>
> >>> Evan.
> >>>
> >>>
> >>> On 09/08/2016 01:32 PM, Scot Needy wrote:
> >>>> Hi All,
> >>>>
> >>>> Is there a better way to archive my syslog data once it?s been
> received and injected into elk,spunk or whatever ?
> >>>>
> >>>>  I see how I can log my data to flat files using date macros then
> compress and archive as normal data but is there a better way to write out
> the data in the same format it was received to facilitate re-injesting ?
> >>>>
> >>>> For example: My syslog data on file does not include the original
> syslog header data .
> >>>>
> >>>> Syslog is just one example.
> >>>>
> >>>>
> >>> ____________________________________________________________
> __________________
> >>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <
> https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> >>> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng <http://www.balabit.com/support/documentation/?
> product=syslog-ng>
> >>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <
> http://www.balabit.com/wiki/syslog-ng-faq>
> >>>
> >> ____________________________________________________________
> __________________
> >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <
> https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> >> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng <http://www.balabit.com/support/documentation/?
> product=syslog-ng>
> >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <
> http://www.balabit.com/wiki/syslog-ng-faq>
> >>
> >
> >
> > --
> > Evan Rempel                                      erempel at uvic.ca
> <mailto:erempel at uvic.ca>
> > Senior Systems Administrator                        250.721.7691
> > Data Centre Services, University Systems, University of Victoria
> > ____________________________________________________________
> __________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: http://lists.balabit.hu/pipermail/syslog-ng/
> attachments/20160908/552cf5bf/attachment.htm
>
> ------------------------------
>
> _______________________________________________
> syslog-ng maillist  -  syslog-ng at lists.balabit.hu
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
>
>
> End of syslog-ng Digest, Vol 137, Issue 11
> ******************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160909/b5e8eafa/attachment-0001.htm 


More information about the syslog-ng mailing list