[syslog-ng] Buffering AF_UNIX Destination, Batch Post Processing Messages

[Balazs Scheidler]

On Mon, 2010-09-27 at 09:38 -0500, Martin Holste wrote:
> That could definitely be helpful, but I think the big one I'd be
> looking for would be something more basic that fires for a log chain
> when the rollover occurs, such as:
> 
> destination d_file { file("/var/log/messages.$MIN" events( on_rotate(
> exec("/usr/local/bin/messages-file-finished") ) ) ) };
> 
> I'm sure you could replicate this by using your example syntax and
> making sure that your time macro in the destination file name matches
> the pseudo-cron entry, but it seems like that might introduce some
> small issues with synchronization or race conditions.

Yes. The problem with this, that there's no such thing as
rollover. :( syslog-ng keeps expanding the template string to find out
which files to write and then times out files that do not get written to
after time_reap() seconds.

The issue is that files may get closed even in the middle of the minute.

So the only sane way to react to "rollovers" is based on time.

... and there's also an issue with nonsynchronized clocks, so the same
file can be written to when the local time is way past that, therefore
you have to calculate with which timestamps to trust.

In our SSB product for example we tend to use the local time now (e.g.
R_DATE) because of this reason.

> 
> On Mon, Sep 27, 2010 at 9:24 AM, Balazs Scheidler <bazsi at balabit.hu> wrote:
> > On Mon, 2010-09-20 at 14:35 -0500, Martin Holste wrote:
> >> > commit 70e91556b6af8724334443347fd6488745405344
> >> > Author: Balazs Scheidler <bazsi at balabit.hu>
> >> > Date:   Mon Sep 20 17:12:27 2010 +0200
> >> >
> >> >    convertfuncs: new plugin to contain conversion template functions
> >> >
> >> >    The plugin now only contains ipv4-to-int which converts an IPv4 address
> >> >    to a long integer.
> >> >
> >> > Usage:
> >> >
> >> > $(ipv4-to-int $SOURCEIP)
> >> >
> >>
> >> Very cool stuff!
> >>
> >> > Expect a  blog post on this topic, a simple correllation engine is now
> >> > built into patterndb.
> >>
> >> Hm, very interesting, I'll be taking a look.
> >>
> >> Regarding detecting the batches being complete:  It seems a little
> >> inelegant to have a baby-sitter script that looks for an appropriately
> >> named file in a given directory and hoping it's the right buffer.  It
> >> would be really nice if Syslog-NG could execute program() on a file
> >> that has just been written to for the last time.
> >
> > I was thinking about adding "events" to sources/destinations which could
> > invoke 3rd party tools/scripts when something happens.
> >
> > Events could be time based, but other setup/teardown style stuff can
> > come in handy.
> >
> > e.g.
> >
> > destination d_file { file("/var/log/messages.$HOUR"
> >                        events(cron(min(5) hour(*) exec("/usr/local/bin/messages-file-finished"));
> > };
> >
> > Not sure about the syntax though. Also I want it to be able to run processes like tail -f:
> >
> > source s_follow { pipe("/var/run/syslog-ng/tail-pipe"
> >                        events(startup(supervise("/usr/bin/tail -f /var/log/apache.log > /var/run/syslog-ng/tail-pipe"))));
> > };
> >
> > I know that syslog-ng is capable for tailing files, but the point is that there
> > are sometimes complex log systems of various applications, and the only sane interface
> > to them to run a process to tail its otherwise binary logfile. I want syslog-ng to
> > manage these processes.
> >
> >
> > --
> > Bazsi
> >
> > ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.campin.net/syslog-ng/faq.html
> >
> >
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
> 
> 

-- 
Bazsi