[syslog-ng] get destination(s) based on log facility and severity

[Dejan Muhamedagic]

On Wed, Jun 08, 2011 at 11:52:46AM +0200, Gergely Nagy wrote:
> Dejan Muhamedagic <dejan at suse.de> writes:
> 
> > On Tue, Jun 07, 2011 at 06:10:24PM +0200, Gergely Nagy wrote:
> >> Dejan Muhamedagic <dejan at suse.de> writes:
> >> 
> >> > No. The task is to get the destination (log file) for the given
> >> > facility. Right now I have brute force discovery like:
> >> >
> >> > # logger -p facility.info unique_message
> >> > # grep -l unique_message /var/log/*
> >> >
> >> > I'd like to be able to query syslog-ng and get a list of
> >> > destinations.
> >> 
> >> I do not think that's possible to do, save parsing the config file.
> >
> > But at some point syslog-ng has to decide where to write the
> > message, right?
> 
> Yep, pretty much right before writing the message to disk. But since the
> destination names can change depending on the message, it's not possible
> to list all the exact destinations. Think templates, for example, like
> the one below:
> 
> destination d_last3 {
>   file("/var/log/r/${HOST}/messages-$(% ${DAY} 3)");
> };
> 
> This basically splits the messages by host, and keeps the last 3 days
> worth of logs. Depending on where the message came from, and the current
> date, this can go to any number of different files, and there's no way
> to tell, until I know the message and the date.

Didn't even know that this was possible. BTW, SLE11SP1 runs 2.0.9.

> There's no way to reliably tell which files messages would go to, as
> syslog-ng does not keep them all open: it closes them when they're idle
> (otherwise bad things may happen, like my systems would run out of file
> descriptors very fast :P).
> 
> (The above only works in syslog-ng 3.4, but similar setups can be done
> even with 3.2, it just takes a lot more effort to construct the
> template.)
> 
> > We'd only need some kind of interface to get that
> > information. I don't know if it's possible to query syslog-ng, I
> > guess that there's no infrustructure for that right now.
> 
> syslog-ng-ctl stats already lists a few things, and there certainly is
> infrastructure in place to talk with syslog-ng.

Great!

> It's just that the
> information you want to extract cannot be easily extracted, since it can
> depend on a lot of things, and vary between messages - or even vary with
> time, as I've shown above.
> 
> > Would you consider such an extension? Alternatively, one could build a
> > new binary for just this purpose which would obviously include
> > configuration parsing. Don't know which approach of the two is less
> > painful.
> 
> The least painful way would be to rethink you setup, in my opinion. It
> sounds very awkward to me, and there _must_ be a less painful way to
> accomplish what you wish.

The setup is not mine :) It depends mainly on the distribution.
SUSE usually sends everything to /var/log/messages, Debian (I
think) to /var/log/syslog or .../daemon, and then there are
shops with /var/log/ha-log. I think that these are the most
common configurations. But nothing is stopping them to devise
their own and it would be difficult to enforce unified logging
configuration (I think actually impossible).

> >> It may be possible to add some debugging stuff to syslog-ng, that would
> >> echo the information you need, but it's not currently present.
> >> 
> >> I also fail to see why this would be useful, but that might just be my
> >> lack of imagination O:)
> >
> > This is for a reporting tool for clusters. It collects all the
> > relevant information from all cluster members and that includes
> > excerpts from log files. People use all kinds of syslog setups so
> > the tool needs to figure out which log file is relevant.
> 
> Wouldn't it be possible to collect the logs with a central syslog-ng,
> and tag them appropriately, based on source, and then filter them to the
> appropriate files, based on that tag?

See above. Whatever the user does, we should make best effort to
find the file where the messages end. Of course, if the setup is
really awkward, then no cigar, but we have to support
"reasonable" configurations (for some values of reasonable :)

I'd be happy to get just the destination before message parsing,
i.e. depending on just facility.severity. I suppose that that
would be easy to do with syslog-ng-ctl?

Cheers,

Dejan

> Just beat the inputs into a standard-ish shape and then you can do the
> rest within syslog-ng.
> 
> -- 
> |8]
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>