[syslog-ng] get destination(s) based on log facility and severity

[Gergely Nagy]

Dejan Muhamedagic <dejan at suse.de> writes:

> On Tue, Jun 07, 2011 at 06:10:24PM +0200, Gergely Nagy wrote:
>> Dejan Muhamedagic <dejan at suse.de> writes:
>> 
>> > No. The task is to get the destination (log file) for the given
>> > facility. Right now I have brute force discovery like:
>> >
>> > # logger -p facility.info unique_message
>> > # grep -l unique_message /var/log/*
>> >
>> > I'd like to be able to query syslog-ng and get a list of
>> > destinations.
>> 
>> I do not think that's possible to do, save parsing the config file.
>
> But at some point syslog-ng has to decide where to write the
> message, right?

Yep, pretty much right before writing the message to disk. But since the
destination names can change depending on the message, it's not possible
to list all the exact destinations. Think templates, for example, like
the one below:

destination d_last3 {
  file("/var/log/r/${HOST}/messages-$(% ${DAY} 3)");
};

This basically splits the messages by host, and keeps the last 3 days
worth of logs. Depending on where the message came from, and the current
date, this can go to any number of different files, and there's no way
to tell, until I know the message and the date.

There's no way to reliably tell which files messages would go to, as
syslog-ng does not keep them all open: it closes them when they're idle
(otherwise bad things may happen, like my systems would run out of file
descriptors very fast :P).

(The above only works in syslog-ng 3.4, but similar setups can be done
even with 3.2, it just takes a lot more effort to construct the
template.)

> We'd only need some kind of interface to get that
> information. I don't know if it's possible to query syslog-ng, I
> guess that there's no infrustructure for that right now.

syslog-ng-ctl stats already lists a few things, and there certainly is
infrastructure in place to talk with syslog-ng. It's just that the
information you want to extract cannot be easily extracted, since it can
depend on a lot of things, and vary between messages - or even vary with
time, as I've shown above.

> Would you consider such an extension? Alternatively, one could build a
> new binary for just this purpose which would obviously include
> configuration parsing. Don't know which approach of the two is less
> painful.

The least painful way would be to rethink you setup, in my opinion. It
sounds very awkward to me, and there _must_ be a less painful way to
accomplish what you wish.

>> It may be possible to add some debugging stuff to syslog-ng, that would
>> echo the information you need, but it's not currently present.
>> 
>> I also fail to see why this would be useful, but that might just be my
>> lack of imagination O:)
>
> This is for a reporting tool for clusters. It collects all the
> relevant information from all cluster members and that includes
> excerpts from log files. People use all kinds of syslog setups so
> the tool needs to figure out which log file is relevant.

Wouldn't it be possible to collect the logs with a central syslog-ng,
and tag them appropriately, based on source, and then filter them to the
appropriate files, based on that tag?

Just beat the inputs into a standard-ish shape and then you can do the
rest within syslog-ng.

-- 
|8]