get destination(s) based on log facility and severity
Hi, We have a kind of machinery which collects syslog logs from several hosts. Currently it works by sending a message using logger(1) to the given facility.severity and then looking up the message in log files in /var/log and a few other directories (using grep). I'm looking for a more robust and faster way to discover the destination. Would that be possible with syslog-ng? And which is the optimal way to achieve that? An extra syslog-ng option? An extra binary? Or perhaps using logger(1)? Cheers, Dejan
Hello again, I'd just like to give a gently nudge, hoping that somebody can comment. Is it out of scope, or plain silly, or just boring :) Cheers, Dejan On Wed, May 04, 2011 at 06:43:41PM +0200, Dejan Muhamedagic wrote:
Hi,
We have a kind of machinery which collects syslog logs from several hosts. Currently it works by sending a message using logger(1) to the given facility.severity and then looking up the message in log files in /var/log and a few other directories (using grep).
I'm looking for a more robust and faster way to discover the destination. Would that be possible with syslog-ng? And which is the optimal way to achieve that? An extra syslog-ng option? An extra binary? Or perhaps using logger(1)?
Cheers,
Dejan ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Dejan Muhamedagic <dejan@suse.de> writes:
We have a kind of machinery which collects syslog logs from several hosts. Currently it works by sending a message using logger(1) to the given facility.severity and then looking up the message in log files in /var/log and a few other directories (using grep).
I'm looking for a more robust and faster way to discover the destination. Would that be possible with syslog-ng? And which is the optimal way to achieve that? An extra syslog-ng option? An extra binary? Or perhaps using logger(1)?
I don't quite get the question, I'm afraid. So, lets clarify what you have, and what you want to accomplish: If I understood you correctly, you have a set of hosts sending logs to wherever, and every hosts' messages end up in the same file (depending on facility.severity), and you want to split them by host? -- |8]
On Tue, Jun 07, 2011 at 05:38:07PM +0200, Gergely Nagy wrote:
Dejan Muhamedagic <dejan@suse.de> writes:
We have a kind of machinery which collects syslog logs from several hosts. Currently it works by sending a message using logger(1) to the given facility.severity and then looking up the message in log files in /var/log and a few other directories (using grep).
I'm looking for a more robust and faster way to discover the destination. Would that be possible with syslog-ng? And which is the optimal way to achieve that? An extra syslog-ng option? An extra binary? Or perhaps using logger(1)?
I don't quite get the question, I'm afraid.
So, lets clarify what you have, and what you want to accomplish:
If I understood you correctly, you have a set of hosts sending logs to wherever, and every hosts' messages end up in the same file (depending on facility.severity), and you want to split them by host?
No. The task is to get the destination (log file) for the given facility. Right now I have brute force discovery like: # logger -p facility.info unique_message # grep -l unique_message /var/log/* I'd like to be able to query syslog-ng and get a list of destinations. Cheers, Dejan
-- |8] ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Dejan Muhamedagic <dejan@suse.de> writes:
No. The task is to get the destination (log file) for the given facility. Right now I have brute force discovery like:
# logger -p facility.info unique_message # grep -l unique_message /var/log/*
I'd like to be able to query syslog-ng and get a list of destinations.
I do not think that's possible to do, save parsing the config file. It may be possible to add some debugging stuff to syslog-ng, that would echo the information you need, but it's not currently present. I also fail to see why this would be useful, but that might just be my lack of imagination O:) -- |8]
On Tue, Jun 07, 2011 at 06:10:24PM +0200, Gergely Nagy wrote:
Dejan Muhamedagic <dejan@suse.de> writes:
No. The task is to get the destination (log file) for the given facility. Right now I have brute force discovery like:
# logger -p facility.info unique_message # grep -l unique_message /var/log/*
I'd like to be able to query syslog-ng and get a list of destinations.
I do not think that's possible to do, save parsing the config file.
But at some point syslog-ng has to decide where to write the message, right? We'd only need some kind of interface to get that information. I don't know if it's possible to query syslog-ng, I guess that there's no infrustructure for that right now. Would you consider such an extension? Alternatively, one could build a new binary for just this purpose which would obviously include configuration parsing. Don't know which approach of the two is less painful.
It may be possible to add some debugging stuff to syslog-ng, that would echo the information you need, but it's not currently present.
I also fail to see why this would be useful, but that might just be my lack of imagination O:)
This is for a reporting tool for clusters. It collects all the relevant information from all cluster members and that includes excerpts from log files. People use all kinds of syslog setups so the tool needs to figure out which log file is relevant. Cheers, Dejan
-- |8] ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
This is for a reporting tool for clusters. It collects all the relevant information from all cluster members and that includes excerpts from log files. People use all kinds of syslog setups so the tool needs to figure out which log file is relevant.
It sounds to me like you are trying to work around a poorly configured environment. You would do much better in the long run to harmonise the logging configuration of your servers so that they are identical, or at least very similar. Have you considered simply using a central logging server and adding something to the configuration on each of the clients to send the logs which you are interested in to that central server? That would be the typical way of achieving what you describe. Regards, Scott
Hi Scott, On Wed, Jun 08, 2011 at 11:29:33AM +1100, Scott Rochford wrote:
This is for a reporting tool for clusters. It collects all the relevant information from all cluster members and that includes excerpts from log files. People use all kinds of syslog setups so the tool needs to figure out which log file is relevant.
It sounds to me like you are trying to work around a poorly configured environment. You would do much better in the long run to harmonise the logging configuration of your servers so that they are identical, or at least very similar.
Well, I wish that would be possible. If you figure out how to make humanity more uniform, let me know ;-) Seriously though, it's amazing the number of different (and most interesting) setups one can see.
Have you considered simply using a central logging server and adding something to the configuration on each of the clients to send the logs which you are interested in to that central server? That would be the typical way of achieving what you describe.
For me it would also be the first option to use a log host, but for whatever reason almost nobody's doing that. At least judging by the traffic on our ML. Of course, in that case too people use varying facilities and destinations. Cheers, Dejan
Regards,
Scott
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Dejan Muhamedagic <dejan@suse.de> writes:
On Tue, Jun 07, 2011 at 06:10:24PM +0200, Gergely Nagy wrote:
Dejan Muhamedagic <dejan@suse.de> writes:
No. The task is to get the destination (log file) for the given facility. Right now I have brute force discovery like:
# logger -p facility.info unique_message # grep -l unique_message /var/log/*
I'd like to be able to query syslog-ng and get a list of destinations.
I do not think that's possible to do, save parsing the config file.
But at some point syslog-ng has to decide where to write the message, right?
Yep, pretty much right before writing the message to disk. But since the destination names can change depending on the message, it's not possible to list all the exact destinations. Think templates, for example, like the one below: destination d_last3 { file("/var/log/r/${HOST}/messages-$(% ${DAY} 3)"); }; This basically splits the messages by host, and keeps the last 3 days worth of logs. Depending on where the message came from, and the current date, this can go to any number of different files, and there's no way to tell, until I know the message and the date. There's no way to reliably tell which files messages would go to, as syslog-ng does not keep them all open: it closes them when they're idle (otherwise bad things may happen, like my systems would run out of file descriptors very fast :P). (The above only works in syslog-ng 3.4, but similar setups can be done even with 3.2, it just takes a lot more effort to construct the template.)
We'd only need some kind of interface to get that information. I don't know if it's possible to query syslog-ng, I guess that there's no infrustructure for that right now.
syslog-ng-ctl stats already lists a few things, and there certainly is infrastructure in place to talk with syslog-ng. It's just that the information you want to extract cannot be easily extracted, since it can depend on a lot of things, and vary between messages - or even vary with time, as I've shown above.
Would you consider such an extension? Alternatively, one could build a new binary for just this purpose which would obviously include configuration parsing. Don't know which approach of the two is less painful.
The least painful way would be to rethink you setup, in my opinion. It sounds very awkward to me, and there _must_ be a less painful way to accomplish what you wish.
It may be possible to add some debugging stuff to syslog-ng, that would echo the information you need, but it's not currently present.
I also fail to see why this would be useful, but that might just be my lack of imagination O:)
This is for a reporting tool for clusters. It collects all the relevant information from all cluster members and that includes excerpts from log files. People use all kinds of syslog setups so the tool needs to figure out which log file is relevant.
Wouldn't it be possible to collect the logs with a central syslog-ng, and tag them appropriately, based on source, and then filter them to the appropriate files, based on that tag? Just beat the inputs into a standard-ish shape and then you can do the rest within syslog-ng. -- |8]
On Wed, Jun 08, 2011 at 11:52:46AM +0200, Gergely Nagy wrote:
Dejan Muhamedagic <dejan@suse.de> writes:
On Tue, Jun 07, 2011 at 06:10:24PM +0200, Gergely Nagy wrote:
Dejan Muhamedagic <dejan@suse.de> writes:
No. The task is to get the destination (log file) for the given facility. Right now I have brute force discovery like:
# logger -p facility.info unique_message # grep -l unique_message /var/log/*
I'd like to be able to query syslog-ng and get a list of destinations.
I do not think that's possible to do, save parsing the config file.
But at some point syslog-ng has to decide where to write the message, right?
Yep, pretty much right before writing the message to disk. But since the destination names can change depending on the message, it's not possible to list all the exact destinations. Think templates, for example, like the one below:
destination d_last3 { file("/var/log/r/${HOST}/messages-$(% ${DAY} 3)"); };
This basically splits the messages by host, and keeps the last 3 days worth of logs. Depending on where the message came from, and the current date, this can go to any number of different files, and there's no way to tell, until I know the message and the date.
Didn't even know that this was possible. BTW, SLE11SP1 runs 2.0.9.
There's no way to reliably tell which files messages would go to, as syslog-ng does not keep them all open: it closes them when they're idle (otherwise bad things may happen, like my systems would run out of file descriptors very fast :P).
(The above only works in syslog-ng 3.4, but similar setups can be done even with 3.2, it just takes a lot more effort to construct the template.)
We'd only need some kind of interface to get that information. I don't know if it's possible to query syslog-ng, I guess that there's no infrustructure for that right now.
syslog-ng-ctl stats already lists a few things, and there certainly is infrastructure in place to talk with syslog-ng.
Great!
It's just that the information you want to extract cannot be easily extracted, since it can depend on a lot of things, and vary between messages - or even vary with time, as I've shown above.
Would you consider such an extension? Alternatively, one could build a new binary for just this purpose which would obviously include configuration parsing. Don't know which approach of the two is less painful.
The least painful way would be to rethink you setup, in my opinion. It sounds very awkward to me, and there _must_ be a less painful way to accomplish what you wish.
The setup is not mine :) It depends mainly on the distribution. SUSE usually sends everything to /var/log/messages, Debian (I think) to /var/log/syslog or .../daemon, and then there are shops with /var/log/ha-log. I think that these are the most common configurations. But nothing is stopping them to devise their own and it would be difficult to enforce unified logging configuration (I think actually impossible).
It may be possible to add some debugging stuff to syslog-ng, that would echo the information you need, but it's not currently present.
I also fail to see why this would be useful, but that might just be my lack of imagination O:)
This is for a reporting tool for clusters. It collects all the relevant information from all cluster members and that includes excerpts from log files. People use all kinds of syslog setups so the tool needs to figure out which log file is relevant.
Wouldn't it be possible to collect the logs with a central syslog-ng, and tag them appropriately, based on source, and then filter them to the appropriate files, based on that tag?
See above. Whatever the user does, we should make best effort to find the file where the messages end. Of course, if the setup is really awkward, then no cigar, but we have to support "reasonable" configurations (for some values of reasonable :) I'd be happy to get just the destination before message parsing, i.e. depending on just facility.severity. I suppose that that would be easy to do with syslog-ng-ctl? Cheers, Dejan
Just beat the inputs into a standard-ish shape and then you can do the rest within syslog-ng.
-- |8] ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Dejan Muhamedagic <dejan@suse.de> writes:
This basically splits the messages by host, and keeps the last 3 days worth of logs. Depending on where the message came from, and the current date, this can go to any number of different files, and there's no way to tell, until I know the message and the date.
Didn't even know that this was possible. BTW, SLE11SP1 runs 2.0.9.
That's some ancient version there :)
It's just that the information you want to extract cannot be easily extracted, since it can depend on a lot of things, and vary between messages - or even vary with time, as I've shown above.
Would you consider such an extension? Alternatively, one could build a new binary for just this purpose which would obviously include configuration parsing. Don't know which approach of the two is less painful.
The least painful way would be to rethink you setup, in my opinion. It sounds very awkward to me, and there _must_ be a less painful way to accomplish what you wish.
The setup is not mine :) It depends mainly on the distribution. SUSE usually sends everything to /var/log/messages, Debian (I think) to /var/log/syslog or .../daemon, and then there are shops with /var/log/ha-log. I think that these are the most common configurations. But nothing is stopping them to devise their own and it would be difficult to enforce unified logging configuration (I think actually impossible).
Well, if you can't enforce centralized logging, then there's not much one can do. If you could set up a system where the various hosts are free to do whatever they want with their logs, as long as they forward a copy to a central server too, then it would be easy to handle. If you can't do that, then grepping around is your best bet. I'd try to persuade them that this way of operation doesn't scale in the long run =)
It may be possible to add some debugging stuff to syslog-ng, that would echo the information you need, but it's not currently present.
I also fail to see why this would be useful, but that might just be my lack of imagination O:)
This is for a reporting tool for clusters. It collects all the relevant information from all cluster members and that includes excerpts from log files. People use all kinds of syslog setups so the tool needs to figure out which log file is relevant.
Wouldn't it be possible to collect the logs with a central syslog-ng, and tag them appropriately, based on source, and then filter them to the appropriate files, based on that tag?
See above. Whatever the user does, we should make best effort to find the file where the messages end. Of course, if the setup is really awkward, then no cigar, but we have to support "reasonable" configurations (for some values of reasonable :)
Well, if users want their logs to be collected, they should forward a copy to a central server, imo. Whatever they do with the log message afterwards, doesn't matter. Just send a copy. With most syslog daemons, that's only a few lines, and even legacy syslogd can do that, even on such esoteric systems as Solaris and AIX. -- |8]
participants (3)
-
Dejan Muhamedagic
-
Gergely Nagy
-
Scott Rochford