Idea for streaming logs to my workstation as well as logserver
I have a cunning idea. When I first set up a logserver, I also had it log to tty12, and a monitor next to my desk of the server with the network logs streaming down the screen. This was very interesting and I managed to pick up quite a lot about my network by noticing when a sudden screen full of information would whizz by instead of the usual incremental logs coming in. I would like to implement something similar by having the logserver also send on it's logs to my linux workstation by way of udp. I could then have my local syslog-ng listen for this udp and write it to a pipe (similar to the way you integrate mysql with syslog-ng). I would then like to have root-tail read from this pipe and output to my X background so that I have a continuous stream of network logs in the background. Are there any pitfalls that you can think of when doing this? I don't think this will risk filling up the bit bucket on the logserver since the logs are immediately sent on via udp, "spray and pray". If they don't get to my workstation for any reason, no harm done, it won't clog up the logserver. I guess the root-tail will take a bit out of my workstation having to draw on the X background all the time but it's a fast machine. I believe that it would block my local syslog-ng on my workstation if the root-tail were to stop reading from it. Which brings me to my next question: Is it better to do root-tail /var/log/logstream.pipe or root-tail - < /var/log/logstream.pipe Since I'm not sure that the first will take the actual throughput away from the pipe to stop the logger from blocking on the sending side. Also, this would require that the my X session not be closed otherwise the root-tail wouldn't be able to run to take away the logs from the other side of the pipe and the local logger would block again. All feedback welcome. Thanks. -h -- Hari Sekhon
On Sun, 2006-10-29 at 17:19 +0000, Hari Sekhon wrote:
Are there any pitfalls that you can think of when doing this? I don't think this will risk filling up the bit bucket on the logserver since the logs are immediately sent on via udp, "spray and pray". If they don't get to my workstation for any reason, no harm done, it won't clog up the logserver.
I guess the root-tail will take a bit out of my workstation having to draw on the X background all the time but it's a fast machine.
I believe that it would block my local syslog-ng on my workstation if the root-tail were to stop reading from it. Which brings me to my next question:
Is it better to do
root-tail /var/log/logstream.pipe
or
root-tail - < /var/log/logstream.pipe
Since I'm not sure that the first will take the actual throughput away from the pipe to stop the logger from blocking on the sending side.
Also, this would require that the my X session not be closed otherwise the root-tail wouldn't be able to run to take away the logs from the other side of the pipe and the local logger would block again.
All feedback welcome.
Another possibility is by using the latest multicasting feature, send the logs to a multicast address, have your workstation joined to the multicast group, and use the packets sent there. If you are not joined, it could mean no traffic (if set up properly), if you are you receive messages. There's an example script in contrib that does the joining and receiving part, but it could be hacked to do what you are up to. On the other hand the setup you are proposiing should work as well, syslog-ng won't block if the reader of a pipe does not read messages. It will simply start dropping messages internally (which you can then see as DROPPED messages in the stats message) -- Bazsi
I'm not entirely clear on the whole dropping thing in syslog-ng. If messages are dropped, I take it this does not affect other destination sources, they still get their messages, right? Also, I remember reading on campin.net's faq that if the logger fills up the global buffer due to one destination not being available, then you effectively lose all buffering. I take it this is what will happen to my local logger if I stop running root-tail for any reason, ie my X session dies or something... Is this what you mean by dropping messages internally? They overflow the global buffer and then are immediately discarded if the destination can't accept them? Which brings me to another point. How exactly do you specify different global buffers, as recommended at the bottom of the campin.net faq? It doesn't explicitly say how to do this. Does having separate destination definitions do this or is some other configuration needed? Thanks -h On 30/10/06, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Sun, 2006-10-29 at 17:19 +0000, Hari Sekhon wrote:
Are there any pitfalls that you can think of when doing this? I don't think this will risk filling up the bit bucket on the logserver since the logs are immediately sent on via udp, "spray and pray". If they don't get to my workstation for any reason, no harm done, it won't clog up the logserver.
I guess the root-tail will take a bit out of my workstation having to draw on the X background all the time but it's a fast machine.
I believe that it would block my local syslog-ng on my workstation if the root-tail were to stop reading from it. Which brings me to my next question:
Is it better to do
root-tail /var/log/logstream.pipe
or
root-tail - < /var/log/logstream.pipe
Since I'm not sure that the first will take the actual throughput away from the pipe to stop the logger from blocking on the sending side.
Also, this would require that the my X session not be closed otherwise the root-tail wouldn't be able to run to take away the logs from the other side of the pipe and the local logger would block again.
All feedback welcome.
Another possibility is by using the latest multicasting feature, send the logs to a multicast address, have your workstation joined to the multicast group, and use the packets sent there. If you are not joined, it could mean no traffic (if set up properly), if you are you receive messages.
There's an example script in contrib that does the joining and receiving part, but it could be hacked to do what you are up to.
On the other hand the setup you are proposiing should work as well, syslog-ng won't block if the reader of a pipe does not read messages. It will simply start dropping messages internally (which you can then see as DROPPED messages in the stats message)
-- Bazsi
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
-- Hari Sekhon
On Mon, 2006-10-30 at 19:29 +0000, Hari Sekhon wrote:
I'm not entirely clear on the whole dropping thing in syslog-ng.
If messages are dropped, I take it this does not affect other destination sources, they still get their messages, right?
Also, I remember reading on campin.net's faq that if the logger fills up the global buffer due to one destination not being available, then you effectively lose all buffering. I take it this is what will happen to my local logger if I stop running root-tail for any reason, ie my X session dies or something... Is this what you mean by dropping messages internally? They overflow the global buffer and then are immediately discarded if the destination can't accept them?
the buffers are not global, in fact no global buffer exists. There's a per-destination buffer however, only the dropped counter is global (at least in 1.6.x)
Which brings me to another point. How exactly do you specify different global buffers, as recommended at the bottom of the campin.net faq? It doesn't explicitly say how to do this. Does having separate destination definitions do this or is some other configuration needed?
If I read the FAQ correctly Nate refers to system architecture and not syslog-ng architecture. He says that instead of using a single pipe to distribute messages to multiple components, use a pipe for each component. For syslog-ng, every destination has its own buffer, the size of this buffer can be set globally (e.g. the same for all destinations), or locally, but whereever you set the buffer size, the buffer will always be there for each destination. -- Bazsi
participants (2)
-
Balazs Scheidler
-
Hari Sekhon