Hi, As I mentioned in the last thread, I've found myself wanting to have better than one second resolution for the follow_freq() option. Attached is a patch which adds a follow_freq_ms option, which takes its value in milliseconds rather than full seconds. follow_freq() continues to work as it did previously. Thanks, Joe
On Thu, 2008-08-21 at 16:00 -0400, Joe Shaw wrote:
Hi,
As I mentioned in the last thread, I've found myself wanting to have better than one second resolution for the follow_freq() option. Attached is a patch which adds a follow_freq_ms option, which takes its value in milliseconds rather than full seconds. follow_freq() continues to work as it did previously.
First of all, thanks for your contibution. I'm not sure about the patch though, is millisecond polling really needed? It might increase the load on the host significantly to check files several times a second. The reason this patch was born was solved in an unrelated manner (e.g. increasing FIFO size). Any other opinions? If this turns out really useful, I'd not add a separate keyword, but would permit the use of floating point numbers, e.g. follow_freq(0.1) instead of follow_freq_ms(100) -- Bazsi
Hi, On Mon, Aug 25, 2008 at 3:07 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
I'm not sure about the patch though, is millisecond polling really needed? It might increase the load on the host significantly to check files several times a second.
The reason this patch was born was solved in an unrelated manner (e.g. increasing FIFO size).
The reasoning behind the patch really isn't increasing the FIFO size (although that is why I did it initially) but because when you are pushing large amounts of data, a one second delay can cause the data to take hours to get from the source to the destination. This is what I was referring to when I was talking about my lack of patience. :) It's true that decreasing the interval increases the load -- and I think anyone who decreased it to, say, 10ms would see that -- but it's another area in which people can make that trade-off and optimize their deployment. Dropping it to 500ms doubles the throughput but doesn't add much load. I was running at 200ms rather happily.
If this turns out really useful, I'd not add a separate keyword, but would permit the use of floating point numbers, e.g. follow_freq(0.1) instead of follow_freq_ms(100)
Makes a lot of sense. Joe
Wouldn't the log_fetch_limit set to a large number allow for 1 second follow to process huge amounts of data. You would have to set the log_fifo_size large enough to host this number as well, but I don't see the requirement for the millisecond polling. Evan. Joe Shaw wrote:
Hi,
On Mon, Aug 25, 2008 at 3:07 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
I'm not sure about the patch though, is millisecond polling really needed? It might increase the load on the host significantly to check files several times a second.
The reason this patch was born was solved in an unrelated manner (e.g. increasing FIFO size).
The reasoning behind the patch really isn't increasing the FIFO size (although that is why I did it initially) but because when you are pushing large amounts of data, a one second delay can cause the data to take hours to get from the source to the destination. This is what I was referring to when I was talking about my lack of patience. :)
It's true that decreasing the interval increases the load -- and I think anyone who decreased it to, say, 10ms would see that -- but it's another area in which people can make that trade-off and optimize their deployment. Dropping it to 500ms doubles the throughput but doesn't add much load. I was running at 200ms rather happily.
If this turns out really useful, I'd not add a separate keyword, but would permit the use of floating point numbers, e.g. follow_freq(0.1) instead of follow_freq_ms(100)
Makes a lot of sense.
Joe ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Mon, 2008-08-25 at 08:22 -0700, Evan Rempel wrote:
Wouldn't the log_fetch_limit set to a large number allow for 1 second follow to process huge amounts of data. You would have to set the log_fifo_size large enough to host this number as well, but I don't see the requirement for the millisecond polling.
Evan.
This is exactly what I wanted to say. Thanks Evan. You could increase log_fetch_limit(), and in case you are using flow-control, you might also increase log_iw_size(). -- Bazsi
Hi, On Mon, Aug 25, 2008 at 11:22 AM, Evan Rempel <erempel@uvic.ca> wrote:
Wouldn't the log_fetch_limit set to a large number allow for 1 second follow to process huge amounts of data. You would have to set the log_fifo_size large enough to host this number as well, but I don't see the requirement for the millisecond polling.
This is a good point, but since increasing the log_fifo_size means additional memory usage allowing millisecond polling allows me to make the tradeoff of CPU vs. memory. It may be that on this host I have more CPU to burn than memory. Joe
On Tue, 2008-08-26 at 15:56 -0400, Joe Shaw wrote:
Hi,
On Mon, Aug 25, 2008 at 11:22 AM, Evan Rempel <erempel@uvic.ca> wrote:
Wouldn't the log_fetch_limit set to a large number allow for 1 second follow to process huge amounts of data. You would have to set the log_fifo_size large enough to host this number as well, but I don't see the requirement for the millisecond polling.
This is a good point, but since increasing the log_fifo_size means additional memory usage allowing millisecond polling allows me to make the tradeoff of CPU vs. memory. It may be that on this host I have more CPU to burn than memory.
Ok, that's a good point. -- Bazsi
Hi, I just wanted to mention that I've pushed a patch to implement support for follow_freq() values less than a second. It's currently in the git archive (or daily snapshots) but the next release will also contain it. On Thu, 2008-08-21 at 16:00 -0400, Joe Shaw wrote:
Hi,
As I mentioned in the last thread, I've found myself wanting to have better than one second resolution for the follow_freq() option. Attached is a patch which adds a follow_freq_ms option, which takes its value in milliseconds rather than full seconds. follow_freq() continues to work as it did previously.
Thanks, Joe ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Bazsi
participants (3)
-
Balazs Scheidler
-
Evan Rempel
-
Joe Shaw