Greetings, I've noticed in 1.6.x that there are tags already built into the configuration file to support inline compression of file output (but they complain if used). Is there any plan to support this? Are there reasons this hasn't been implemented yet? Given my knowledge of the libgzip, I can't imagine its terribly hard. The ability to write logs compressed would be -very- useful. -Aaron
On Wed, 2008-10-01 at 15:35 -0400, Aaron Wiebe wrote:
Greetings,
I've noticed in 1.6.x that there are tags already built into the configuration file to support inline compression of file output (but they complain if used). Is there any plan to support this? Are there reasons this hasn't been implemented yet? Given my knowledge of the libgzip, I can't imagine its terribly hard.
The ability to write logs compressed would be -very- useful.
It is not so easy, as syslog-ng needs to reopen the same file multiple times even if the previous shutdown was not clean. if syslog-ng crashed, you still want to append logs to the same file. This cannot be cleanly done with libz alone. And the requirement that the same file cannot be written to any time again is too much restriction. You could do something like this though: destination d_gzip { program("gzip -c >> logfile.gz"); }; This is somewhat better as gzip runs as a separate process and has time to close the logfile properly. Then you only need a cat program that understands that multiple gzip streams are concatenated. Hmmm, I've just checked simple zcat works. So as it seems this is much more robust to do with an external program than building it into syslog-ng, on the other hand this also means that you can use a 2nd CPU in the system for compression. -- Bazsi
On Thu, Oct 2, 2008 at 4:06 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
You could do something like this though:
destination d_gzip { program("gzip -c >> logfile.gz"); };
While I see the advantages, I don't think the program handler works with things like: $YEAR $MONTH $DAY $HOUR $HOST Now, obviously some of the date related items could be handled by writing something that goes based on the local system time, but if there are logs from remote systems that are slightly off in their timestamps (ie, getting remote logs 5 seconds behind or something), they'd end up in the wrong files. At the same time, $HOST would require parsing the logfile line - which could get messy. Having it inline would make me really comfortable in using it. I'm guessing it was originally in the plan given the tags in the code for a file option "compress(yes)"... If I submitted a patch, would you consider including it in a future release? -Aaron
This is somewhat better as gzip runs as a separate process and has time to close the logfile properly. Then you only need a cat program that understands that multiple gzip streams are concatenated. Hmmm, I've just checked simple zcat works.
So as it seems this is much more robust to do with an external program than building it into syslog-ng, on the other hand this also means that you can use a 2nd CPU in the system for compression.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Thu, 2008-10-02 at 11:35 -0400, Aaron Wiebe wrote:
On Thu, Oct 2, 2008 at 4:06 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
You could do something like this though:
destination d_gzip { program("gzip -c >> logfile.gz"); };
While I see the advantages, I don't think the program handler works with things like:
$YEAR $MONTH $DAY $HOUR $HOST
Now, obviously some of the date related items could be handled by writing something that goes based on the local system time, but if there are logs from remote systems that are slightly off in their timestamps (ie, getting remote logs 5 seconds behind or something), they'd end up in the wrong files. At the same time, $HOST would require parsing the logfile line - which could get messy.
Having it inline would make me really comfortable in using it. I'm guessing it was originally in the plan given the tags in the code for a file option "compress(yes)"... If I submitted a patch, would you consider including it in a future release?
Yes, I would, but please note the "Contributory License Agreement" on our website. On the technical side, I think doing it properly and robust enough, the solution needs the "separate" process idea, otherwise you'd end up with corrupt files whenever syslog-ng crashes. So what I propose is to create a program() destination that can handle macros, it would be useful for other purposes as well. affile.c has code that you can pattern your implementation. -- Bazsi
participants (2)
-
Aaron Wiebe
-
Balazs Scheidler