Example 3.4. Script to feed a growing logfile into syslog-ng
Followed http://www.syslog.org/syslog-ng/v2/#id2535001 and created some tail logger combos to feed some growing logfiles into syslog-ng. Issue that I am need to get around is that while I haven't missed capturing any messages this way the time stamps in syslog-ng seem to be when logger actually gets the line into syslog rather than that of the tailed files line. I have a very quick-filling file (logged 143 messages yesterday all at 14:15:18) and when I view syslog-ng these are stamped up to 14:15:36 and no longer in any order. Does anyone have any suggestions? Thanks.
Chris Pratt (pratt70@gmail.com) wrote on 24 July 2008 13:10:
Followed http://www.syslog.org/syslog-ng/v2/#id2535001 and created some tail logger combos to feed some growing logfiles into syslog-ng.
Issue that I am need to get around is that while I haven't missed capturing any messages this way the time stamps in syslog-ng seem to be when logger actually gets the line into syslog rather than that of the tailed files line. I have a very quick-filling file (logged 143 messages yesterday all at 14:15:18) and when I view syslog-ng these are stamped up to 14:15:36 and no longer in any order.
I suggest you make your disk-eater write to a fifo, and syslog-ng read from it with a pipe source. I do it for apache...
Thanks for your suggestion. This is for Tomcat. Unfortunately I am stuck reading from the log because the syslog appender in the log4j version on our app does not capture stack traces while the file appenders do. In other cases we want to capture garbage collections which are only written to stdout. I tried sending my file appends to /dev/log but that didn't work so I am stuck trying to read from files. Only my central log server runs syslog-ng. All of my Tomcat hosts just use syslogd and send *.* @mylogserver. Otherwise I'd probably try # source s_tail { file("log.log" follow_freq(1) flags(no-parse)); }; to see if it kept it in order any better. What I could really use is a logger that has the ability to read and forward to syslog the existing timestamp... Carlos Carvalho wrote:
Chris Pratt (pratt70@gmail.com) wrote on 24 July 2008 13:10:
Followed http://www.syslog.org/syslog-ng/v2/#id2535001 and created some tail logger combos to feed some growing logfiles into syslog-ng.
Issue that I am need to get around is that while I haven't missed capturing any messages this way the time stamps in syslog-ng seem to be when logger actually gets the line into syslog rather than that of the tailed files line. I have a very quick-filling file (logged 143 messages yesterday all at 14:15:18) and when I view syslog-ng these are stamped up to 14:15:36 and no longer in any order.
I suggest you make your disk-eater write to a fifo, and syslog-ng read from it with a pipe source. I do it for apache... ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Chris Pratt (pratt70@gmail.com) wrote on 24 July 2008 16:10:
Thanks for your suggestion. This is for Tomcat. Unfortunately I am stuck reading from the log because the syslog appender in the log4j version on our app does not capture stack traces while the file appenders do. In other cases we want to capture garbage collections which are only written to stdout.
Couldn't you redirect stdout >> fifo?
I tried sending my file appends to /dev/log but that didn't work
Yes, doesn't work. I think I tried it myself once... Use the fifo and the pipe source.
Only my central log server runs syslog-ng.
Ah, the real reason finally :-) :-)
On Thu, 2008-07-24 at 16:10 -0400, Chris Pratt wrote:
Thanks for your suggestion. This is for Tomcat. Unfortunately I am stuck reading from the log because the syslog appender in the log4j version on our app does not capture stack traces while the file appenders do. In other cases we want to capture garbage collections which are only written to stdout. I tried sending my file appends to /dev/log but that didn't work so I am stuck trying to read from files. Only my central log server runs syslog-ng. All of my Tomcat hosts just use syslogd and send *.* @mylogserver. Otherwise I'd probably try # source s_tail { file("log.log" follow_freq(1) flags(no-parse)); }; to see if it kept it in order any better.
What I could really use is a logger that has the ability to read and forward to syslog the existing timestamp...
syslog-ng could that if it was running on the client hosts too. but otherwise what about this dirty trick: tail -f <logfile> | sed -e 's/^/<133>/' | nc syslog-server 514 If the timestamps are otherwise in syslog-ng processable format, the above scriptlet would work. The sed sets local6.notice as a priority. However the difficult part is in doing the "tail -f" part robust enough, but I guess you achieved that already. -- Bazsi
On Thu, Jul 24, 2008 at 04:10:01PM -0400, Chris Pratt wrote:
What I could really use is a logger that has the ability to read and forward to syslog the existing timestamp...
Take a look at relogger.pl in the syslog-ng source directory. It will be much more efficient than calling the "logger" command and might be able to keep up with your traffic. Failing that, you could hack the Perl Sys::Syslog.pm module to allow you to specify the timestamp. Or ask the current maintainer, maybe he'll be in a good mood and do it for you.
participants (4)
-
Balazs Scheidler
-
carlos@fisica.ufpr.br
-
Chris Pratt
-
Ed Ravin