manual snippet: "syslog-ng always reads its incoming log channels to prevent the running daemons from blocking." log_fifo_size() is offered as mitigation strategy. is there a way to force syslog-ng not to do this - option i've missed (or undocumented), preprocessor DEFINE, etc? why: i want to re-bin some historical logs into a $YEAR/$MONTH/$DAY/ $HOUR/$HOST file layout. syslog-ng is configured with a source pipe and destination as above, and fed via a fifo: find /historical/log/dir -type f -exec cat {} /tmp/myfifo \; but the resulting output dir is only 1/3 the size of the input dir. syslog-ng internal() messages include: syslog-ng[425]: Cannot open file /scratch/ptest/2005/05/13/13/sn5 for writing (No such file or directory) syslog-ng[425]: STATS: dropped 0 "dropped 0" seems inaccurate. this continues to happen even when log_fifo_size() is set higher than the maximum actual number of lines in an output file (which i happen to know for this test case). i could throttle feeding the pipe, but then it's trial-and-error to find the zero-loss rate. is there a better way? i'm running syslog-ng 1.6.0 on an old debian box, but since 1.6.x and 1.9.x manuals both contain the statement above i doubted an upgrade would behave differently in this respect? so i thought i'd ask. many thanks! -- +--------------------------------------------------------------+ | Jon Stearley (505) 845-7571 (FAX 844-9297) | | Sandia National Laboratories Scalable Systems Integration | +--------------------------------------------------------------+
On Apr 12, 2006, at 10:02 AM, Jon Stearley wrote:
i'm running syslog-ng 1.6.0 on an old debian box, but since 1.6.x and 1.9.x manuals both contain the statement above i doubted an upgrade would behave differently in this respect? so i thought i'd ask. many thanks!
sorry to be lazy - 1.9.x log{flags(flow-control)} seem appropriate... upgrading <:P -jon
On Apr 12, 2006, at 10:02 AM, Jon Stearley wrote:
why: i want to re-bin some historical logs into a $YEAR/$MONTH/$DAY/ $HOUR/$HOST file layout. syslog-ng is configured with a source pipe and destination as above, and fed via a fifo: find /historical/log/dir -type f -exec cat {} /tmp/myfifo \; but the resulting output dir is only 1/3 the size of the input dir. syslog-ng internal() messages include: syslog-ng[425]: Cannot open file /scratch/ptest/2005/05/13/13/sn5 for writing (No such file or directory) syslog-ng[425]: STATS: dropped 0
ok, i've upgraded and the same thing happens with 1.9.9. the real problem is revealed by adding two-lines of code: -------------------------------------------------------- $ diff -c affile.c.orig affile.c *** affile.c.orig Thu Apr 13 10:28:09 2006 --- affile.c Thu Apr 13 16:44:57 2006 *************** *** 78,83 **** --- 78,85 ---- } *fd = open(name, flags, mode); } + if (*fd == -1) + return *fd != -1; if (uid != -1 || gid != -1) chown(name, uid, gid); if (mode != -1) -------------------------------------------------------- without these, errno is set by chown to mean "No such file". but with these root problem is exposed: Apr 13 16:44:03 s_local@mercy syslog-ng[29159]: Error opening file for writing; error='Too many open files (24)', filename='/tmp/p/logs/ host1018' i don't know the best way to force some fd's to close (eg before they timeout naturally), but would suggest that as an appropriate strategy. please advise. thx. -jon ############### i run the below perl as `./mklog > /tmp/p/myfifo` to generate the messages: #!/usr/bin/perl foreach $i (1..2000) { print "Apr 13 09:09:09 host".$i." msg\n"; } ############### and here is the syslog-ng.conf: options { create_dirs(yes); keep_hostname(yes); stats(60); }; source s_ptest { pipe("/tmp/p/myfifo"); }; source s_local { internal(); }; destination d_local { file("/tmp/p/stats"); }; destination d_ptest { file("/tmp/p/logs/$HOST"); }; log { source(s_local); destination(d_local); }; log { source(s_ptest); destination(d_ptest); flags(flow-control); };
On Thu, 2006-04-13 at 17:04 -0600, Jon Stearley wrote:
On Apr 12, 2006, at 10:02 AM, Jon Stearley wrote:
why: i want to re-bin some historical logs into a $YEAR/$MONTH/$DAY/ $HOUR/$HOST file layout. syslog-ng is configured with a source pipe and destination as above, and fed via a fifo: find /historical/log/dir -type f -exec cat {} /tmp/myfifo \; but the resulting output dir is only 1/3 the size of the input dir. syslog-ng internal() messages include: syslog-ng[425]: Cannot open file /scratch/ptest/2005/05/13/13/sn5 for writing (No such file or directory) syslog-ng[425]: STATS: dropped 0
ok, i've upgraded and the same thing happens with 1.9.9. the real problem is revealed by adding two-lines of code: -------------------------------------------------------- $ diff -c affile.c.orig affile.c *** affile.c.orig Thu Apr 13 10:28:09 2006 --- affile.c Thu Apr 13 16:44:57 2006 *************** *** 78,83 **** --- 78,85 ---- } *fd = open(name, flags, mode); } + if (*fd == -1) + return *fd != -1; if (uid != -1 || gid != -1) chown(name, uid, gid); if (mode != -1) --------------------------------------------------------
without these, errno is set by chown to mean "No such file". but with these root problem is exposed: Apr 13 16:44:03 s_local@mercy syslog-ng[29159]: Error opening file for writing; error='Too many open files (24)', filename='/tmp/p/logs/ host1018'
i don't know the best way to force some fd's to close (eg before they timeout naturally), but would suggest that as an appropriate strategy. please advise. thx.
-jon
Thanks for tracking this down. I have applied a similar patch to my syslog-ng 1.9.x tree, 1.6.x will soon to follow. -- Bazsi
participants (2)
-
Balazs Scheidler
-
Jon Stearley