[syslog-ng] Re: zero-loss from source{pipe()}

[Balazs Scheidler]

On Thu, 2006-04-13 at 17:04 -0600, Jon Stearley wrote:
> On Apr 12, 2006, at 10:02 AM, Jon Stearley wrote:
> > why: i want to re-bin some historical logs into a $YEAR/$MONTH/$DAY/ 
> > $HOUR/$HOST file layout.  syslog-ng is configured with a source  
> > pipe and destination as above, and fed via a fifo:
> >   find /historical/log/dir -type f -exec cat {} /tmp/myfifo \;
> > but the resulting output dir is only 1/3 the size of the input  
> > dir.  syslog-ng internal() messages include:
> >   syslog-ng[425]: Cannot open file /scratch/ptest/2005/05/13/13/sn5  
> > for writing (No such file or directory)
> >   syslog-ng[425]: STATS: dropped 0
> 
> ok, i've upgraded and the same thing happens with 1.9.9.  the real  
> problem is revealed by adding two-lines of code:
> --------------------------------------------------------
> $ diff -c affile.c.orig affile.c
> *** affile.c.orig       Thu Apr 13 10:28:09 2006
> --- affile.c    Thu Apr 13 16:44:57 2006
> ***************
> *** 78,83 ****
> --- 78,85 ----
>          }
>          *fd = open(name, flags, mode);
>        }
> +   if (*fd == -1)
> +     return *fd != -1;
>      if (uid != -1 || gid != -1)
>        chown(name, uid, gid);
>      if (mode != -1)
> --------------------------------------------------------
> 
> without these, errno is set by chown to mean "No such file".  but  
> with these root problem is exposed:
> Apr 13 16:44:03 s_local at mercy syslog-ng[29159]: Error opening file  
> for writing; error='Too many open files (24)', filename='/tmp/p/logs/ 
> host1018'
> 
> i don't know the best way to force some fd's to close (eg before they  
> timeout naturally), but would suggest that as an appropriate  
> strategy.  please advise.  thx.
> 
> -jon

Thanks for tracking this down. I have applied a similar patch to my
syslog-ng 1.9.x tree, 1.6.x will soon to follow.

-- 
Bazsi