[syslog-ng] Re: zero-loss from source{pipe()}

[Jon Stearley]

On Apr 12, 2006, at 10:02 AM, Jon Stearley wrote:
> why: i want to re-bin some historical logs into a $YEAR/$MONTH/$DAY/ 
> $HOUR/$HOST file layout.  syslog-ng is configured with a source  
> pipe and destination as above, and fed via a fifo:
>   find /historical/log/dir -type f -exec cat {} /tmp/myfifo \;
> but the resulting output dir is only 1/3 the size of the input  
> dir.  syslog-ng internal() messages include:
>   syslog-ng[425]: Cannot open file /scratch/ptest/2005/05/13/13/sn5  
> for writing (No such file or directory)
>   syslog-ng[425]: STATS: dropped 0

ok, i've upgraded and the same thing happens with 1.9.9.  the real  
problem is revealed by adding two-lines of code:
--------------------------------------------------------
$ diff -c affile.c.orig affile.c
*** affile.c.orig       Thu Apr 13 10:28:09 2006
--- affile.c    Thu Apr 13 16:44:57 2006
***************
*** 78,83 ****
--- 78,85 ----
         }
         *fd = open(name, flags, mode);
       }
+   if (*fd == -1)
+     return *fd != -1;
     if (uid != -1 || gid != -1)
       chown(name, uid, gid);
     if (mode != -1)
--------------------------------------------------------

without these, errno is set by chown to mean "No such file".  but  
with these root problem is exposed:
Apr 13 16:44:03 s_local at mercy syslog-ng[29159]: Error opening file  
for writing; error='Too many open files (24)', filename='/tmp/p/logs/ 
host1018'

i don't know the best way to force some fd's to close (eg before they  
timeout naturally), but would suggest that as an appropriate  
strategy.  please advise.  thx.

-jon


############### i run the below perl as `./mklog > /tmp/p/myfifo` to  
generate the messages:
#!/usr/bin/perl
foreach $i (1..2000) {
         print "Apr 13 09:09:09 host".$i." msg\n";
}

############### and here is the syslog-ng.conf:
options {
     create_dirs(yes);
     keep_hostname(yes);
     stats(60);
};

source s_ptest {
     pipe("/tmp/p/myfifo");
};

source s_local {
     internal();
};

destination d_local {
     file("/tmp/p/stats");
};

destination d_ptest {
     file("/tmp/p/logs/$HOST");
};

log { source(s_local); destination(d_local); };
log { source(s_ptest); destination(d_ptest); flags(flow-control); };