[syslog-ng] Transmitting a Huge Log File

[Balazs Scheidler]

On Wed, 2011-03-09 at 16:21 +0100, Zoltán Pallagi wrote:
> Hi,
> 
> If the problem is permanent after using Sandor solution as well, maybe
> you should use flow-control in server, because it can happen that the
> client syslog-ng will forward the logs as fast as it can, but the
> server will drop the incoming lines if it cannot process them in time
> and all buffers are full.
> 
> Use this config:
> 
> log {
> source(s_mysrc);
> filter(f_filter);
> flags(flow-control);
> destination(d_mydest);
> };

I guess this is important on the client, not on the server, right? ;) Or
preferably both.

> 
> 
> On 2011-03-09 15:49, Sandor Geller wrote: 
> > Hi,
> > 
> > On Wed, Mar 9, 2011 at 2:50 PM, Hidayath Basha
> > <hidayath.basha at saventech.com> wrote:
> > > Hi all,
> > > 
> > > I'm trying to transmit a huge log file (of about 80k lines) to a centralized
> > > syslog server over TCP
> > 80k lines is piece of cake unless the average line length is quite big :)

yeah, I could see syslog-ng chewing 800k messages/second on my test
environment. I only need to get some time to push that out. :( But I
guess 3.3alpha2 will come first.

> > 
> > > But, on the syslog server, I'm receiver only the last part of the log file
> > > (of about 7000 lines)
> > > 
> > > How can I transmit the whole log file
> > syslog-ng keeps track where it left off reading a file to avoid
> > sending the whole file again when it gets restarted so my guess is
> > that you fired up syslog-ng a few times.
> > 
> > To confirm this could you stop syslog-ng, delete
> > /var/lib/syslog-ng.persist (or where your persist file lives) and
> > start syslog-ng?
> > 
> > If the problem persists then run syslog-ng under strace and show the
> > relevant parts (file opens, seeks, reads).
> > 

-- 
Bazsi