Transmitting a Huge Log File
Hi all, I'm trying to transmit a huge log file (of about 80k lines) to a centralized syslog server over TCP But, on the syslog server, I'm receiver only the last part of the log file (of about 7000 lines) How can I transmit the whole log file On client, the syslog-ng.conf file ... destination d_mydest { tcp("192.168.18.7" port(5140)); }; source s_src { file("/some/huge/file.log" follow_freq(1) log_prefix("mylogs:")); }; log { source(s_src); destination(d_mydest); }; On server (192.168.18.7) , the conf file ... source s_mysrc{ tcp(ip("192.168.18.7") port(5140) keep-alive(yes)); }; filter f_filter{ match("mylogs:"); }; destination d_mydest { file ("/var/log/hosts/$HOST/$YEAR-$MONTH/$YEAR-$MONTH-$DAY/hugefile.log" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes) ); }; log { source(s_mysrc); filter(f_filter); destination(d_mydest); }; Thanks and regards, Hidayath
Hi, On Wed, Mar 9, 2011 at 2:50 PM, Hidayath Basha <hidayath.basha@saventech.com> wrote:
Hi all,
I'm trying to transmit a huge log file (of about 80k lines) to a centralized syslog server over TCP
80k lines is piece of cake unless the average line length is quite big :)
But, on the syslog server, I'm receiver only the last part of the log file (of about 7000 lines)
How can I transmit the whole log file
syslog-ng keeps track where it left off reading a file to avoid sending the whole file again when it gets restarted so my guess is that you fired up syslog-ng a few times. To confirm this could you stop syslog-ng, delete /var/lib/syslog-ng.persist (or where your persist file lives) and start syslog-ng? If the problem persists then run syslog-ng under strace and show the relevant parts (file opens, seeks, reads). Regards, Sandor
Hi, If the problem is permanent after using Sandor solution as well, maybe you should use flow-control in server, because it can happen that the client syslog-ng will forward the logs as fast as it can, but the server will drop the incoming lines if it cannot process them in time and all buffers are full. Use this config: log { source(s_mysrc); filter(f_filter); flags(flow-control); destination(d_mydest); }; On 2011-03-09 15:49, Sandor Geller wrote:
Hi,
On Wed, Mar 9, 2011 at 2:50 PM, Hidayath Basha <hidayath.basha@saventech.com> wrote:
Hi all,
I'm trying to transmit a huge log file (of about 80k lines) to a centralized syslog server over TCP 80k lines is piece of cake unless the average line length is quite big :)
But, on the syslog server, I'm receiver only the last part of the log file (of about 7000 lines)
How can I transmit the whole log file syslog-ng keeps track where it left off reading a file to avoid sending the whole file again when it gets restarted so my guess is that you fired up syslog-ng a few times.
To confirm this could you stop syslog-ng, delete /var/lib/syslog-ng.persist (or where your persist file lives) and start syslog-ng?
If the problem persists then run syslog-ng under strace and show the relevant parts (file opens, seeks, reads).
Regards,
Sandor ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Wed, 2011-03-09 at 16:21 +0100, Zoltán Pallagi wrote:
Hi,
If the problem is permanent after using Sandor solution as well, maybe you should use flow-control in server, because it can happen that the client syslog-ng will forward the logs as fast as it can, but the server will drop the incoming lines if it cannot process them in time and all buffers are full.
Use this config:
log { source(s_mysrc); filter(f_filter); flags(flow-control); destination(d_mydest); };
I guess this is important on the client, not on the server, right? ;) Or preferably both.
On 2011-03-09 15:49, Sandor Geller wrote:
Hi,
On Wed, Mar 9, 2011 at 2:50 PM, Hidayath Basha <hidayath.basha@saventech.com> wrote:
Hi all,
I'm trying to transmit a huge log file (of about 80k lines) to a centralized syslog server over TCP 80k lines is piece of cake unless the average line length is quite big :)
yeah, I could see syslog-ng chewing 800k messages/second on my test environment. I only need to get some time to push that out. :( But I guess 3.3alpha2 will come first.
But, on the syslog server, I'm receiver only the last part of the log file (of about 7000 lines)
How can I transmit the whole log file syslog-ng keeps track where it left off reading a file to avoid sending the whole file again when it gets restarted so my guess is that you fired up syslog-ng a few times.
To confirm this could you stop syslog-ng, delete /var/lib/syslog-ng.persist (or where your persist file lives) and start syslog-ng?
If the problem persists then run syslog-ng under strace and show the relevant parts (file opens, seeks, reads).
-- Bazsi
Thanks Sandor Removing .persist file really worked... Thanks Thanks and regards, Hidayath Saven Technologies: A leader in technology driven business solutions ---------------------------------------------------------------------------- ----------------------------------- #302, 3rd Floor, MyHome Sarovar Plaza, Hyderabad- 500 063, India Off: +91 040 23237303/7304 ext 115 email: hidayath.basha @saventech.com <mailto:hidayath.basha@saventech.com> | www.saventech.com <http://www.saventech.com> follow me: http://twitter.com/hidayath26 <http://twitter.com/hidayath26> ---------------------------------------------------------------------------- ----------------------------------- -----Original Message----- From: Sandor Geller [mailto:Sandor.Geller@morganstanley.com] Sent: Wednesday, March 09, 2011 8:19 PM To: Syslog-ng users' and developers' mailing list Cc: Hidayath Basha Subject: Re: [syslog-ng] Transmitting a Huge Log File Hi, On Wed, Mar 9, 2011 at 2:50 PM, Hidayath Basha <hidayath.basha@saventech.com> wrote:
Hi all,
I'm trying to transmit a huge log file (of about 80k lines) to a centralized syslog server over TCP
80k lines is piece of cake unless the average line length is quite big :)
But, on the syslog server, I'm receiver only the last part of the log file (of about 7000 lines)
How can I transmit the whole log file
syslog-ng keeps track where it left off reading a file to avoid sending the whole file again when it gets restarted so my guess is that you fired up syslog-ng a few times. To confirm this could you stop syslog-ng, delete /var/lib/syslog-ng.persist (or where your persist file lives) and start syslog-ng? If the problem persists then run syslog-ng under strace and show the relevant parts (file opens, seeks, reads). Regards, Sandor
participants (4)
-
Balazs Scheidler
-
Hidayath Basha
-
Sandor Geller
-
Zoltán Pallagi