Hi, I have 3 syslog listeners (solaris 10 ) collecting log events from about 20,000 devices and fwding them on a central syslog box (solaris 10, 16 gb RAM and 3TB SAN storage) . the central syslog box stores them in oracle db and on filesystem. I notice that events from many devices are being droped on central syslog box. I thought it might be because of udp protocol, but even after enabling tcp its droping events. I can see that events come on listeners boxes (I created temp file logs there) please tell me what could be missing. Regards, -Manish ____________________________________________________________________________________ Want to start your own business? Learn how on Yahoo! Small Business. http://smallbusiness.yahoo.com/r-index
Hi, Arya, Manish Kumar <m.arya@yahoo.com> [20061110 01:11:54 -0800]:
Hi,
I have 3 syslog listeners (solaris 10 ) collecting log events from about 20,000 devices and fwding them on a central syslog box (solaris 10, 16 gb RAM and 3TB SAN storage) . the central syslog box stores them in oracle db and on filesystem.
I notice that events from many devices are being droped on central syslog box. I thought it might be because of udp protocol, but even after enabling tcp its droping events. I can see that events come on listeners boxes (I created temp file logs there)
please tell me what could be missing.
Munching through the mailing list archives ;) https://lists.balabit.hu/pipermail/syslog-ng/2006-May/008836.html Sounds like, what DB monkeys call, 'SQL Exhaustion' or something. You SQL database is slowing up the whole process...mainly as SQL is not suitable for *live* realtime processing, thats my opinion though. Cheers Alex
Regards, -Manish
____________________________________________________________________________________ Want to start your own business? Learn how on Yahoo! Small Business. http://smallbusiness.yahoo.com/r-index _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
actually this was cuz of limitation on number of opened files in solaris, i have increated it from 256 to 65536 i see one more error 10.11.2006 09:46:39 syslog-ng syslog err Connection broken to AF_INET(XXXXXXX:514), reopening in 10 seconds 10.11.2006 09:46:49 syslog-ng syslog err io.c: do_write: write() failed (errno 146), Connection refused 10.11.2006 09:46:49 syslog-ng syslog err Connection broken to AF_INET(XXXXXX:514), reopening in 10 seconds 10.11.2006 09:46:59 syslog-ng syslog err io.c: do_write: write() failed (errno 146), Connection refused 10.11.2006 09:46:59 syslog-ng syslog err Connection broken to AF_INET(XXXXXXXX:514), reopening in 10 second -manish --- Alexander Clouter <ac56@soas.ac.uk> wrote:
Hi,
Arya, Manish Kumar <m.arya@yahoo.com> [20061110 01:11:54 -0800]:
Hi,
I have 3 syslog listeners (solaris 10 )
collecting
log events from about 20,000 devices and fwding them on a central syslog box (solaris 10, 16 gb RAM and 3TB SAN storage) . the central syslog box stores them in oracle db and on filesystem.
I notice that events from many devices are being droped on central syslog box. I thought it might be because of udp protocol, but even after enabling tcp its droping events. I can see that events come on listeners boxes (I created temp file logs there)
please tell me what could be missing.
Munching through the mailing list archives ;)
https://lists.balabit.hu/pipermail/syslog-ng/2006-May/008836.html
Sounds like, what DB monkeys call, 'SQL Exhaustion' or something. You SQL database is slowing up the whole process...mainly as SQL is not suitable for *live* realtime processing, thats my opinion though.
Cheers
Alex
Regards, -Manish
____________________________________________________________________________________
Want to start your own business? Learn how on Yahoo! Small Business. http://smallbusiness.yahoo.com/r-index _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu
https://lists.balabit.hu/mailman/listinfo/syslog-ng
Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
____________________________________________________________________________________ Cheap talk? Check out Yahoo! Messenger's low PC-to-Phone call rates. http://voice.yahoo.com
On Nov 10, 2006, at 4:24 AM, Alexander Clouter wrote:
Sounds like, what DB monkeys call, 'SQL Exhaustion' or something. You SQL database is slowing up the whole process...mainly as SQL is not suitable for *live* realtime processing, thats my opinion though.
We have one very hot process that we log to a file first, then in batches stuff into a database. Doing one single insert transaction per log line is death to a DB. If you can do them in batches all in one transaction, it can help a lot.
On Fri, 2006-11-10 at 01:11 -0800, Arya, Manish Kumar wrote:
Hi,
I have 3 syslog listeners (solaris 10 ) collecting log events from about 20,000 devices and fwding them on a central syslog box (solaris 10, 16 gb RAM and 3TB SAN storage) . the central syslog box stores them in oracle db and on filesystem.
I notice that events from many devices are being droped on central syslog box. I thought it might be because of udp protocol, but even after enabling tcp its droping events. I can see that events come on listeners boxes (I created temp file logs there)
Where does it drop messages? Some events are missing from the SQL database, or some are missing from simple files too? Message dropping can happen for two reasons: * messages are dropped from the input receive buffer because syslog-ng is too slow to receive messages, _or_ * messages are actively dropped by syslog-ng after being successfully received, because a specific target cannot cope with the load You need to identify which scenario is happening and then take appropriate measures. (increase socket buffer, increase log_fifo_size(), increase DB performance, or enable flow_control and ensure the complete log patch is flow-controlled) -- Bazsi
participants (4)
-
Alexander Clouter
-
Arya, Manish Kumar
-
Balazs Scheidler
-
Vivek Khera