Hi, I've planned to setup a Central Logging system using the great Syslog-NG with MySQL. All is fine but just need a couple of clarifications to optimize performance. Basically, I look at logging close to 40 Network devices which can potentially log as much 10Gigs or more of data. I want to use the DNS hostnames to be displayed when viewed in php-syslog-ng interface so I enabled DNS on syslog-ng conf file. I've changed the nsswitch.conf to look at hosts file and then dns server so that I can populate the individual device details into the hosts file so this server doesnt have to query the DNS server for the host details everytime it receives a log entry. Am I doing the right thing or is there a better way of accomplishing this? Also, planning to archive all of the logs onto files and not to simple files so that I can rotate the old logs DB every month. Again, am I doing the right thing or is there an alternate way to do this? Any suggestions and recommendations are gratefully accepted. -- Kumaran Find your IP @ www.itsyourip.com
On Fri, Dec 02, 2005 at 03:04:25PM +0000, Kumaran Babu wrote:
Basically, I look at logging close to 40 Network devices which can potentially log as much 10Gigs or more of data. I want to use the DNS hostnames to be displayed when viewed in php-syslog-ng interface so I enabled DNS on syslog-ng conf file. I've changed the nsswitch.conf to look at hosts file and then dns server so that I can populate the individual device details into the hosts file so this server doesnt have to query the DNS server for the host details everytime it receives a log entry.
Am I doing the right thing or is there a better way of accomplishing this?
This is good, speed up lookups using /etc/hosts, I'd also enable DNS caching in syslog-ng to help performance there (or run a local caching nameserver that's only listening on a loopback interface).
Also, planning to archive all of the logs onto files and not to simple files so that I can rotate the old logs DB every month. Again, am I doing the right thing or is there an alternate way to do this?
I don't know what you mean by this. What kind of log files are you using? -- Nate A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable.
Thanks for that. It looks I've completed confused u in the 2nd question. Right now, I've set all logs to go to the MySQL DB and also to the files so that they can be compressed and archived which allows me to rotate the MySQL DB once a month. This sounds a good idea but has a lot of negatives like, a. Archived files will not be on the web interface (simply because its not on DB anymore). Although, the archived logs may be read once in a blue moon or not read at all. b. This propels the Disk I/O to almost double as you write to the DB and to files at the same time. c. Storage space turns out to be an issue as well. What I also thought of was to see if I can archive the DB itself once in a month and store it on a Tape media which means anytime the archived DB can be mounted for reference and with little Adm task the syslogs can be viewed on the web interface. But, I'm no DB expert hence have no clue of issues in here. Any thoughts from experts here would be great. -- Kumaran Find your IP @ www.itsyourip.com On 12/8/05, Nate Campi <nate@campin.net> wrote:
On Fri, Dec 02, 2005 at 03:04:25PM +0000, Kumaran Babu wrote:
Basically, I look at logging close to 40 Network devices which can potentially log as much 10Gigs or more of data. I want to use the DNS hostnames to be displayed when viewed in php-syslog-ng interface so I enabled DNS on syslog-ng conf file. I've changed the nsswitch.conf to
look
at hosts file and then dns server so that I can populate the individual device details into the hosts file so this server doesnt have to query the DNS server for the host details everytime it receives a log entry.
Am I doing the right thing or is there a better way of accomplishing this?
This is good, speed up lookups using /etc/hosts, I'd also enable DNS caching in syslog-ng to help performance there (or run a local caching nameserver that's only listening on a loopback interface).
Also, planning to archive all of the logs onto files and not to simple files so that I can rotate the old logs DB every month. Again, am I doing the right thing or is there an alternate way to do this?
I don't know what you mean by this. What kind of log files are you using? -- Nate
A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable.
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
participants (2)
-
Kumaran Babu
-
Nate Campi