[syslog-ng] Store syslog occurrence frequency instead of adding all of them to the DB

[Marcos Tang]

Hi,

I am using Syslog-NG which stores the Syslog to a remote MySQL server for central logging. 

Now, I am running into a situation a lot of records inside the MySQL database is "almost" exactly the same, just except the timestamp is different. For example, I get "file system is full" every minute for a host and this message repeats itself periodically.

In order to better manage the MySQL database, do you have any suggestions how I can reduce the number of overall records inside the DB, but add the occurrence frequency instead? 

For example, syslog message "file system is full" occurs every minutes on the system. There will be 24 x 60 "file system is full" records on my MySQL DB after 1 day. If no one fix it, I will get 10 x 24 x 60 "file system is full" records  for 10 hosts having the same problem for 1 day.

Can those records being "processed" some how and when I search the MySQL DB, I only see ONE record list the following only? 

Total occurrenceMessage content
=====================
14,400File system is full

If I can do that, it will definitely reduce a log of space of my DB and speed up the query.

It seems this is a database problem as well. Please let me know your idea at anytime.

 
Regards,
Marcos
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20110818/df490a17/attachment.htm