[syslog-ng]Over zealous syslog-ng problem

[Ben Russo]

There are a few ways to look at this problem...

1. The box sending the messages..
	Do the 16,000,000 messages all have the same facility.priority?
	traditional syslog on solaris can only decide what to send based
	on facility and priority (and maybe the "tag" IIRC).
	So you may or may not be able to filter them at the sending side
	depending on whether the facility.priority of the messages is 
	unique to what you want to filter.

2.  The syslog-ng receiving the messages...
	Have your syslog-ng use the "match (regexp)" rule to filter
	out certain messages, but not others.  Maybe that will work?

3.  Have your perl program decide what to insert and what not to..

As far as losing the messages...  Syslog-ng doesn't buffer, so if your
mysql database isn't able to keep up with the flood of messages that are
coming in to the pipe and from there to your perl program then syslog-ng
drops them. (AFAIK)

The way that I have handled this in my situation is documented at
http://www.muppethouse.com/~ben/

I had syslog-ng format my incomming messages into SQL insert statements
in batches by second.  Then I have a program come by and pick up each
batch to be inserted and delete the batch file when it finishes.
This way if there is a flood of messages, they queue up in the directory
and get pushed into the database ASAP until the queue is empty.

-Ben.

On Tue, 2002-12-31 at 11:44, Aaron Jackson wrote:
> My Setup:
> I have syslog-ng running on several Solaris 8 machines.  Each machine 
> writes log messages to their local disks and also forwards the messages 
> to a central log server via a UPD connection, also a Solaris 8 machine. 
>   The central log server stores everything into a mysql database via a 
> perl script I wrote.
> 
> My Problem:
> I am running the UNIX version of Cisco Secure on one of the Solaris 
> boxes.  A couple of times it has blown up.  When this happens, it 
> generates millions of log messages in a very short period.  The problem 
> is that syslog-ng logs most of these messages (I also get the mangled 
> message problem during these heavy loads).  The most recent episode 
> generated 1,930,974 messages that made it into the mysql database and 
> 49,573 mangled messages on the central log server, but 16,040,886 
> messages were written to disk on the local machine (see below).
> 
> My Questions:
> Is there any way to throttle syslog-ng, or make syslog-ng not accept all 
>   log messages when an app goes crazy?  I want to log some of these 
> messages, so I know when to restart the service, but I don't want all 16 
> million.  Also, it seems that around 15 million log messages didn't make 
> it to my central server.  Where were they lost?  Is this a problem with 
> the UDP transport?
> 
> Aaron
> 
> # cat local0.log | grep -c 'ERROR - error on accept'
> 16040886
> 
> jackson@auth:/tmp {5} cat sql_errors | grep -c 'INSERT INTO'
> 49573
> 
> mysql> delete from logs where host='acs' and facility='local0' and 
> priority='err' and msg like '%ERROR - error on accept%';
> Query OK, 1943387 rows affected (1 hour 40.16 sec)
> 
> 
> _______________________________________________
> syslog-ng maillist  -  syslog-ng@lists.balabit.hu
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
-- 
Ben Russo <ben@umialumni.com>