[syslog-ng] organizing logs over network

[Sandor Geller]

Hi,

On 06/11/2015 08:15 AM, westlake wrote:
> if i use source-> system(facility(local0)); and use an application that
> sends to local0, what's the most effective way of categorizing this into
> a special log file if the destination is over network?
>
> I'm using two dns machines and they both send output to one logging
> machine.  So far I'm using the host() directive with filter{} and two
> log{} entries on the recipient, this way I can create a log depending on
> the ip.  Is there a better way to organize the logs instead of using
> host() with filter{} ?

Without seeing your configuration it isn't easy to tell that your 
solution is the most efficient or not so let's speak about filtering in 
general.

Fields in the header (including host and syslog priority) are parsed 
when the log gets received (unless you use the no-parse flag) so these 
are available in the first place and it is usually very efficient to 
filter on these fields. Filtering on syslog priority means numerical 
operations on a single byte so it is the fastest. When it comes to 
hostnames it could be a string up to 255 bytes but using an anchored 
regexp could be also fast.

Do you want to take into account the message payload as well like 
collecting certain lookup types / results into a common logfile? Take a 
look at patterndb in this case as it is a bit more complex but much more 
efficient than using dozens of regexp-based filters.

Multiple filters in a log statement are executed in the order as the 
filters are present in the configuration. These filters are logically 
'OR'-ed together so one can play with the ordering of filters but this 
can only be done when enough sample logs are available to see which 
filter is the most efficient (has small overhead and excludes the most 
logs so other filters will process significantly less data).

I usually add the final flag to all log sections to ensure that the log 
won't get evaluated in additional log sections.

Ordering the log sections would be another optimisation.

If you don't need anything fancy then have a separated network source 
and in the log section reference only this source. The first filter 
should be the syslog priority filter optionally followed by message 
payload-based filters and I'd use embedded log sections for separating 
logs based on the sender. If embedded log sections cover all logs 
processed by the log section then use the final flag in the outer log 
statement and you're done.

Regards,

Sandor