[syslog-ng] Mass text log files processing over network.

[Balazs Scheidler]

On Wed, 2010-09-08 at 12:26 +0200, Tomasz Wrona wrote:
> Hello,
> 
> Could somebody point me if is it possible to configure syslog-ng to 
> automatic, mass, text log files processing over network?
> Which version eventually support following case?
> 
> 
> My case:
> Webservers run virtual servers, which logs their own php error logs 
> [text files].
> I want to send this logs to central log server keeping orginal 
> distribution based on virtual servers in the simplest way possible.
> 
> 
> I would like to have something like this virtual setup:
> 
> 
> #*** Client config ***#
> 
> source s_php {
>   # Wildcard match [only for commercial edition?]  and TAG it somehow:
>  file("/var/log/php/*-error.log" follow_freq(1) flags(no-parse) 
> TAG("$FILENAME: "));

as of now wildcards are only supported by the commercial edition. will
be published in the OSE version in the future though, probably at the
OSE-PE core merger (which is going to be syslog-ng PE 4.1 and syslog-ng
OSE 4.0, but I might decide to sync the version numbers to clear up
confusion).

See just released roadmap for the OSE here:
http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/features/roadmap

The tags() option cannot accept macros right now, but you could use a
rewrite rule to prefix the the message with the filename:

rewrite r_prog { set("$FILE_NAME: $MSG"); };

>   # eventually manual config for each file if above not possible:
>  file("/var/log/php/service1-error.log" follow_freq(1) flags(no-parse) 
> log_prefix("service1: "));
>  file("/var/log/php/service2-error.log" follow_freq(1) flags(no-parse) 
> log_prefix("service2: "));
> };

log_prefix() is deprecated in favor of program_override() and
host_override().

> 
> destination d_collector_php { udp("collector" port(5501)); };

This would add a proper syslog prefix to your originally non-syslog
formatted message ($DATE $HOST $MSG) and if you overwrote PROGRAM then
that'd get inserted too.

> 
> log { source(s_php); destination(d_collector_php); };
> 
> 
> 
> 
> #*** Collector Server config ***#
> 
> source s_network_php {
>     udp(ip(0.0.0.0) port(5501));
> };
> 
>  # Can I split log stream using MACRO based on custom information send in 
> log [or syslog header]?
> destination d_php { 
> file("/var/log/hosts/webservers/php/$HOST/$YEAR$MONTH$DAY/$TAG.log"); };
>  # or maybe:
> destination d_php { 
> file("/var/log/hosts/webservers/php/$HOST/$YEAR$MONTH$DAY/$log_prefix.log"); 
> };
> 
> log { source(s_network_php); destination(d_php); };

If you used program_override() on the client, that would get passed to
the server side, so you could simply use $PROGRAM on the server to get
the original program name.

If you construct more complex formats, then you could use either the
csv-parser() or the db-parser() or plain regexps groups to get the
necessary information from the message payload and use the values in
filenames.


-- 
Bazsi