On Wed, 2010-09-08 at 12:26 +0200, Tomasz Wrona wrote:
Hello,
Could somebody point me if is it possible to configure syslog-ng to automatic, mass, text log files processing over network? Which version eventually support following case?
My case: Webservers run virtual servers, which logs their own php error logs [text files]. I want to send this logs to central log server keeping orginal distribution based on virtual servers in the simplest way possible.
I would like to have something like this virtual setup:
#*** Client config ***#
source s_php { # Wildcard match [only for commercial edition?] and TAG it somehow: file("/var/log/php/*-error.log" follow_freq(1) flags(no-parse) TAG("$FILENAME: "));
as of now wildcards are only supported by the commercial edition. will be published in the OSE version in the future though, probably at the OSE-PE core merger (which is going to be syslog-ng PE 4.1 and syslog-ng OSE 4.0, but I might decide to sync the version numbers to clear up confusion). See just released roadmap for the OSE here: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/... The tags() option cannot accept macros right now, but you could use a rewrite rule to prefix the the message with the filename: rewrite r_prog { set("$FILE_NAME: $MSG"); };
# eventually manual config for each file if above not possible: file("/var/log/php/service1-error.log" follow_freq(1) flags(no-parse) log_prefix("service1: ")); file("/var/log/php/service2-error.log" follow_freq(1) flags(no-parse) log_prefix("service2: ")); };
log_prefix() is deprecated in favor of program_override() and host_override().
destination d_collector_php { udp("collector" port(5501)); };
This would add a proper syslog prefix to your originally non-syslog formatted message ($DATE $HOST $MSG) and if you overwrote PROGRAM then that'd get inserted too.
log { source(s_php); destination(d_collector_php); };
#*** Collector Server config ***#
source s_network_php { udp(ip(0.0.0.0) port(5501)); };
# Can I split log stream using MACRO based on custom information send in log [or syslog header]? destination d_php { file("/var/log/hosts/webservers/php/$HOST/$YEAR$MONTH$DAY/$TAG.log"); }; # or maybe: destination d_php { file("/var/log/hosts/webservers/php/$HOST/$YEAR$MONTH$DAY/$log_prefix.log"); };
log { source(s_network_php); destination(d_php); };
If you used program_override() on the client, that would get passed to the server side, so you could simply use $PROGRAM on the server to get the original program name. If you construct more complex formats, then you could use either the csv-parser() or the db-parser() or plain regexps groups to get the necessary information from the message payload and use the values in filenames. -- Bazsi