On 09/08/2010 06:26 AM, Tomasz Wrona wrote:

Hello,

Could somebody point me if is it possible to configure syslog-ng to automatic, mass, text log files processing over network? Which version eventually support following case?

My case: Webservers run virtual servers, which logs their own php error logs [text files]. I want to send this logs to central log server keeping orginal distribution based on virtual servers in the simplest way possible.

I would like to have something like this virtual setup:

#*** Client config ***#

source s_php { # Wildcard match [only for commercial edition?] and TAG it somehow: file("/var/log/php/*-error.log" follow_freq(1) flags(no-parse) TAG("$FILENAME: ")); # eventually manual config for each file if above not possible: file("/var/log/php/service1-error.log" follow_freq(1) flags(no-parse) log_prefix("service1: ")); file("/var/log/php/service2-error.log" follow_freq(1) flags(no-parse) log_prefix("service2: ")); };

destination d_collector_php { udp("collector" port(5501)); };

log { source(s_php); destination(d_collector_php); };

#*** Collector Server config ***#

source s_network_php { udp(ip(0.0.0.0) port(5501)); };

# Can I split log stream using MACRO based on custom information send in log [or syslog header]? destination d_php { file("/var/log/hosts/webservers/php/$HOST/$YEAR$MONTH$DAY/$TAG.log"); }; # or maybe: destination d_php { file("/var/log/hosts/webservers/php/$HOST/$YEAR$MONTH$DAY/$log_prefix.log"); };

log { source(s_network_php); destination(d_php); };

Regards Tom

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

Probably easier to just pipe the configs to logger, since you can set the facility and log level with it. This is what I do for my apache stuff, pretty much a standard practice among the guys I know. -- Joe McDonagh AIM: YoosingYoonickz IRC: joe-mac on freenode "When the going gets weird, the weird turn pro."

On Wed, 2010-09-08 at 12:26 +0200, Tomasz Wrona wrote:

Hello,

Could somebody point me if is it possible to configure syslog-ng to automatic, mass, text log files processing over network? Which version eventually support following case?

My case: Webservers run virtual servers, which logs their own php error logs [text files]. I want to send this logs to central log server keeping orginal distribution based on virtual servers in the simplest way possible.

I would like to have something like this virtual setup:

#*** Client config ***#

source s_php { # Wildcard match [only for commercial edition?] and TAG it somehow: file("/var/log/php/*-error.log" follow_freq(1) flags(no-parse) TAG("$FILENAME: "));

as of now wildcards are only supported by the commercial edition. will be published in the OSE version in the future though, probably at the OSE-PE core merger (which is going to be syslog-ng PE 4.1 and syslog-ng OSE 4.0, but I might decide to sync the version numbers to clear up confusion). See just released roadmap for the OSE here: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/... The tags() option cannot accept macros right now, but you could use a rewrite rule to prefix the the message with the filename: rewrite r_prog { set("$FILE_NAME: $MSG"); };

# eventually manual config for each file if above not possible: file("/var/log/php/service1-error.log" follow_freq(1) flags(no-parse) log_prefix("service1: ")); file("/var/log/php/service2-error.log" follow_freq(1) flags(no-parse) log_prefix("service2: ")); };

log_prefix() is deprecated in favor of program_override() and host_override().

destination d_collector_php { udp("collector" port(5501)); };

This would add a proper syslog prefix to your originally non-syslog formatted message ($DATE $HOST $MSG) and if you overwrote PROGRAM then that'd get inserted too.

log { source(s_php); destination(d_collector_php); };

#*** Collector Server config ***#

source s_network_php { udp(ip(0.0.0.0) port(5501)); };

# Can I split log stream using MACRO based on custom information send in log [or syslog header]? destination d_php { file("/var/log/hosts/webservers/php/$HOST/$YEAR$MONTH$DAY/$TAG.log"); }; # or maybe: destination d_php { file("/var/log/hosts/webservers/php/$HOST/$YEAR$MONTH$DAY/$log_prefix.log"); };

log { source(s_network_php); destination(d_php); };

If you used program_override() on the client, that would get passed to the server side, so you could simply use $PROGRAM on the server to get the original program name. If you construct more complex formats, then you could use either the csv-parser() or the db-parser() or plain regexps groups to get the necessary information from the message payload and use the values in filenames. -- Bazsi