Follow-freq(1) log-fifo-size(10) log-fetch-limit(10) does it sound like a viable setup? Basically I do not need to be completely synchronized to the other servers to the second but if I schedule syslog-ng to only transfer the data every minute or so then I'm really bound to have data loss cause I don’t know what to set the buffer variables to. Is there some norm in this situation ? I'm using syslog-ng 2.0.0-1etch1 on debian 4 It seems to work so far. Thanks. -----הודעה מקורית----- מאת: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] בשם Balazs Scheidler נשלח: ג 02 ספטמבר 2008 17:54 אל: Syslog-ng users' and developers' mailing list נושא: Re: [syslog-ng] apache logs over syslog-ng sychronized On Tue, 2008-09-02 at 17:39 +0300, Eli Shemer wrote:

Hey there,

I would like to know if this configuration sounds viable to you.

Right now it's not really operational performance wise.

What I'm basically trying to do is to have my syslog server completely synchronized to the apache access logs of my other servers.

On a client I have this configured:

source s_apachelogs { file("/etc/apache2/logs/test/2008-09-02-test.log"); };

destination df_apachelogs { tcp("192.168.200.4" port(1999)); };

log { source(s_apachelogs); destination(df_apachelogs); };

and on the server:

source s_apachelogs { tcp(ip(192.168.200.4) port(1999)); };

destination df_apachelogs { file("/var/log/apachelogs.log"); };

log {

source(s_apachelogs);

destination(df_apachelogs);

};

<

This is just a sample case. Later on I would like to have my server keep an /var/log/apachelogs/ directory completely identical to the logs of the other servers.

For some reason I don’t see any traffic passed over the line unless I run a /etc/init.d/syslog-ng reload which is really bizzar.

Well, I don't know which syslog-ng version you are running, latest 2.0.x or 2.1.x should be ok, but you should specify follow-freq(XX) in your config to tell syslog-ng that you want to follow the specified file and not read it from the beginning. Performance wise you will probably need to tune log-fetch-limit() and maybe log-fifo-size(), but see the recent thread titled "lost messages with follow_freq()?" on this mailing list. -- Bazsi ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

Well, perhaps, but I rather also keep the logs on the local machine incase there's some error in the log transfers over syslog-ng, so I'm more comfortable with the current setup. Right now i'm reading all the data from /etc/apache2/logs/*/*.log as defined by me and send them over the wire to syslog-ng. Then the goal is to reconstruct all of the logs from the simple log file syslog-ng is writing to on the server side. Possibly using log_prefix in syslog-ng and cronolog. It's vital for me to have on the server side apache logs in this structure: domain-date.log I'm hoping it wont require any recompilation of any binary file. -----הודעה מקורית----- מאת: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] בשם Patrice Levesque נשלח: ד 03 ספטמבר 2008 03:41 אל: Syslog-ng users' and developers' mailing list נושא: Re: [syslog-ng] apache logs over syslog-ng sychronized

IMHO it is better to let Apache log via Syslog in the first place. Try a httpd.conf line like: CustomLog "|/usr/bin/logger -p local1.info -t apache" complete

If you choose that direct-to-syslog path, maybe named pipes create less overhead and give you the flexibility you need to deal with multiple virtual hosts. -- --====|====-- --------================|================-------- Patrice Levesque http://ptaff.ca/ syslog-ng.wayne@ptaff.ca --------================|================-------- --====|====-- --

On Wed, 2008-09-03 at 18:02 +0300, Eli Shemer wrote:

source s_apachelogs { file("/etc/apache2/logs/test/2008-09-03-test.log" follow-freq(1) log_prefix("test") ); };

is it possible to somehow get the log_prefix string from the string in the file() ? otherwise I will have to write down close to 200 source definitions.

Not currently with 2.0 or 2.1 I'm afraid. With the next major release currently in the works it is. You can do things like: source s_apachelogs { file("/etc/apache2/logs/test/2008-09-03-test.log" follow-freq(1)); }; parser p_csvparser { csv_parser(columns('APACHE.VHOST_NAME', 'APACHE.CLIENT_IP', ...)); }; destination d_apache_files { file("/var/log/apache/access.log-${APACHE.VHOST_NAME}"); }; log { source(s_apachelogs); parser(p_csvparser); destination(d_apache_files); }; The key is the parser here, it extracts information from the log message and puts them into "values". A value is a name-value pair, associated with a message. You can use these user-defined values anywhere you could use template() before. You can also rewrite these values, or the message itself: rewrite r_sample { set("$FILE_NAME $MSG" value("MESSAGE")); }; This changes the contents of the "MESSAGE" value (that is the message payload itself), or for that matter, you can change custom values: rewrite r_sample { set("$FILE_NAME $MSG" value("USER.DEFINED.VALUE")); }; And then reference ${USER.DEFINED.VALUE} in your config. I'd really want to publish these features now, I'd only need about 1 day of silence which is difficult to get, as I'm involved in a load of other projects. However I'm leaving for a holiday on Friday, and probably will be able to do this. -- Bazsi