Are those destination names (d_applogs and d_syslog) used in the client configuration? No, I did only cut out log names in the cfg. All other configurations are the same.

Another thought: what if you are reading _back_ a stats message from file? It may be _very_ long and could easily be longer than log-msg-size() which defaults to 8k. That could explain why this happens. I had a problem with 17k character loglines (don't ask... I don’t like it either) where the too small configured log_msg_size cause corrupted filenames too...

Does this ring a bell? Yes... With stats_level(2);" the syslog stats beat my log_msg_size limit by a few thousand characters. Didn’t even bother to check this before because I thought a 128k character limit is ridiculous already. Thanks I will now watch the servers the next days and hope this will not pop up again.

Would be a nice feature if the msg just get cut and doesn't cause such problems :) -----Ursprüngliche Nachricht----- Von: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] Im Auftrag von Balazs Scheidler Gesendet: Montag, 20. August 2012 15:34 An: Syslog-ng users' and developers' mailing list Betreff: Re: [syslog-ng] special characters in filenames On Fri, 2012-08-17 at 14:24 +0000, Daniel Neubacher wrote:

Hello there,

I’ve got a pretty annoying problem with syslog-ng 3.3.5 and log filenames. I think my syslog-ng.cfg is a little unconventional because I’ve configured for each log file a file statement and a uniq program name to keep the filenames from the client on the server. The cfg I’ve attached bellow is only a small part of it because I can’t show you everything. In the original syslog-ng.cfg I’ve configured about 120 individual log files.

Now to the problem. Each day I’m getting log file names like this on my log server:

/log/applogs/2012/08/17/perl-1.hk.fra1.xing.com/t.file(d_applogs#0,/lo g/applogs/2012/08/17/mx2-2.xing.com/syslog-ng)=0',

/log/applogs/2012/08/15/syslog-2.log.fra2.xing.com/ropped='dst.file(d_ syslog#0,/log/syslog

/log/syslog/2012/08/17/ fai-1.ops.fra1 fai?

This is pretty annoying because my script which is compressing the logs is failing on these files. Does anybody know this problem?

These seem to be part of the stats message of syslog-ng, as if the $PROGRAM portion of those wouldn't be properly set. It's interesting too, as these seem not be complete. To diagnose this issue, it'd be nice to pinpoint which server generate the partial messages in the first place. Judging the identifiers in there (d_applogs#0, d_syslog#0), it seems to relate to the server. Are those destination names (d_applogs and d_syslog) used in the client configuration? I'm assuming that no, only the server has these destinations. This would mean, that somehow the stats message, that gets emitted by internal() is mangled. I've reviewed the code that generates the statistics message, and it is manually setting the various portions of the message $MESSSAGE, $PROGRAM, etc. No real parsing occurs, this would rule out the possibility that the statistics message itself would be mangled. Another thought: what if you are reading _back_ a stats message from file? It may be _very_ long and could easily be longer than log-msg-size() which defaults to 8k. That could explain why this happens. Does this ring a bell?

Server cfg:

@version: 3.3

options {

threaded(yes);

owner("root");

group("root");

perm(0660);

dir_owner("root");

dir_group("root");

dir_perm(0770);

create_dirs(yes);

stats_freq(600);

stats_level(2);

chain_hostnames(no);

check_hostname(yes);

keep_hostname(no);

dns_cache(yes);

dns_cache_size(16384);

dns_cache_expire(3600);

dns_cache_expire_failed(60);

use_fqdn(yes);

log_msg_size(128000);

log_fifo_size(1000000);

};

template t_plain {

template("$MSG\n");

template_escape(no);

};

filter f_syslog {

program(^auth.log)

or program(^cron.log)

or program(^daemon.log)

or program(^kern.log)

or program(^lpr.log)

or program(^mail.log)

or program(^syslog.log)

or program(^user.log)

or program(^uucp.log)

or program(^mail.info)

or program(^mail.warn)

or program(^mail.err)

or program(^news.crit)

or program(^news.err)

or program(^news.notice)

or program(^debug.log)

or program(^error.log)

or program(^messages.log)

or program(^ppp.log);

};

filter f_applogs {

not filter(f_syslog)

and program(".*.log");

};

source s_src {

unix-dgram("/dev/log" max-connections(500));

internal();

file("/proc/kmsg" program_override("kernel"));

};

source s_net {

udp(

log_fetch_limit(400)

so_rcvbuf(51200000)

keep_timestamp(yes)

port(514)

);

tcp(

max-connections(1000)

so_rcvbuf(51200000)

so_keepalive(yes)

keep_timestamp(yes)

port(514)

);

syslog();

};

destination d_messages { file("/var/log/messages"); };

destination d_syslog { file("/log/syslog/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST/$PROGRAM" template(t_plain)); };

destination d_syslog_onefile { file("/log/syslog_onefile/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST"); };

destination d_applogs { file("/log/applogs/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST/$PROGRAM" template(t_plain)); };

destination d_perlhk {tcp("perl-1.hk.fra1.xing.com" port(514));};

log {

source(s_src);

destination(d_messages);

};

log {

source(s_net);

filter(f_syslog);

destination(d_syslog);

};

log {

source(s_net);

filter(f_syslog);

destination(d_syslog_onefile);

};

log {

source(s_net);

filter(f_applogs);

destination(d_applogs);

};

log {

source(s_net);

filter(f_applogs);

destination(d_perlhk);

};

Client:

@version: 3.3

options {

threaded(yes);

use_dns(yes);

use_fqdn(yes);

dns_cache(yes);

dns_cache_size(16384);

dns_cache_expire(3600);

dns_cache_expire_failed(10);

log_msg_size(128000);

log_fifo_size(100000);

normalize_hostnames(yes);

check_hostname(yes);

bad_hostname("^gconfd$");

create_dirs(yes);

owner("root");

group("root");

perm(0640);

stats_freq(3600);

time_reopen(30);

};

# Applogs

source s_perl_applogs {

file(/www/applogs/admin.log follow_freq(1) flags(no-parse) program_override(admin.log));

file(/www/applogs/fcgid.log follow_freq(1) flags(no-parse) program_override(fcgid.log));

};

##Ruby

source s_ruby_applogs {

file("/virtual/cra/shared/log/production.log" follow_freq(1) flags(no-parse) program_override(production.log));

};

source s_syslog {

file("/var/log/auth.log" follow_freq(1) flags(no-parse) program_override("auth.log"));

file("/var/log/cron.log" follow_freq(1) flags(no-parse) program_override("cron.log"));

file("/var/log/daemon.log" follow_freq(1) flags(no-parse) program_override("daemon.log"));

file("/var/log/kern.log" follow_freq(1) flags(no-parse) program_override("kern.log"));

file("/var/log/lpr.log" follow_freq(1) flags(no-parse) program_override("lpr.log"));

file("/var/log/mail.log" follow_freq(1) flags(no-parse) program_override("mail.log"));

file("/var/log/syslog" follow_freq(1) flags(no-parse) program_override("syslog.log"));

file("/var/log/user.log" follow_freq(1) flags(no-parse) program_override("user.log"));

file("/var/log/uucp.log" follow_freq(1) flags(no-parse) program_override("uucp.log"));

file("/var/log/mail/mail.info" follow_freq(1) flags(no-parse) program_override("mail.info"));

file("/var/log/mail/mail.warn" follow_freq(1) flags(no-parse) program_override("mail.warn"));

file("/var/log/mail/mail.err" follow_freq(1) flags(no-parse) program_override("mail.err"));

file("/var/log/news/news.crit" follow_freq(1) flags(no-parse) program_override("news.crit"));

file("/var/log/news/news.err" follow_freq(1) flags(no-parse) program_override("news.err"));

file("/var/log/news/news.notice" follow_freq(1) flags(no-parse) program_override("news.notice"));

file("/var/log/debug" follow_freq(1) flags(no-parse) program_override("debug.log"));

file("/var/log/error" follow_freq(1) flags(no-parse) program_override("error.log"));

file("/var/log/messages" follow_freq(1) flags(no-parse) program_override("messages.log"));

file("/var/log/ppp.log" follow_freq(1) flags(no-parse) program_override("ppp.log"));

};

destination syslog-1.log.fra1 {

udp("syslog-1.log.fra1.xing.com" port(514));

};

destination syslog-2.log.fra1 {

tcp("syslog-2.log.fra1.xing.com" port(514));

};

destination syslog-1.log.fra2 {

tcp("syslog-1.log.fra2.xing.com" port(514));

};

destination syslog-2.log.fra2 {

tcp("syslog-2.log.fra2.xing.com" port(514));

};

log {

source(s_all);

destination(syslog-1.log.fra1);

};

log {

source(s_syslog);

source(s_perl_applogs);

source(s_ruby_applogs);

destination(syslog-2.log.fra1);

destination(syslog-1.log.fra2);

destination(syslog-2.log.fra2);

};

--

Daniel Neubacher, Network Administrator

daniel.neubacher@xing.com

XING AG

Gaensemarkt 43, 20354 Hamburg, Germany

Tel. +49 40 419131-28, Fax +49 40 419131-11

Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 98807

Exec. Board (Vorstand): Dr. Stefan Groß-Selbeck (Vorsitzender), Dr. Thomas Vollmoeller, Ingo Chu, Dr. Helmut Becker, Jens Pape

Chairman of the Supervisory Board (Aufsichtsratsvorsitzender): Dr. Neil Sunderland

This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden and may be unlawful.

______________________________________________________________________ ________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq