[syslog-ng] special characters in filenames
Balazs Scheidler
bazsi77 at gmail.com
Mon Aug 20 15:34:08 CEST 2012
On Fri, 2012-08-17 at 14:24 +0000, Daniel Neubacher wrote:
> Hello there,
>
> I’ve got a pretty annoying problem with syslog-ng 3.3.5 and log
> filenames. I think my syslog-ng.cfg is a little unconventional because
> I’ve configured for each log file a file statement and a uniq program
> name to keep the filenames from the client on the server. The cfg I’ve
> attached bellow is only a small part of it because I can’t show you
> everything. In the original syslog-ng.cfg I’ve configured about 120
> individual log files.
>
> Now to the problem. Each day I’m getting log file names like this on
> my log server:
>
>
>
> /log/applogs/2012/08/17/perl-1.hk.fra1.xing.com/t.file(d_applogs#0,/log/applogs/2012/08/17/mx2-2.xing.com/syslog-ng)=0',
>
> /log/applogs/2012/08/15/syslog-2.log.fra2.xing.com/ropped='dst.file(d_syslog#0,/log/syslog
>
> /log/syslog/2012/08/17/ fai-1.ops.fra1 fai?
>
>
>
> This is pretty annoying because my script which is compressing the
> logs is failing on these files. Does anybody know this problem?
These seem to be part of the stats message of syslog-ng, as if the
$PROGRAM portion of those wouldn't be properly set. It's interesting
too, as these seem not be complete.
To diagnose this issue, it'd be nice to pinpoint which server generate
the partial messages in the first place. Judging the identifiers in
there (d_applogs#0, d_syslog#0), it seems to relate to the server. Are
those destination names (d_applogs and d_syslog) used in the client
configuration?
I'm assuming that no, only the server has these destinations. This would
mean, that somehow the stats message, that gets emitted by internal() is
mangled.
I've reviewed the code that generates the statistics message, and it is
manually setting the various portions of the message $MESSSAGE,
$PROGRAM, etc. No real parsing occurs, this would rule out the
possibility that the statistics message itself would be mangled.
Another thought: what if you are reading _back_ a stats message from
file? It may be _very_ long and could easily be longer than
log-msg-size() which defaults to 8k. That could explain why this
happens.
Does this ring a bell?
>
>
>
> Server cfg:
>
>
>
> @version: 3.3
>
> options {
>
> threaded(yes);
>
> owner("root");
>
> group("root");
>
> perm(0660);
>
>
>
> dir_owner("root");
>
> dir_group("root");
>
> dir_perm(0770);
>
> create_dirs(yes);
>
>
>
> stats_freq(600);
>
> stats_level(2);
>
> chain_hostnames(no);
>
> check_hostname(yes);
>
> keep_hostname(no);
>
>
>
> dns_cache(yes);
>
> dns_cache_size(16384);
>
> dns_cache_expire(3600);
>
> dns_cache_expire_failed(60);
>
> use_fqdn(yes);
>
>
>
> log_msg_size(128000);
>
> log_fifo_size(1000000);
>
>
>
> };
>
>
>
> template t_plain {
>
> template("$MSG\n");
>
> template_escape(no);
>
> };
>
>
>
> filter f_syslog {
>
> program(^auth.log)
>
> or program(^cron.log)
>
> or program(^daemon.log)
>
> or program(^kern.log)
>
> or program(^lpr.log)
>
> or program(^mail.log)
>
> or program(^syslog.log)
>
> or program(^user.log)
>
> or program(^uucp.log)
>
> or program(^mail.info)
>
> or program(^mail.warn)
>
> or program(^mail.err)
>
> or program(^news.crit)
>
> or program(^news.err)
>
> or program(^news.notice)
>
> or program(^debug.log)
>
> or program(^error.log)
>
> or program(^messages.log)
>
> or program(^ppp.log);
>
> };
>
>
>
> filter f_applogs {
>
> not filter(f_syslog)
>
> and program(".*.log");
>
> };
>
>
>
> source s_src {
>
> unix-dgram("/dev/log" max-connections(500));
>
> internal();
>
> file("/proc/kmsg" program_override("kernel"));
>
> };
>
>
>
> source s_net {
>
> udp(
>
> log_fetch_limit(400)
>
> so_rcvbuf(51200000)
>
> keep_timestamp(yes)
>
> port(514)
>
> );
>
> tcp(
>
> max-connections(1000)
>
> so_rcvbuf(51200000)
>
> so_keepalive(yes)
>
> keep_timestamp(yes)
>
> port(514)
>
>
>
> );
>
> syslog();
>
> };
>
>
>
> destination d_messages { file("/var/log/messages"); };
>
> destination d_syslog
> { file("/log/syslog/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST/$PROGRAM"
> template(t_plain)); };
>
> destination d_syslog_onefile
> { file("/log/syslog_onefile/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST"); };
>
> destination d_applogs
> { file("/log/applogs/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST/$PROGRAM"
> template(t_plain)); };
>
> destination d_perlhk {tcp("perl-1.hk.fra1.xing.com" port(514));};
>
>
>
> log {
>
> source(s_src);
>
> destination(d_messages);
>
> };
>
>
>
> log {
>
> source(s_net);
>
> filter(f_syslog);
>
> destination(d_syslog);
>
> };
>
>
>
> log {
>
> source(s_net);
>
> filter(f_syslog);
>
> destination(d_syslog_onefile);
>
> };
>
>
>
> log {
>
> source(s_net);
>
> filter(f_applogs);
>
> destination(d_applogs);
>
> };
>
>
>
> log {
>
> source(s_net);
>
> filter(f_applogs);
>
> destination(d_perlhk);
>
> };
>
>
>
>
>
> Client:
>
>
>
> @version: 3.3
>
> options {
>
> threaded(yes);
>
>
>
> use_dns(yes);
>
> use_fqdn(yes);
>
> dns_cache(yes);
>
> dns_cache_size(16384);
>
> dns_cache_expire(3600);
>
> dns_cache_expire_failed(10);
>
>
>
> log_msg_size(128000);
>
> log_fifo_size(100000);
>
>
>
> normalize_hostnames(yes);
>
> check_hostname(yes);
>
> bad_hostname("^gconfd$");
>
>
>
> create_dirs(yes);
>
> owner("root");
>
> group("root");
>
> perm(0640);
>
>
>
> stats_freq(3600);
>
> time_reopen(30);
>
> };
>
>
>
>
>
> # Applogs
>
> source s_perl_applogs {
>
> file(/www/applogs/admin.log follow_freq(1) flags(no-parse)
> program_override(admin.log));
>
> file(/www/applogs/fcgid.log follow_freq(1) flags(no-parse)
> program_override(fcgid.log));
>
> };
>
>
>
> ##Ruby
>
> source s_ruby_applogs {
>
> file("/virtual/cra/shared/log/production.log" follow_freq(1)
> flags(no-parse) program_override(production.log));
>
>
>
> };
>
>
>
> source s_syslog {
>
> file("/var/log/auth.log" follow_freq(1) flags(no-parse)
> program_override("auth.log"));
>
> file("/var/log/cron.log" follow_freq(1) flags(no-parse)
> program_override("cron.log"));
>
> file("/var/log/daemon.log" follow_freq(1) flags(no-parse)
> program_override("daemon.log"));
>
> file("/var/log/kern.log" follow_freq(1) flags(no-parse)
> program_override("kern.log"));
>
> file("/var/log/lpr.log" follow_freq(1) flags(no-parse)
> program_override("lpr.log"));
>
> file("/var/log/mail.log" follow_freq(1) flags(no-parse)
> program_override("mail.log"));
>
> file("/var/log/syslog" follow_freq(1) flags(no-parse)
> program_override("syslog.log"));
>
> file("/var/log/user.log" follow_freq(1) flags(no-parse)
> program_override("user.log"));
>
> file("/var/log/uucp.log" follow_freq(1) flags(no-parse)
> program_override("uucp.log"));
>
> file("/var/log/mail/mail.info" follow_freq(1) flags(no-parse)
> program_override("mail.info"));
>
> file("/var/log/mail/mail.warn" follow_freq(1) flags(no-parse)
> program_override("mail.warn"));
>
> file("/var/log/mail/mail.err" follow_freq(1) flags(no-parse)
> program_override("mail.err"));
>
> file("/var/log/news/news.crit" follow_freq(1) flags(no-parse)
> program_override("news.crit"));
>
> file("/var/log/news/news.err" follow_freq(1) flags(no-parse)
> program_override("news.err"));
>
> file("/var/log/news/news.notice" follow_freq(1)
> flags(no-parse) program_override("news.notice"));
>
> file("/var/log/debug" follow_freq(1) flags(no-parse)
> program_override("debug.log"));
>
> file("/var/log/error" follow_freq(1) flags(no-parse)
> program_override("error.log"));
>
> file("/var/log/messages" follow_freq(1) flags(no-parse)
> program_override("messages.log"));
>
> file("/var/log/ppp.log" follow_freq(1) flags(no-parse)
> program_override("ppp.log"));
>
> };
>
>
>
>
>
> destination syslog-1.log.fra1 {
>
> udp("syslog-1.log.fra1.xing.com" port(514));
>
> };
>
>
>
> destination syslog-2.log.fra1 {
>
> tcp("syslog-2.log.fra1.xing.com" port(514));
>
> };
>
>
>
> destination syslog-1.log.fra2 {
>
> tcp("syslog-1.log.fra2.xing.com" port(514));
>
> };
>
>
>
> destination syslog-2.log.fra2 {
>
> tcp("syslog-2.log.fra2.xing.com" port(514));
>
> };
>
>
>
> log {
>
> source(s_all);
>
> destination(syslog-1.log.fra1);
>
> };
>
>
>
> log {
>
> source(s_syslog);
>
> source(s_perl_applogs);
>
> source(s_ruby_applogs);
>
> destination(syslog-2.log.fra1);
>
> destination(syslog-1.log.fra2);
>
> destination(syslog-2.log.fra2);
>
> };
>
>
>
>
>
> --
>
> Daniel Neubacher, Network Administrator
>
> daniel.neubacher at xing.com
>
>
>
> XING AG
>
> Gaensemarkt 43, 20354 Hamburg, Germany
>
> Tel. +49 40 419131-28, Fax +49 40 419131-11
>
>
>
> Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 98807
>
> Exec. Board (Vorstand): Dr. Stefan Groß-Selbeck (Vorsitzender), Dr.
> Thomas Vollmoeller, Ingo Chu, Dr. Helmut Becker, Jens Pape
>
> Chairman of the Supervisory Board (Aufsichtsratsvorsitzender): Dr.
> Neil Sunderland
>
>
>
> This e-mail may contain confidential and/or privileged information. If
> you are not the intended recipient (or have received this e-mail in
> error) please notify the sender immediately and destroy this e-mail.
> Any unauthorised copying, disclosure or distribution of the material
> in this e-mail is strictly forbidden and may be unlawful.
>
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
More information about the syslog-ng
mailing list