[syslog-ng] special characters in filenames
Daniel Neubacher
daniel.neubacher at xing.com
Tue Aug 21 11:21:30 CEST 2012
>Are those destination names (d_applogs and d_syslog) used in the client configuration?
No, I did only cut out log names in the cfg. All other configurations are the same.
>Another thought: what if you are reading _back_ a stats message from file? It may be _very_ long and could easily be longer than
>log-msg-size() which defaults to 8k. That could explain why this happens.
I had a problem with 17k character loglines (don't ask... I don’t like it either) where the too small configured log_msg_size cause corrupted filenames too...
>Does this ring a bell?
Yes... With stats_level(2);" the syslog stats beat my log_msg_size limit by a few thousand characters. Didn’t even bother to check this before because I thought a 128k character limit is ridiculous already. Thanks I will now watch the servers the next days and hope this will not pop up again.
Would be a nice feature if the msg just get cut and doesn't cause such problems :)
-----Ursprüngliche Nachricht-----
Von: syslog-ng-bounces at lists.balabit.hu [mailto:syslog-ng-bounces at lists.balabit.hu] Im Auftrag von Balazs Scheidler
Gesendet: Montag, 20. August 2012 15:34
An: Syslog-ng users' and developers' mailing list
Betreff: Re: [syslog-ng] special characters in filenames
On Fri, 2012-08-17 at 14:24 +0000, Daniel Neubacher wrote:
> Hello there,
>
> I’ve got a pretty annoying problem with syslog-ng 3.3.5 and log
> filenames. I think my syslog-ng.cfg is a little unconventional because
> I’ve configured for each log file a file statement and a uniq program
> name to keep the filenames from the client on the server. The cfg I’ve
> attached bellow is only a small part of it because I can’t show you
> everything. In the original syslog-ng.cfg I’ve configured about 120
> individual log files.
>
> Now to the problem. Each day I’m getting log file names like this on
> my log server:
>
>
>
> /log/applogs/2012/08/17/perl-1.hk.fra1.xing.com/t.file(d_applogs#0,/lo
> g/applogs/2012/08/17/mx2-2.xing.com/syslog-ng)=0',
>
> /log/applogs/2012/08/15/syslog-2.log.fra2.xing.com/ropped='dst.file(d_
> syslog#0,/log/syslog
>
> /log/syslog/2012/08/17/ fai-1.ops.fra1 fai?
>
>
>
> This is pretty annoying because my script which is compressing the
> logs is failing on these files. Does anybody know this problem?
These seem to be part of the stats message of syslog-ng, as if the $PROGRAM portion of those wouldn't be properly set. It's interesting too, as these seem not be complete.
To diagnose this issue, it'd be nice to pinpoint which server generate the partial messages in the first place. Judging the identifiers in there (d_applogs#0, d_syslog#0), it seems to relate to the server. Are those destination names (d_applogs and d_syslog) used in the client configuration?
I'm assuming that no, only the server has these destinations. This would mean, that somehow the stats message, that gets emitted by internal() is mangled.
I've reviewed the code that generates the statistics message, and it is manually setting the various portions of the message $MESSSAGE, $PROGRAM, etc. No real parsing occurs, this would rule out the possibility that the statistics message itself would be mangled.
Another thought: what if you are reading _back_ a stats message from file? It may be _very_ long and could easily be longer than
log-msg-size() which defaults to 8k. That could explain why this happens.
Does this ring a bell?
>
>
>
> Server cfg:
>
>
>
> @version: 3.3
>
> options {
>
> threaded(yes);
>
> owner("root");
>
> group("root");
>
> perm(0660);
>
>
>
> dir_owner("root");
>
> dir_group("root");
>
> dir_perm(0770);
>
> create_dirs(yes);
>
>
>
> stats_freq(600);
>
> stats_level(2);
>
> chain_hostnames(no);
>
> check_hostname(yes);
>
> keep_hostname(no);
>
>
>
> dns_cache(yes);
>
> dns_cache_size(16384);
>
> dns_cache_expire(3600);
>
> dns_cache_expire_failed(60);
>
> use_fqdn(yes);
>
>
>
> log_msg_size(128000);
>
> log_fifo_size(1000000);
>
>
>
> };
>
>
>
> template t_plain {
>
> template("$MSG\n");
>
> template_escape(no);
>
> };
>
>
>
> filter f_syslog {
>
> program(^auth.log)
>
> or program(^cron.log)
>
> or program(^daemon.log)
>
> or program(^kern.log)
>
> or program(^lpr.log)
>
> or program(^mail.log)
>
> or program(^syslog.log)
>
> or program(^user.log)
>
> or program(^uucp.log)
>
> or program(^mail.info)
>
> or program(^mail.warn)
>
> or program(^mail.err)
>
> or program(^news.crit)
>
> or program(^news.err)
>
> or program(^news.notice)
>
> or program(^debug.log)
>
> or program(^error.log)
>
> or program(^messages.log)
>
> or program(^ppp.log);
>
> };
>
>
>
> filter f_applogs {
>
> not filter(f_syslog)
>
> and program(".*.log");
>
> };
>
>
>
> source s_src {
>
> unix-dgram("/dev/log" max-connections(500));
>
> internal();
>
> file("/proc/kmsg" program_override("kernel"));
>
> };
>
>
>
> source s_net {
>
> udp(
>
> log_fetch_limit(400)
>
> so_rcvbuf(51200000)
>
> keep_timestamp(yes)
>
> port(514)
>
> );
>
> tcp(
>
> max-connections(1000)
>
> so_rcvbuf(51200000)
>
> so_keepalive(yes)
>
> keep_timestamp(yes)
>
> port(514)
>
>
>
> );
>
> syslog();
>
> };
>
>
>
> destination d_messages { file("/var/log/messages"); };
>
> destination d_syslog
> { file("/log/syslog/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST/$PROGRAM"
> template(t_plain)); };
>
> destination d_syslog_onefile
> { file("/log/syslog_onefile/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST"); };
>
> destination d_applogs
> { file("/log/applogs/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST/$PROGRAM"
> template(t_plain)); };
>
> destination d_perlhk {tcp("perl-1.hk.fra1.xing.com" port(514));};
>
>
>
> log {
>
> source(s_src);
>
> destination(d_messages);
>
> };
>
>
>
> log {
>
> source(s_net);
>
> filter(f_syslog);
>
> destination(d_syslog);
>
> };
>
>
>
> log {
>
> source(s_net);
>
> filter(f_syslog);
>
> destination(d_syslog_onefile);
>
> };
>
>
>
> log {
>
> source(s_net);
>
> filter(f_applogs);
>
> destination(d_applogs);
>
> };
>
>
>
> log {
>
> source(s_net);
>
> filter(f_applogs);
>
> destination(d_perlhk);
>
> };
>
>
>
>
>
> Client:
>
>
>
> @version: 3.3
>
> options {
>
> threaded(yes);
>
>
>
> use_dns(yes);
>
> use_fqdn(yes);
>
> dns_cache(yes);
>
> dns_cache_size(16384);
>
> dns_cache_expire(3600);
>
> dns_cache_expire_failed(10);
>
>
>
> log_msg_size(128000);
>
> log_fifo_size(100000);
>
>
>
> normalize_hostnames(yes);
>
> check_hostname(yes);
>
> bad_hostname("^gconfd$");
>
>
>
> create_dirs(yes);
>
> owner("root");
>
> group("root");
>
> perm(0640);
>
>
>
> stats_freq(3600);
>
> time_reopen(30);
>
> };
>
>
>
>
>
> # Applogs
>
> source s_perl_applogs {
>
> file(/www/applogs/admin.log follow_freq(1) flags(no-parse)
> program_override(admin.log));
>
> file(/www/applogs/fcgid.log follow_freq(1) flags(no-parse)
> program_override(fcgid.log));
>
> };
>
>
>
> ##Ruby
>
> source s_ruby_applogs {
>
> file("/virtual/cra/shared/log/production.log" follow_freq(1)
> flags(no-parse) program_override(production.log));
>
>
>
> };
>
>
>
> source s_syslog {
>
> file("/var/log/auth.log" follow_freq(1) flags(no-parse)
> program_override("auth.log"));
>
> file("/var/log/cron.log" follow_freq(1) flags(no-parse)
> program_override("cron.log"));
>
> file("/var/log/daemon.log" follow_freq(1) flags(no-parse)
> program_override("daemon.log"));
>
> file("/var/log/kern.log" follow_freq(1) flags(no-parse)
> program_override("kern.log"));
>
> file("/var/log/lpr.log" follow_freq(1) flags(no-parse)
> program_override("lpr.log"));
>
> file("/var/log/mail.log" follow_freq(1) flags(no-parse)
> program_override("mail.log"));
>
> file("/var/log/syslog" follow_freq(1) flags(no-parse)
> program_override("syslog.log"));
>
> file("/var/log/user.log" follow_freq(1) flags(no-parse)
> program_override("user.log"));
>
> file("/var/log/uucp.log" follow_freq(1) flags(no-parse)
> program_override("uucp.log"));
>
> file("/var/log/mail/mail.info" follow_freq(1) flags(no-parse)
> program_override("mail.info"));
>
> file("/var/log/mail/mail.warn" follow_freq(1) flags(no-parse)
> program_override("mail.warn"));
>
> file("/var/log/mail/mail.err" follow_freq(1) flags(no-parse)
> program_override("mail.err"));
>
> file("/var/log/news/news.crit" follow_freq(1) flags(no-parse)
> program_override("news.crit"));
>
> file("/var/log/news/news.err" follow_freq(1) flags(no-parse)
> program_override("news.err"));
>
> file("/var/log/news/news.notice" follow_freq(1)
> flags(no-parse) program_override("news.notice"));
>
> file("/var/log/debug" follow_freq(1) flags(no-parse)
> program_override("debug.log"));
>
> file("/var/log/error" follow_freq(1) flags(no-parse)
> program_override("error.log"));
>
> file("/var/log/messages" follow_freq(1) flags(no-parse)
> program_override("messages.log"));
>
> file("/var/log/ppp.log" follow_freq(1) flags(no-parse)
> program_override("ppp.log"));
>
> };
>
>
>
>
>
> destination syslog-1.log.fra1 {
>
> udp("syslog-1.log.fra1.xing.com" port(514));
>
> };
>
>
>
> destination syslog-2.log.fra1 {
>
> tcp("syslog-2.log.fra1.xing.com" port(514));
>
> };
>
>
>
> destination syslog-1.log.fra2 {
>
> tcp("syslog-1.log.fra2.xing.com" port(514));
>
> };
>
>
>
> destination syslog-2.log.fra2 {
>
> tcp("syslog-2.log.fra2.xing.com" port(514));
>
> };
>
>
>
> log {
>
> source(s_all);
>
> destination(syslog-1.log.fra1);
>
> };
>
>
>
> log {
>
> source(s_syslog);
>
> source(s_perl_applogs);
>
> source(s_ruby_applogs);
>
> destination(syslog-2.log.fra1);
>
> destination(syslog-1.log.fra2);
>
> destination(syslog-2.log.fra2);
>
> };
>
>
>
>
>
> --
>
> Daniel Neubacher, Network Administrator
>
> daniel.neubacher at xing.com
>
>
>
> XING AG
>
> Gaensemarkt 43, 20354 Hamburg, Germany
>
> Tel. +49 40 419131-28, Fax +49 40 419131-11
>
>
>
> Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 98807
>
> Exec. Board (Vorstand): Dr. Stefan Groß-Selbeck (Vorsitzender), Dr.
> Thomas Vollmoeller, Ingo Chu, Dr. Helmut Becker, Jens Pape
>
> Chairman of the Supervisory Board (Aufsichtsratsvorsitzender): Dr.
> Neil Sunderland
>
>
>
> This e-mail may contain confidential and/or privileged information. If
> you are not the intended recipient (or have received this e-mail in
> error) please notify the sender immediately and destroy this e-mail.
> Any unauthorised copying, disclosure or distribution of the material
> in this e-mail is strictly forbidden and may be unlawful.
>
>
>
>
> ______________________________________________________________________
> ________ Member info:
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq
More information about the syslog-ng
mailing list