<div dir="ltr"><div><div><div><div>Hi,<br><br>You should switch to unix-dgram for /dev/log for a few reasons:<br></div><br>- using dgram logs are "properly" terminated, one read will give back an entire message so there is no need of scanning for embedded nuls / newlines<br>
</div><div>- it doesn't have a connection limit<br>- it uses a single file descriptor instead of one fd per connected client (no need to poll hundreds of fds)<br></div></div>- FIFO so the ordering of logs (originating from multiple clients) won't change<br>
</div><div>- last but not at least: everyone else is using unix-dgram :)<br><br></div>hth,<br><br>Sandor<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Sep 12, 2013 at 4:25 AM, Micah Anderson <span dir="ltr"><<a href="mailto:micah@riseup.net" target="_blank">micah@riseup.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
Hello,<br>
<br>
I recently ugpraded a server to Debian Wheezy, which means I get 3.3.5-4<br>
of syslog-ng. I did my best to upgrade the config to the 3.3 format, and<br>
then started it up and it seemed to be working fine. So I started<br>
providing services there, but under load the system seemed to be paused.<br>
<br>
Load is quite low on the system, but what was happening was logins via<br>
dovecot or ssh were pausing for a very long time. Typically, this<br>
indicates a DNS issue, but I quickly ruled that out. Eventually, I was<br>
reduced to using strace and I found that strace was pausing for a really<br>
long time trying to write a log message to file descriptor 4, and<br>
looking at what that FD was, I found:<br>
<br>
[pid 9051] connect(4, {sa_family=AF_FILE, path="/dev/log"}, 110) = 0<br>
<br>
I stopped syslog-ng, and then voila, everything was working as it should<br>
be. I tried to increase my log_fifo_size(2048); to be 4096, but that<br>
didn't help, and so I started searching around and I found a similar bug<br>
in 2009 that was fixed[0].<br>
<br>
I figured that bug was properly fixed, but maybe something came back, or<br>
something is wrong in my config. I noticed in the bug log that one of<br>
the configuration options that they had I did not, so I tried adding<br>
a max-connections argument to the unix-stream for the /dev/log:<br>
<br>
unix-stream("/dev/log" max-connections(2048));<br>
<br>
so far that seems to have made things function ok, although the<br>
syslog-ng startup message produces a warning that has me puzzled:<br>
<br>
WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='12', new_log_iw_size='100', min_log_fifo_size='204800'<br>
<br>
So... I'm wondering, is this a bug in syslog-ng, or a bug in my config?<br>
<br>
Thanks for any help, or suggestions of things to try!<br>
<br>
Here is my config:<br>
<br>
@version: 3.3<br>
<br>
# First, set some global options.<br>
options {<br>
# default is zero<br>
flush_lines(25);<br>
# default is 10000(10s)<br>
flush_timeout(20000);<br>
use_dns(no);<br>
use_fqdn(no);<br>
owner("root");<br>
group("adm");<br>
perm(0640);<br>
stats_freq(0);<br>
time_reopen(10);<br>
time_reap(360);<br>
create_dirs(yes);<br>
log_fifo_size(4096);<br>
};<br>
<br>
########################<br>
# Sources<br>
########################<br>
source s_all {<br>
internal();<br>
unix-stream("/dev/log" max-connections(2048));<br>
file("/proc/kmsg" program_override("kernel"));<br>
};<br>
<br>
########################<br>
# Rewrites<br>
########################<br>
<br>
rewrite r_ip { subst('(25[0-5]|2[0-4][0-9]|[0-1]?[0-9]?[0-9])([\.\-](25[0-5]|2[0-4][0-9]|[0-1]?[0-9]?[0-9])){3}', "0.0.0.0", value("MESSAGE"), type("pcre"), flags("global" "nobackref")); };<br>
<br>
########################<br>
# Filters<br>
########################<br>
<br>
filter f_dbg { level(debug); };<br>
filter f_info { level(info); };<br>
filter f_notice { level(notice); };<br>
filter f_warn { level(warn); };<br>
filter f_err { level(err); };<br>
filter f_crit { level(crit .. emerg); };<br>
<br>
filter f_debug { level(debug) and not facility(auth, authpriv, news, mail); };<br>
filter f_error { level(err .. emerg) ; };<br>
filter f_cron { facility(cron) and not filter(f_debug); };<br>
filter f_local { facility(local0, local1, local3, local4, local5,<br>
local6, local7) and not filter(f_debug); };<br>
filter f_syslog3 { not facility(auth, authpriv, mail) and not filter(f_debug); };<br>
filter f_cother { level(debug, info, notice, warn) or facility(daemon, mail); };<br>
filter f_console { level(warn .. emerg); };<br>
<br>
########################<br>
# Destinations<br>
########################<br>
<br>
############################################################<br>
## apache.log<br>
<br>
filter f_apache { program("apache") and level(info); };<br>
destination d_apache { file("/var/log/apache/access.log"); };<br>
log {<br>
source(s_all);<br>
filter(f_apache);<br>
rewrite(r_ip);<br>
destination(d_apache);<br>
flags(final);<br>
};<br>
<br>
filter f_apache_err { program("apache") and level(err); };<br>
destination d_apache_err { file("/var/log/apache/error.log"); };<br>
log {<br>
source(s_all);<br>
filter(f_apache_err);<br>
rewrite(r_ip);<br>
destination(d_apache_err);<br>
flags(final);<br>
};<br>
<br>
############################################################<br>
## auth.log<br>
<br>
filter f_auth { facility(auth, authpriv); };<br>
destination df_auth { file("/var/log/auth.log"); };<br>
log {<br>
source(s_all);<br>
filter(f_auth);<br>
destination(df_auth);<br>
};<br>
<br>
############################################################<br>
## clamav.log<br>
filter f_clamav { program("^clamav-milter|clamd"); };<br>
destination d_clamav { file("/var/log/clamav.log"); };<br>
log {<br>
source(s_all);<br>
filter(f_clamav);<br>
destination(d_clamav);<br>
flags(final);<br>
};<br>
<br>
############################################################<br>
## courier.log<br>
<br>
filter f_courier { program("courier|imap|pop"); };<br>
destination d_courier { file("/var/log/courier.log"); };<br>
log {<br>
source(s_all);<br>
filter(f_courier);<br>
rewrite(r_ip);<br>
destination(d_courier);<br>
flags(final);<br>
};<br>
<br>
############################################################<br>
## dovecot.log<br>
filter f_dovecot { program("dovecot.*") and not message("^lda"); };<br>
destination d_dovecot { file("/var/log/dovecot/dovecot.log", group("dovecot")); };<br>
log {<br>
source(s_all);<br>
filter(f_dovecot);<br>
rewrite(r_ip);<br>
destination(d_dovecot);<br>
flags(final);<br>
};<br>
<br>
############################################################<br>
## dovecot-lda.log<br>
<br>
filter f_dovecot-lda { program("dovecot.*") and message("^lda"); };<br>
destination d_dovecot-lda { file("/var/log/dovecot/dovecot-lda.log", group("dovecot")); };<br>
<br>
log {<br>
source(s_all);<br>
filter(f_dovecot-lda);<br>
rewrite(r_ip);<br>
destination(d_dovecot-lda);<br>
flags(final);<br>
};<br>
############################################################<br>
## daemon.log<br>
<br>
filter f_daemon { facility(daemon); };<br>
destination df_daemon { file("/var/log/daemon.log"); };<br>
log {<br>
source(s_all);<br>
filter(f_daemon);<br>
rewrite(r_ip);<br>
destination(df_daemon);<br>
};<br>
<br>
############################################################<br>
## errors.log<br>
<br>
filter f_errors {<br>
level(warn,err,crit,alert,emerg)<br>
and not facility(auth,authpriv,daemon,mail,user,kern,local1);<br>
};<br>
destination df_errors { file("/var/log/errors.log"); };<br>
log {<br>
source(s_all);<br>
filter(f_errors);<br>
destination(df_errors);<br>
};<br>
<br>
############################################################<br>
## emergencies<br>
<br>
filter f_emerg { level(emerg); };<br>
destination du_all { usertty("*"); };<br>
log {<br>
source(s_all);<br>
filter(f_emerg);<br>
destination(du_all);<br>
};<br>
<br>
############################################################<br>
## kern.log<br>
<br>
filter f_kern { facility(kern); };<br>
destination df_kern { file("/var/log/kern.log"); };<br>
log {<br>
source(s_all);<br>
filter(f_kern);<br>
destination(df_kern);<br>
};<br>
<br>
############################################################<br>
## ldap.log<br>
<br>
filter f_ldap { program("slapd"); };<br>
destination d_ldap { file("/var/log/ldap.log"); };<br>
log {<br>
source(s_all);<br>
filter(f_ldap);<br>
destination(d_ldap);<br>
flags(final);<br>
};<br>
<br>
############################################################<br>
## maildrop.log<br>
<br>
filter f_maildrop { program("^maildrop"); };<br>
destination d_maildrop { file("/var/log/maildrop.log"); };<br>
log {<br>
source(s_all);<br>
filter(f_maildrop);<br>
destination(d_maildrop);<br>
flags(final);<br>
};<br>
<br>
############################################################<br>
## mail.log<br>
<br>
filter f_mail { facility(mail) and not facility(local1) and not program("^postfix/"); };<br>
destination df_mail { file("/var/log/mail.log"); };<br>
<br>
log {<br>
source(s_all);<br>
filter(f_mail);<br>
destination(df_mail);<br>
};<br>
<br>
############################################################<br>
## messages.log<br>
<br>
filter f_messages {<br>
level(debug,info,notice)<br>
and not facility(auth,authpriv,daemon,mail,user,kern,local1);<br>
};<br>
destination df_messages { file("/var/log/messages.log"); };<br>
log {<br>
source(s_all);<br>
filter(f_messages);<br>
destination(df_messages);<br>
};<br>
<br>
############################################################<br>
## perdition.log<br>
<br>
filter f_perdition { program("perdition"); };<br>
destination d_perdition { file("/var/log/perdition.log"); };<br>
log {<br>
source(s_all);<br>
filter(f_perdition);<br>
destination(d_perdition);<br>
flags(final);<br>
};<br>
<br>
############################################################<br>
## postfix.log<br>
<br>
filter f_postfix { program("^postfix/"); };<br>
# We increase the flush lines/timeout since postfix writes so much<br>
destination d_postfix { file("/var/log/postfix.log" flush_lines(500) flush_timeout(60000)); };<br>
log {<br>
source(s_all);<br>
filter(f_postfix);<br>
rewrite(r_ip);<br>
destination(d_postfix);<br>
flags(final);<br>
};<br>
<br>
############################################################<br>
## rails logging<br>
<br>
filter f_rails { program("rails"); };<br>
destination d_rails { file("/var/log/rails.log"); };<br>
log {<br>
source(s_all);<br>
filter(f_rails);<br>
destination(d_rails);<br>
flags(final);<br>
};<br>
<br>
############################################################<br>
## user.log<br>
<br>
filter f_user { facility(user) and not facility(local1); };<br>
destination df_user { file("/var/log/user.log"); };<br>
log {<br>
source(s_all);<br>
filter(f_user);<br>
destination(df_user);<br>
};<br>
<br>
# Some `catch-all' logfiles.<br>
#<br>
destination d_debug { file("/var/log/debug"); };<br>
destination d_error { file("/var/log/error"); };<br>
destination d_messages { file("/var/log/messages"); };<br>
<br>
# The root's console.<br>
#<br>
destination d_console { usertty("root"); };<br>
<br>
# Virtual console.<br>
#<br>
destination d_console_all { file("/dev/tty10"); };<br>
<br>
0. <a href="https://bugzilla.balabit.com/show_bug.cgi?id=48" target="_blank">https://bugzilla.balabit.com/show_bug.cgi?id=48</a><br>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div><br></div>