guys<br>
<br>
I've been pulling my hair out here. I gonna go insane....<br>
<br>
Here's the story.<br>
<br>
I first setup syslog-ng-1.6.6 for remote syslogging a year ago.<br>
<br>
I seemed to work out great - all my stuff appeared to be logged remotely and everything was dandy.<br>
Then I decided to come up with some sort of check that would tell me if a machine was not logging remotely anymore<br>
cause of network problems or whatever. I have a perl logchecker which I basically told to check for the cron.hourly log message<br>
and if it finds it to grep for the source host and add this host to a
file. I was then gonna check this list against a pre-defined array <br>
of hosts that should be logging blablaba....<br>
This is how I discovered that not everything is really logged. All my clients have this conf file:<br>
<br>
options { chain_hostnames(0);<br>
time_reopen(10);<br>
time_reap(360);<br>
log_fifo_size(2048);<br>
sync(1);<br>
stats(0);<br>
create_dirs(yes);<br>
owner(root);<br>
group(root);<br>
perm(0644);<br>
dir_perm(0755);<br>
use_dns(yes);<br>
dns_cache(yes); };<br>
<br>
source src { internal();<br>
unix-stream("/dev/log");<br>
file("/proc/kmsg"); };<br>
<br>
destination messages { file("/var/log/messages"); };<br>
destination auth { file("/var/log/auth"); };<br>
destination cron { file("/var/log/cron"); };<br>
destination mail { file("/var/log/mail"); };<br>
destination loghost { tcp("**********" port(5000)); };<br>
<br>
filter auth { facility(auth, authpriv); };<br>
filter cron { facility(cron); };<br>
filter mail { facility(mail); };<br>
filter nagios { not ( match("Accepted publickey for nagios from ********") or<br>
match("COMMAND=/usr/local/nagios/home/check_duplex") or<br>
match("session opened for user nagios") or<br>
match("session closed for user nagios")); };<br>
<br>
log { source(src); filter(cron); destination(cron); destination(loghost); flags(final); };<br>
log { source(src); filter(mail); destination(mail); destination(loghost); flags(final); };<br>
log { source(src); filter(auth); filter(nagios); destination(auth); destination(loghost); flags(final); };<br>
log { source(src); filter(nagios); destination(messages); destination(loghost); };<br>
<br clear="all">If I check the cronlog locally on a client I can see that all the hourly logs are there:<br>
<br>
Feb 27 12:01:01 sbeta crond[31964]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 27 13:01:01 sbeta crond[9232]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 27 14:01:01 sbeta crond[18969]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 27 15:01:01 sbeta crond[28630]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 27 16:01:01 sbeta crond[5311]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 27 17:01:01 sbeta crond[14977]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 27 18:01:01 sbeta crond[22335]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 27 19:01:01 sbeta crond[31995]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 27 20:01:01 sbeta crond[9336]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 27 21:01:01 sbeta crond[19002]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 27 22:01:01 sbeta crond[28663]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 27 23:01:01 sbeta crond[5928]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 28 00:01:01 sbeta crond[15495]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 28 01:01:01 sbeta crond[25160]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 28 02:01:01 sbeta crond[2394]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 28 03:01:01 sbeta crond[12087]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 28 04:01:01 sbeta crond[21753]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 28 05:01:01 sbeta crond[31573]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 28 06:01:01 sbeta crond[8840]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 28 07:01:01 sbeta crond[18506]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 28 08:01:01 sbeta crond[28166]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 28 09:01:01 sbeta crond[5433]: (root) CMD (run-parts /etc/cron.hourly)<br>
<br>
However, checking the remote cron logs for that machine for that whole of February shows<br>
this:<br>
<br>
Feb 21 12:01:01 sbeta crond[23754]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 24 05:01:01 sbeta crond[5012]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 24 17:01:01 sbeta crond[26525]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 25 03:01:01 sbeta crond[28212]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 25 13:01:01 sbeta crond[30890]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 26 01:01:01 sbeta crond[20620]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 16 19:01:01 sbeta crond[6299]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 17 00:01:01 sbeta crond[22138]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 17 04:01:01 sbeta crond[28292]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 17 08:01:01 sbeta crond[1917]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 17 12:01:01 sbeta crond[7812]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 17 16:01:01 sbeta crond[14136]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 17 19:01:01 sbeta crond[10669]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 18 00:01:01 sbeta crond[26489]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 18 04:01:01 sbeta crond[32742]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 18 08:01:01 sbeta crond[6852]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 18 12:01:01 sbeta crond[13106]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 18 15:01:01 sbeta crond[9768]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 18 20:01:01 sbeta crond[25588]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 12 08:01:01 sbeta crond[22554]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 12 12:01:01 sbeta crond[28808]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 12 16:01:01 sbeta crond[2735]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 12 20:01:01 sbeta crond[9063]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 13 00:01:01 sbeta crond[15268]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 14 17:01:01 sbeta crond[22845]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 14 21:01:02 sbeta crond[29177]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 15 22:01:01 sbeta crond[12945]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 16 05:01:01 sbeta crond[17917]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 16 16:01:01 sbeta crond[29758]: (root) CMD (run-parts /etc/cron.hourly)<br>
Feb 16 18:01:01 sbeta crond[16687]: (root) CMD (run-parts /etc/cron.hourly)<br>
<br>
As you can see it seemingly randomly logs this stuff now and then. So this machine hasn't been<br>
sending logs remotely since Feb 21. ? As I said all clients run the same version of syslog-ng and the<br>
exact same config - yet some of then seem to log cron stuff every hour and others don't.<br>
I get a log ot those 'Error connecting to remote host AF_INET' messages BUT i get those for EVERY host so<br>
why is there no consistency ?<br>
I know what everbody is gonna this - your network is hosed but here is the problem with that theory.<br>
<br>
I can log on to that box and I will immediately see this on the loghost in /var/log/auth<br>
<br>
Feb 28 10:43:06 sbeta sshd[21815]: Accepted publickey for stucky from ******** port 47427 ssh2<br>
Feb 28 10:43:06 sbeta sshd(pam_unix)[21819]: session opened for user stucky by (uid=0)<br>
<br>
This will work every time for all other machines as well. I'd understand if it didn't log at all or with a slight delay<br>
like say 1 minute. Besides it appears cron stuff really gets lost but then why doesn't any of the auth/authpriv<br>
stuff get lost ? I can't help feeling it has something to do with logging cron but looking at the config I clearly told<br>
it to log to the local file first and then again to the remote host just like auth/authpriv stuff.<br>
<br>
Here is my logservers config:<br>
<br>
options { chain_hostnames(0);<br>
time_reopen(10);<br>
time_reap(360);<br>
log_fifo_size(2048);<br>
use_fqdn(no);<br>
use_dns(yes);<br>
dns_cache(yes);<br>
keep_hostname(yes);<br>
long_hostnames(off);<br>
sync(1);<br>
stats(0);<br>
create_dirs(yes);<br>
perm(0640);<br>
dir_perm(0750);<br>
};<br>
<br>
source src { internal();<br>
unix-stream("/dev/log");<br>
file("/proc/kmsg");<br>
udp (ip("*********"));<br>
tcp (ip("*********")<br>
port(5000)<br>
max-connections(1000)<br>
keep-alive(yes)); };<br>
<br>
destination messages { file("/var/log/messages"); };<br>
destination auth { file("/var/log/auth"); };<br>
destination cron { file("/var/log/cron"); };<br>
destination mail { file("/var/log/mail"); };<br>
destination arch { file("/usr/local/var/log_archive/$HOST/$YEAR/$MONTH/$DAY/archive"); };<br>
<br>
filter auth { facility(auth, authpriv); };<br>
filter cron { facility(cron); };<br>
filter mail { facility(mail); };<br>
filter messages { not facility(auth, authpriv, cron, mail); };<br>
filter nagios { not ( match("Accepted publickey for nagios from *******") or<br>
match("COMMAND=/usr/local/nagios/home/check_duplex") or<br>
match("session opened for user nagios") or<br>
match("session closed for user nagios")); };<br>
filter junk { not ( match("Accepted publickey for oracle from ***********") or<br>
match("Accepted publickey for oracle from ***********") or<br>
match("Accepted publickey for stucky from ***********") or<br>
match("COMMAND=/usr/local/nagios/home/check_duplex") or<br>
match(".+reconnecting to LDAP server.+sleeping") or<br>
match("session opened for user root") or<br>
match(".+AF_INET client connected from .+") or<br>
match(".+AF_INET client dropped connection from.+") or<br>
match(".+Connection broken to AF_INET.+")); };<br>
<br>
log { source(src); filter(cron); destination(cron); destination(arch); flags(final); };<br>
log { source(src); filter(mail); destination(mail); destination(arch); flags(final); };<br>
log { source(src); filter(auth); filter(nagios); destination(auth); destination(arch); flags(final); };<br>
log { source(src); filter(junk); filter(nagios); destination(messages); destination(arch); };<br>
<br>
And here's the really messed up thing. EVERY time I set up a tcpdump on the loghost at the full hour <br>
to see whether the box actually sends stuff to it - it does ! Yet, if I wait again for a while and check the logs<br>
I can see again that a log of hourly cronlogs are skipped. <br>
Can anybody please take a look at my configs and at least tell me that they are not completely wrong ?<br>
I'm running out of ideas.<br>-- <br>stucky