Logrotate Not Rotating All - Need Help Matching Rules
My logrotate.d/syslog-ng file contains a few entries for many different logs, see below. My syslog-ng.conf automatically creates entries for many different hosts, each with unique hostnames, depending on their purpose. The "purpose" for the hosts each have a different subdir under /var/log/Myhosts/, i.e. /var/log/Myhosts/Switches and /var/log/Myhosts/Firewalls. Logrotate does not work unless I've added each host separately under my logrotate conf, which is impossible since I dynamically create these entries via mysyslog-ng.conf . Any help greatly appreciated. .vp /var/log/Myhosts/*/*.log \ /var/log/Myhosts/Switches/*/*.log \ /var/log/Myhosts/*.log /var/log/messages \ /var/log/secure \ /var/log/maillog \ /var/log/spooler \ /var/log/boot.log \ /var/log/cron \ /var/log/mail/* { missingok weekly rotate 4 create compress copytruncate ifempty # olddir /var/log/Myhosts/Archived-Logs/ sharedscripts postrotate sharedscripts postrotate n/kill -HUP `cat /var/run/syslog-ng.pid 2> /dev/null` 2> /dev/null || true endscript
Quick workaround: Write a bash script to generate your logrotate files just before your logrotate is called by looking at what files exist. This will then logrotate all your dynamic files that exist at the time. You might want to have this script delete the logrotate configs for files which no longer exist as well if using one config per file... although I think it would be easier to have one big auto-generated config for all those dynamic files and have it replaced by the script once a day just before logrotate is run... -h -- Hari Sekhon wiskbroom@hotmail.com wrote:
My logrotate.d/syslog-ng file contains a few entries for many different logs, see below.
My syslog-ng.conf automatically creates entries for many different hosts, each with unique hostnames, depending on their purpose. The "purpose" for the hosts each have a different subdir under /var/log/Myhosts/, i.e. /var/log/Myhosts/Switches and /var/log/Myhosts/Firewalls.
Logrotate does not work unless I've added each host separately under my logrotate conf, which is impossible since I dynamically create these entries via mysyslog-ng.conf .
Any help greatly appreciated.
.vp
/var/log/Myhosts/*/*.log \ /var/log/Myhosts/Switches/*/*.log \ /var/log/Myhosts/*.log /var/log/messages \ /var/log/secure \ /var/log/maillog \ /var/log/spooler \ /var/log/boot.log \ /var/log/cron \ /var/log/mail/* { missingok weekly rotate 4 create compress copytruncate ifempty # olddir /var/log/Myhosts/Archived-Logs/ sharedscripts postrotate sharedscripts postrotate n/kill -HUP `cat /var/run/syslog-ng.pid 2> /dev/null` 2> /dev/null || true endscript
Thanks!I *think* I have a solution:1. Created a script named make-new-config.sh (see below make-new-config.sh)2. I created a file which will be appended to the dynamically generated one (see logrotate-append.txt below)3. I've added make-new-config.sh to /etc/cron.daily/logrotate on the line directly above /usr/sbin/logrotate /etc/logrotate.conf.What it does? Basically re-generate a new config for logrotate based on what is contained in /var/log/MyHosts/ with more than zero byte size.Any suggestions would be greatly appreciated..vp### make-new-config.sh ####!/bin/sh# This for recordkeeping/troubleshooting, until the next run that is.find /var/log/MyHosts/ -name "*.log" -empty -exec ls -al {} \; > /var/log/MyHosts/Deleted-Empty-Logs# This one to remove unwanted files, presumably from hosts no longer existent?find /var/log/MyHosts/ -name "*.log" -empty -exec rm {} \;# Generate list of valid/current logsfind /var/log/MyHosts/ -name "*.log" -print > /var/log/MyHosts/Latest-Logs-Prepend# Begin creating a new syslog-ng in logrotate.decho "#" > /etc/logrotate.d/syslog-ngecho "# DO NOT EDIT THIS FILE MANUALLY - SEE CRONJOB WHICH CREATED IT INSTEAD" >> /etc/logrotate.d/syslog-ngecho "#" >> /etc/logrotate.d/syslog-ng# Append two files together, making one final /etc/logrotate.d/syslog-ng.# The logrotate-append.txt is shown below.cat /var/log/MyHosts/Latest-Logs-Prepend /etc/syslog-ng/logrotate-append.txt >> /etc/logrotate.d/syslog-ng### logrotate-append.txt ##### DO NOT EDIT THIS FILE - IT IS AUTOMATICALLY CREATED VIA SCRIPT IN CRON## The non-MyHosts nodes will be placed below this line.# Basically, anything that does *NOT* live under /var/log/MyHosts//var/log/messages/var/log/secure/var/log/maillog/var/log/spooler/var/log/boot.log/var/log/cron { missingok weekly rotate 4 create compress copytruncate ifempty# olddir /var/log/MyHosts/Archived-Logs/ sharedscripts postrotate sharedscripts postrotate n/kill -HUP `cat /var/run/syslog-ng.pid 2> /dev/null` 2> /dev/null || true endscript}# END OF FILE> Date: Wed, 7 May 2008 13:00:05 +0100> From: hpsekhon@googlemail.com> To: syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Logrotate Not Rotating All - Need Help Matching Rules> > Quick workaround:> > Write a bash script to generate your logrotate files just before your > logrotate is called by looking at what files exist. This will then > logrotate all your dynamic files that exist at the time. You might want > to have this script delete the logrotate configs for files which no > longer exist as well if using one config per file... although I think it > would be easier to have one big auto-generated config for all those > dynamic files and have it replaced by the script once a day just before > logrotate is run...> > -h> > -- > Hari Sekhon> > > wiskbroom@hotmail.com wrote:> >> > My logrotate.d/syslog-ng file contains a few entries for many> > different logs, see below.> >> > My syslog-ng.conf automatically creates entries for many different> > hosts, each with unique hostnames, depending on their purpose. > > The "purpose" for the hosts each have a different subdir under> > /var/log/Myhosts/, i.e. /var/log/Myhosts/Switches and> > /var/log/Myhosts/Firewalls.> >> > Logrotate does not work unless I've added each host separately> > under my logrotate conf, which is impossible since I dynamically> > create these entries via mysyslog-ng.conf .> >> > Any help greatly appreciated.> >> > .vp
Greetings; My setup works well with one exception, my filtering rules contained in my syslog-ng.conf do not appear to work properly. My logs are not lost, instead they end up in a directory which I did not intend them to be in. Background: I log to a MySql DB, flatfiles and finally, to SEC, which parses stuff and takes various actions (almost working ;-) For ease of reading, I will simply add the contents of my config file which pertains to just one filter.Many thanks in advance for taking the time to read and help me. .vp ############ # OPTIONS # ############ options { chain_hostnames(no); create_dirs (yes); dir_perm(0755); use_dns (yes); dns_cache(yes); dns_cache_size(1000); dns_cache_expire(604800); keep_hostname(yes); log_fifo_size(10000); log_msg_size(8192); long_hostnames(on); perm(0644); stats(3600); sync(0); time_reopen (10); use_dns(yes); use_fqdn(yes); }; ############ # SOURCES # ############ source S_dgram { unix-dgram("/dev/log"); }; source S_internal { internal(); }; source S_kernel { file("/proc/kmsg" log_prefix("kernel: ")); }; source S_tcp { tcp(port(4800) keep-alive(yes) max_connections(100)); }; source S_udp { udp(ip("0.0.0.0") port(514)); }; ############### # DEST SQL DB # ############### destination D_db_mysql { pipe("/var/log/mysql.pipe" template("INSERT INTO logs (host, facility, priority, level, tag, datetime, program, msg) VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n") template-escape(yes)); }; ############## # DESTINATIONS # ############## destination D_authlog { file("/var/log/auth.log"); }; destination D_bootlog { file("/var/log/boot.log"); }; destination D_debug { file("/var/log/debug"); }; destination D_explan { file("/var/log/explanations"); }; destination D_messages { file("/var/log/messages"); }; destination D_secure { file("/var/log/secure"); }; destination D_spooler { file("/var/log/spooler"); }; destination D_syslog { file("/var/log/syslog"); }; destination D_user { file("/var/log/user.log"); }; destination D_switch { file("/var/log/MyHosts/Switches/$FULLHOST.log" perm(0644)); }; destination D_edge { file("/var/log/MyHosts/EdgeDevices/$FULLHOST.log" perm(0644)); }; destination D_firewall { file("/var/log/MyHosts/Firewalls/$FULLHOST.log" owner(root) group(root) perm(0644) dir_perm(0700) create_dirs(yes)); }; destination D_router { file("/var/log/MyHosts/Routers/$FULLHOST.log" perm(0644)); }; destination D_accesspoints { file("/var/log/MyHosts/AccessPoints/$FULLHOST.log" perm(0644)); }; destination D_mailservers { file("/var/log/MyHosts/MailServers/$FULLHOST.log" perm(0644)); }; ########### # FILTERS # ########### filter F_auth { facility(auth, authpriv); }; filter F_authpriv { facility(authpriv); }; filter F_cron { facility(cron); }; filter F_daemon { facility(daemon); }; filter F_kern { facility(kern); }; filter F_local1 { facility(local1); }; filter F_local2 { facility(local2); }; filter F_local3 { facility(local3); }; filter F_local4 { facility(local4); }; filter F_local5 { facility(local5); }; filter F_local6 { facility(local6); }; filter F_local7 { facility(local7); }; filter F_lpr { facility(lpr); }; filter F_mail { facility(mail); }; filter F_messages { facility(daemon, kern, user); }; filter F_news { facility(news); }; filter F_spooler { facility(uucp,news) and level(crit); }; filter F_syslog { not facility(auth, authpriv) and not facility(mail); }; filter F_user { facility(user); }; filter F_crit { level(crit); }; filter F_debug { level(debug); }; filter F_emergency { level(emerg); }; filter F_err { level(err); }; filter F_info { level(info); }; filter F_notice { level(notice); }; filter F_warn { level(warn); }; filter F_edge { host("edge*") or host("122.21.*"); }; filter F_router { host("gw*") or host("rtr") or host("mmsc"); }; filter F_switch { host("sw*") or host("sw1") or host("sw2"); }; filter F_firewall { host("^fw*") or host("^mlm*-*") or host("^cm*"); }; filter F_dc { host("^mydc*") or host("^dc*"); }; filter F_accesspoints { host("^melanie*"); }; filter F_mailservers { host("^mail*") or host("^smtpgw*"); }; filter F_proxies { host("^proxygw*"); }; filter F_InternetIP { host("161.17.10.*"); }; ############## # LOGS # ############## log { source(S_udp); destination(D_db_mysql); }; # Send ALL logs to SEC # log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_auth); destination(D_sec); }; # log { source(S_udp); source(S_tcp); destination(D_sec); }; log { source(S_udp); destination(D_sec); }; ### log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_auth); destination(D_authlog); flags(final); }; log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_local7); destination(D_bootlog); flags(final); }; log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_local1); destination(D_explan); flags(final); }; log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_local5); destination(D_router); flags(final); }; log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_messages); destination(D_messages); flags(final); }; log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_authpriv); destination(D_secure); flags(final); }; log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_spooler); destination(D_spooler); flags(final); }; log { source(S_dgram); source(S_internal); source(S_kernel); source(S_tcp); filter(F_syslog); destination(D_syslog); flags(final); }; log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_user); destination(D_user); flags(final); }; log { source(S_dgram); source(S_internal); source(S_kernel); source(S_tcp); destination(D_hosts); flags(final); }; log { source(S_udp); filter(F_switch); destination(D_switch); flags(final); }; log { source(S_udp); filter(F_firewall); destination(D_firewall); flags(final); }; log { source(S_udp); filter(F_router); destination(D_router); flags(final); }; log { source(S_udp); filter(F_edge); destination(D_edge); flags(final); }; log { source(S_udp); filter(F_dc); destination(D_dc); flags(final); }; log { source(S_udp); filter(F_accesspoints); destination(D_accesspoints); flags(final); }; log { source(S_udp); filter(F_proxies); destination(D_proxies); flags(final); }; log { source(S_udp); filter(F_mailservers); destination(D_mailservers); flags(final); }; log { source(S_udp); destination(D_udp);};
Your config seems a bit odd and maybe overly complex. Here's how I read it, please let us know if this is how you intend it to work. Your sources: S_dgram = only messages from local processes that call syslog() S_internal = only internal messages from syslog-ng on this node S_kernel = only messages from the kernel on this node S_tcp = only TCP messages from other nodes S_udp = only UDP messages from other nodes Your log paths tell syslog-ng to behave as: log { source(S_udp); destination(D_db_mysql); }; All UDP messages from other nodes should be sent to mysql. log { source(S_udp); destination(D_sec); }; All UDP messages from other nodes should be sent to D_sec. log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_auth); destination(D_authlog); flags(final); }; All messages matching F_auth and coming either from local syslog(), local syslog-ng internal or via TCP from other nodes (but not via UDP) should be sent to D_authlog. If you send a message down this path then don't bother evaluating any other paths. log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_local7); destination(D_bootlog); flags(final); }; All messages matching F_local7 and coming either from local syslog(), local syslog-ng internal or via TCP from other nodes (but not via UDP) should be sent to D_bootlog. If you send a message down this path then don't bother evaluating any other paths. Then there's a bunch of other log paths in the same vein. ________________________________ From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of wiskbroom@hotmail.com Sent: 08 May 2008 15:13 To: Syslog-ng users' and developers' mailing list Subject: [syslog-ng] Problems With Filter Rules - Using First Rule, Not One Intended Greetings; My setup works well with one exception, my filtering rules contained in my syslog-ng.conf do not appear to work properly. My logs are not lost, instead they end up in a directory which I did not intend them to be in. Background: I log to a MySql DB, flatfiles and finally, to SEC, which parses stuff and takes various actions (almost working ;-) For ease of reading, I will simply add the contents of my config file which pertains to just one filter. Many thanks in advance for taking the time to read and help me. .vp ############ # OPTIONS # ############ options { chain_hostnames(no); create_dirs (yes); dir_perm(0755); use_dns (yes); dns_cache(yes); dns_cache_size(1000); dns_cache_expire(604800); keep_hostname(yes); log_fifo_size(10000); log_msg_size(8192); long_hostnames(on); perm(0644); stats(3600); sync(0); time_reopen (10); use_dns(yes); use_fqdn(yes); }; ############ # SOURCES # ############ source S_dgram { unix-dgram("/dev/log"); }; source S_internal { internal(); }; source S_kernel { file("/proc/kmsg" log_prefix("kernel: ")); }; source S_tcp { tcp(port(4800) keep-alive(yes) max_connections(100)); }; source S_udp { udp(ip("0.0.0.0") port(514)); }; ############### # DEST SQL DB # ############### destination D_db_mysql { pipe("/var/log/mysql.pipe" template("INSERT INTO logs (host, facility, priority, level, tag, datetime, program, msg) VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n") template-escape(yes)); }; ############## # DESTINATIONS # ############## destination D_authlog { file("/var/log/auth.log"); }; destination D_bootlog { file("/var/log/boot.log"); }; destination D_debug { file("/var/log/debug"); }; destination D_explan { file("/var/log/explanations"); }; destination D_messages { file("/var/log/messages"); }; destination D_secure { file("/var/log/secure"); }; destination D_spooler { file("/var/log/spooler"); }; destination D_syslog { file("/var/log/syslog"); }; destination D_user { file("/var/log/user.log"); }; destination D_switch { file("/var/log/MyHosts/Switches/$FULLHOST.log" perm(0644)); }; destination D_edge { file("/var/log/MyHosts/EdgeDevices/$FULLHOST.log" perm(0644)); }; destination D_firewall { file("/var/log/MyHosts/Firewalls/$FULLHOST.log" owner(root) group(root) perm(0644) dir_perm(0700) create_dirs(yes)); }; destination D_router { file("/var/log/MyHosts/Routers/$FULLHOST.log" perm(0644)); }; destination D_accesspoints { file("/var/log/MyHosts/AccessPoints/$FULLHOST.log" perm(0644)); }; destination D_mailservers { file("/var/log/MyHosts/MailServers/$FULLHOST.log" perm(0644)); }; ########### # FILTERS # ########### filter F_auth { facility(auth, authpriv); }; filter F_authpriv { facility(authpriv); }; filter F_cron { facility(cron); }; filter F_daemon { facility(daemon); }; filter F_kern { facility(kern); }; filter F_local1 { facility(local1); }; filter F_local2 { facility(local2); }; filter F_local3 { facility(local3); }; filter F_local4 { facility(local4); }; filter F_local5 { facility(local5); }; filter F_local6 { facility(local6); }; filter F_local7 { facility(local7); }; filter F_lpr { facility(lpr); }; filter F_mail { facility(mail); }; filter F_messages { facility(daemon, kern, user); }; filter F_news { facility(news); }; filter F_spooler { facility(uucp,news) and level(crit); }; filter F_syslog { not facility(auth, authpriv) and not facility(mail); }; filter F_user { facility(user); }; filter F_crit { level(crit); }; filter F_debug { level(debug); }; filter F_emergency { level(emerg); }; filter F_err { level(err); }; filter F_info { level(info); }; filter F_notice { level(notice); }; filter F_warn { level(warn); }; filter F_edge { host("edge*") or host("122.21.*"); }; filter F_router { host("gw*") or host("rtr") or host("mmsc"); }; filter F_switch { host("sw*") or host("sw1") or host("sw2"); }; filter F_firewall { host("^fw*") or host("^mlm*-*") or host("^cm*"); }; filter F_dc { host("^mydc*") or host("^dc*"); }; filter F_accesspoints { host("^melanie*"); }; filter F_mailservers { host("^mail*") or host("^smtpgw*"); }; filter F_proxies { host("^proxygw*"); }; filter F_InternetIP { host("161.17.10.*"); }; ############## # LOGS # ############## log { source(S_udp); destination(D_db_mysql); }; # Send ALL logs to SEC # log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_auth); destination(D_sec); }; # log { source(S_udp); source(S_tcp); destination(D_sec); }; log { source(S_udp); destination(D_sec); }; ### log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_auth); destination(D_authlog); flags(final); }; log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_local7); destination(D_bootlog); flags(final); }; log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_local1); destination(D_explan); flags(final); }; log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_local5); destination(D_router); flags(final); }; log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_messages); destination(D_messages); flags(final); }; log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_authpriv); destination(D_secure); flags(final); }; log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_spooler); destination(D_spooler); flags(final); }; log { source(S_dgram); source(S_internal); source(S_kernel); source(S_tcp); filter(F_syslog); destination(D_syslog); flags(final); }; log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_user); destination(D_user); flags(final); }; log { source(S_dgram); source(S_internal); source(S_kernel); source(S_tcp); destination(D_hosts); flags(final); }; log { source(S_udp); filter(F_switch); destination(D_switch); flags(final); }; log { source(S_udp); filter(F_firewall); destination(D_firewall); flags(final); }; log { source(S_udp); filter(F_router); destination(D_router); flags(final); }; log { source(S_udp); filter(F_edge); destination(D_edge); flags(final); }; log { source(S_udp); filter(F_dc); destination(D_dc); flags(final); }; log { source(S_udp); filter(F_accesspoints); destination(D_accesspoints); flags(final); }; log { source(S_udp); filter(F_proxies); destination(D_proxies); flags(final); }; log { source(S_udp); filter(F_mailservers); destination(D_mailservers); flags(final); }; log { source(S_udp); destination(D_udp);}; <!--[if !supportLineBreakNewLine]--> <!--[endif]-->
Joe; Many thanks for the fast reply! My problems lie with the other filters, the ones at the end: filter F_edge { host("edge*") or host("122.21.*"); }; filter F_router { host("gw*") or host("rtr") or host("mmsc"); }; filter F_switch { host("sw*") or host("sw1") or host("sw2"); }; filter F_firewall { host("^fw*") or host("^mlm*-*") or host("^cm*"); }; filter F_dc { host("^mydc*") or host("^dc*"); }; filter F_accesspoints { host("^melanie*"); }; filter F_mailservers { host("^mail*") or host("^smtpgw*"); }; filter F_proxies { host("^proxygw*"); }; filter F_InternetIP { host("161.17.10.*"); }; The above, based on the filter rule for F_mailservers, should place anything coming in from a host named mailserver1, or smtpgw1 into destination D_mailservers, which in turn should save logs into file named /var/log/MyHosts/MailServers/$FULLHOST.log. Instead I find those logs in /var/log/MyHosts/Switches/$FULLHOST.log (which is really /var/log/MyHosts/Switches/mailserver1.mycorp.net/mailserver1.mycorp.net.log) I need to figure out a way to write the differences for hosts that begin with pattern xxx (^xxx)? and those with xxx at the end (*xxx) and those with xxx in the middle (*xxx)?, and for the life of me, I can't fifure out why the above is sending into Switches :-( Sorry if I am confusing this matter, and the reason for the overcomplication on my config is largely due to me being very new at this great tool. Thanks! .vp From: Joe.Fegan@hp.com To: syslog-ng@lists.balabit.hu Date: Thu, 8 May 2008 16:08:03 +0000 Subject: Re: [syslog-ng] Problems With Filter Rules - Using First Rule, Not One Intended Your config seems a bit odd and maybe overly complex. Here's how I read it, please let us know if this is how you intend it to work. Your sources: S_dgram = only messages from local processes that call syslog() S_internal = only internal messages from syslog-ng on this node S_kernel = only messages from the kernel on this node S_tcp = only TCP messages from other nodes S_udp = only UDP messages from other nodes Your log paths tell syslog-ng to behave as: log { source(S_udp); destination(D_db_mysql); }; All UDP messages from other nodes should be sent to mysql. log { source(S_udp); destination(D_sec); }; All UDP messages from other nodes should be sent to D_sec. log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_auth); destination(D_authlog); flags(final); }; All messages matching F_auth and coming either from local syslog(), local syslog-ng internal or via TCP from other nodes (but not via UDP) should be sent to D_authlog. If you send a message down this path then don't bother evaluating any other paths. log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_local7); destination(D_bootlog); flags(final); }; All messages matching F_local7 and coming either from local syslog(), local syslog-ng internal or via TCP from other nodes (but not via UDP) should be sent to D_bootlog. If you send a message down this path then don't bother evaluating any other paths. Then there's a bunch of other log paths in the same vein. From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of wiskbroom@hotmail.com Sent: 08 May 2008 15:13 To: Syslog-ng users' and developers' mailing list Subject: [syslog-ng] Problems With Filter Rules - Using First Rule, Not One Intended Greetings; My setup works well with one exception, my filtering rules contained in my syslog-ng.conf do not appear to work properly. My logs are not lost, instead they end up in a directory which I did not intend them to be in. Background: I log to a MySql DB, flatfiles and finally, to SEC, which parses stuff and takes various actions (almost working ;-) For ease of reading, I will simply add the contents of my config file which pertains to just one filter. Many thanks in advance for taking the time to read and help me. .vp ############ # OPTIONS # ############ options { chain_hostnames(no); create_dirs (yes); dir_perm(0755); use_dns (yes); dns_cache(yes); dns_cache_size(1000); dns_cache_expire(604800); keep_hostname(yes); log_fifo_size(10000); log_msg_size(8192); long_hostnames(on); perm(0644); stats(3600); sync(0); time_reopen (10); use_dns(yes); use_fqdn(yes); }; ############ # SOURCES # ############ source S_dgram { unix-dgram("/dev/log"); }; source S_internal { internal(); }; source S_kernel { file("/proc/kmsg" log_prefix("kernel: ")); }; source S_tcp { tcp(port(4800) keep-alive(yes) max_connections(100)); }; source S_udp { udp(ip("0.0.0.0") port(514)); }; ############### # DEST SQL DB # ############### destination D_db_mysql { pipe("/var/log/mysql.pipe" template("INSERT INTO logs (host, facility, priority, level, tag, datetime, program, msg) VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n") template-escape(yes)); }; ############## # DESTINATIONS # ############## destination D_authlog { file("/var/log/auth.log"); }; destination D_bootlog { file("/var/log/boot.log"); }; destination D_debug { file("/var/log/debug"); }; destination D_explan { file("/var/log/explanations"); }; destination D_messages { file("/var/log/messages"); }; destination D_secure { file("/var/log/secure"); }; destination D_spooler { file("/var/log/spooler"); }; destination D_syslog { file("/var/log/syslog"); }; destination D_user { file("/var/log/user.log"); }; destination D_switch { file("/var/log/MyHosts/Switches/$FULLHOST.log" perm(0644)); }; destination D_edge { file("/var/log/MyHosts/EdgeDevices/$FULLHOST.log" perm(0644)); }; destination D_firewall { file("/var/log/MyHosts/Firewalls/$FULLHOST.log" owner(root) group(root) perm(0644) dir_perm(0700) create_dirs(yes)); }; destination D_router { file("/var/log/MyHosts/Routers/$FULLHOST.log" perm(0644)); }; destination D_accesspoints { file("/var/log/MyHosts/AccessPoints/$FULLHOST.log" perm(0644)); }; destination D_mailservers { file("/var/log/MyHosts/MailServers/$FULLHOST.log" perm(0644)); }; ########### # FILTERS # ########### filter F_auth { facility(auth, authpriv); }; filter F_authpriv { facility(authpriv); }; filter F_cron { facility(cron); }; filter F_daemon { facility(daemon); }; filter F_kern { facility(kern); }; filter F_local1 { facility(local1); }; filter F_local2 { facility(local2); }; filter F_local3 { facility(local3); }; filter F_local4 { facility(local4); }; filter F_local5 { facility(local5); }; filter F_local6 { facility(local6); }; filter F_local7 { facility(local7); }; filter F_lpr { facility(lpr); }; filter F_mail { facility(mail); }; filter F_messages { facility(daemon, kern, user); }; filter F_news { facility(news); }; filter F_spooler { facility(uucp,news) and level(crit); }; filter F_syslog { not facility(auth, authpriv) and not facility(mail); }; filter F_user { facility(user); }; filter F_crit { level(crit); }; filter F_debug { level(debug); }; filter F_emergency { level(emerg); }; filter F_err { level(err); }; filter F_info { level(info); }; filter F_notice { level(notice); }; filter F_warn { level(warn); }; filter F_edge { host("edge*") or host("122.21.*"); }; filter F_router { host("gw*") or host("rtr") or host("mmsc"); }; filter F_switch { host("sw*") or host("sw1") or host("sw2"); }; filter F_firewall { host("^fw*") or host("^mlm*-*") or host("^cm*"); }; filter F_dc { host("^mydc*") or host("^dc*"); }; filter F_accesspoints { host("^melanie*"); }; filter F_mailservers { host("^mail*") or host("^smtpgw*"); }; filter F_proxies { host("^proxygw*"); }; filter F_InternetIP { host("161.17.10.*"); }; ############## # LOGS # ############## log { source(S_udp); destination(D_db_mysql); }; # Send ALL logs to SEC # log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_auth); destination(D_sec); }; # log { source(S_udp); source(S_tcp); destination(D_sec); }; log { source(S_udp); destination(D_sec); }; ### log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_auth); destination(D_authlog); flags(final); }; log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_local7); destination(D_bootlog); flags(final); }; log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_local1); destination(D_explan); flags(final); }; log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_local5); destination(D_router); flags(final); }; log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_messages); destination(D_messages); flags(final); }; log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_authpriv); destination(D_secure); flags(final); }; log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_spooler); destination(D_spooler); flags(final); }; log { source(S_dgram); source(S_internal); source(S_kernel); source(S_tcp); filter(F_syslog); destination(D_syslog); flags(final); }; log { source(S_dgram); source(S_internal); source(S_tcp); filter(F_user); destination(D_user); flags(final); }; log { source(S_dgram); source(S_internal); source(S_kernel); source(S_tcp); destination(D_hosts); flags(final); }; log { source(S_udp); filter(F_switch); destination(D_switch); flags(final); }; log { source(S_udp); filter(F_firewall); destination(D_firewall); flags(final); }; log { source(S_udp); filter(F_router); destination(D_router); flags(final); }; log { source(S_udp); filter(F_edge); destination(D_edge); flags(final); }; log { source(S_udp); filter(F_dc); destination(D_dc); flags(final); }; log { source(S_udp); filter(F_accesspoints); destination(D_accesspoints); flags(final); }; log { source(S_udp); filter(F_proxies); destination(D_proxies); flags(final); }; log { source(S_udp); filter(F_mailservers); destination(D_mailservers); flags(final); }; log { source(S_udp); destination(D_udp);}; <!--[if !supportLineBreakNewLine]--> <!--[endif]-->
Hi,
My problems lie with the other filters, the ones at the end:
filter F_edge { host("edge*") or host("122.21.*"); }; filter F_router { host("gw*") or host("rtr") or host("mmsc"); }; filter F_switch { host("sw*") or host("sw1") or host("sw2"); }; filter F_firewall { host("^fw*") or host("^mlm*-*") or host("^cm*"); }; filter F_dc { host("^mydc*") or host("^dc*"); }; filter F_accesspoints { host("^melanie*"); }; filter F_mailservers { host("^mail*") or host("^smtpgw*"); }; filter F_proxies { host("^proxygw*"); }; filter F_InternetIP { host("161.17.10.*"); };
The above, based on the filter rule for F_mailservers, should place anything coming in from a host named mailserver1, or smtpgw1 into destination D_mailservers, which in turn should save logs into file named /var/log/MyHosts/MailServers/$FULLHOST.log. Instead I find those logs in /var/log/MyHosts/Switches/$FULLHOST.log (which is really /var/log/MyHosts/Switches/mailserver1.mycorp.net/mailserver1.m ycorp.net.log)
It would be nice to see at least a log entry from the file. BTW how did the hostname appear twice in the destination filename? Either I overlooked something or you're not using exactly the same config you sent.
I need to figure out a way to write the differences for hosts that begin with pattern xxx (^xxx)? and those with xxx at the end (*xxx) and those with xxx in the middle (*xxx)?, and for the life of me, I can't fifure out why the above is sending into Switches :-(
You've anchors in your filter regexps already. "^xxx", "xxx$", ".xxx." are what you need if I understand you correctly. Regards, Sandor
It's usually best to put the "always do this" rules first and the "stop if this rule is matched" rules afterwards. Syslog-ng works down the list of rules from the top like this pseudo-code: foreach rule in ruleslist { if (message matches rule) { send message down this path if (rule has flags(final) in it) { break } } } So if a flags(final) rule gets satisfied then no subsequent rule will even be evaluated. -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Geller, Sandor (IT) Sent: 08 May 2008 18:05 To: 'Syslog-ng users' and developers' mailing list' Subject: Re: [syslog-ng] Problems With Filter Rules - Using First Rule, Not One Intended Hi,
My problems lie with the other filters, the ones at the end:
filter F_edge { host("edge*") or host("122.21.*"); }; filter F_router { host("gw*") or host("rtr") or host("mmsc"); }; filter F_switch { host("sw*") or host("sw1") or host("sw2"); }; filter F_firewall { host("^fw*") or host("^mlm*-*") or host("^cm*"); }; filter F_dc { host("^mydc*") or host("^dc*"); }; filter F_accesspoints { host("^melanie*"); }; filter F_mailservers { host("^mail*") or host("^smtpgw*"); }; filter F_proxies { host("^proxygw*"); }; filter F_InternetIP { host("161.17.10.*"); };
The above, based on the filter rule for F_mailservers, should place anything coming in from a host named mailserver1, or smtpgw1 into destination D_mailservers, which in turn should save logs into file named /var/log/MyHosts/MailServers/$FULLHOST.log. Instead I find those logs in /var/log/MyHosts/Switches/$FULLHOST.log (which is really /var/log/MyHosts/Switches/mailserver1.mycorp.net/mailserver1.m ycorp.net.log)
It would be nice to see at least a log entry from the file. BTW how did the hostname appear twice in the destination filename? Either I overlooked something or you're not using exactly the same config you sent.
I need to figure out a way to write the differences for hosts that begin with pattern xxx (^xxx)? and those with xxx at the end (*xxx) and those with xxx in the middle (*xxx)?, and for the life of me, I can't fifure out why the above is sending into Switches :-(
You've anchors in your filter regexps already. "^xxx", "xxx$", ".xxx." are what you need if I understand you correctly. Regards, Sandor ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
That is *exactly* what I am doing, no? log { source(S_udp); filter(F_mailservers); destination(D_mailservers); flags(final); }; # above catches inbound, from a remote host udp only, with pattern matching F_mailservers, sending it down the path of D_mailservers. log { source(S_udp); destination(D_udp);}; # The above is a catchall, nothing seems to go into here. ??? .vp
From: Joe.Fegan@hp.com To: syslog-ng@lists.balabit.hu Date: Thu, 8 May 2008 17:44:53 +0000 Subject: Re: [syslog-ng] Problems With Filter Rules - Using First Rule, Not One Intended
It's usually best to put the "always do this" rules first and the "stop if this rule is matched" rules afterwards. Syslog-ng works down the list of rules from the top like this pseudo-code:
foreach rule in ruleslist { if (message matches rule) { send message down this path if (rule has flags(final) in it) { break } } }
So if a flags(final) rule gets satisfied then no subsequent rule will even be evaluated.
-----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Geller, Sandor (IT) Sent: 08 May 2008 18:05 To: 'Syslog-ng users' and developers' mailing list' Subject: Re: [syslog-ng] Problems With Filter Rules - Using First Rule, Not One Intended
Hi,
My problems lie with the other filters, the ones at the end:
filter F_edge { host("edge*") or host("122.21.*"); }; filter F_router { host("gw*") or host("rtr") or host("mmsc"); }; filter F_switch { host("sw*") or host("sw1") or host("sw2"); }; filter F_firewall { host("^fw*") or host("^mlm*-*") or host("^cm*"); }; filter F_dc { host("^mydc*") or host("^dc*"); }; filter F_accesspoints { host("^melanie*"); }; filter F_mailservers { host("^mail*") or host("^smtpgw*"); }; filter F_proxies { host("^proxygw*"); }; filter F_InternetIP { host("161.17.10.*"); };
The above, based on the filter rule for F_mailservers, should place anything coming in from a host named mailserver1, or smtpgw1 into destination D_mailservers, which in turn should save logs into file named /var/log/MyHosts/MailServers/$FULLHOST.log. Instead I find those logs in /var/log/MyHosts/Switches/$FULLHOST.log (which is really /var/log/MyHosts/Switches/mailserver1.mycorp.net/mailserver1.m ycorp.net.log)
It would be nice to see at least a log entry from the file. BTW how did the hostname appear twice in the destination filename? Either I overlooked something or you're not using exactly the same config you sent.
I need to figure out a way to write the differences for hosts that begin with pattern xxx (^xxx)? and those with xxx at the end (*xxx) and those with xxx in the middle (*xxx)?, and for the life of me, I can't fifure out why the above is sending into Switches :-(
You've anchors in your filter regexps already. "^xxx", "xxx$", ".xxx." are what you need if I understand you correctly.
Regards,
Sandor ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Here are some recent logs. May 8 13:48:41 mailserver1.mycorp.net/mailserver1.mycorp.net postfix/smtp[22079]: [ID 197553 mail.info] BBBF66CB1E: to=<b.smith@nodomain.net>, relay=192.168.12.1[192.168.12.1]:25, delay=0.48, delays=0.31/0.02/0.01/0.14, dsn=2.6.0, status=sent (250 2.6.0 <B7C2C6BA798F3C4DBDD78BEDC1F8AD5732046E44@ns2.someotherdomain.com> Queued mail for delivery) May 8 13:48:41 mailserver1.mycorp.net/mailserver1.mycorp.net postfix/smtp[22079]: [ID 197553 mail.info] BBBF66CB1E: to=<m.jackson@nodomain.net>, relay=192.168.12.1[192.168.12.1]:25, delay=0.48, delays=0.31/0.02/0.01/0.14, dsn=2.6.0, status=sent (250 2.6.0 <B7C2C6BA798F3C4DBDD78BEDC1F8AD5732046E44@ ns2.someotherdomain.com> Queued mail for delivery) May 8 13:48:41 mailserver1.mycorp.net/mailserver1.mycorp.net postfix/smtp[22079]: [ID 197553 mail.info] BBBF66CB1E: to=<r.lindsay@nodomain.net>, relay=192.168.12.1[192.168.12.1]:25, delay=0.48, delays=0.31/0.02/0.01/0.14, dsn=2.6.0, status=sent (250 2.6.0 <B7C2C6BA798F3C4DBDD78BEDC1F8AD5732046E44@ ns2.someotherdomain.com> Queued mail for delivery) May 8 13:48:41 mailserver1.mycorp.net/mailserver1.mycorp.net postfix/qmgr[13267]: [ID 197553 mail.info] BBBF66CB1E: removed I *believe* the double hostname is die to chain_hostnames=yes? Don't remember. Regards, .vp
From: Sandor.Geller@morganstanley.com To: syslog-ng@lists.balabit.hu Date: Thu, 8 May 2008 18:05:28 +0100 Subject: Re: [syslog-ng] Problems With Filter Rules - Using First Rule, Not One Intended
Hi,
My problems lie with the other filters, the ones at the end:
filter F_edge { host("edge*") or host("122.21.*"); }; filter F_router { host("gw*") or host("rtr") or host("mmsc"); }; filter F_switch { host("sw*") or host("sw1") or host("sw2"); }; filter F_firewall { host("^fw*") or host("^mlm*-*") or host("^cm*"); }; filter F_dc { host("^mydc*") or host("^dc*"); }; filter F_accesspoints { host("^melanie*"); }; filter F_mailservers { host("^mail*") or host("^smtpgw*"); }; filter F_proxies { host("^proxygw*"); }; filter F_InternetIP { host("161.17.10.*"); };
The above, based on the filter rule for F_mailservers, should place anything coming in from a host named mailserver1, or smtpgw1 into destination D_mailservers, which in turn should save logs into file named /var/log/MyHosts/MailServers/$FULLHOST.log. Instead I find those logs in /var/log/MyHosts/Switches/$FULLHOST.log (which is really /var/log/MyHosts/Switches/mailserver1.mycorp.net/mailserver1.m ycorp.net.log)
It would be nice to see at least a log entry from the file. BTW how did the hostname appear twice in the destination filename? Either I overlooked something or you're not using exactly the same config you sent.
I need to figure out a way to write the differences for hosts that begin with pattern xxx (^xxx)? and those with xxx at the end (*xxx) and those with xxx in the middle (*xxx)?, and for the life of me, I can't fifure out why the above is sending into Switches :-(
You've anchors in your filter regexps already. "^xxx", "xxx$", ".xxx." are what you need if I understand you correctly.
Regards,
Sandor
Hi,
Here are some recent logs.
May 8 13:48:41 mailserver1.mycorp.net/mailserver1.mycorp.net postfix/smtp[22079]: [ID 197553 mail.info] BBBF66CB1E: to=<b.smith@nodomain.net>, relay=192.168.12.1[192.168.12.1]:25, delay=0.48, delays=0.31/0.02/0.01/0.14, dsn=2.6.0, status=sent (250 2.6.0
<B7C2C6BA798F3C4DBDD78BEDC1F8AD5732046E44@ns2.someotherdomain. com> Queued mail for delivery)
Which version of syslog-ng are you using? I remember that postfix (more precisely postfix/daemonname-like program names) caused problems for older versions, although this might be unrelated.
I *believe* the double hostname is die to chain_hostnames=yes? Don't remember.
No, there would be an '@' between the hostnames. I still don't see how 'sw' could match your logs :( regards, Sandor -------------------------------------------------------- NOTICE: If received in error, please destroy and notify sender. Sender does not intend to waive confidentiality or privilege. Use of this email is prohibited when received in error.
syslog-ng 2.0.7 I remember needing that in order to see the hostname from a WAP not too long ago.
From: Sandor.Geller@morganstanley.com To: syslog-ng@lists.balabit.hu Date: Thu, 8 May 2008 19:19:05 +0100 Subject: Re: [syslog-ng] Problems With Filter Rules - Using First Rule, Not One Intended
Hi,
Here are some recent logs.
May 8 13:48:41 mailserver1.mycorp.net/mailserver1.mycorp.net postfix/smtp[22079]: [ID 197553 mail.info] BBBF66CB1E: to=<b.smith@nodomain.net>, relay=192.168.12.1[192.168.12.1]:25, delay=0.48, delays=0.31/0.02/0.01/0.14, dsn=2.6.0, status=sent (250 2.6.0
<B7C2C6BA798F3C4DBDD78BEDC1F8AD5732046E44@ns2.someotherdomain. com> Queued mail for delivery)
Which version of syslog-ng are you using? I remember that postfix (more precisely postfix/daemonname-like program names) caused problems for older versions, although this might be unrelated.
I *believe* the double hostname is die to chain_hostnames=yes? Don't remember.
No, there would be an '@' between the hostnames. I still don't see how 'sw' could match your logs :(
regards,
Sandor --------------------------------------------------------
NOTICE: If received in error, please destroy and notify sender. Sender does not intend to waive confidentiality or privilege. Use of this email is prohibited when received in error. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
syslog-ng 2.0.7 I remember needing that in order to see the hostname from a WAP not too long ago.
From: Sandor.Geller@morganstanley.com To: syslog-ng@lists.balabit.hu Date: Thu, 8 May 2008 19:19:05 +0100 Subject: Re: [syslog-ng] Problems With Filter Rules - Using First Rule, Not One Intended
Hi,
Here are some recent logs.
May 8 13:48:41 mailserver1.mycorp.net/mailserver1.mycorp.net postfix/smtp[22079]: [ID 197553 mail.info] BBBF66CB1E: to=<b.smith@nodomain.net>, relay=192.168.12.1[192.168.12.1]:25, delay=0.48, delays=0.31/0.02/0.01/0.14, dsn=2.6.0, status=sent (250 2.6.0
<B7C2C6BA798F3C4DBDD78BEDC1F8AD5732046E44@ns2.someotherdomain. com> Queued mail for delivery)
Which version of syslog-ng are you using? I remember that postfix (more precisely postfix/daemonname-like program names) caused problems for older versions, although this might be unrelated.
I *believe* the double hostname is die to chain_hostnames=yes? Don't remember.
No, there would be an '@' between the hostnames. I still don't see how 'sw' could match your logs :(
regards,
Sandor --------------------------------------------------------
NOTICE: If received in error, please destroy and notify sender. Sender does not intend to waive confidentiality or privilege. Use of this email is prohibited when received in error. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
With regards to filters for host, which one of the two is correct? (if any) filter F_mailservers { host ("^mail$*") or ("^smtp$); }; filter F_mailservers { host("^mail$") or host("^smtp$"); }; Again, I am trying to find hosts named mailserver1, mailserver2, mailserver-gw, smtpgw1 and smtp-gw Thanks in advance, .vp From: wiskbroom@hotmail.com To: syslog-ng@lists.balabit.hu Date: Thu, 8 May 2008 15:08:48 -0400 Subject: Re: [syslog-ng] Problems With Filter Rules - Using First Rule, Not One Intended syslog-ng 2.0.7 I remember needing that in order to see the hostname from a WAP not too long ago.
From: Sandor.Geller@morganstanley.com To: syslog-ng@lists.balabit.hu Date: Thu, 8 May 2008 19:19:05 +0100 Subject: Re: [syslog-ng] Problems With Filter Rules - Using First Rule, Not One Intended
Hi,
Here are some recent logs.
May 8 13:48:41 mailserver1.mycorp.net/mailserver1.mycorp.net postfix/smtp[22079]: [ID 197553 mail.info] BBBF66CB1E: to=<b.smith@nodomain.net>, relay=192.168.12.1[192.168.12.1]:25, delay=0.48, delays=0.31/0.02/0.01/0.14, dsn=2.6.0, status=sent (250 2.6.0
<B7C2C6BA798F3C4DBDD78BEDC1F8AD5732046E44@ns2.someotherdomain. com> Queued mail for delivery)
Which version of syslog-ng are you using? I remember that postfix (more precisely postfix/daemonname-like program names) caused problems for older versions, although this might be unrelated.
I *believe* the double hostname is die to chain_hostnames=yes? Don't remember.
No, there would be an '@' between the hostnames. I still don't see how 'sw' could match your logs :(
regards,
Sandor --------------------------------------------------------
NOTICE: If received in error, please destroy and notify sender. Sender does not intend to waive confidentiality or privilege. Use of this email is prohibited when received in error. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
participants (4)
-
Fegan, Joe
-
Geller, Sandor (IT)
-
Hari Sekhon
-
wiskbroom＠hotmail.com