[syslog-ng] Missing hosts
Balazs Scheidler
bazsi at balabit.hu
Tue Jul 14 09:02:21 CEST 2009
On Mon, 2009-07-13 at 12:00 -0400, Christopher Bland wrote:
> Balazs,
>
> Networking is not my specialty, I am a little unclear on your response
> for the netmask filter. If hosts are on the same network there
> shouldn't be any hops so what does netmask look for or can I properly do
> a netmask filter. Also if I understand you correctly if Host1 on subnet
> A would send a message to syslog-ng on subnet B the message goes though
> subnet A gateway then out subnet B gateway. So, syslog-ng sees the
> message coming from subnet B since that is the last hop?
Under the "hop" I meant syslog relay. If there's no relay the source IP
address of the message is the same as the host sending the syslog
message.
Simple routers do not change the source IP address, but syslog relays do
(at least by default).
Did you try to enable debug to see which filters match and which do not?
>
> -Chris
>
>
> Balazs Scheidler wrote:
> > On Wed, 2009-07-08 at 13:24 -0400, Christopher Bland wrote:
> >
> >> Hi guys,
> >>
> >> I have a problem that needs guru magic. When I first setup syslog-ng I
> >> had too many host/log entries for the system I was using to handle. I
> >> had tens of thousands of entries and each day created a 10G database
> >> table. I remedied this logging certain hosts to flat files and
> >> separating host and network equipment into different databases. I left
> >> my original catch all DB in place for comparison, to verify that all
> >> hosts were being logged properly. When I do queries I find that a
> >> number of hosts are in the catch all but not in the individual
> >> databases. Many of the missing hosts should be caught by the netmask
> >> entries but aren't.
> >>
> >
> > Although I didn't try to comprehend your filters completely, but here
> > are some tips:
> >
> > 1) netmask() is filtering against the last-hop relay, that actually
> > sent the message to syslog-ng, not the contents of the "HOST" field in
> > the message
> > 2) your "catch-all" database does not exactly negate the filter of the
> > first two statements
> > 3) you could enable --verbose --debug and have syslog-ng log to your
> > terminal (redirecting syslog-ng to your terminal is quite important in
> > this case as debug may generate recursive messages if you are not using
> > syslog-ng 3.0), then you should see messages that help you debug your
> > filter statements:
> >
> > This message is printed to aid filter evaluation:
> >
> > msg_debug("Filter node evaluation result",
> > evt_tag_str("filter_result", res ? "match" : "not-match"),
> > evt_tag_str("filter_type", self->type),
> > NULL);
> >
> > msg_debug("Filter rule evaluation result",
> > evt_tag_str("filter_result", res ? "match" : "not-match"),
> > evt_tag_str("filter_rule", self->super.name),
> > NULL);
> >
> >
> >
> >
> >>
> >> options {
> >> sync (0);
> >> time_reopen (10);
> >> log_fifo_size (10000);
> >> time_sleep(50);
> >> log_fetch_limit(100);
> >> chain_hostnames(no);
> >> long_hostnames (off);
> >> keep_hostname(no);
> >> use_dns (yes);
> >> dns_cache(yes);
> >> use_fqdn (no);
> >> create_dirs (no);
> >> keep_hostname (yes);
> >> };
> >>
> >> source s_sys {
> >> file ("/proc/kmsg" log_prefix("kernel: "));
> >> unix-stream ("/dev/log");
> >> internal();
> >> };
> >>
> >> source s_everything {
> >> file ("/proc/kmsg" log_prefix("kernel: "));
> >> unix-stream ("/dev/log");
> >> internal();
> >> udp();
> >> };
> >>
> >> destination d_database { pipe("/tmp/mysql.pipe" template("INSERT INTO
> >> logs (host, facility, priority, level, tag, datetime, program, msg)
> >> VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG',
> >> '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n")
> >> template-escape(yes)); };
> >>
> >> destination d_networkdb { pipe("/tmp/mysql-network.pipe"
> >> template("INSERT INTO logs (host, facility, priority, level, tag,
> >> datetime, program, msg) VALUES ( '$HOST', '$FACILITY', '$PRIORITY',
> >> '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC', '$PROGRAM',
> >> '$MSG' );\n") template-escape(yes)); };
> >>
> >> destination d_hostsdb { pipe("/tmp/mysql-hosts.pipe" template("INSERT
> >> INTO logs (host, facility, priority, level, tag, datetime, program, msg)
> >> VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG',
> >> '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n")
> >> template-escape(yes)); };
> >>
> >> filter f_nofw { not host("fwhost"); };
> >>
> >> filter f_hosts { netmask("10.16.5.0/255.255.255.0") or
> >> netmask("10.16.6.0/255.255.255.0") or
> >> host("hostA") or
> >> host("hostB") or
> >> host("hostC");
> >> };
> >> filter f_network { not host("fwhost") and (
> >> host("host1") or
> >> host("host2") or
> >> host("host3") or
> >> netmask("10.16.57.0/255.255.255.0") or
> >> netmask("10.16.36.0/255.255.255.0") or
> >> netmask("10.16.120.0/255.255.255.0") or
> >> netmask("10.16.217.0/255.255.255.0")
> >> );
> >> };
> >>
> >>
> >> log { source(s_everything); filter(f_hosts); destination(d_hostsdb); };
> >> log { source(s_everything); filter(f_network); destination(d_networkdb); };
> >> log { source(s_everything); filter(f_nofw); destination(d_database); };
> >>
> >>
> >> ______________________________________________________________________________
> >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> >> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> >> FAQ: http://www.campin.net/syslog-ng/faq.html
> >>
> >>
> >>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>
--
Bazsi
More information about the syslog-ng
mailing list