[syslog-ng] Missing hosts

[Christopher Bland]

Balazs,

Networking is not my specialty, I am a little unclear on your response 
for the netmask filter.  If hosts are on the same network there 
shouldn't be any hops so what does netmask look for or can I properly do 
a netmask filter.  Also if I understand you correctly if Host1 on subnet 
A would send a message to syslog-ng on subnet B the message goes though 
subnet A gateway then out subnet B gateway.  So, syslog-ng sees the 
message coming from subnet B since that is the last hop?

-Chris


Balazs Scheidler wrote:
> On Wed, 2009-07-08 at 13:24 -0400, Christopher Bland wrote:
>   
>> Hi guys,
>>
>> I have a problem that needs guru magic.  When I first setup syslog-ng I 
>> had too many host/log entries for the system I was using to handle.  I 
>> had tens of thousands of entries and each day created a 10G database 
>> table.  I remedied this logging certain hosts to flat files and 
>> separating host and network equipment into different databases.  I left 
>> my original catch all DB in place for comparison, to verify that all 
>> hosts were being logged properly.  When I do queries I find that a 
>> number of hosts are in the catch all but not in the individual 
>> databases.  Many of the missing hosts should be caught by the netmask 
>> entries but aren't.
>>     
>
> Although I didn't try to comprehend your filters completely, but here
> are some tips:
>
>  1) netmask() is filtering against the last-hop relay, that actually
> sent the message to syslog-ng, not the contents of the "HOST" field in
> the message
>  2) your "catch-all" database does not exactly negate the filter of the
> first two statements
>  3) you could enable --verbose --debug and have syslog-ng log to your
> terminal (redirecting syslog-ng to your terminal is quite important in
> this case as debug may generate recursive messages if you are not using
> syslog-ng 3.0), then you should see messages that help you debug your
> filter statements:
>
> This message is printed to aid filter evaluation:
>
>   msg_debug("Filter node evaluation result",
>             evt_tag_str("filter_result", res ? "match" : "not-match"),
>             evt_tag_str("filter_type", self->type),
>             NULL);
>
>   msg_debug("Filter rule evaluation result",
>             evt_tag_str("filter_result", res ? "match" : "not-match"),
>             evt_tag_str("filter_rule", self->super.name),
>             NULL);
>
>
>
>   
>>
>> options {
>>     sync (0);
>>     time_reopen (10);
>>     log_fifo_size (10000);
>>     time_sleep(50);
>>     log_fetch_limit(100);
>>     chain_hostnames(no);
>>     long_hostnames (off);
>>     keep_hostname(no);
>>     use_dns (yes);
>>     dns_cache(yes);
>>     use_fqdn (no);
>>     create_dirs (no);
>>     keep_hostname (yes);
>> };
>>
>> source s_sys {
>>     file ("/proc/kmsg" log_prefix("kernel: "));
>>     unix-stream ("/dev/log");
>>     internal();
>> };
>>
>> source s_everything {
>>     file ("/proc/kmsg" log_prefix("kernel: "));
>>     unix-stream ("/dev/log");
>>     internal();
>>     udp();
>> };
>>
>> destination d_database { pipe("/tmp/mysql.pipe" template("INSERT INTO 
>> logs (host, facility, priority, level, tag, datetime, program, msg) 
>> VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', 
>> '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n") 
>> template-escape(yes)); };
>>
>> destination d_networkdb { pipe("/tmp/mysql-network.pipe" 
>> template("INSERT INTO logs (host, facility, priority, level, tag, 
>> datetime, program, msg) VALUES ( '$HOST', '$FACILITY', '$PRIORITY', 
>> '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC', '$PROGRAM', 
>> '$MSG' );\n") template-escape(yes)); };
>>
>> destination d_hostsdb { pipe("/tmp/mysql-hosts.pipe" template("INSERT 
>> INTO logs (host, facility, priority, level, tag, datetime, program, msg) 
>> VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', 
>> '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n") 
>> template-escape(yes)); };
>>
>> filter f_nofw            { not host("fwhost"); };
>>
>> filter f_hosts            { netmask("10.16.5.0/255.255.255.0") or
>>                                   netmask("10.16.6.0/255.255.255.0") or
>>                                   host("hostA") or
>>                                   host("hostB") or
>>                                   host("hostC");
>>                                 };
>> filter f_network        { not host("fwhost") and (
>>                                   host("host1") or
>>                                   host("host2") or
>>                                   host("host3") or
>>                                   netmask("10.16.57.0/255.255.255.0") or
>>                                   netmask("10.16.36.0/255.255.255.0") or
>>                                   netmask("10.16.120.0/255.255.255.0") or
>>                                   netmask("10.16.217.0/255.255.255.0")
>>                                   );
>>                                 };
>>
>>
>> log { source(s_everything); filter(f_hosts); destination(d_hostsdb); };
>> log { source(s_everything); filter(f_network); destination(d_networkdb); };
>> log { source(s_everything); filter(f_nofw); destination(d_database); };
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>
>>
>>