[syslog-ng] hostname issue with aggregation from aggregator of end-system

[Balazs Scheidler]

On Mon, 2012-02-27 at 14:25 -0800, Ian Veach wrote:
>  
> We're currently running loghosts/aggregators for each data center
> and two OS types.  This works fine.  We are now trying to forward all
> syslogs from our aggregators to another single aggregator (a security
> loghost, sec-loghost, which is proprietary).  We are having a problem
> retaining hostname when forwarding from a loghost aggregator to
> ANOTHER loghost - but only from one OS type / syslog-ng install.
>  
> We've been running syslog-ng for some time on RHEL (and previously
> SLES) and AIX, in a loghost configuration, with no issues.  Our
> current architecture (example):
>  
> data center 1:
>         rhelhostX -> rhel-loghost1
>         aixhostX -> aix-loghost1
> data center 2:
>         rhelhostX -> rhel-loghost2
>         aixhostX -> aix-loghost2
>  
> aix-loghost's are running syslog-ng 3.0.5, RHEL loghost's are running
> syslog-ng 3.1.4.  Both are similarly configured (see below), and are
> working fine as advertised by themselves - e.g. each loghost gets
> $HOST_FROM and properly stores the logs for that host in our
> architecture (/logs/$YEAR/$MONTH/$DAY/$HOST_FROM/).
>  
> However, given data center 1, if syslogs follow this path:
>         aixhostX -> aix-loghost1 -> sec-loghost
>         rhelhostX -> rhel-loghost1 -> sec-loghost
>         rhelhostX -> rhel-loghost1 -> rhel-loghost2
> sec-loghost sees syslogs from all aixhostXs (even though they passed
> through aix-loghost1) correctly.  However, sec-loghost (and we also
> tested going to rhel-loghost2) sees ALL syslog entries as coming from
> rhel-loghost, even though it's simply aggregating from rhelhostX's.
> loghost1's all correctly get $HOST_FROM and see syslog record hosts
> correctly.  This issue only occurs when forwarding again to another
> loghost.
>  
> Our nutshell configuration on BOTH loghosts (related to this) is:
>  
> options {
>         chain_hostnames(no);
>         keep_hostname(yes);
>         use_fqdn(no);
>         use_dns(yes);
> };

The best way to diagnose such problems is to look into what goes on the
wire between various hosts and check if the hostnames in them are valid.

You can look at that information by using either of
tcpdump/wireshark/ngrep or something similar tool, or by enabling the
debug logs from syslog-ng (which generates a _lot_ of information and
would probably require running a separate instance of syslog-ng and not
munging with the production one).

Anyway, some tips:

$HOST_FROM is only set on the _receiving_ machine to indicate where the
message was received from. It is not included in the message itself and
is not relayed either. 

$HOST_FROM will be aixhostX on aix-loghost, and aix-loghost1 on
sec-loghost.

If you use the $HOST macro, that refers to the hostname as included in
the syslog header. The best way to work is to define the
"trusted-syslog-boundary", probably the first relay in the chain, and
set that up this way:

1) keep-hostname(no) => this will trigger setting the value of $HOST
based on the reverse-DNS name, instead of letting the client specify it.

2) Then on any further relays/syslog servers in the chain, use
keep-hostname(yes).

This means that you don't trust leaf systems to specify their own
hostname, you'd rather trust the first element in the relaying
structure. But then on, you trust this value and have that available on
all hops.

> 
> Any ideas on how to fix this?  The only hint I've gotten is from our
> networks group, who uses kiwi syslog and had to make sure a checkbox
> for RFC 3164 headers compliance was turned on.  I don't see an
> equivalent of that not already set with the above config.

It is really the best to diagnose such issues is to look at the actual
traffic, if you have first-hand knowledge, it is trivial to figure out
what went wrong.

-- 
Bazsi