On Mon, 2012-02-27 at 14:25 -0800, Ian Veach wrote:
We're currently running loghosts/aggregators for each data center and two OS types. This works fine. We are now trying to forward all syslogs from our aggregators to another single aggregator (a security loghost, sec-loghost, which is proprietary). We are having a problem retaining hostname when forwarding from a loghost aggregator to ANOTHER loghost - but only from one OS type / syslog-ng install.
We've been running syslog-ng for some time on RHEL (and previously SLES) and AIX, in a loghost configuration, with no issues. Our current architecture (example):
data center 1: rhelhostX -> rhel-loghost1 aixhostX -> aix-loghost1 data center 2: rhelhostX -> rhel-loghost2 aixhostX -> aix-loghost2
aix-loghost's are running syslog-ng 3.0.5, RHEL loghost's are running syslog-ng 3.1.4. Both are similarly configured (see below), and are working fine as advertised by themselves - e.g. each loghost gets $HOST_FROM and properly stores the logs for that host in our architecture (/logs/$YEAR/$MONTH/$DAY/$HOST_FROM/).
However, given data center 1, if syslogs follow this path: aixhostX -> aix-loghost1 -> sec-loghost rhelhostX -> rhel-loghost1 -> sec-loghost rhelhostX -> rhel-loghost1 -> rhel-loghost2 sec-loghost sees syslogs from all aixhostXs (even though they passed through aix-loghost1) correctly. However, sec-loghost (and we also tested going to rhel-loghost2) sees ALL syslog entries as coming from rhel-loghost, even though it's simply aggregating from rhelhostX's. loghost1's all correctly get $HOST_FROM and see syslog record hosts correctly. This issue only occurs when forwarding again to another loghost.
Our nutshell configuration on BOTH loghosts (related to this) is:
options { chain_hostnames(no); keep_hostname(yes); use_fqdn(no); use_dns(yes); };
The best way to diagnose such problems is to look into what goes on the wire between various hosts and check if the hostnames in them are valid. You can look at that information by using either of tcpdump/wireshark/ngrep or something similar tool, or by enabling the debug logs from syslog-ng (which generates a _lot_ of information and would probably require running a separate instance of syslog-ng and not munging with the production one). Anyway, some tips: $HOST_FROM is only set on the _receiving_ machine to indicate where the message was received from. It is not included in the message itself and is not relayed either. $HOST_FROM will be aixhostX on aix-loghost, and aix-loghost1 on sec-loghost. If you use the $HOST macro, that refers to the hostname as included in the syslog header. The best way to work is to define the "trusted-syslog-boundary", probably the first relay in the chain, and set that up this way: 1) keep-hostname(no) => this will trigger setting the value of $HOST based on the reverse-DNS name, instead of letting the client specify it. 2) Then on any further relays/syslog servers in the chain, use keep-hostname(yes). This means that you don't trust leaf systems to specify their own hostname, you'd rather trust the first element in the relaying structure. But then on, you trust this value and have that available on all hops.
Any ideas on how to fix this? The only hint I've gotten is from our networks group, who uses kiwi syslog and had to make sure a checkbox for RFC 3164 headers compliance was turned on. I don't see an equivalent of that not already set with the above config.
It is really the best to diagnose such issues is to look at the actual traffic, if you have first-hand knowledge, it is trivial to figure out what went wrong. -- Bazsi