[syslog-ng] trouble with chain_hostnames

[Balazs Scheidler]

On Wed, 2012-01-04 at 17:32 -0500, Matt Van Mater wrote:
> Hi all,
> 
> Many thanks to the developers, I've been a user for many years and
> have run into a deployment that I can't quite get working the way I
> want to.
> 
> Please correct me if I am wrong, but the chain_hostnames global option
> is intended to append the current syslog-ng hostname to every message
> processed, no matter how many hosts a message chains through?
> 
> My messages are being relayed through these hosts:
> 1) webserver: Ubuntu LTS 10.4.3, syslog-ng 2.0.9 (from standard
> apt-get sources)
> 2) proxy: OpenBSD 5.0, syslog-ng 3.1.4 (from prebuilt packages)
> 3) logserver: Ubuntu LTS 10.4.3, syslog-ng 2.0.9 (from standard
> apt-get sources)
> 
> The traffic flows through them this way:
> webserver --> proxy (tcp 514) --> proxy (tcp 1514) --> ssh tunnel -->
> logserver (tcp 514)
> 
> I have chain_hostnames(yes) in the options global variable section all
> three syslog-ng.conf files, and messages relay through all hosts
> properly EXCEPT that the proxy server does not seem to append its
> hostname or IP address to the syslog message when forwarding it on to
> the logserver.  I have experimented with keep_hostnames as both yes
> and no on all hosts and have not seen what i am expecting... (example:
> timestamp s_all at webserver_ip/proxy_ip/logserver_ip
> actual_log_message)

The hostname format should become:

s_all at webserver_ip/webserver_ip/proxy_ip/localhost

^^^^^^^^^^^^^^^^^^ Sent by the webserver

                   ^^^^^^^^^^^^ Appended by the first proxy hop

                                ^^^^^^^^ Appended by the 2nd proxy hop

                                         ^^^^^^^^^ Appended by the logserver, 
                                                   assuming the SSH tunnel is 
                                                   visible from 127.0.0.1

The hostname update mechanism requires that you set keep-hostname(no),
otherwise syslog-ng will stick to the hostname it receives.

The basic algorithm is: concatenate the host sent by the client and the
resolved IP address of the sender

Basically equivalent to "$HOST/$HOST_FROM" evaluated on all hops.
> 
> An example of what i see today is this:
> On webserver # echo "webserver sending message to localhost tcp 514" |
> nc localhsot 514
> On loghost# tail -n1 /var/log/messages
> Jan  4 17:22:16 webserver/webserver/loghost_ip webserver sending
> message to localhost tcp 514

The implementation is in logsource.c:log_source_mangle_hostname()
function. I'd recommend to study the various conditions there to see why
syslog-ng formats the hostname this way.

The function is really simple, the problem is either that
keep-hostname() is not properly set, or that one of the hops doesn't
send the $HOST value for some reason.

> 
> Note the fact that the message gets transported through all 3 hosts
> but the proxy hostname isn't being inserted into the chain.
> 
> So my questions are:
> 1) Is chain_hostnames supposed to append the hostname to the message,
> no matter how many hosts a message is relayed through?

We always append the HOST we've received the message from. So it is not
inserting _itself_, but rather the sender.

> 1a) Suggestion: can you update the FAQ here:
> http://www.campin.net/syslog-ng/faq.html#hostname to include an
> example of a log message relayed through multiple hosts and the
> expected behavior

That FAQ has been unmaintained for some time. Some of the entries was
moved to www.balabit.com and updated there, but not all.

@Peter, can you check if this is included there and update it with this
information please?

> 2) Is there a known bug in syslog-ng v3.1.4 and / or OpenBSD's example
> syslog-ng.conf file that is the cause of hostname chaining not
> working?


Not that I remember of.

> 2a) Suggestion: can you publish the syslog-ng changelogs in a single
> file appending changes, or a searchable database?  As far as I can
> tell, I currently have to download the changelog for each individual
> syslog-ng version released in order to search and see if an issue was
> released and that is a huge PITA.  

The easiest way to do this is to check the git history, I try to write
readable commit logs and that's easy to search using gitk.

The NEWS file is wiped out in every major release. Probably not the best
idea, but that's how I did it in the past 14 years, primarily because it
is a source for merge failures when merging in an older branch to the
new one.

Still, the best option is gitk.

-- 
Bazsi