Chained changing IP addresses to FQDN
Hello, I have a syslog infrastructure using syslog-ng comprising of three tiers: probes that accept messages and forwards them to the broker, a broker that routes and/or duplicates messages to the appropriate storage devices, and storage devices. All forward using the tcp("hostname"); as a destination. Basically it looks like this: machine1 ==udp==> +-------+ +--------+ +-----------+ machine2 ==udp==> | probe | ==tcp==> | broker | ==tcp==> | storage 1 | machine3 ==udp==> +-------+ +--------+ +-----------+ || +-----------+ ++=======tcp==> | storage 2 | +-----------+ The probes rewrite the hostname to the IP address that the packet came from using the chain_hostnames(no) and keep_hostname(no) options. One of the requirements of one specific storage device is to change the IP addresses into hostnames. I'm having difficulty on the storage device switching the IP address in the log message to the FQDN. I have tried all combinations I can think of of chain_hostnames and keep_hostname and cannot seem to get the IP address replaced with the FQDN. Is there a way to replace the IP address with the FQDN while receiving the messages from an intermediary? Thank you, Jason
On Tue, 2009-12-22 at 15:28 -0500, Jason Carr wrote:
Hello,
I have a syslog infrastructure using syslog-ng comprising of three tiers: probes that accept messages and forwards them to the broker, a broker that routes and/or duplicates messages to the appropriate storage devices, and storage devices. All forward using the tcp("hostname"); as a destination.
Basically it looks like this:
machine1 ==udp==> +-------+ +--------+ +-----------+ machine2 ==udp==> | probe | ==tcp==> | broker | ==tcp==> | storage 1 | machine3 ==udp==> +-------+ +--------+ +-----------+ || +-----------+ ++=======tcp==> | storage 2 | +-----------+
The probes rewrite the hostname to the IP address that the packet came from using the chain_hostnames(no) and keep_hostname(no) options. One of the requirements of one specific storage device is to change the IP addresses into hostnames. I'm having difficulty on the storage device switching the IP address in the log message to the FQDN. I have tried all combinations I can think of of chain_hostnames and keep_hostname and cannot seem to get the IP address replaced with the FQDN.
Is there a way to replace the IP address with the FQDN while receiving the messages from an intermediary?
Thank you,
Jason
uhm, don't re-write it to ip addresses in the first place? Seems logical, but I may not fully understand your implementation. -C
Our security team wants IP addresses from the packet for SIEM tools among other things and the sysadmins want FQDN instead of IP addresses since they know machines by hostname not IP. Each team would have their own storage server. On Dec 27, 2009, at 12:21 AM, Christopher Barry wrote:
On Tue, 2009-12-22 at 15:28 -0500, Jason Carr wrote:
Hello,
I have a syslog infrastructure using syslog-ng comprising of three tiers: probes that accept messages and forwards them to the broker, a broker that routes and/or duplicates messages to the appropriate storage devices, and storage devices. All forward using the tcp("hostname"); as a destination.
Basically it looks like this:
machine1 ==udp==> +-------+ +--------+ +-----------+ machine2 ==udp==> | probe | ==tcp==> | broker | ==tcp==> | storage 1 | machine3 ==udp==> +-------+ +--------+ +-----------+ || +-----------+ ++=======tcp==> | storage 2 | +-----------+
The probes rewrite the hostname to the IP address that the packet came from using the chain_hostnames(no) and keep_hostname(no) options. One of the requirements of one specific storage device is to change the IP addresses into hostnames. I'm having difficulty on the storage device switching the IP address in the log message to the FQDN. I have tried all combinations I can think of of chain_hostnames and keep_hostname and cannot seem to get the IP address replaced with the FQDN.
Is there a way to replace the IP address with the FQDN while receiving the messages from an intermediary?
Thank you,
Jason
uhm, don't re-write it to ip addresses in the first place? Seems logical, but I may not fully understand your implementation.
-C
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Sun, 2009-12-27 at 12:41 -0500, Jason Carr wrote:
Our security team wants IP addresses from the packet for SIEM tools among other things and the sysadmins want FQDN instead of IP addresses since they know machines by hostname not IP. Each team would have their own storage server.
On Dec 27, 2009, at 12:21 AM, Christopher Barry wrote:
On Tue, 2009-12-22 at 15:28 -0500, Jason Carr wrote:
Hello,
I have a syslog infrastructure using syslog-ng comprising of three tiers: probes that accept messages and forwards them to the broker, a broker that routes and/or duplicates messages to the appropriate storage devices, and storage devices. All forward using the tcp("hostname"); as a destination.
Basically it looks like this:
machine1 ==udp==> +-------+ +--------+ +-----------+ machine2 ==udp==> | probe | ==tcp==> | broker | ==tcp==> | storage 1 | machine3 ==udp==> +-------+ +--------+ +-----------+ || +-----------+ ++=======tcp==> | storage 2 | +-----------+
The probes rewrite the hostname to the IP address that the packet came from using the chain_hostnames(no) and keep_hostname(no) options. One of the requirements of one specific storage device is to change the IP addresses into hostnames. I'm having difficulty on the storage device switching the IP address in the log message to the FQDN. I have tried all combinations I can think of of chain_hostnames and keep_hostname and cannot seem to get the IP address replaced with the FQDN.
Is there a way to replace the IP address with the FQDN while receiving the messages from an intermediary?
Thank you,
Jason
uhm, don't re-write it to ip addresses in the first place? Seems logical, but I may not fully understand your implementation.
-C
First, I'm absolutely no expert with syslog-ng, but since nobody is piping up, I'll try to help you. Don't get upset if you go on a wild goose chase :) If it was me, I would see if I could do the splitting before the conversion to all IP addresses. If this is simply not possible architecturally, then I would look to see if syslog-ng had the ability to route through a program as a filter (I'm betting it can, although I can't say for sure.) If so, you could write a simple script to nslookup or dig the IP address token in the stream, find the FQDN, and rewrite it that way. If you do that, you'll definitely want the machine doing the re-writing to have a good dnscache on it, so you're not going over the wire all the time for lookups. Other than that I have no clue. Hopefully someone with actual knowledge can chime in now :) -C
On Monday 28 of December 2009, Christopher Barry wrote:
I have a syslog infrastructure using syslog-ng comprising of three tiers: probes that accept messages and forwards them to the broker, a broker that routes and/or duplicates messages to the appropriate storage devices, and storage devices. All forward using the tcp("hostname"); as a destination.
[...]
The probes rewrite the hostname to the IP address that the packet came from using the chain_hostnames(no) and keep_hostname(no) options. One of the requirements of one specific storage device is to change the IP addresses into hostnames. I'm having difficulty on the storage device switching the IP address in the log message to the FQDN. I have tried all combinations I can think of of chain_hostnames and keep_hostname and cannot seem to get the IP address replaced with the FQDN.
Is there a way to replace the IP address with the FQDN while receiving the messages from an intermediary?
uhm, don't re-write it to ip addresses in the first place? Seems logical, but I may not fully understand your implementation.
Our security team wants IP addresses from the packet for SIEM tools among other things and the sysadmins want FQDN instead of IP addresses since they know machines by hostname not IP. Each team would have their own storage server. [...] If it was me, I would see if I could do the splitting before the conversion to all IP addresses. If this is simply not possible architecturally, then I would look to see if syslog-ng had the ability to route through a program as a filter (I'm betting it can, although I can't say for sure.) If so, you could write a simple script to nslookup or dig the IP address token in the stream, find the FQDN, and rewrite it that way. If you do that, you'll definitely want the machine doing the re-writing to have a good dnscache on it, so you're not going over the wire all the time for lookups.
See syslog-ng admin guide, look for use_dns(). You might find persist-only useful, combined with dns_cache_hosts(). HTH -- Jakub Jankowski|shasta@toxcorp.com|http://toxcorp.com/ GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D
On Sun, Dec 27, 2009 at 4:04 PM, Jakub Jankowski <shasta@toxcorp.com> wrote:
On Monday 28 of December 2009, Christopher Barry wrote:
I have a syslog infrastructure using syslog-ng comprising of three tiers: probes that accept messages and forwards them to the broker, a broker that routes and/or duplicates messages to the appropriate storage devices, and storage devices. All forward using the tcp("hostname"); as a destination.
[...]
The probes rewrite the hostname to the IP address that the packet came from using the chain_hostnames(no) and keep_hostname(no) options. One of the requirements of one specific storage device is to change the IP addresses into hostnames. I'm having difficulty on the storage device switching the IP address in the log message to the FQDN. I have tried all combinations I can think of of chain_hostnames and keep_hostname and cannot seem to get the IP address replaced with the FQDN.
Is there a way to replace the IP address with the FQDN while receiving the messages from an intermediary?
uhm, don't re-write it to ip addresses in the first place? Seems logical, but I may not fully understand your implementation.
Our security team wants IP addresses from the packet for SIEM tools among other things and the sysadmins want FQDN instead of IP addresses since they know machines by hostname not IP. Each team would have their own storage server. [...] If it was me, I would see if I could do the splitting before the conversion to all IP addresses. If this is simply not possible architecturally, then I would look to see if syslog-ng had the ability to route through a program as a filter (I'm betting it can, although I can't say for sure.) If so, you could write a simple script to nslookup or dig the IP address token in the stream, find the FQDN, and rewrite it that way. If you do that, you'll definitely want the machine doing the re-writing to have a good dnscache on it, so you're not going over the wire all the time for lookups.
See syslog-ng admin guide, look for use_dns(). You might find persist-only useful, combined with dns_cache_hosts().
HTH
-- Jakub Jankowski|shasta@toxcorp.com|http://toxcorp.com/ GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email ______________________________________________________________________
Hello, Since you have a relay(broker) box in between the original log sender and the storage device, you are going to run into a problem when you use_dns(yes). Using the option keep_hostname(yes) simply tells syslog-ng to set the $host to whatever it says in the second column of the log message it is receiving. When you use keep_hostname(yes), use_dns() will be completely ignored. When you set keep_hostname(no) and use_dns(yes), your logs are simply going to be written out as coming from your probe or broker (whichever relayed it to your storage device) because syslog will look at the source IP of the packet it received the message from (broker) and then do a reverse lookup on that in order to set the $host variable. There are a couple of things you can do in your scenario. Your best bet may be to recompile your syslog-ng instances with the --enable-spoof-source option and use spoof_source(yes), and then use_dns(yes) at the storage box; however, it only works with UDP, and i'm not sure of its behaviour over a couple of hops of syslog boxes. Your next option is hacky, and that would be to run your destinations to a script that does reverse lookups and rewrites the host column. This option is rather terrifying to me, but I don't know your log volume. I had thought there was somthing else you can do but I can't seem to remember right now...I'll reply later if I do. In the meantime grab the admin guide ( http://www.balabit.com/dl/guides/syslog-ng-v3.0-guide-admin-en.pdf ) and look at the spoof-source option. Keep in mind syslog-ng 3.x+ lets you use keep_hostname and use_dns on a per-source basis, so you can set it for some sources but not for others. Also, syslog-ng blocks on DNS lookups - if your dns goes down, your logs (that have sources that use_dns) won't get written out. I don't know how many hosts you have, so putting them in /etc/hosts then using the use_dns(persist-only); dns_cache_hosts('/etc/hosts') options might be ok, otherwise just set up a load of PTR records at your resolver (which is a good idea anyways). There's also a use_fqdn option if your admins want that. -- Lance Laursen Demonware Systems Engineer
participants (4)
-
Christopher Barry
-
Jakub Jankowski
-
Jason Carr
-
Lance Laursen