[syslog-ng] Having hostname and sender's IP address in header

[Balazs Scheidler]

On Mon, 2010-10-25 at 19:31 -0400, Worsham, Michael wrote:
> We are trying to streamline the syslog-ng v3.x data from the RHEL
> server-clients to the RHEL DCS/Syslog server while also providing
> another destination to a local Tripwire Log Center server. 
> 
>  
> 
> I previously had the version 3.0.8 options (as seen below), which
> recorded the data files correctly on the DCS server with the correct
> hostname attribute:
> 
>  
> 
> options {
> 
>         chain_hostnames(no);
> 
>         time_reopen(10);
> 
>         time_reap(360);
> 
>         log_fifo_size(1000);
> 
>         group(adm);
> 
>         perm(0644);
> 
>         dir_perm(0755);
> 
>         use_dns(yes);
> 
>         use_fqdn(yes);
> 
>         create_dirs(yes);
> 
>         keep_hostname(yes);
> 
>         log_msg_size(32768);
> 
>         stats_freq(0);
> 
>         flush_lines(0);
> 
> };
> 
>  
> 
> Destination (directory and log file structure): 
> 
>  
> 
> destination d_general {
> 
>           file
> ("/var/log/syslog/general/$HOST/$FULLHOST-$MONTH.$DAY.$YEAR.log"
> 
>           template("$DATE $HOST <$FACILITY.$PRIORITY> $MSGHDR $MSG\n")
> 
>           template_escape(no)
> 
>         );
> 
> };
> 
>  
> 
> Example: /var/log/syslog/general/CH33Test-WebCO/CH33Test-WebCO-10.25.2010.log
> 
> 
> 
>  
> 
> However, we found today that Tripwire Log Center (which we are now
> required to use for log aggregation/reporting needs) doesn’t have the
> ability to do hostname translation (i.e. regex the sender’s hostname
> seen in the datastream and give it an IP address so that it can be
> tracked). So this means the data file that is being sent from the RHEL
> clients will need to keep the sender’s IP address in the actual data
> stream. So if I do keep_hostname(no) and still enable the
> facility.priority template on the DCS server, it will show the IP
> address rather than the hostname of the reporting client:
> 
>  
> 
> Oct 25 18:51:12 10.153.13.70 <syslog.err> syslog-ng[2820]:  EOF
> occurred while idle; fd='9'
> 
> Oct 25 18:51:12 10.153.13.70 <syslog.notice> syslog-ng[2820]:  Syslog
> connection broken; fd='9', server='AF_INET(10.153.29.235:514)',
> time_reopen='10'
> 
> Oct 25 18:51:22 10.153.13.70 <syslog.notice> syslog-ng[2820]:  Syslog
> connection established; fd='9', server='AF_INET(10.153.29.235:514)',
> local='AF_INET(0.0.0.0:0)'
> 
>  
> 
>  
> 
> However, this now causes a problem on the Syslog/DCS server with the
> local log file storage requirements:
> 
>  
> 
> Old
> Version: /var/log/syslog/general/CH33Test-WebCO/CH33Test-WebCO-10.25.2010.log
> New Version:
>  /var/log/syslog/general/10.153.13.70/10.153.13.70-10.25.2010.log
> 
>  
> 
>  
> 
> NOTE: We don’t have a local DNS that all of the servers can be queried
> against and that putting in nearly 100+ sender hosts in the /etc/hosts
> really isn’t a viable option.
> 
>  
> 
> How do I satisfy each requirement:
> 
>  
> 
> 1) Allow the incoming data stream to be saved in a format based on the
> incoming sender’s hostname for the directory structure (ex:
> "/var/log/syslog/general/$HOST/$FULLHOST-$MONTH.$DAY.$YEAR.log”)?
> 
> 2) Allow the IP address of the sender to be seen in the actual data
> file once saved to the DCS server (ex: “Oct 25 18:51:12 10.153.13.70
> <syslog.err>…”)?
> 
> 3) Is there an option to have the incoming sender’s data saved with
> both the hostname and IP address in the same header format (ex: “Oct
> 25 18:51:12 CH33Test-WebCO 10.153.13.70 <syslog.err>…”)?

I've read the complete thread and to say the least I'm confused. Do I
understand it right that:


1) you have RHEL clients sending messages to a syslog-ng server
2) you have a syslog-ng server (on DCS) accepting these log messages.
you create a file for each client, using the hostname of the client.

how does the tripwire agent come into the picture? it reads the files
stored on the DCS server?

Because if that's the case you can always use $HOST for the filename
template and $SOURCEIP as the content template



destination d_general {

          file
("/var/log/syslog/general/$HOST/$FULLHOST-$MONTH.$DAY.$YEAR.log"

          template("$DATE $SOURCEIP <$FACILITY.$PRIORITY> $MSGHDR $MSG
\n")

          template_escape(no)

        );

};

This way the content has an IP whereas the filename has the hostname. Is
this what you want? Or, since you are already using a non-standard file
format (because of the facility/priority values) you can add both, as
long as tripwire is able to process it.

template("$DATE $SOURCEIP $HOST <$FACILITY.$PRIORITY> $MSGHDR $MSG\n")

-- 
Bazsi