Having hostname and sender's IP address in header
We are trying to streamline the syslog-ng v3.x data from the RHEL server-clients to the RHEL DCS/Syslog server while also providing another destination to a local Tripwire Log Center server. I previously had the version 3.0.8 options (as seen below), which recorded the data files correctly on the DCS server with the correct hostname attribute: options { chain_hostnames(no); time_reopen(10); time_reap(360); log_fifo_size(1000); group(adm); perm(0644); dir_perm(0755); use_dns(yes); use_fqdn(yes); create_dirs(yes); keep_hostname(yes); log_msg_size(32768); stats_freq(0); flush_lines(0); }; Destination (directory and log file structure): destination d_general { file ("/var/log/syslog/general/$HOST/$FULLHOST-$MONTH.$DAY.$YEAR.log" template("$DATE $HOST <$FACILITY.$PRIORITY> $MSGHDR $MSG\n") template_escape(no) ); }; Example: /var/log/syslog/general/CH33Test-WebCO/CH33Test-WebCO-10.25.2010.log However, we found today that Tripwire Log Center (which we are now required to use for log aggregation/reporting needs) doesn't have the ability to do hostname translation (i.e. regex the sender's hostname seen in the datastream and give it an IP address so that it can be tracked). So this means the data file that is being sent from the RHEL clients will need to keep the sender's IP address in the actual data stream. So if I do keep_hostname(no) and still enable the facility.priority template on the DCS server, it will show the IP address rather than the hostname of the reporting client: Oct 25 18:51:12 10.153.13.70 <syslog.err> syslog-ng[2820]: EOF occurred while idle; fd='9' Oct 25 18:51:12 10.153.13.70 <syslog.notice> syslog-ng[2820]: Syslog connection broken; fd='9', server='AF_INET(10.153.29.235:514)', time_reopen='10' Oct 25 18:51:22 10.153.13.70 <syslog.notice> syslog-ng[2820]: Syslog connection established; fd='9', server='AF_INET(10.153.29.235:514)', local='AF_INET(0.0.0.0:0)' However, this now causes a problem on the Syslog/DCS server with the local log file storage requirements: Old Version: /var/log/syslog/general/CH33Test-WebCO/CH33Test-WebCO-10.25.2010.log New Version: /var/log/syslog/general/10.153.13.70/10.153.13.70-10.25.2010.log NOTE: We don't have a local DNS that all of the servers can be queried against and that putting in nearly 100+ sender hosts in the /etc/hosts really isn't a viable option. How do I satisfy each requirement: 1) Allow the incoming data stream to be saved in a format based on the incoming sender's hostname for the directory structure (ex: "/var/log/syslog/general/$HOST/$FULLHOST-$MONTH.$DAY.$YEAR.log")? 2) Allow the IP address of the sender to be seen in the actual data file once saved to the DCS server (ex: "Oct 25 18:51:12 10.153.13.70 <syslog.err>...")? 3) Is there an option to have the incoming sender's data saved with both the hostname and IP address in the same header format (ex: "Oct 25 18:51:12 CH33Test-WebCO 10.153.13.70 <syslog.err>...")? -- Michael ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
On Mon, Oct 25, 2010 at 07:31:00PM -0400, Worsham, Michael wrote:
How do I satisfy each requirement:
1) Allow the incoming data stream to be saved in a format based on the incoming sender's hostname for the directory structure (ex: "/var/log/syslog/general/$HOST/$FULLHOST-$MONTH.$DAY.$YEAR.log")?
2) Allow the IP address of the sender to be seen in the actual data file once saved to the DCS server (ex: "Oct 25 18:51:12 10.153.13.70 <syslog.err>...")?
3) Is there an option to have the incoming sender's data saved with both the hostname and IP address in the same header format (ex: "Oct 25 18:51:12 CH33Test-WebCO 10.153.13.70 <syslog.err>...")?
I think your best bet would be something like setting it up to receive the messages on the port the servers are logging onto using a log source which has flags(final). Then you will send the messages back to yourself. Using TCP, UDP, UNIX, or pipe would work for this part. To do this you will configure a destination, which is used to relay the messages as they come in from the servers, and a source which is used to allow syslog-ng another chance to process the messages. 1) one socket which is set with flags to use DNS for local storage 2) one socket which is set with flags to use IPs for Tripwire That way you could apply different properties to each socket. You will probably need to experiment with the template used for relaying the messages from yourself to yourself to be sure nothing weird gets added when they are reforwarded back to you. For this part, using a UDP 127.0.0.1 socket might be helpful at first because you can spy on the traffic using wireshark / tshark / tcpdump along with "udp port XXXX" filter to be sure everything looks like it should.
-- Michael
HTH, Matthew Hall.
Umm... what? Totally lost me there. --- Attached is the syslog-ng.conf that is currently running on the DCS/Syslog server (the data repository): @version: 3.0 # syslog-ng configuration file. # # This should behave pretty much like the original syslog on RedHat. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # options { chain_hostnames(no); time_reopen(10); time_reap(360); log_fifo_size(1000); group(adm); perm(0644); dir_perm(0755); use_dns(yes); use_fqdn(yes); create_dirs(yes); keep_hostname(no); log_msg_size(32768); stats_freq(0); flush_lines(0); }; # # Connectivity needs, TLS encryption also available # source s_localhost { file ("/proc/kmsg" program_override("kernel: ")); unix-stream ("/dev/log"); internal(); }; source s_general { tcp(ip(10.153.29.235) port (514) max-connections(1000) ); }; source s_weblogic { tcp(ip(10.153.29.235) port (515) max-connections(1000) ); }; source s_apache { tcp(ip(10.153.29.235) port (516) max-connections(1000) ); }; source s_windows { tcp(ip(10.153.29.235) port (517) max-connections(1000) ); }; destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages" template("$DATE $HOST <$FACILITY.$PRIORITY> $MSG\n") template_escape(no) ); }; destination d_auth { file("/var/log/secure" template("$DATE $HOST <$FACILITY.$PRIORITY> $MSG\n") template_escape(no) ); }; destination d_mail { file("/var/log/maillog" flush_lines(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_kern { file("/var/log/kern"); }; destination d_mlal { usertty("*"); }; filter notdebug { level(info...emerg); }; filter f_kernel { facility(kern); }; filter f_default { level(info..emerg) and not (facility(mail) or facility(authpriv) or facility(cron)); }; filter f_auth { facility(authpriv); }; filter f_mail { facility(mail); }; filter f_emergency { level(emerg); }; filter f_news { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_boot { facility(local7); }; filter f_cron { facility(cron); }; # # Filters needed for RHEL platforms # filter M_audit { not message("Audit daemon rotating log files"); }; filter M_repeat { not message("last message repeated"); }; filter M_stats { not message("Log statistics"); }; filter M_cron { not message("CMD"); }; filter M_snmp { not message("SNMP"); }; filter M_snmp2 { not message("UDP"); }; filter windows { program(MSWinEventLog); }; #log { source(s_localhost); filter(f_kernel); destination(d_cons); }; log { source(s_localhost); filter(f_kernel); destination(d_kern); }; log { source(s_localhost); filter(f_default); filter(M_snmp); filter(M_snmp2); destination(d_mesg); }; log { source(s_localhost); filter(f_auth); destination(d_auth); }; log { source(s_localhost); filter(f_mail); destination(d_mail); }; log { source(s_localhost); filter(f_emergency); destination(d_mlal); }; log { source(s_localhost); filter(f_news); destination(d_spol); }; log { source(s_localhost); filter(f_boot); destination(d_boot); }; log { source(s_localhost); filter(f_cron); destination(d_cron); }; # vim:ft=syslog-ng:ai:si:ts=4:sw=4:et: # # Destinations for storing data logs # destination d_general { file ("/var/log/syslog/general/$HOST/$FULLHOST-$MONTH.$DAY.$YEAR.log" template("$DATE $HOST <$FACILITY.$PRIORITY> $MSGHDR $MSG\n") template_escape(no) ); }; destination d_weblogic { file ("/var/log/syslog/weblogic/$HOST/$FULLHOST-$MONTH.$DAY.$YEAR.log"); }; destination d_apache { file ("/var/log/syslog/apache/$HOST/$FULLHOST-$MONTH.$DAY.$YEAR.log"); }; destination d_windows { file("/var/log/syslog/windows/$HOST/$FULLHOST-$MONTH.$DAY.$YEAR.log" template("$DATE <$FACILITY.$PRIORITY> $HOST $MSGHDR $MSG\n") template_escape(no) ); }; destination d_tripwire_udp { udp("10.153.29.249" port (514)); }; # # How to log and apply filters, if necessary # log { source(s_localhost); filter(M_audit); filter(M_repeat); filter(M_stats); filter(M_cron); filter(M_snmp); filter(M_snmp2); filter(notdebug); destination(d_general); }; log { source(s_general); filter(M_audit); filter(M_repeat); filter(M_stats); filter(M_cron); filter(M_snmp); filter(M_snmp2); destination(d_general); destination(d_tripwire_udp); }; log { source(s_weblogic); destination(d_weblogic); }; log { source(s_apache); destination(d_apache); }; log { source(s_windows); filter(windows); destination(d_windows); flags(final); }; -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Matthew Hall Sent: Monday, October 25, 2010 7:48 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Having hostname and sender's IP address in header On Mon, Oct 25, 2010 at 07:31:00PM -0400, Worsham, Michael wrote:
How do I satisfy each requirement:
1) Allow the incoming data stream to be saved in a format based on the incoming sender's hostname for the directory structure (ex: "/var/log/syslog/general/$HOST/$FULLHOST-$MONTH.$DAY.$YEAR.log")?
2) Allow the IP address of the sender to be seen in the actual data file once saved to the DCS server (ex: "Oct 25 18:51:12 10.153.13.70 <syslog.err>...")?
3) Is there an option to have the incoming sender's data saved with both the hostname and IP address in the same header format (ex: "Oct 25 18:51:12 CH33Test-WebCO 10.153.13.70 <syslog.err>...")?
I think your best bet would be something like setting it up to receive the messages on the port the servers are logging onto using a log source which has flags(final). Then you will send the messages back to yourself. Using TCP, UDP, UNIX, or pipe would work for this part. To do this you will configure a destination, which is used to relay the messages as they come in from the servers, and a source which is used to allow syslog-ng another chance to process the messages. 1) one socket which is set with flags to use DNS for local storage 2) one socket which is set with flags to use IPs for Tripwire That way you could apply different properties to each socket. You will probably need to experiment with the template used for relaying the messages from yourself to yourself to be sure nothing weird gets added when they are reforwarded back to you. For this part, using a UDP 127.0.0.1 socket might be helpful at first because you can spy on the traffic using wireshark / tshark / tcpdump along with "udp port XXXX" filter to be sure everything looks like it should.
-- Michael
HTH, Matthew Hall. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
Michael, Let's say for the sake of discussion all syslogs arrive from here:
source s_general { tcp(ip(10.153.29.235) port (514) max-connections(1000) ); };
Now you could do this:
source s_general { tcp(ip(10.153.29.235) port (514) max-connections(1000) ); };
source s_loopback_local { udp(ip(127.0.0.1) port(5140)); # set properties to preserve hostnames } source s_loopback_tripwire { udp(ip(127.0.0.1) port(5141)); # set properties to preserve IP addresses } destination d_loopback_local { udp(ip(127.0.0.1) port(5140)); # set appropriate template } destination d_loopback_tripwire { udp(ip(127.0.0.1) port(5141)); # set appropriate template } log { source(s_general); destination(d_loopback_local); destination(d_loopback_tripwire); flags(final); } log { source(s_loopback_local); destination(***local storage log files***); } log { source(s_loopback_tripwire); destination(***remote relay to tripwire using d_tripwire_udp***); }
destination d_tripwire_udp { udp("10.153.29.249" port (514)); };
Is this description more clear? Cheers, Matthew. On Mon, Oct 25, 2010 at 07:57:08PM -0400, Worsham, Michael wrote:
Umm... what? Totally lost me there.
Almost. I am confused on the "set properties to preserve hostnames" and "set appropriate template" pieces. I am still rather new to this still. I am just happy I got TLS encryption working and integrated TLS with Syslog4j for Weblogic support. So I guess back to the original question, is it not possible to pass over the hostname and IP address at the same time in the header? -- M -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Matthew Hall Sent: Monday, October 25, 2010 10:05 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Having hostname and sender's IP address in header Michael, Let's say for the sake of discussion all syslogs arrive from here:
source s_general { tcp(ip(10.153.29.235) port (514) max-connections(1000) ); };
Now you could do this:
source s_general { tcp(ip(10.153.29.235) port (514) max-connections(1000) ); };
source s_loopback_local { udp(ip(127.0.0.1) port(5140)); # set properties to preserve hostnames } source s_loopback_tripwire { udp(ip(127.0.0.1) port(5141)); # set properties to preserve IP addresses } destination d_loopback_local { udp(ip(127.0.0.1) port(5140)); # set appropriate template } destination d_loopback_tripwire { udp(ip(127.0.0.1) port(5141)); # set appropriate template } log { source(s_general); destination(d_loopback_local); destination(d_loopback_tripwire); flags(final); } log { source(s_loopback_local); destination(***local storage log files***); } log { source(s_loopback_tripwire); destination(***remote relay to tripwire using d_tripwire_udp***); }
destination d_tripwire_udp { udp("10.153.29.249" port (514)); };
Is this description more clear? Cheers, Matthew. On Mon, Oct 25, 2010 at 07:57:08PM -0400, Worsham, Michael wrote:
Umm... what? Totally lost me there.
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
I'm not entirely sure I'm following this either, but I will point out that the IP address is inherent in the message, not the header. The only way to preserve it is to use spoof_source(yes) on one of your destination blocks. That way your $SOURCEIP macro will stay intact. I would use this to forward on to Tripwire, as long as there isn't a router using reverse source-path verify (on Cisco). Then you can proceed normally with $HOST for your local logging and Tripwire will get an exact replica of them, IP and all. I use this basic spoofing to copy all my incoming messages to dev log servers, which get an intact $SOURCEIP. Does that help? On Mon, Oct 25, 2010 at 10:07 PM, Worsham, Michael <mworsham@scires.com> wrote:
Almost. I am confused on the "set properties to preserve hostnames" and "set appropriate template" pieces. I am still rather new to this still. I am just happy I got TLS encryption working and integrated TLS with Syslog4j for Weblogic support.
So I guess back to the original question, is it not possible to pass over the hostname and IP address at the same time in the header?
-- M
-----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Matthew Hall Sent: Monday, October 25, 2010 10:05 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Having hostname and sender's IP address in header
Michael,
Let's say for the sake of discussion all syslogs arrive from here:
source s_general { tcp(ip(10.153.29.235) port (514) max-connections(1000) ); };
Now you could do this:
source s_general { tcp(ip(10.153.29.235) port (514) max-connections(1000) ); };
source s_loopback_local { udp(ip(127.0.0.1) port(5140)); # set properties to preserve hostnames }
source s_loopback_tripwire { udp(ip(127.0.0.1) port(5141)); # set properties to preserve IP addresses }
destination d_loopback_local { udp(ip(127.0.0.1) port(5140)); # set appropriate template }
destination d_loopback_tripwire { udp(ip(127.0.0.1) port(5141)); # set appropriate template }
log { source(s_general); destination(d_loopback_local); destination(d_loopback_tripwire); flags(final); }
log { source(s_loopback_local); destination(***local storage log files***); }
log { source(s_loopback_tripwire); destination(***remote relay to tripwire using d_tripwire_udp***); }
destination d_tripwire_udp { udp("10.153.29.249" port (514)); };
Is this description more clear?
Cheers, Matthew.
On Mon, Oct 25, 2010 at 07:57:08PM -0400, Worsham, Michael wrote:
Umm... what? Totally lost me there.
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments.
EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Where would I put the spoof_source(yes) variable/setting -- on the syslog/DCS server or on each RHEL clients? -- M -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Martin Holste Sent: Monday, October 25, 2010 11:36 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Having hostname and sender's IP address in header I'm not entirely sure I'm following this either, but I will point out that the IP address is inherent in the message, not the header. The only way to preserve it is to use spoof_source(yes) on one of your destination blocks. That way your $SOURCEIP macro will stay intact. I would use this to forward on to Tripwire, as long as there isn't a router using reverse source-path verify (on Cisco). Then you can proceed normally with $HOST for your local logging and Tripwire will get an exact replica of them, IP and all. I use this basic spoofing to copy all my incoming messages to dev log servers, which get an intact $SOURCEIP. Does that help? On Mon, Oct 25, 2010 at 10:07 PM, Worsham, Michael <mworsham@scires.com> wrote:
Almost. I am confused on the "set properties to preserve hostnames" and "set appropriate template" pieces. I am still rather new to this still. I am just happy I got TLS encryption working and integrated TLS with Syslog4j for Weblogic support.
So I guess back to the original question, is it not possible to pass over the hostname and IP address at the same time in the header?
-- M
-----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Matthew Hall Sent: Monday, October 25, 2010 10:05 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Having hostname and sender's IP address in header
Michael,
Let's say for the sake of discussion all syslogs arrive from here:
source s_general { tcp(ip(10.153.29.235) port (514) max-connections(1000) ); };
Now you could do this:
source s_general { tcp(ip(10.153.29.235) port (514) max-connections(1000) ); };
source s_loopback_local { udp(ip(127.0.0.1) port(5140)); # set properties to preserve hostnames }
source s_loopback_tripwire { udp(ip(127.0.0.1) port(5141)); # set properties to preserve IP addresses }
destination d_loopback_local { udp(ip(127.0.0.1) port(5140)); # set appropriate template }
destination d_loopback_tripwire { udp(ip(127.0.0.1) port(5141)); # set appropriate template }
log { source(s_general); destination(d_loopback_local); destination(d_loopback_tripwire); flags(final); }
log { source(s_loopback_local); destination(***local storage log files***); }
log { source(s_loopback_tripwire); destination(***remote relay to tripwire using d_tripwire_udp***); }
destination d_tripwire_udp { udp("10.153.29.249" port (514)); };
Is this description more clear?
Cheers, Matthew.
On Mon, Oct 25, 2010 at 07:57:08PM -0400, Worsham, Michael wrote:
Umm... what? Totally lost me there.
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments.
EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
You would put that on the syslog/DCS server, as it will be spoofing the original RHEL client messages to Tripwire so that Tripwire thinks it is getting them directly from the RHEL clients. On Mon, Oct 25, 2010 at 10:39 PM, Worsham, Michael <mworsham@scires.com> wrote:
Where would I put the spoof_source(yes) variable/setting -- on the syslog/DCS server or on each RHEL clients?
-- M
-----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Martin Holste Sent: Monday, October 25, 2010 11:36 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Having hostname and sender's IP address in header
I'm not entirely sure I'm following this either, but I will point out that the IP address is inherent in the message, not the header. The only way to preserve it is to use spoof_source(yes) on one of your destination blocks. That way your $SOURCEIP macro will stay intact. I would use this to forward on to Tripwire, as long as there isn't a router using reverse source-path verify (on Cisco). Then you can proceed normally with $HOST for your local logging and Tripwire will get an exact replica of them, IP and all. I use this basic spoofing to copy all my incoming messages to dev log servers, which get an intact $SOURCEIP. Does that help?
On Mon, Oct 25, 2010 at 10:07 PM, Worsham, Michael <mworsham@scires.com> wrote:
Almost. I am confused on the "set properties to preserve hostnames" and "set appropriate template" pieces. I am still rather new to this still. I am just happy I got TLS encryption working and integrated TLS with Syslog4j for Weblogic support.
So I guess back to the original question, is it not possible to pass over the hostname and IP address at the same time in the header?
-- M
-----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Matthew Hall Sent: Monday, October 25, 2010 10:05 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Having hostname and sender's IP address in header
Michael,
Let's say for the sake of discussion all syslogs arrive from here:
source s_general { tcp(ip(10.153.29.235) port (514) max-connections(1000) ); };
Now you could do this:
source s_general { tcp(ip(10.153.29.235) port (514) max-connections(1000) ); };
source s_loopback_local { udp(ip(127.0.0.1) port(5140)); # set properties to preserve hostnames }
source s_loopback_tripwire { udp(ip(127.0.0.1) port(5141)); # set properties to preserve IP addresses }
destination d_loopback_local { udp(ip(127.0.0.1) port(5140)); # set appropriate template }
destination d_loopback_tripwire { udp(ip(127.0.0.1) port(5141)); # set appropriate template }
log { source(s_general); destination(d_loopback_local); destination(d_loopback_tripwire); flags(final); }
log { source(s_loopback_local); destination(***local storage log files***); }
log { source(s_loopback_tripwire); destination(***remote relay to tripwire using d_tripwire_udp***); }
destination d_tripwire_udp { udp("10.153.29.249" port (514)); };
Is this description more clear?
Cheers, Matthew.
On Mon, Oct 25, 2010 at 07:57:08PM -0400, Worsham, Michael wrote:
Umm... what? Totally lost me there.
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments.
EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments.
EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Monday, October 25, 2010 20:36:24 Martin Holste wrote:
I'm not entirely sure I'm following this either, but I will point out that the IP address is inherent in the message, not the header.
Not always true. In my past company our Ethernet switches sent the switch primary IP address in the place where the hostname goes, unless you configured it differently.
The only way to preserve it is to use spoof_source(yes) on one of your destination blocks. That way your $SOURCEIP macro will stay intact.
Very important point indeed.
I would use this to forward on to Tripwire, as long as there isn't a router using reverse source-path verify (on Cisco).
Good thing to check!
Then you can proceed normally with $HOST for your local logging and Tripwire will get an exact replica of them, IP and all. I use this basic spoofing to copy all my incoming messages to dev log servers, which get an intact $SOURCEIP. Does that help?
Good advice. Overall, very good points as always, Martin. These comments mesh with the comments I was making about the templates to use on the normal disk destinations and the tripwire destinations. If you use a different template for Tripwire, where you reference $SOURCEIP instead of $HOST, and you make sure spoof_source is enabled, this will probably help a lot, as the Tripwire will always be fed IP addresses. -- Matthew Hall
On Monday, October 25, 2010 20:07:08 Worsham, Michael wrote:
Almost. I am confused on the "set properties to preserve hostnames" and "set appropriate template" pieces. I am still rather new to this still. I am just happy I got TLS encryption working and integrated TLS with Syslog4j for Weblogic support.
So I guess back to the original question, is it not possible to pass over the hostname and IP address at the same time in the header?
-- M
Some of the items from your "options" clause are what you are using to get hostnames / IPs to appear or not appear in your log messages during your testing if I have understood properly. You will want to move them from being "global" where they affect to all logging, into the sources where they are needed, to prevent them from spilling over when you are looping the messages back through syslog-ng a second time. options { chain_hostnames(no); time_reopen(10); time_reap(360); log_fifo_size(1000); group(adm); perm(0644); dir_perm(0755); use_dns(yes); use_fqdn(yes); create_dirs(yes); keep_hostname(yes); log_msg_size(32768); stats_freq(0); flush_lines(0); }; The set appropriate template part means using the right template string to output the message in the correct format when forwarding it. According to the administrator guide: $ISODATE $HOST $MSGHDR$MSG\n is the default template. It will probably work to reforward the messages back to syslog-ng just fine but could need some changes. I don't think you can pass the host and IP at the same time because there is only one spot in the packet where the $HOST can be written. I think you will need to make two flows of log messages where one contains hosts and the other contains IPs. I thought of another way to do it which could be worth trying. You could try using set(...) function of syslog-ng to create your own macros: $host_dns and $host_ip perhaps, where you have saved the values you want for each field. Then you could change your template of your destination to reference these macro values of your own instead of the $HOST. To figure this one out you will need to spend some time in the manual as it's not complete how to do it yet in my mind. Regards, Matthew Hall.
On Mon, 2010-10-25 at 19:31 -0400, Worsham, Michael wrote:
We are trying to streamline the syslog-ng v3.x data from the RHEL server-clients to the RHEL DCS/Syslog server while also providing another destination to a local Tripwire Log Center server.
I previously had the version 3.0.8 options (as seen below), which recorded the data files correctly on the DCS server with the correct hostname attribute:
options {
chain_hostnames(no);
time_reopen(10);
time_reap(360);
log_fifo_size(1000);
group(adm);
perm(0644);
dir_perm(0755);
use_dns(yes);
use_fqdn(yes);
create_dirs(yes);
keep_hostname(yes);
log_msg_size(32768);
stats_freq(0);
flush_lines(0);
};
Destination (directory and log file structure):
destination d_general {
file ("/var/log/syslog/general/$HOST/$FULLHOST-$MONTH.$DAY.$YEAR.log"
template("$DATE $HOST <$FACILITY.$PRIORITY> $MSGHDR $MSG\n")
template_escape(no)
);
};
Example: /var/log/syslog/general/CH33Test-WebCO/CH33Test-WebCO-10.25.2010.log
However, we found today that Tripwire Log Center (which we are now required to use for log aggregation/reporting needs) doesn’t have the ability to do hostname translation (i.e. regex the sender’s hostname seen in the datastream and give it an IP address so that it can be tracked). So this means the data file that is being sent from the RHEL clients will need to keep the sender’s IP address in the actual data stream. So if I do keep_hostname(no) and still enable the facility.priority template on the DCS server, it will show the IP address rather than the hostname of the reporting client:
Oct 25 18:51:12 10.153.13.70 <syslog.err> syslog-ng[2820]: EOF occurred while idle; fd='9'
Oct 25 18:51:12 10.153.13.70 <syslog.notice> syslog-ng[2820]: Syslog connection broken; fd='9', server='AF_INET(10.153.29.235:514)', time_reopen='10'
Oct 25 18:51:22 10.153.13.70 <syslog.notice> syslog-ng[2820]: Syslog connection established; fd='9', server='AF_INET(10.153.29.235:514)', local='AF_INET(0.0.0.0:0)'
However, this now causes a problem on the Syslog/DCS server with the local log file storage requirements:
Old Version: /var/log/syslog/general/CH33Test-WebCO/CH33Test-WebCO-10.25.2010.log New Version: /var/log/syslog/general/10.153.13.70/10.153.13.70-10.25.2010.log
NOTE: We don’t have a local DNS that all of the servers can be queried against and that putting in nearly 100+ sender hosts in the /etc/hosts really isn’t a viable option.
How do I satisfy each requirement:
1) Allow the incoming data stream to be saved in a format based on the incoming sender’s hostname for the directory structure (ex: "/var/log/syslog/general/$HOST/$FULLHOST-$MONTH.$DAY.$YEAR.log”)?
2) Allow the IP address of the sender to be seen in the actual data file once saved to the DCS server (ex: “Oct 25 18:51:12 10.153.13.70 <syslog.err>…”)?
3) Is there an option to have the incoming sender’s data saved with both the hostname and IP address in the same header format (ex: “Oct 25 18:51:12 CH33Test-WebCO 10.153.13.70 <syslog.err>…”)?
I've read the complete thread and to say the least I'm confused. Do I understand it right that: 1) you have RHEL clients sending messages to a syslog-ng server 2) you have a syslog-ng server (on DCS) accepting these log messages. you create a file for each client, using the hostname of the client. how does the tripwire agent come into the picture? it reads the files stored on the DCS server? Because if that's the case you can always use $HOST for the filename template and $SOURCEIP as the content template destination d_general { file ("/var/log/syslog/general/$HOST/$FULLHOST-$MONTH.$DAY.$YEAR.log" template("$DATE $SOURCEIP <$FACILITY.$PRIORITY> $MSGHDR $MSG \n") template_escape(no) ); }; This way the content has an IP whereas the filename has the hostname. Is this what you want? Or, since you are already using a non-standard file format (because of the facility/priority values) you can add both, as long as tripwire is able to process it. template("$DATE $SOURCEIP $HOST <$FACILITY.$PRIORITY> $MSGHDR $MSG\n") -- Bazsi
participants (4)
-
Balazs Scheidler
-
Martin Holste
-
Matthew Hall
-
Worsham, Michael