Troubleshooting Question
Hello All, I'd like to see if we can get some troubleshooting help with Syslog-NG OSE. Here's some background: Our environment collects logs from various network locations (F5, checkpoint, plixer, etc) and sends them to our Syslog-NG cluster through two F5 load balancers. We have four netlog boxes in round-robin that are running version 5 of the PE version of Syslog-NG on RHEL6. We also have a dev server running rhel7 and the 3.19.1 OSE edition of Syslog-NG from the COPR repository. On each of these servers, we run the Splunk Universal Forwarder that then sends the logs over to our indexer cluster. The problem we are having is that the dev server will not listen to any traffic except via localhost. We can see the traffic just fine on a TCPDUMP as it comes out of the load balancer into the dev box, but watching Syslog-NG in the foreground with -Fevd the traffic never registers at all. We can send test messages with loggen or netcat, etc, from localhost and Syslog-NG will see it and log it to disk as expected. Anywhere else it just never sees the traffic nor logs anything to disk even though we have confirmed that Syslog-NG is listening on the 9999 port to UDP with netstat. Here is our syslog-ng.conf file from the working RHEL6 boxes: https://gist.github.com/MrTink76/9ee1e88f93a313f953e4033560af463a This is the syslog-ng.conf file on the OSE box: https://gist.github.com/MrTink76/e181f4c0bf052077440d7bdfaf418e02 This is an example of our testing CONF file located in conf.d on the OSE box: https://gist.github.com/MrTink76/a0433b9e908ba683e36cb6199f9cc43f We send our test traffic to the F5 load balancer vip using UDP 9999. Like I said above, when we send on 9999 localhost with loggen or netcat, Syslog-NG sees it just fine and logs it to disk, but anywhere else it never registers nor records the test message to disk. We currently have SELinux disabled and there is no firewall running on the dev box (we see the traffic fine via tcpdump). Any help/suggestions would be greatly appreciated. Please let me know if I need to provide further information. Thanks much, Walter Tienken walter.tienken@asu.edu<https://ex2010.asu.edu/owa/redir.aspx?SURL=sS2_o_WV6gQ_JAkG-_VgxIDZLGj9-EeBZIHMzfX5pjLCAxsj0_bSCG0AYQBpAGwAdABvADoAdwBhAGwAdABlAHIALgB0AGkAZQBuAGsAZQBuAEAAYQBzAHUALgBlAGQAdQA.&URL=mailto%3awalter.tienken%40asu.edu> Cloud and Advanced Network Engineering Services
Hello, without digging any deeper into you configuration, just a small note: In most of the cases (maybe your will be different!) these kind of problems are network related and has nothing to do with Syslog-ng. Packet showing up in TCPDUMP doesn't mean necessary that it will eventually reach the application. I would make some tests with a listening netcat application first. Best regards, Laci On Fri, May 10, 2019 at 2:55 AM Walter Tienken <Walter.Tienken@asu.edu> wrote:
Hello All,
I'd like to see if we can get some troubleshooting help with Syslog-NG OSE. Here's some background:
Our environment collects logs from various network locations (F5, checkpoint, plixer, etc) and sends them to our Syslog-NG cluster through two F5 load balancers. We have four netlog boxes in round-robin that are running version 5 of the PE version of Syslog-NG on RHEL6. We also have a dev server running rhel7 and the 3.19.1 OSE edition of Syslog-NG from the COPR repository. On each of these servers, we run the Splunk Universal Forwarder that then sends the logs over to our indexer cluster.
The problem we are having is that the dev server will not listen to any traffic except via localhost. We can see the traffic just fine on a TCPDUMP as it comes out of the load balancer into the dev box, but watching Syslog-NG in the foreground with -Fevd the traffic never registers at all. We can send test messages with loggen or netcat, etc, from localhost and Syslog-NG will see it and log it to disk as expected. Anywhere else it just never sees the traffic nor logs anything to disk even though we have confirmed that Syslog-NG is listening on the 9999 port to UDP with netstat.
Here is our syslog-ng.conf file from the working RHEL6 boxes:
https://gist.github.com/MrTink76/9ee1e88f93a313f953e4033560af463a
This is the syslog-ng.conf file on the OSE box:
https://gist.github.com/MrTink76/e181f4c0bf052077440d7bdfaf418e02
This is an example of our testing CONF file located in conf.d on the OSE box:
https://gist.github.com/MrTink76/a0433b9e908ba683e36cb6199f9cc43f
We send our test traffic to the F5 load balancer vip using UDP 9999. Like I said above, when we send on 9999 localhost with loggen or netcat, Syslog-NG sees it just fine and logs it to disk, but anywhere else it never registers nor records the test message to disk. We currently have SELinux disabled and there is no firewall running on the dev box (we see the traffic fine via tcpdump).
Any help/suggestions would be greatly appreciated. Please let me know if I need to provide further information.
Thanks much,
Walter Tienken
walter.tienken@asu.edu <https://ex2010.asu.edu/owa/redir.aspx?SURL=sS2_o_WV6gQ_JAkG-_VgxIDZLGj9-EeBZIHMzfX5pjLCAxsj0_bSCG0AYQBpAGwAdABvADoAdwBhAGwAdABlAHIALgB0AGkAZQBuAGsAZQBuAEAAYQBzAHUALgBlAGQAdQA.&URL=mailto%3awalter.tienken%40asu.edu>
Cloud and Advanced Network Engineering Services
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Fri, May 10, 2019 at 09:06:19AM +0200, Szemere, László wrote:
Hello, without digging any deeper into you configuration, just a small note: In most of the cases (maybe your will be different!) these kind of problems are network related and has nothing to do with Syslog-ng. Packet showing up in TCPDUMP doesn't mean necessary that it will eventually reach the application. I would make some tests with a listening netcat application first.
I second that: tcpdump will happily bypass iptables
Hi, Thanks for the suggestions. I ran "nc -lvu 9999" on both the prod and dev machines, I never saw the traffic hit netcat on either machine, but the prod running version 5 PE picked up the test traffic fine and logged it to disk. The dev box running 3.19.1 OSE did not pick up the test traffic nor log it to disk. Admittedly I am not very well versed in using netcat, but I believe that is the correct way to run it (if not please elighten me!). I also verified that no firewall is running on the dev box ... I checked both iptables and firewalld and they are both not running or active. What am I missing? Thanks, Walter Tienken walter.tienken@asu.edu <https://ex2010.asu.edu/owa/redir.aspx?SURL=sS2_o_WV6gQ_JAkG-_VgxIDZLGj9-EeBZIHMzfX5pjLCAxsj0_bSCG0AYQBpAGwAdABvADoAdwBhAGwAdABlAHIALgB0AGkAZQBuAGsAZQBuAEAAYQBzAHUALgBlAGQAdQA.&URL=mailto%3awalter.tienken%40asu.edu> Cloud and Advanced Network Engineering Services On 5/10/19, 12:40 AM, "syslog-ng on behalf of Fabien Wernli" <syslog-ng-bounces@lists.balabit.hu on behalf of wernli@in2p3.fr> wrote: On Fri, May 10, 2019 at 09:06:19AM +0200, Szemere, László wrote: > Hello, > without digging any deeper into you configuration, just a small note: In > most of the cases (maybe your will be different!) these kind of problems > are network related and has nothing to do with Syslog-ng. > Packet showing up in TCPDUMP doesn't mean necessary that it will > eventually reach the application. I would make some tests with a listening > netcat application first. I second that: tcpdump will happily bypass iptables
Does your box have multiple interfaces? Does it run a dual-stack? If yes, does the program (nc or syslogng) listen on the correct interface? (netstat -ulp)
participants (3)
-
Fabien Wernli
-
Szemere, László
-
Walter Tienken