hostname not appearing correctly when receiving logs from switches
Interesting! We’ve been getting a lot of support tickets for this very problem. I can easily recreate the issue. Balabit Team: is this a new bug? [cid:image001.png@01D306E3.0FEBC990] Clayton Dukes Founder & CEO LogZilla Corporation 2900 N. Quinlan Park Rd, B240-341 Austin, TX, 78732 Tel: 936-4NetOps (463-8677) Web: www.logzilla.net<http://www.logzilla.net/> [cid:image002.png@01D306E3.0FEBC990]<https://twitter.com/logzilla>[cid:image003.png@01D306E3.0FEBC990]<https://youtu.be/drg5wv_mgfA>[cid:image004.png@01D306E3.0FEBC990]<https://www.linkedin.com/in/lzcdukes/> For NetOps, By NetOps! From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Joshua <aces621@yahoo.com> Reply-To: Joshua <aces621@yahoo.com>, Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Date: Monday, April 30, 2018 at 7:09 PM To: "syslog-ng@lists.balabit.hu" <syslog-ng@lists.balabit.hu> Subject: [Suspected Spam] [syslog-ng] hostname not appearing correctly when receiving logs from switches Hi All, I am pretty new to syslog-ng but do have some basic knowledge. I have deployed syslog-ng v3.14 on a newly deployed Linux server because syslog-ng v3.5 is working very well on another syslog server. On this new deployment, the syslogs received from most of the servers are able to show IP/host, however, the syslogs from our switches contains IP/host showing as ":" (colons). I copied the current working custom build .conf from another syslog server into our new server. Can someone help me figure out what I am missing? It is working for some components but not for switches. I tested the same switch by sending syslog to another syslog server and the hostname is appearing but just not appearing on the new syslog server. The only difference between the two server is that one uses v3.5 (the working one) and the other uses syslog-ng v3.14. I have set: "keep_hostname (yes)" but it still doesn't work. Can someone please help? Am I missing something here? Thanks Joshua Lai
Interesting that I saw this message the first time in your response, and not the original one. Anyhow, to understand the problem we would need an exact byte-by-byte representation of what syslog-ng is receiving from the switch together with the configuration that is used to process it. A tcpdump or an "Incoming message" from syslog debug outout should work. We haven't intentionally changed the syslog parser as far as I remember. On May 1, 2018 22:50, "Clayton Dukes" <cdukes@logzilla.net> wrote:
Interesting! We’ve been getting a lot of support tickets for this very problem.
I can easily recreate the issue.
Balabit Team: is this a new bug?
*[image: cid:image001.png@01D306E3.0FEBC990]*
*Clayton Dukes*
Founder & CEO
LogZilla Corporation 2900 N. Quinlan Park Rd <https://maps.google.com/?q=2900+N.+Quinlan+Park+Rd&entry=gmail&source=g>, B240-341 Austin, TX, 78732
Tel: 936-4NetOps (463-8677)
Web: www.logzilla.net
[image: cid:image002.png@01D306E3.0FEBC990] <https://twitter.com/logzilla>[image: cid:image003.png@01D306E3.0FEBC990] <https://youtu.be/drg5wv_mgfA>[image: cid:image004.png@01D306E3.0FEBC990] <https://www.linkedin.com/in/lzcdukes/>
*For NetOps, By NetOps!*
*From: *syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Joshua <aces621@yahoo.com> *Reply-To: *Joshua <aces621@yahoo.com>, Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> *Date: *Monday, April 30, 2018 at 7:09 PM *To: *"syslog-ng@lists.balabit.hu" <syslog-ng@lists.balabit.hu> *Subject: *[Suspected Spam] [syslog-ng] hostname not appearing correctly when receiving logs from switches
Hi All,
I am pretty new to syslog-ng but do have some basic knowledge. I have deployed syslog-ng v3.14 on a newly deployed Linux server because syslog-ng v3.5 is working very well on another syslog server.
On this new deployment, the syslogs received from most of the servers are able to show IP/host, however, the syslogs from our switches contains IP/host showing as ":" (colons). I copied the current working custom build .conf from another syslog server into our new server. Can someone help me figure out what I am missing? It is working for some components but not for switches. I tested the same switch by sending syslog to another syslog server and the hostname is appearing but just not appearing on the new syslog server. The only difference between the two server is that one uses v3.5 (the working one) and the other uses syslog-ng v3.14.
I have set: "keep_hostname (yes)" but it still doesn't work.
Can someone please help? Am I missing something here?
Thanks
*Joshua Lai*
For me the original showed up in my junk folder for some reason, so thats probably where it is in yours. Ronald Fenner Network Architect Game Circus LLC. rfenner@gamecircus.com
On May 1, 2018, at 7:45 PM, Scheidler, Balázs <balazs.scheidler@balabit.com> wrote:
Interesting that I saw this message the first time in your response, and not the original one.
Anyhow, to understand the problem we would need an exact byte-by-byte representation of what syslog-ng is receiving from the switch together with the configuration that is used to process it. A tcpdump or an "Incoming message" from syslog debug outout should work.
We haven't intentionally changed the syslog parser as far as I remember.
On May 1, 2018 22:50, "Clayton Dukes" <cdukes@logzilla.net <mailto:cdukes@logzilla.net>> wrote: Interesting! We’ve been getting a lot of support tickets for this very problem.
I can easily recreate the issue.
Balabit Team: is this a new bug?
<image001.png>
Clayton Dukes
Founder & CEO
LogZilla Corporation 2900 N. Quinlan Park Rd <https://maps.google.com/?q=2900+N.+Quinlan+Park+Rd&entry=gmail&source=g>, B240-341 Austin, TX, 78732
Tel: 936-4NetOps (463-8677)
Web: www.logzilla.net <http://www.logzilla.net/> <image002.png> <https://twitter.com/logzilla><image003.png> <https://youtu.be/drg5wv_mgfA><image004.png> <https://www.linkedin.com/in/lzcdukes/>
For NetOps, By NetOps!
From: syslog-ng <syslog-ng-bounces@lists.balabit.hu <mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Joshua <aces621@yahoo.com <mailto:aces621@yahoo.com>> Reply-To: Joshua <aces621@yahoo.com <mailto:aces621@yahoo.com>>, Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu <mailto:syslog-ng@lists.balabit.hu>> Date: Monday, April 30, 2018 at 7:09 PM To: "syslog-ng@lists.balabit.hu <mailto:syslog-ng@lists.balabit.hu>" <syslog-ng@lists.balabit.hu <mailto:syslog-ng@lists.balabit.hu>> Subject: [Suspected Spam] [syslog-ng] hostname not appearing correctly when receiving logs from switches
Hi All, <>
I am pretty new to syslog-ng but do have some basic knowledge. I have deployed syslog-ng v3.14 on a newly deployed Linux server because syslog-ng v3.5 is working very well on another syslog server.
On this new deployment, the syslogs received from most of the servers are able to show IP/host, however, the syslogs from our switches contains IP/host showing as ":" (colons). I copied the current working custom build .conf from another syslog server into our new server. Can someone help me figure out what I am missing? It is working for some components but not for switches. I tested the same switch by sending syslog to another syslog server and the hostname is appearing but just not appearing on the new syslog server. The only difference between the two server is that one uses v3.5 (the working one) and the other uses syslog-ng v3.14.
I have set: "keep_hostname (yes)" but it still doesn't work.
Can someone please help? Am I missing something here?
Thanks
Joshua Lai
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Unfortunately, I do not have the luxury to perform any testing since we do not have any test switch setup and due to resources. I will be removing v3.14 and installing v3.5.. Just to be clear, I did receive the syslog messages into the directory where I want the logs to be at. The only issue is the $HOST not displaying correctly from my Switch's syslog. From what I can see, it looks like the $HOST displayed was from the first word of the received syslog message. Joshua Lai On Tuesday, May 1, 2018, 5:45:10 PM PDT, Scheidler, Balázs <balazs.scheidler@balabit.com> wrote: Interesting that I saw this message the first time in your response, and not the original one. Anyhow, to understand the problem we would need an exact byte-by-byte representation of what syslog-ng is receiving from the switch together with the configuration that is used to process it. A tcpdump or an "Incoming message" from syslog debug outout should work. We haven't intentionally changed the syslog parser as far as I remember. On May 1, 2018 22:50, "Clayton Dukes" <cdukes@logzilla.net> wrote: Interesting! We’ve been getting a lot of support tickets for this very problem. I can easily recreate the issue. Balabit Team: is this a new bug? | | | Clayton Dukes Founder & CEO LogZilla Corporation 2900 N. Quinlan Park Rd, B240-341 Austin, TX, 78732 Tel: 936-4NetOps (463-8677) Web:www.logzilla.net For NetOps, By NetOps! | From: syslog-ng <syslog-ng-bounces@lists. balabit.hu> on behalf of Joshua <aces621@yahoo.com> Reply-To: Joshua <aces621@yahoo.com>, Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Date: Monday, April 30, 2018 at 7:09 PM To: "syslog-ng@lists.balabit.hu" <syslog-ng@lists.balabit.hu> Subject: [Suspected Spam] [syslog-ng] hostname not appearing correctly when receiving logs from switches Hi All, I am pretty new to syslog-ng but do have some basic knowledge. I have deployed syslog-ng v3.14 on a newly deployed Linux server because syslog-ng v3.5 is working very well on another syslog server. On this new deployment, the syslogs received from most of the servers are able to show IP/host, however, the syslogs from our switches contains IP/host showing as ":" (colons). I copied the current working custom build .conf from another syslog server into our new server. Can someone help me figure out what I am missing? It is working for some components but not for switches. I tested the same switch by sending syslog to another syslog server and the hostname is appearing but just not appearing on the new syslog server. The only difference between the two server is that one uses v3.5 (the working one) and the other uses syslog-ng v3.14. I have set: "keep_hostname (yes)" but it still doesn't work. Can someone please help? Am I missing something here? Thanks Joshua Lai
Hi Joshua, "Joshua" <aces621@yahoo.com> írta 2018-05-03 16:40-kor:
Unfortunately, I do not have the luxury to perform any testing since we do not have any test switch setup and due to resources. I will be removing v3.14 and installing v3.5.. Just to be clear, I did receive the syslog messages into the directory where I want the logs to be at. The only issue is the $HOST not displaying correctly from my Switch's syslog. From what I can see, it looks like the $HOST displayed was from the first word of the received syslog message. Joshua Lai
I don't understand why didn't they suggested to use the syslog-debun script from the contrib dir. #1. reproduction doesn't need a test switch. it needs interaction on the syslog "collector" side. #2. if you can run another "instance" (eg in a docker, lxc, pod, domU, whatever you have) and send a copy of every log messages from at least one switch where the effect is given that could help to resolve the problem As far as I remember you can define more then one log servers in a Cisco device's config and they will send copies to every defined syslog server. #3. the problem on this field is usually that rfc3164 is not a protocol, rather a description based on various garbage how network vendors implemented their logging. On the other hands this rfc were made almost two decades ago. One of the most incosequent vendor on the market I saw is Cisco. It's not easy to implement a parser which fits all of their "flavour". I think Balabit folks does a great job on that field. It worths that extra few minutes to help them out with some extra pcap. Don't misunderstand me: I like both Cisco products and Balabit products. I workd a lot with both of them. Just I'm realistic. With the help of that script you could minimize the downtime of your syslog server, if they want to see the "debug log". Anyway: If you just collect some pcap content to have examples for this case, and the pcap file would contain the whole log packets unharmed, that could be a big help too: Based on that they can reproduce the problem by resending the packets to syslog-ng, and check their debug mode instance. Btw.: the debun script is "version independent". So I suggest to use the latest version from github: To look into the script: https://github.com/balabit/syslog-ng/blob/master/contrib/syslog-ng-debun To download via curl: https://raw.githubusercontent.com/balabit/syslog-ng/master/contrib/syslog-ng... Some docs, and example for usage: https://github.com/balabit/syslog-ng/blob/master/contrib/README.syslog-ng-de... Btw2.: It could be also a useful information if you could share which Cisco device produces this effects, and what is the exact logging related configuration of that device. eg. Catalyst 6500 with x. type firmware, version v, and the output of ```sh ru | inc log ''' is ... Cheers, Gyu
Hi Balazs, Sorry for the delay, I don’t get a lot of free time these days :) I have attached a pcap as well as a raw log. The log is prior to any manipulation of LogZilla rules, etc. This is easily reproduceable. Also, if I add show-timezone to the device config, the host field shows up. The problem, of course, is that we can’t tell all of our customers to re-configure all of their cisco devices. We have documented the work-around here (search the page for “hostname missing”): http://demo.logzilla.net/help/receiving_data/cisco_ios_configuration This problem never existed before, but I am not certain which syslog-ng version it started occurring in. From: "Scheidler, Balázs" <balazs.scheidler@balabit.com> Date: Tuesday, May 1, 2018 at 8:45 PM To: Clayton Dukes <cdukes@logzilla.net> Cc: Joshua <aces621@yahoo.com>, Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] hostname not appearing correctly when receiving logs from switches Interesting that I saw this message the first time in your response, and not the original one. Anyhow, to understand the problem we would need an exact byte-by-byte representation of what syslog-ng is receiving from the switch together with the configuration that is used to process it. A tcpdump or an "Incoming message" from syslog debug outout should work. We haven't intentionally changed the syslog parser as far as I remember. On May 1, 2018 22:50, "Clayton Dukes" <cdukes@logzilla.net<mailto:cdukes@logzilla.net>> wrote: Interesting! We’ve been getting a lot of support tickets for this very problem. I can easily recreate the issue. Balabit Team: is this a new bug? [cid:image001.png@01D306E3.0FEBC990] Clayton Dukes Founder & CEO LogZilla Corporation 2900 N. Quinlan Park Rd<https://maps.google.com/?q=2900+N.+Quinlan+Park+Rd&entry=gmail&source=g>, B240-341 Austin, TX, 78732 Tel: 936-4NetOps (463-8677) Web: www.logzilla.net<http://www.logzilla.net/> [cid:image002.png@01D306E3.0FEBC990]<https://twitter.com/logzilla>[cid:image003.png@01D306E3.0FEBC990]<https://youtu.be/drg5wv_mgfA>[cid:image004.png@01D306E3.0FEBC990]<https://www.linkedin.com/in/lzcdukes/> For NetOps, By NetOps! From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Joshua <aces621@yahoo.com<mailto:aces621@yahoo.com>> Reply-To: Joshua <aces621@yahoo.com<mailto:aces621@yahoo.com>>, Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Date: Monday, April 30, 2018 at 7:09 PM To: "syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>" <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: [Suspected Spam] [syslog-ng] hostname not appearing correctly when receiving logs from switches Hi All, I am pretty new to syslog-ng but do have some basic knowledge. I have deployed syslog-ng v3.14 on a newly deployed Linux server because syslog-ng v3.5 is working very well on another syslog server. On this new deployment, the syslogs received from most of the servers are able to show IP/host, however, the syslogs from our switches contains IP/host showing as ":" (colons). I copied the current working custom build .conf from another syslog server into our new server. Can someone help me figure out what I am missing? It is working for some components but not for switches. I tested the same switch by sending syslog to another syslog server and the hostname is appearing but just not appearing on the new syslog server. The only difference between the two server is that one uses v3.5 (the working one) and the other uses syslog-ng v3.14. I have set: "keep_hostname (yes)" but it still doesn't work. Can someone please help? Am I missing something here? Thanks Joshua Lai
participants (5)
-
Clayton Dukes
-
Joshua
-
PÁSZTOR György
-
Ronald Fenner
-
Scheidler, Balázs