processing stops after config change
I have a pretty simple configuration file on my syslog-ng-2.1.3.1 running on CentOS 5.4 64bit and 8GB of RAM. I'm sending logs to 3 destinations. One is a mysql database, one is a flat file, and one is to a log correlation engine. All was running fine and out of nowhere one day, I noticed that syslog-ng kept hanging. I would restart the process and it would be fine for a few seconds then crash. I noticed that when I would start it, it would jump to 100% CPU and stay there until crash. I did some log viewing and straced the process and I noticed that logs were getting to the process, but they never got to their destinations. I started disabling one destination at a time until I found the culprit. The problem was with the destination going to the log correlation via udp 514. If I comment out that source and destination, the system runs like a champ to the other destinations. I'm confused as to what happened, because this was working for at least 3 months solid. I am posting the config here, if anyone can see something or has some suggestions, I would appreciate it. #begin syslog-ng.conf source s_all{ unix-stream("/dev/log"); internal(); udp(ip(0.0.0.0) port(514)); }; destination d_lce { udp("xxx.xxx.xxx.xxx"); } log { source(s_all); destination(d_lce); }; Just to reiterate, I do have two other destinations setup and they run just fine if I comment out this destination and log statement. Every time I uncomment this destination and log statement and restart syslog-ng, I reproduce the problem. Any suggestions would be great, and if you need more info, just ask and I can present it. The first thing I wanted to check is that the config looks sane, though again it worked fine for months. Thanks Curt
In the 2.1.4 release notes, there is this comment: * Fixed a possible DoS condition triggered by a destination port unreaachable ICMP packet received from a UDP destination. syslog-ng started eating all available memory and CPU until it crashed if this happened. It certainly sounds like what you were seeing. Regards, Jerry On 3/26/2010 8:20 AM, Curt Shaffer wrote:
I have a pretty simple configuration file on my syslog-ng-2.1.3.1 running on CentOS 5.4 64bit and 8GB of RAM.
I'm sending logs to 3 destinations. One is a mysql database, one is a flat file, and one is to a log correlation engine. All was running fine and out of nowhere one day, I noticed that syslog-ng kept hanging. I would restart the process and it would be fine for a few seconds then crash. I noticed that when I would start it, it would jump to 100% CPU and stay there until crash. I did some log viewing and straced the process and I noticed that logs were getting to the process, but they never got to their destinations. I started disabling one destination at a time until I found the culprit. The problem was with the destination going to the log correlation via udp 514. If I comment out that source and destination, the system runs like a champ to the other destinations.
I'm confused as to what happened, because this was working for at least 3 months solid. I am posting the config here, if anyone can see something or has some suggestions, I would appreciate it.
#begin syslog-ng.conf
source s_all{ unix-stream("/dev/log"); internal(); udp(ip(0.0.0.0) port(514)); };
destination d_lce { udp("xxx.xxx.xxx.xxx"); }
log { source(s_all); destination(d_lce); };
Just to reiterate, I do have two other destinations setup and they run just fine if I comment out this destination and log statement. Every time I uncomment this destination and log statement and restart syslog-ng, I reproduce the problem.
Any suggestions would be great, and if you need more info, just ask and I can present it. The first thing I wanted to check is that the config looks sane, though again it worked fine for months.
Thanks
Curt ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
I should have looked there. It didn't occur to me as things were working great for so long. That does seem to have fixed it for now anyway. Thanks! Curt On Mar 26, 2010, at 9:16 AM, Jerry Bell wrote:
In the 2.1.4 release notes, there is this comment:
* Fixed a possible DoS condition triggered by a destination port unreaachable ICMP packet received from a UDP destination. syslog-ng started eating all available memory and CPU until it crashed if this happened.
It certainly sounds like what you were seeing.
Regards,
Jerry On 3/26/2010 8:20 AM, Curt Shaffer wrote:
I have a pretty simple configuration file on my syslog-ng-2.1.3.1 running on CentOS 5.4 64bit and 8GB of RAM.
I'm sending logs to 3 destinations. One is a mysql database, one is a flat file, and one is to a log correlation engine. All was running fine and out of nowhere one day, I noticed that syslog-ng kept hanging. I would restart the process and it would be fine for a few seconds then crash. I noticed that when I would start it, it would jump to 100% CPU and stay there until crash. I did some log viewing and straced the process and I noticed that logs were getting to the process, but they never got to their destinations. I started disabling one destination at a time until I found the culprit. The problem was with the destination going to the log correlation via udp 514. If I comment out that source and destination, the system runs like a champ to the other destinations.
I'm confused as to what happened, because this was working for at least 3 months solid. I am posting the config here, if anyone can see something or has some suggestions, I would appreciate it.
#begin syslog-ng.conf
source s_all{ unix-stream("/dev/log"); internal(); udp(ip(0.0.0.0) port(514)); };
destination d_lce { udp("xxx.xxx.xxx.xxx"); }
log { source(s_all); destination(d_lce); };
Just to reiterate, I do have two other destinations setup and they run just fine if I comment out this destination and log statement. Every time I uncomment this destination and log statement and restart syslog-ng, I reproduce the problem.
Any suggestions would be great, and if you need more info, just ask and I can present it. The first thing I wanted to check is that the config looks sane, though again it worked fine for months.
Thanks
Curt ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
participants (2)
-
Curt Shaffer
-
Jerry Bell