[Bug 46] New: Syslog-ng can go into and endless loop, consume all memory, and crash
https://bugzilla.balabit.com/show_bug.cgi?id=46 Summary: Syslog-ng can go into and endless loop, consume all memory, and crash Product: syslog-ng Version: 2.0.x Platform: PC OS/Version: Linux Status: NEW Severity: critical Priority: unspecified Component: syslog-ng AssignedTo: bazsi@balabit.hu ReportedBy: support@sikkerhed.org Type of the Report: bug Estimated Hours: 0.0 I've discovered a scenario where syslog-ng will eat all memory, and then either crash or get OOM-killed. I'm using the following config: {{{ options { chain_hostnames(0); time_reopen(10); time_reap(360); log_fifo_size(2048); create_dirs(yes); owner(root); group(adm); perm(0640); dir_owner(root); dir_group(adm); dir_perm(0755); use_dns(yes); use_fqdn(yes); stats_freq(3600); }; source s_all { internal(); unix-stream("/dev/log"); file("/proc/kmsg" log_prefix("kernel: ")); }; destination du_all { usertty("*"); }; destination df_logs { file("/var/log/$FACILITY/log-$YEAR-$MONTH-$DAY"); file("/var/log/$FACILITY/current"); }; destination df_everything { file("/var/log/everything/log-$YEAR-$MONTH-$DAY"); file("/var/log/everything/current"); }; filter f_messages { level(info,notice,warn) and not facility(auth,authpriv); }; filter f_emerg { level(emerg); }; log { source(s_all); destination(df_logs); }; log { source(s_all); filter(f_messages); destination(df_everything); }; log { source(s_all); filter(f_emerg); destination(du_all); }; }}} The problem is the "/var/log/$FACILITY/current" output. I had some old output left over from the earlier logger, which meant that /var/log/syslog was a file. When syslog-ng expects this node to be a directory, not a file, it tries to mkdir() it, and fails. This fail generates a syslog message, which needs to get logged to /var/log/syslog/current... which doesn't exist, is mkdir()'ed, fails, is syslogged, etc. In other words, it never starts up, and doesn't log anything useful about what is going on. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=46 Balazs Scheidler <bazsi@balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution| |WONTFIX --- Comment #1 from Balazs Scheidler <bazsi@balabit.hu> 2009-05-04 10:35:53 --- This is a positive feedback loop between the internal() source and one of the destinations (e.g. internal feeds messages to a given destination which in turn generate error messages that originate from internal()). There's a proper solution for this in syslog-ng 3.0, in which case it'll detect this case and drop internal messages accordingly. Since the change is rather big, it cannot really be backported to either 2.0 or 2.1, so please use 3.0 if you want this fixed, or don't feed internal() logs to anywhere but a local static file, then read that file using the file() source. This is only a workaround I know. I set this to WONTFIX on 2.0, a fix is available in later versions. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=46 --- Comment #2 from Sikkerhed.org ApS <support@sikkerhed.org> 2009-05-05 14:50:20 --- (In reply to comment #1)
This is a positive feedback loop between the internal() source and one of the destinations (e.g. internal feeds messages to a given destination which in turn generate error messages that originate from internal()).
There's a proper solution for this in syslog-ng 3.0, in which case it'll detect this case and drop internal messages accordingly.
Since the change is rather big, it cannot really be backported to either 2.0 or 2.1, so please use 3.0 if you want this fixed, or don't feed internal() logs to anywhere but a local static file, then read that file using the file() source. This is only a workaround I know.
I set this to WONTFIX on 2.0, a fix is available in later versions.
Ok. We've rolled out syslog-ng on all our servers now anyway, so it shouldn't be a problem from here on. It was, however, a little unnerving when it first happened, before I figured out why :-) Are there other major reasons to use version 3.x over 2.x? By the way, there's a problem with your mail server configuration. When replying to a bug report, it says: <bugzilla@bugzilla.balabit.com>: host bugzilla.balabit.com[195.70.41.85] said: 554 5.1.1 <bugzilla@bugzilla.balabit.com>: Recipient address rejected: undeliverable address: mail for bugzilla.balabit.com loops back to myself (in reply to RCPT TO command) -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=46 --- Comment #3 from Balazs Scheidler <bazsi@balabit.hu> 2009-05-07 16:43:15 --- (In reply to comment #2)
(In reply to comment #1)
This is a positive feedback loop between the internal() source and one of the destinations (e.g. internal feeds messages to a given destination which in turn generate error messages that originate from internal()).
There's a proper solution for this in syslog-ng 3.0, in which case it'll detect this case and drop internal messages accordingly.
Since the change is rather big, it cannot really be backported to either 2.0 or 2.1, so please use 3.0 if you want this fixed, or don't feed internal() logs to anywhere but a local static file, then read that file using the file() source. This is only a workaround I know.
I set this to WONTFIX on 2.0, a fix is available in later versions.
Ok. We've rolled out syslog-ng on all our servers now anyway, so it shouldn't be a problem from here on.
if there's a feedback loop, and you don't apply the workaround I recommended you might have the same problem if the central server goes down.
It was, however, a little unnerving when it first happened, before I figured out why :-)
Are there other major reasons to use version 3.x over 2.x?
Sure there are various, especially if you also want to take care about the contents of your messages, rather than just store them. http://bazsi.blogs.balabit.com/2009/03/as-promised-on-mailing-list-here-come...
By the way, there's a problem with your mail server configuration. When replying to a bug report, it says:
<bugzilla@bugzilla.balabit.com>: host bugzilla.balabit.com[195.70.41.85] said: 554 5.1.1 <bugzilla@bugzilla.balabit.com>: Recipient address rejected: undeliverable address: mail for bugzilla.balabit.com loops back to myself (in reply to RCPT TO command)
bugzilla@bugzilla.balabit.com is not for replying messages, you should rather post your questions back to the ticket that you opened. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=46 --- Comment #4 from Sikkerhed.org ApS <support@sikkerhed.org> 2009-05-07 23:17:56 --- (In reply to comment #3)
(In reply to comment #2)
(In reply to comment #1)
This is a positive feedback loop between the internal() source and one of the destinations (e.g. internal feeds messages to a given destination which in turn generate error messages that originate from internal()).
There's a proper solution for this in syslog-ng 3.0, in which case it'll detect this case and drop internal messages accordingly.
Since the change is rather big, it cannot really be backported to either 2.0 or 2.1, so please use 3.0 if you want this fixed, or don't feed internal() logs to anywhere but a local static file, then read that file using the file() source. This is only a workaround I know.
I set this to WONTFIX on 2.0, a fix is available in later versions.
Ok. We've rolled out syslog-ng on all our servers now anyway, so it shouldn't be a problem from here on.
if there's a feedback loop, and you don't apply the workaround I recommended you might have the same problem if the central server goes down.
Well, your solution was to save to a local static file. That is what I am already doing, if I understand you correctly. The static file is /var/log/syslog/current. I guess that by "static" you mean a file that doesn't have any macro expansions in the name? Now that all the machines have the dir /var/log/syslog in place, I don't see how this feedback loop could be triggered again?
It was, however, a little unnerving when it first happened, before I figured out why :-)
Are there other major reasons to use version 3.x over 2.x?
Sure there are various, especially if you also want to take care about the contents of your messages, rather than just store them.
http://bazsi.blogs.balabit.com/2009/03/as-promised-on-mailing-list-here-come...
I'll read that, thanks.
By the way, there's a problem with your mail server configuration. When replying to a bug report, it says:
<bugzilla@bugzilla.balabit.com>: host bugzilla.balabit.com[195.70.41.85] said: 554 5.1.1 <bugzilla@bugzilla.balabit.com>: Recipient address rejected: undeliverable address: mail for bugzilla.balabit.com loops back to myself (in reply to RCPT TO command)
bugzilla@bugzilla.balabit.com is not for replying messages, you should rather post your questions back to the ticket that you opened.
Wouldn't it make more sense to call it do-not-reply@balabit.com.invalid then? Suppling a valid-looking email address that can't be used is not so user-friendly :) -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
participants (1)
-
bugzilla＠bugzilla.balabit.com