[syslog-ng] rewrite problem

[micah anderson]

On Mon, 02 May 2011 17:41:41 +0200, Balazs Scheidler <bazsi at balabit.hu> wrote:
> On Sun, 2011-05-01 at 12:43 -0400, micah anderson wrote:
> > On Sat, 30 Apr 2011 22:17:36 +0200, Balazs Scheidler <bazsi at balabit.hu> wrote:
> > > On Tue, 2011-04-26 at 14:51 -0400, Micah Anderson wrote:
> > > > Hi, 
> > > > 
> > > > I've been trying to use the rewrite functionality using 3.1.3, but I am
> > > > having some problems:
> > > > 
> > > > 1. when having a rewrite setup, I can denial of service syslog-ng :(
> > > > 
> > > > if I do this:
> > > > 
> > > > rewrite r_ip { subst("([[:digit:]]{1,3}\.){3}[[:digit:]]{1,3}", "\[REDACTED\]", value("MESSAGE"), type("pcre"), flags("global")); };
> > > > 
> > > > filter f_mail { facility(mail) and not facility(local1) and not program("^postfix/"); };
> > > > destination df_mail { file("/var/log/mail.log"); };
> > > > 
> > > > log {
> > > >     source(s_src);
> > > >     filter(f_mail);
> > > >     rewrite(r_ip);
> > > >     destination(df_mail);
> > > >     flags(final);
> > > > };
> > > > 
> > > > and then I do:
> > > > 
> > > > $ logger -p mail.info hi
> > > > 
> > > > syslog-ng starts going crazy, it eats 100% of the CPU and the memory. If
> > > > you are lucky, the OOM killer kicks in and stops it before your system
> > > > goes down.
> > > > 
> > > > I originally thought that this was due to a bad regexp I had, but the
> > > > above regexp works fine for normal mail syslog messages. However, in the
> > > > process of trying to make a better rewrite rule, I found that I've
> > > > created ones where normal mail messages do this too.
> > > 
> > > Hmm... well, it is still probably a regexp problem, but it'd probably
> > > make sense to validate it.
> > 
> > It could be a regexp problem, although I have not found it
> > yet. Validating would be better than having the system brought down by
> > syslog-ng, which doesn't look so good :)
> > 
> > > Hmm.. reading the code again, there's indeed a loop when "global"
> > > replacements are made. However an infinite loop is only possible, when
> > > the match has a zero length.
> > 
> > That seems like a problem, because the log message I send, if I
> > understand it correctly, has no match.
> > 
> > > Can you please give me a log sample that causes syslog-ng to spin? I
> > > couldn't reproduce it with a simplistic sample.
> > 
> > Well the above works flawlessly for me. This is the bare-minimum
> > syslog-ng.conf that causes this:
> > 
> > @version: 3.1
> > 
> > source s_src { unix-dgram("/dev/log"); internal();
> >        	     file("/proc/kmsg" program_override("kernel"));
> > };
> > 
> > rewrite r_ip { subst("([[:digit:]]{1,3}\.){3}[[:digit:]]{1,3}", "\[REDACTED\]", value("MESSAGE"), type("pcre"), flags("global")); };
> > 
> > filter f_mail { facility(mail); };
> > 
> > destination df_mail { file("/var/log/mail.log"); };
> > 
> > log {
> >     source(s_src);
> >     filter(f_mail);
> >     rewrite(r_ip);
> >     destination(df_mail);
> >     flags(final);
> > };
> > 
> > Then restart syslog-ng, and then do this (even as a regular user):
> > 
> > $ logger -p mail.info hi
> > 
> > you might want to be ready to kill the syslog-ng process, because if you
> > wait too long, then things will become very slow as you start to swap
> > and then you have to just hope the OOM killer makes the right choice.
> 
> Thanks for the detailed testcase. I've tried to reproduce the problem,
> but couldn't, with both 3.2 and 3.1. Which OS are you running syslog-ng
> on? PCRE version? Did you compile it on your own?

This is 3.1.3 on Debian sid, I did not compile it on my own. The package
depends on libpcre3 (>= 8.10) and build-depends on libpcre3-dev which
also depends on libpcrecpp0. Looking at its build logs, it was built
against 8.12-3 (-3 is the debian revision).

If you would like to look at the build log, this is it:

https://buildd.debian.org/status/fetch.php?pkg=syslog-ng&arch=amd64&ver=3.1.3-3%2Bb1&stamp=1302672422

micah
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20110502/ab6cf167/attachment.pgp