[syslog-ng] rewrite problem

[Balazs Scheidler]

On Sun, 2011-05-01 at 12:43 -0400, micah anderson wrote:
> On Sat, 30 Apr 2011 22:17:36 +0200, Balazs Scheidler <bazsi at balabit.hu> wrote:
> > On Tue, 2011-04-26 at 14:51 -0400, Micah Anderson wrote:
> > > Hi, 
> > > 
> > > I've been trying to use the rewrite functionality using 3.1.3, but I am
> > > having some problems:
> > > 
> > > 1. when having a rewrite setup, I can denial of service syslog-ng :(
> > > 
> > > if I do this:
> > > 
> > > rewrite r_ip { subst("([[:digit:]]{1,3}\.){3}[[:digit:]]{1,3}", "\[REDACTED\]", value("MESSAGE"), type("pcre"), flags("global")); };
> > > 
> > > filter f_mail { facility(mail) and not facility(local1) and not program("^postfix/"); };
> > > destination df_mail { file("/var/log/mail.log"); };
> > > 
> > > log {
> > >     source(s_src);
> > >     filter(f_mail);
> > >     rewrite(r_ip);
> > >     destination(df_mail);
> > >     flags(final);
> > > };
> > > 
> > > and then I do:
> > > 
> > > $ logger -p mail.info hi
> > > 
> > > syslog-ng starts going crazy, it eats 100% of the CPU and the memory. If
> > > you are lucky, the OOM killer kicks in and stops it before your system
> > > goes down.
> > > 
> > > I originally thought that this was due to a bad regexp I had, but the
> > > above regexp works fine for normal mail syslog messages. However, in the
> > > process of trying to make a better rewrite rule, I found that I've
> > > created ones where normal mail messages do this too.
> > 
> > Hmm... well, it is still probably a regexp problem, but it'd probably
> > make sense to validate it.
> 
> It could be a regexp problem, although I have not found it
> yet. Validating would be better than having the system brought down by
> syslog-ng, which doesn't look so good :)
> 
> > Hmm.. reading the code again, there's indeed a loop when "global"
> > replacements are made. However an infinite loop is only possible, when
> > the match has a zero length.
> 
> That seems like a problem, because the log message I send, if I
> understand it correctly, has no match.
> 
> > Can you please give me a log sample that causes syslog-ng to spin? I
> > couldn't reproduce it with a simplistic sample.
> 
> Well the above works flawlessly for me. This is the bare-minimum
> syslog-ng.conf that causes this:
> 
> @version: 3.1
> 
> source s_src { unix-dgram("/dev/log"); internal();
>        	     file("/proc/kmsg" program_override("kernel"));
> };
> 
> rewrite r_ip { subst("([[:digit:]]{1,3}\.){3}[[:digit:]]{1,3}", "\[REDACTED\]", value("MESSAGE"), type("pcre"), flags("global")); };
> 
> filter f_mail { facility(mail); };
> 
> destination df_mail { file("/var/log/mail.log"); };
> 
> log {
>     source(s_src);
>     filter(f_mail);
>     rewrite(r_ip);
>     destination(df_mail);
>     flags(final);
> };
> 
> Then restart syslog-ng, and then do this (even as a regular user):
> 
> $ logger -p mail.info hi
> 
> you might want to be ready to kill the syslog-ng process, because if you
> wait too long, then things will become very slow as you start to swap
> and then you have to just hope the OOM killer makes the right choice.

Thanks for the detailed testcase. I've tried to reproduce the problem,
but couldn't, with both 3.2 and 3.1. Which OS are you running syslog-ng
on? PCRE version? Did you compile it on your own?

-- 
Bazsi