<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#0050d0">
<font size="-1"><font face="Helvetica, Arial, sans-serif">We turned on
syslog-ng for our production environment today and ran into a problem
which I think the solution might also solve this one.<br>
The problem I ran into is that with full production logging on,
syslog-ng is using up about 50% of the CPU, and thats writing
everything out to /dev/null (a few hundred machines). With a single
regex turned on, it was using 90% CPU. So what I'm going to end up
doing is running multiple syslog-ng processes so they can run in
parallel on separate CPU cores. This way they can process data
simultaneously. There will be a master process which basically sends
everthing with the mail facility to two other syslog-ng processes, one
of which will be parsing out data to insert into a database, and
another will be parsing out data to write to files (different regexes
for each).<br>
<br>
Now, the way this might also solve the failover issue is to make
multi-process capability part of syslog-ng. So one might end up with a
config like:<br>
<br>
</font></font><font size="-1"><tt><font color="#990000">@version: 3.0<br>
options {<br>
use_dns(no);<br>
log_iw_size(10000);<br>
};<br>
<br>
</font></tt></font><tt><font color="#990000" size="-1">source s_sys {<br>
unix-stream('/dev/log');<br>
};<br>
source s_net {<br>
tcp(ip(0.0.0.0) port(514) max-connections(1000));<br>
udp(ip(0.0.0.0) port(514));<br>
};</font><font color="#990000"><br>
</font></tt><font size="-1"><tt><font color="#990000">filter f_mail {
facility(mail); };<br>
process p_msgid {<br>
filter f_msgid {<br>
message('MsgID: (?<MESSAGEID>\S+)', type('pcre')
flags('nobackref','store-matches'));<br>
};<br>
destination d_oracle { sql(...); };<br>
destination d_oracle_fallback { sql(...); };<br>
log { filter(f_msgid); destination(d_oracle); };<br>
log { filter(f_msgid); destination(d_oracle_fallback);
flags(fallback); };<br>
};<br>
process p_foobar {<br>
options {<br>
flush_timeout(1000);<br>
}<br>
filter f_foobar { ... };<br>
destination d_foobar { ... };<br>
destination d_foobar_fallback { ... };<br>
log { filter(f_foobar); destination(d_foobar); };<br>
log { filter(f_foobar); destination(d_foobar_fallback);
flags(fallback); };<br>
};<br>
<br>
log { source(s_sys); source(s_net); filter(f_mail);
destination(p_msgid); destination(p_foobar); };</font></tt></font><font
size="-1"><font face="Helvetica, Arial, sans-serif"><br>
<br>
This will launch 3 processes, a master control process that does very
basic filtering and uses some sort of IPC to send the data to 2 other
processes which inherit the global options section and accept every
config statement the master process does except sources. This way each
process can run in parallel on the different CPU cores, and can have
fallback destinations that wont interfere with the other processes.<br>
<br>
</font></font><br>
Sent: Wednesday, March 17, 2010 7:54:42 AM<br>
From: Zoltán Pallagi <a class="moz-txt-link-rfc2396E" href="mailto:pzolee@balabit.hu"><pzolee@balabit.hu></a><br>
To: Syslog-ng users' and developers' mailing list
<a class="moz-txt-link-rfc2396E" href="mailto:syslog-ng@lists.balabit.hu"><syslog-ng@lists.balabit.hu></a>, <a class="moz-txt-link-abbreviated" href="mailto:syslogng@feystorm.net">syslogng@feystorm.net</a> <br>
Subject: Re: [syslog-ng] log failback groups
<blockquote cite="mid:4BA0DF22.8040103@balabit.hu" type="cite">
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
Hi Patrick,<br>
<br>
As far as I know, we would like to solve the failover/failback problem
in syslog-ng PE v3.2 (and perhaps in OSE v3.2), but we are still
working on it.<br>
However, we are planning to support it only in case of tcp/syslog
destinations.<br>
<br>
Patrick H. wrote:
<blockquote cite="mid:4BA0192E.80701@feystorm.net" type="cite"><big><font
size="-1"><big><font face="Helvetica, Arial, sans-serif">How
do you do groups of failbacks? For example<br>
<br>
log { filter(f_filter1); destination(d_file1); };<br>
log { filter(f_filter1); destination(d_file2); }; <-- log here only
if the above fails<br>
log { filter(f_filter2); destination(d_sql1); }; <-- may include
messages from the above 2 lines<br>
log { filter(f_filter2); destination(d_sql2); }; <-- log here only
if d_sql1 fails<br>
<br>
So, say lines 1 and 3 both fail, lines 2 and 3 should both start
working. If I put a fallback flag on d_file2 and d_sql2, and d_sql1
fails, d_sql2 wont kick in on matches that d_file1 is still taking.<br>
It seems like there should be a way to do "</font></big></font><font
size="-1"><big><tt>log { destination(...) or destination(...); };</tt></big></font><font
size="-1"><big><font face="Helvetica, Arial, sans-serif">", or "</font></big></font><font
size="-1"><big><tt>log {...} or log {...};</tt></big></font><font
size="-1"><big><font face="Helvetica, Arial, sans-serif">".<br>
<br>
</font></big></font></big>
<pre wrap=""><hr size="4" width="90%">
______________________________________________________________________________
Member info: <a moz-do-not-send="true" class="moz-txt-link-freetext"
href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://www.campin.net/syslog-ng/faq.html">http://www.campin.net/syslog-ng/faq.html</a>
</pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
pzolee
</pre>
</blockquote>
</body>
</html>