How do you do groups of failbacks? For example log { filter(f_filter1); destination(d_file1); }; log { filter(f_filter1); destination(d_file2); }; <-- log here only if the above fails log { filter(f_filter2); destination(d_sql1); }; <-- may include messages from the above 2 lines log { filter(f_filter2); destination(d_sql2); }; <-- log here only if d_sql1 fails So, say lines 1 and 3 both fail, lines 2 and 3 should both start working. If I put a fallback flag on d_file2 and d_sql2, and d_sql1 fails, d_sql2 wont kick in on matches that d_file1 is still taking. It seems like there should be a way to do "log { destination(...) or destination(...); };", or "log {...} or log {...};".
Hi Patrick, Currently this is not supported in syslog-ng, but should be in the (hopefully near) future. Bazsi, can you give an estimate on that? Regards, Robert Patrick H. wrote:
How do you do groups of failbacks? For example
log { filter(f_filter1); destination(d_file1); }; log { filter(f_filter1); destination(d_file2); }; <-- log here only if the above fails log { filter(f_filter2); destination(d_sql1); }; <-- may include messages from the above 2 lines log { filter(f_filter2); destination(d_sql2); }; <-- log here only if d_sql1 fails
So, say lines 1 and 3 both fail, lines 2 and 3 should both start working. If I put a fallback flag on d_file2 and d_sql2, and d_sql1 fails, d_sql2 wont kick in on matches that d_file1 is still taking. It seems like there should be a way to do "log { destination(...) or destination(...); };", or "log {...} or log {...};".
------------------------------------------------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Hi Patrick, As far as I know, we would like to solve the failover/failback problem in syslog-ng PE v3.2 (and perhaps in OSE v3.2), but we are still working on it. However, we are planning to support it only in case of tcp/syslog destinations. Patrick H. wrote:
How do you do groups of failbacks? For example
log { filter(f_filter1); destination(d_file1); }; log { filter(f_filter1); destination(d_file2); }; <-- log here only if the above fails log { filter(f_filter2); destination(d_sql1); }; <-- may include messages from the above 2 lines log { filter(f_filter2); destination(d_sql2); }; <-- log here only if d_sql1 fails
So, say lines 1 and 3 both fail, lines 2 and 3 should both start working. If I put a fallback flag on d_file2 and d_sql2, and d_sql1 fails, d_sql2 wont kick in on matches that d_file1 is still taking. It seems like there should be a way to do "log { destination(...) or destination(...); };", or "log {...} or log {...};".
------------------------------------------------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- pzolee
We turned on syslog-ng for our production environment today and ran into a problem which I think the solution might also solve this one. The problem I ran into is that with full production logging on, syslog-ng is using up about 50% of the CPU, and thats writing everything out to /dev/null (a few hundred machines). With a single regex turned on, it was using 90% CPU. So what I'm going to end up doing is running multiple syslog-ng processes so they can run in parallel on separate CPU cores. This way they can process data simultaneously. There will be a master process which basically sends everthing with the mail facility to two other syslog-ng processes, one of which will be parsing out data to insert into a database, and another will be parsing out data to write to files (different regexes for each). Now, the way this might also solve the failover issue is to make multi-process capability part of syslog-ng. So one might end up with a config like: @version: 3.0 options { use_dns(no); log_iw_size(10000); }; source s_sys { unix-stream('/dev/log'); }; source s_net { tcp(ip(0.0.0.0) port(514) max-connections(1000)); udp(ip(0.0.0.0) port(514)); }; filter f_mail { facility(mail); }; process p_msgid { filter f_msgid { message('MsgID: (?<MESSAGEID>\S+)', type('pcre') flags('nobackref','store-matches')); }; destination d_oracle { sql(...); }; destination d_oracle_fallback { sql(...); }; log { filter(f_msgid); destination(d_oracle); }; log { filter(f_msgid); destination(d_oracle_fallback); flags(fallback); }; }; process p_foobar { options { flush_timeout(1000); } filter f_foobar { ... }; destination d_foobar { ... }; destination d_foobar_fallback { ... }; log { filter(f_foobar); destination(d_foobar); }; log { filter(f_foobar); destination(d_foobar_fallback); flags(fallback); }; }; log { source(s_sys); source(s_net); filter(f_mail); destination(p_msgid); destination(p_foobar); }; This will launch 3 processes, a master control process that does very basic filtering and uses some sort of IPC to send the data to 2 other processes which inherit the global options section and accept every config statement the master process does except sources. This way each process can run in parallel on the different CPU cores, and can have fallback destinations that wont interfere with the other processes. Sent: Wednesday, March 17, 2010 7:54:42 AM From: Zoltán Pallagi <pzolee@balabit.hu> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu>, syslogng@feystorm.net Subject: Re: [syslog-ng] log failback groups
Hi Patrick,
As far as I know, we would like to solve the failover/failback problem in syslog-ng PE v3.2 (and perhaps in OSE v3.2), but we are still working on it. However, we are planning to support it only in case of tcp/syslog destinations.
Patrick H. wrote:
How do you do groups of failbacks? For example
log { filter(f_filter1); destination(d_file1); }; log { filter(f_filter1); destination(d_file2); }; <-- log here only if the above fails log { filter(f_filter2); destination(d_sql1); }; <-- may include messages from the above 2 lines log { filter(f_filter2); destination(d_sql2); }; <-- log here only if d_sql1 fails
So, say lines 1 and 3 both fail, lines 2 and 3 should both start working. If I put a fallback flag on d_file2 and d_sql2, and d_sql1 fails, d_sql2 wont kick in on matches that d_file1 is still taking. It seems like there should be a way to do "log { destination(...) or destination(...); };", or "log {...} or log {...};".
------------------------------------------------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- pzolee
How many messages per second is the system attempting to handle? I'm very surprised that you're seeing that level of utilization. In our setup we've never had a problem pushing up through 30,000 messages per second written to disk with Syslog-NG in production, and I've pushed more than 70,000 per second in development. Your forked process idea seems like a good one, but I haven't seen cases where Syslog-NG is the bottleneck. --Martin On Thu, Mar 18, 2010 at 7:49 PM, Patrick H. <syslogng@feystorm.net> wrote:
We turned on syslog-ng for our production environment today and ran into a problem which I think the solution might also solve this one. The problem I ran into is that with full production logging on, syslog-ng is using up about 50% of the CPU, and thats writing everything out to /dev/null (a few hundred machines). With a single regex turned on, it was using 90% CPU. So what I'm going to end up doing is running multiple syslog-ng processes so they can run in parallel on separate CPU cores. This way they can process data simultaneously. There will be a master process which basically sends everthing with the mail facility to two other syslog-ng processes, one of which will be parsing out data to insert into a database, and another will be parsing out data to write to files (different regexes for each).
Now, the way this might also solve the failover issue is to make multi-process capability part of syslog-ng. So one might end up with a config like:
@version: 3.0 options { use_dns(no); log_iw_size(10000); };
source s_sys { unix-stream('/dev/log'); }; source s_net { tcp(ip(0.0.0.0) port(514) max-connections(1000)); udp(ip(0.0.0.0) port(514)); }; filter f_mail { facility(mail); }; process p_msgid { filter f_msgid { message('MsgID: (?<MESSAGEID>\S+)', type('pcre') flags('nobackref','store-matches')); }; destination d_oracle { sql(...); }; destination d_oracle_fallback { sql(...); }; log { filter(f_msgid); destination(d_oracle); }; log { filter(f_msgid); destination(d_oracle_fallback); flags(fallback); }; }; process p_foobar { options { flush_timeout(1000); } filter f_foobar { ... }; destination d_foobar { ... }; destination d_foobar_fallback { ... }; log { filter(f_foobar); destination(d_foobar); }; log { filter(f_foobar); destination(d_foobar_fallback); flags(fallback); }; };
log { source(s_sys); source(s_net); filter(f_mail); destination(p_msgid); destination(p_foobar); };
This will launch 3 processes, a master control process that does very basic filtering and uses some sort of IPC to send the data to 2 other processes which inherit the global options section and accept every config statement the master process does except sources. This way each process can run in parallel on the different CPU cores, and can have fallback destinations that wont interfere with the other processes.
Sent: Wednesday, March 17, 2010 7:54:42 AM From: Zoltán Pallagi <pzolee@balabit.hu> <pzolee@balabit.hu> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> <syslog-ng@lists.balabit.hu>, syslogng@feystorm.net Subject: Re: [syslog-ng] log failback groups
Hi Patrick,
As far as I know, we would like to solve the failover/failback problem in syslog-ng PE v3.2 (and perhaps in OSE v3.2), but we are still working on it. However, we are planning to support it only in case of tcp/syslog destinations.
Patrick H. wrote:
How do you do groups of failbacks? For example
log { filter(f_filter1); destination(d_file1); }; log { filter(f_filter1); destination(d_file2); }; <-- log here only if the above fails log { filter(f_filter2); destination(d_sql1); }; <-- may include messages from the above 2 lines log { filter(f_filter2); destination(d_sql2); }; <-- log here only if d_sql1 fails
So, say lines 1 and 3 both fail, lines 2 and 3 should both start working. If I put a fallback flag on d_file2 and d_sql2, and d_sql1 fails, d_sql2 wont kick in on matches that d_file1 is still taking. It seems like there should be a way to do "log { destination(...) or destination(...); };", or "log {...} or log {...};".
------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- pzolee
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
I'll check tomorrow when I'm back at work, but when outputting to files, it writes roughly 1.5 mb/s. Processor is a 2.8ghz 64-bit xeon. Sent: Thursday, March 18, 2010 8:42:21 PM From: Martin Holste <mcholste@gmail.com> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] log failback groups
How many messages per second is the system attempting to handle? I'm very surprised that you're seeing that level of utilization. In our setup we've never had a problem pushing up through 30,000 messages per second written to disk with Syslog-NG in production, and I've pushed more than 70,000 per second in development. Your forked process idea seems like a good one, but I haven't seen cases where Syslog-NG is the bottleneck.
--Martin
On Thu, Mar 18, 2010 at 7:49 PM, Patrick H. <syslogng@feystorm.net <mailto:syslogng@feystorm.net>> wrote:
We turned on syslog-ng for our production environment today and ran into a problem which I think the solution might also solve this one. The problem I ran into is that with full production logging on, syslog-ng is using up about 50% of the CPU, and thats writing everything out to /dev/null (a few hundred machines). With a single regex turned on, it was using 90% CPU. So what I'm going to end up doing is running multiple syslog-ng processes so they can run in parallel on separate CPU cores. This way they can process data simultaneously. There will be a master process which basically sends everthing with the mail facility to two other syslog-ng processes, one of which will be parsing out data to insert into a database, and another will be parsing out data to write to files (different regexes for each).
Now, the way this might also solve the failover issue is to make multi-process capability part of syslog-ng. So one might end up with a config like:
@version: 3.0 options { use_dns(no); log_iw_size(10000); };
source s_sys { unix-stream('/dev/log'); }; source s_net { tcp(ip(0.0.0.0) port(514) max-connections(1000)); udp(ip(0.0.0.0) port(514)); }; filter f_mail { facility(mail); }; process p_msgid { filter f_msgid { message('MsgID: (?<MESSAGEID>\S+)', type('pcre') flags('nobackref','store-matches')); }; destination d_oracle { sql(...); }; destination d_oracle_fallback { sql(...); }; log { filter(f_msgid); destination(d_oracle); }; log { filter(f_msgid); destination(d_oracle_fallback); flags(fallback); }; }; process p_foobar { options { flush_timeout(1000); } filter f_foobar { ... }; destination d_foobar { ... }; destination d_foobar_fallback { ... }; log { filter(f_foobar); destination(d_foobar); }; log { filter(f_foobar); destination(d_foobar_fallback); flags(fallback); }; };
log { source(s_sys); source(s_net); filter(f_mail); destination(p_msgid); destination(p_foobar); };
This will launch 3 processes, a master control process that does very basic filtering and uses some sort of IPC to send the data to 2 other processes which inherit the global options section and accept every config statement the master process does except sources. This way each process can run in parallel on the different CPU cores, and can have fallback destinations that wont interfere with the other processes.
Sent: Wednesday, March 17, 2010 7:54:42 AM From: Zoltán Pallagi <pzolee@balabit.hu> <mailto:pzolee@balabit.hu> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> <mailto:syslog-ng@lists.balabit.hu>, syslogng@feystorm.net <mailto:syslogng@feystorm.net> Subject: Re: [syslog-ng] log failback groups
Hi Patrick,
As far as I know, we would like to solve the failover/failback problem in syslog-ng PE v3.2 (and perhaps in OSE v3.2), but we are still working on it. However, we are planning to support it only in case of tcp/syslog destinations.
Patrick H. wrote:
How do you do groups of failbacks? For example
log { filter(f_filter1); destination(d_file1); }; log { filter(f_filter1); destination(d_file2); }; <-- log here only if the above fails log { filter(f_filter2); destination(d_sql1); }; <-- may include messages from the above 2 lines log { filter(f_filter2); destination(d_sql2); }; <-- log here only if d_sql1 fails
So, say lines 1 and 3 both fail, lines 2 and 3 should both start working. If I put a fallback flag on d_file2 and d_sql2, and d_sql1 fails, d_sql2 wont kick in on matches that d_file1 is still taking. It seems like there should be a way to do "log { destination(...) or destination(...); };", or "log {...} or log {...};".
------------------------------------------------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- pzolee
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
------------------------------------------------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Martin Holste <mcholste@gmail.com> wrote:
How many messages per second is the system attempting to handle? I'm very surprised that you're seeing that level of utilization. In our setup we've never had a problem pushing up through 30,000 messages per second written to disk with Syslog-NG in production, and I've pushed more than 70,000 per second in development.
Could you provide your configuration for these systems (including sysctls or kernel tunables etc.)? I've so far not been able to get my systems to accept and process (without any regex matching) more than approximagely 25K - 30K UDP messages/s. -Jan
Ok, so finally got time to look at this again. I tried the time_sleep option, and it was a horrible failure. With it enabled syslog-ng started losing about 98% of all incoming lines (and no, thats not an exaggeration). I'm guessing that time_sleep does not play well with udp as thats what incoming data is being sent over. However I do have the master-slave multi-process thing going and its working really good. I was able to put time_sleep on the child processes (the one doing regex matches), and it dropped their cpu utilization from around 40% to about 20% (master process uses tcp to talk to slave processes, so no drops). Another thing is that when I tried using the syslog protocol to talk to the child processes, the slaves were terminating the connection within seconds of being established. I poked and prodded and could not get this to work without constantly dropping the connection, so I had to switch back to plain tcp. Anyway, the attached config is what it looked like when I had all regexes run within a single process (the config that was utilizing over 90% cpu). Sent: Thursday, March 18, 2010 10:56:16 PM From: Jan Schaumann <jschauma@netmeister.org> To: syslog-ng@lists.balabit.hu Subject: Re: [syslog-ng] log failback groups
Martin Holste <mcholste@gmail.com> wrote:
How many messages per second is the system attempting to handle? I'm very surprised that you're seeing that level of utilization. In our setup we've never had a problem pushing up through 30,000 messages per second written to disk with Syslog-NG in production, and I've pushed more than 70,000 per second in development.
Could you provide your configuration for these systems (including sysctls or kernel tunables etc.)? I've so far not been able to get my systems to accept and process (without any regex matching) more than approximagely 25K - 30K UDP messages/s.
-Jan
------------------------------------------------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
@version: 3.0 # syslog-ng configuration file. # # This should behave pretty much like the original syslog on RedHat. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # options { time_reopen(10); use_dns(no); use_fqdn(no); keep_hostname(yes); create_dirs(yes); perm(0644); dir_perm(0755); log_iw_size(10000); log_fifo_size(20000); }; source s_sys { file("/proc/kmsg" program-override("kernel")); unix-stream ("/dev/log"); internal(); }; source s_net { tcp(ip(0.0.0.0) port(514) max-connections(1000) flags('syslog-protocol')); udp(ip(0.0.0.0) port(514)); }; destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog"); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_null { file("/dev/null" perm(0666)); }; destination d_syslog { file('/var/log/syslog'); }; filter f_kernel { facility(kern); }; filter f_default { level(info..emerg) and not (facility(mail) or facility(authpriv) or facility(cron) or facility(user)); }; filter f_auth { facility(authpriv); }; filter f_mail { facility(mail); }; filter f_emergency { level(emerg); }; filter f_news { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_boot { facility(local7); }; filter f_cron { facility(cron); }; filter f_user { facility(user); }; filter f_syslog { facility(syslog); }; log { source(s_sys); filter(f_kernel); destination(d_cons); }; log { source(s_sys); filter(f_default); destination(d_mesg); }; log { source(s_sys); filter(f_auth); destination(d_auth); }; log { source(s_sys); filter(f_mail); destination(d_mail); }; log { source(s_sys); filter(f_emergency); destination(d_cons); destination(d_mesg); }; log { source(s_sys); filter(f_news); destination(d_spol); }; log { source(s_sys); filter(f_boot); destination(d_boot); }; log { source(s_sys); filter(f_cron); destination(d_cron); }; log { source(s_sys); filter(f_user); destination(d_mesg); }; log { source(s_sys); filter(f_syslog); destination(d_syslog); }; # legacy logging format filter f_usa_app { #not level(notice) and program('^(?<PBASE>smtad|mtad|mrmad|bbqd|cbqd|mrad|scand)' flags('nobackref','store-matches') type('pcre')) and message('^(?<PEXT>\w{4}): (?<TID>\[\d+\]) (?<MSGTAIL>.+)$' flags('nobackref','store-matches') type('pcre')); }; template t_usa_app { template("$PID $TID $DATE $MSGTAIL\n"); }; destination d_usa_app { file("/var/log/hosts/$HOST/$PBASE/$PROGRAM.$MONTH$DAY.$PEXT" template(t_usa_app) flush_lines(10) flush_timeout(5000)); }; log { source(s_sys); source(s_net); filter(f_usa_app); destination(d_usa_app); flags('final'); }; # these are apps that get their own log directory & files filter f_apps { program('^(postfix)' flags('store-matches')) or program('^(amavis)' flags('store-matches')); }; destination d_apps { file("/var/log/hosts/$HOST/$1/$1.$LEVEL" flush_lines(10) flush_timeout(5000)); }; log { source(s_sys); source(s_net); filter(f_apps); destination(d_apps); flags('final'); }; # vim:ft=syslog-ng:ai:si:ts=4:sw=4:et:
On Fri, 2010-03-19 at 00:56 -0400, Jan Schaumann wrote:
Martin Holste <mcholste@gmail.com> wrote:
How many messages per second is the system attempting to handle? I'm very surprised that you're seeing that level of utilization. In our setup we've never had a problem pushing up through 30,000 messages per second written to disk with Syslog-NG in production, and I've pushed more than 70,000 per second in development.
Could you provide your configuration for these systems (including sysctls or kernel tunables etc.)? I've so far not been able to get my systems to accept and process (without any regex matching) more than approximagely 25K - 30K UDP messages/s.
udp() may be the problem here, in fact since syslog-ng is not thread based, its latency to poll for the udp socket may be a bit too long. increasing the udp socket buffer to insane values (like 256MB) could help alleviate the problem somewhat, but the latency is the root cause. adding time_sleep() to the mix increases latency even further. It only helps if you have a lot of connections (and udp is only one even if you have thousands of clients, while tcp keeps a separate connection for each client). After I get to the end of integrating the plugins branch, I intend to work on enabling multiple threads, thus decreasing the latency. -- Bazsi
That's definitely some heavy-duty regex. You'd be a good candidate for the pattern db, as the pattern matching engine is orders of magnitude faster than PCRE because it uses trie-based pattern searching. It also allows for extracting the matches and using them in the output macros, so you wouldn't have to sacrifice any functionality. I would estimate that it would drop your CPU usage down to around 25-30% while doing all of the work in a single thread. @Balabit: You know what goes great with pattern matching? CUDA support with Nvidia cards for GPU-based pattern matching acceleration. They've got preliminary support for it in the Open Information Security Foundation's (OISF) Suricata IDS engine. That project is GPL, so you could port most of that code directly into the pattern db matcher for the OSE version of SyslogNG. $500 USD will buy a GPU with 480 stream processors, so you could match 480 patterns simultaneously, per card. You can link up to four cards together, so you could match 1920 patterns in parallel, offloaded from the CPU, on commodity hardware. So, a server costing under $5,000 could probably process (maybe not store) 250,000+ messages per second. Even if there wasn't much speed increase, the CPU offload alone would probably be worth it for busy log servers. On Mon, Mar 22, 2010 at 11:50 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Fri, 2010-03-19 at 00:56 -0400, Jan Schaumann wrote:
Martin Holste <mcholste@gmail.com> wrote:
How many messages per second is the system attempting to handle? I'm very surprised that you're seeing that level of utilization. In our setup we've never had a problem pushing up through 30,000 messages per second written to disk with Syslog-NG in production, and I've pushed more than 70,000 per second in development.
Could you provide your configuration for these systems (including sysctls or kernel tunables etc.)? I've so far not been able to get my systems to accept and process (without any regex matching) more than approximagely 25K - 30K UDP messages/s.
udp() may be the problem here, in fact since syslog-ng is not thread based, its latency to poll for the udp socket may be a bit too long.
increasing the udp socket buffer to insane values (like 256MB) could help alleviate the problem somewhat, but the latency is the root cause.
adding time_sleep() to the mix increases latency even further. It only helps if you have a lot of connections (and udp is only one even if you have thousands of clients, while tcp keeps a separate connection for each client).
After I get to the end of integrating the plugins branch, I intend to work on enabling multiple threads, thus decreasing the latency.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Mon, 2010-03-22 at 12:54 -0500, Martin Holste wrote:
That's definitely some heavy-duty regex. You'd be a good candidate for the pattern db, as the pattern matching engine is orders of magnitude faster than PCRE because it uses trie-based pattern searching. It also allows for extracting the matches and using them in the output macros, so you wouldn't have to sacrifice any functionality. I would estimate that it would drop your CPU usage down to around 25-30% while doing all of the work in a single thread.
@Balabit: You know what goes great with pattern matching? CUDA support with Nvidia cards for GPU-based pattern matching acceleration. They've got preliminary support for it in the Open Information Security Foundation's (OISF) Suricata IDS engine. That project is GPL, so you could port most of that code directly into the pattern db matcher for the OSE version of SyslogNG. $500 USD will buy a GPU with 480 stream processors, so you could match 480 patterns simultaneously, per card. You can link up to four cards together, so you could match 1920 patterns in parallel, offloaded from the CPU, on commodity hardware. So, a server costing under $5,000 could probably process (maybe not store) 250,000+ messages per second. Even if there wasn't much speed increase, the CPU offload alone would probably be worth it for busy log servers.
Thanks for the hint, I've downloaded the docs, I'll definitely look into it. -- Bazsi
Thank you for sharing your idea with me, but I'm afraid, it is a little bit more complicated to do it well. If I understand your idea correctly, it can handle only the performance problem of the output side but doesn't solve the high traffic problem on input side (in this case, the master process will be overloaded and the other processes will be starved). Also, there are other problems relation in multiple syslog-ng processes/threads as well (e.g.: handling if more processes want to write to the same destination or storing the correct state of every processes and so on). Back to your performance problem: Will you send me your configuration? Perhaps, it's possible to optimize it a bit. Patrick H. wrote:
We turned on syslog-ng for our production environment today and ran into a problem which I think the solution might also solve this one. The problem I ran into is that with full production logging on, syslog-ng is using up about 50% of the CPU, and thats writing everything out to /dev/null (a few hundred machines). With a single regex turned on, it was using 90% CPU. So what I'm going to end up doing is running multiple syslog-ng processes so they can run in parallel on separate CPU cores. This way they can process data simultaneously. There will be a master process which basically sends everthing with the mail facility to two other syslog-ng processes, one of which will be parsing out data to insert into a database, and another will be parsing out data to write to files (different regexes for each).
Now, the way this might also solve the failover issue is to make multi-process capability part of syslog-ng. So one might end up with a config like:
@version: 3.0 options { use_dns(no); log_iw_size(10000); };
source s_sys { unix-stream('/dev/log'); }; source s_net { tcp(ip(0.0.0.0) port(514) max-connections(1000)); udp(ip(0.0.0.0) port(514)); }; filter f_mail { facility(mail); }; process p_msgid { filter f_msgid { message('MsgID: (?<MESSAGEID>\S+)', type('pcre') flags('nobackref','store-matches')); }; destination d_oracle { sql(...); }; destination d_oracle_fallback { sql(...); }; log { filter(f_msgid); destination(d_oracle); }; log { filter(f_msgid); destination(d_oracle_fallback); flags(fallback); }; }; process p_foobar { options { flush_timeout(1000); } filter f_foobar { ... }; destination d_foobar { ... }; destination d_foobar_fallback { ... }; log { filter(f_foobar); destination(d_foobar); }; log { filter(f_foobar); destination(d_foobar_fallback); flags(fallback); }; };
log { source(s_sys); source(s_net); filter(f_mail); destination(p_msgid); destination(p_foobar); };
This will launch 3 processes, a master control process that does very basic filtering and uses some sort of IPC to send the data to 2 other processes which inherit the global options section and accept every config statement the master process does except sources. This way each process can run in parallel on the different CPU cores, and can have fallback destinations that wont interfere with the other processes.
Sent: Wednesday, March 17, 2010 7:54:42 AM From: Zoltán Pallagi <pzolee@balabit.hu> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu>, syslogng@feystorm.net Subject: Re: [syslog-ng] log failback groups
Hi Patrick,
As far as I know, we would like to solve the failover/failback problem in syslog-ng PE v3.2 (and perhaps in OSE v3.2), but we are still working on it. However, we are planning to support it only in case of tcp/syslog destinations.
Patrick H. wrote:
How do you do groups of failbacks? For example
log { filter(f_filter1); destination(d_file1); }; log { filter(f_filter1); destination(d_file2); }; <-- log here only if the above fails log { filter(f_filter2); destination(d_sql1); }; <-- may include messages from the above 2 lines log { filter(f_filter2); destination(d_sql2); }; <-- log here only if d_sql1 fails
So, say lines 1 and 3 both fail, lines 2 and 3 should both start working. If I put a fallback flag on d_file2 and d_sql2, and d_sql1 fails, d_sql2 wont kick in on matches that d_file1 is still taking. It seems like there should be a way to do "log { destination(...) or destination(...); };", or "log {...} or log {...};".
------------------------------------------------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- pzolee
-- pzolee
On Thu, 2010-03-18 at 18:49 -0600, Patrick H. wrote:
@version: 3.0 options { use_dns(no); log_iw_size(10000); };
Try turn syslog-ng to "batch" mode by time_sleep. It can decrease the system log. Please see the doc: http://www.balabit.hu/dl/html/syslog-ng-ose-v3.1-guide-admin-en.html/ch05s02... Peter -- Höltzl Péter CISA, IT biztonsági tanácsadó holtzl.peter@balabit.hu +36 20 366 9667 BalaBit IT Security 1115 Budapest XI. Bártfai u. 54. Tel +36 1 371 0540 Fax +36 1 208 0875 Az üzenet és annak bármely csatolt anyaga bizalmas, jogi védelem alatt áll, a nyilvános közléstől védett. Az üzenetet kizárólag a címzett, illetve az általa meghatalmazottak használhatják fel. Ha Ön nem az üzenet címzettje, úgy kérjük, hogy telefonon, vagy e-mail-ben értesítse erről az üzenet küldőjét és törölje az üzenetet, valamint annak összes csatolt mellékletét a rendszeréből. Ha Ön nem az üzenet címzettje, abban az esetben tilos az üzenetet vagy annak bármely csatolt mellékletét lemásolnia, elmentenie, az üzenet tartalmát bárkivel közölnie vagy azzal visszaélnie.
participants (7)
-
Balazs Scheidler
-
Fekete Robert
-
HÖLTZL Péter
-
Jan Schaumann
-
Martin Holste
-
Patrick H.
-
Zoltán Pallagi