Multi-threaded in open source?
Hi everyone, I'm running an infrastructure with several hundred hosts logging many hundreds of Gb of logs to a central server. Syslog-ng on the central server is using 100% CPU on a single core, and I see here: http://bazsi.blogs.balabit.com/2011/07/syslog-ng-multithreaded-performance/ that there is some multi-threading option available! Is this only in the PE version or is there an OSE edition with this feature?
Yes, the 3.3 branch supports threading. Note though that 3.3 hasnt had a great track record so far (lots of bugs). However 3.3.3 might be stable enough for production use. -Patrick Sent: Fri Nov 18 2011 19:46:48 GMT-0700 (MST) From: Avleen Vig <avleen@gmail.com> To: syslog-ng@lists.balabit.hu Subject: [syslog-ng] Multi-threaded in open source?
Hi everyone,
I'm running an infrastructure with several hundred hosts logging many hundreds of Gb of logs to a central server. Syslog-ng on the central server is using 100% CPU on a single core, and I see here: http://bazsi.blogs.balabit.com/2011/07/syslog-ng-multithreaded-performance/ that there is some multi-threading option available!
Is this only in the PE version or is there an OSE edition with this feature? ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Fri, 2011-11-18 at 19:52 -0700, Patrick H. wrote:
Yes, the 3.3 branch supports threading. Note though that 3.3 hasnt had a great track record so far (lots of bugs). However 3.3.3 might be stable enough for production use.
The track record is true enough, however the positive feedback has started too. So at least give it a try, send feedback and fall back to older in case an issue happens. I only had a single 3.3.x fix ever since 3.3.3, which is more than a week old now :) -- Bazsi
Sent: Fri Nov 25 2011 05:04:01 GMT-0700 (MST) From: Balazs Scheidler <bazsi@balabit.hu> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Multi-threaded in open source?
On Fri, 2011-11-18 at 19:52 -0700, Patrick H. wrote:
Yes, the 3.3 branch supports threading. Note though that 3.3 hasnt had a great track record so far (lots of bugs). However 3.3.3 might be stable enough for production use. The track record is true enough, however the positive feedback has started too. So at least give it a try, send feedback and fall back to older in case an issue happens.
I only had a single 3.3.x fix ever since 3.3.3, which is more than a week old now :)
Actually I have found a bug in 3.3.3, I just havent had time to go track it down. I ran into a case where I had a filter that was setting a macro called 'FEXT' which I was using in a file destination for the filename extension. However it was acting as if the FEXT macro didnt exist at all (as filenames had no extension) and the message text was replaced with the contents of the FEXT macro. I downgraded syslog-ng to 3.2.4 with the exact same config (except with the threading config option removed) and it behaved as expected there.
On Fri, 2011-11-25 at 05:12 -0700, Patrick H. wrote:
Sent: Fri Nov 25 2011 05:04:01 GMT-0700 (MST) From: Balazs Scheidler <bazsi@balabit.hu> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Multi-threaded in open source?
On Fri, 2011-11-18 at 19:52 -0700, Patrick H. wrote:
Yes, the 3.3 branch supports threading. Note though that 3.3 hasnt had a great track record so far (lots of bugs). However 3.3.3 might be stable enough for production use. The track record is true enough, however the positive feedback has started too. So at least give it a try, send feedback and fall back to older in case an issue happens.
I only had a single 3.3.x fix ever since 3.3.3, which is more than a week old now :)
Actually I have found a bug in 3.3.3, I just havent had time to go track it down. I ran into a case where I had a filter that was setting a macro called 'FEXT' which I was using in a file destination for the filename extension. However it was acting as if the FEXT macro didnt exist at all (as filenames had no extension) and the message text was replaced with the contents of the FEXT macro. I downgraded syslog-ng to 3.2.4 with the exact same config (except with the threading config option removed) and it behaved as expected there.
I've just applied a patch for a rewrite specific issue: in case a rewrite rule was used from multiple locations, then some of the references fell back to using $MESSAGE instead of the value specified. Can you post your filter/rewrite rule that is not seemingly applied? Thanks. -- Bazsi
Sent: Fri Nov 25 2011 05:21:38 GMT-0700 (MST) From: Balazs Scheidler <bazsi@balabit.hu> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Multi-threaded in open source?
On Fri, 2011-11-25 at 05:12 -0700, Patrick H. wrote:
Sent: Fri Nov 25 2011 05:04:01 GMT-0700 (MST) From: Balazs Scheidler<bazsi@balabit.hu> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Multi-threaded in open source?
On Fri, 2011-11-18 at 19:52 -0700, Patrick H. wrote:
Yes, the 3.3 branch supports threading. Note though that 3.3 hasnt had a great track record so far (lots of bugs). However 3.3.3 might be stable enough for production use. The track record is true enough, however the positive feedback has started too. So at least give it a try, send feedback and fall back to older in case an issue happens.
I only had a single 3.3.x fix ever since 3.3.3, which is more than a week old now :)
Actually I have found a bug in 3.3.3, I just havent had time to go track it down. I ran into a case where I had a filter that was setting a macro called 'FEXT' which I was using in a file destination for the filename extension. However it was acting as if the FEXT macro didnt exist at all (as filenames had no extension) and the message text was replaced with the contents of the FEXT macro. I downgraded syslog-ng to 3.2.4 with the exact same config (except with the threading config option removed) and it behaved as expected there. I've just applied a patch for a rewrite specific issue: in case a rewrite rule was used from multiple locations, then some of the references fell back to using $MESSAGE instead of the value specified.
Can you post your filter/rewrite rule that is not seemingly applied?
Thanks. Youre good. I applied the logrewrite.c patch and the issue is gone. Thanks. I'll try not to break anything else for a while :-)
Patrick Hemmer <syslogng@feystorm.net> writes:
Youre good. I applied the logrewrite.c patch and the issue is gone. Thanks. I'll try not to break anything else for a while :-)
I'd like to ask you to do try and break stuff, so they can be fixed sooner than later. ;) -- |8]
participants (5)
-
Avleen Vig
-
Balazs Scheidler
-
Gergely Nagy
-
Patrick H.
-
Patrick Hemmer