memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347
Hi , I am getting the following error when starting syslog-ng.conf file which has more than 5000 lines in it and getting the following error: # service syslog-ng start memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347. syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng [FAILED] When I truncate the file to < 5000 lines it starts up fine. This is running version syslog-ng-3.0.8-1.rhel5.x86_64 Mailing list does not have any clues to it. Anything in the syslog-ng.conf file that needs to be done to accept more lines? Any ideas would be appreciated. Thanks # mukarram Mukarram Syed
You seriously have a config thats 5000+ lines??? May I ask why? -Patrick Sent: Mon Nov 08 2010 16:54:45 GMT-0700 (Mountain Standard Time) From: Mukarram Syed <msyed@walmart.com> To: 'syslog-ng@lists.balabit.hu' <syslog-ng@lists.balabit.hu> Subject: [syslog-ng] memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347
Hi ,
I am getting the following error when starting syslog-ng.conf file which has more than 5000 lines in it and getting the following error:
# service syslog-ng start
memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347.
syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng
[FAILED]
When I truncate the file to < 5000 lines it starts up fine.
This is running version syslog-ng-3.0.8-1.rhel5.x86_64
Mailing list does not have any clues to it.
Anything in the syslog-ng.conf file that needs to be done to accept more lines?
Any ideas would be appreciated.
Thanks
# mukarram
*Mukarram Syed*
------------------------------------------------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
I was migrating from syslog-ng-1.6 to a new version of syslog-ng in a veritas cluster. The original config file has >5000 lines and we want to keep it that way since it's Production. I got around it using syslog-ng-2.0 but 3.0 did not work due to this error. Thanks for the response. # mukarram From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Patrick H. Sent: Monday, November 08, 2010 4:31 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347 You seriously have a config thats 5000+ lines??? May I ask why? -Patrick Sent: Mon Nov 08 2010 16:54:45 GMT-0700 (Mountain Standard Time) From: Mukarram Syed <msyed@walmart.com><mailto:msyed@walmart.com> To: 'syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>' <syslog-ng@lists.balabit.hu><mailto:syslog-ng@lists.balabit.hu> Subject: [syslog-ng] memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347 Hi , I am getting the following error when starting syslog-ng.conf file which has more than 5000 lines in it and getting the following error: # service syslog-ng start memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347. syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng [FAILED] When I truncate the file to < 5000 lines it starts up fine. This is running version syslog-ng-3.0.8-1.rhel5.x86_64 Mailing list does not have any clues to it. Anything in the syslog-ng.conf file that needs to be done to accept more lines? Any ideas would be appreciated. Thanks # mukarram Mukarram Syed ________________________________ ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
I imagine a config that large has got to make syslog-ng really slow, and I cant imagine why it would need to be that large in the first place. Do you have any sort of ulimit in place controlling memory? (ulimit -m on most shells) Are you actually running out of memory (watch top while the process is starting up)? The only other case I can think that would cause this is an internal limit in the code that prevents it from using more than a certain amount, and that I have no clue about. -Patrick Sent: Mon Nov 08 2010 17:33:35 GMT-0700 (Mountain Standard Time) From: Mukarram Syed <msyed@walmart.com> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347
I was migrating from syslog-ng-1.6 to a new version of syslog-ng in a veritas cluster. The original config file has >5000 lines and we want to keep it that way since it's Production.
I got around it using syslog-ng-2.0 but 3.0 did not work due to this error.
Thanks for the response.
# mukarram
*From:* syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] *On Behalf Of *Patrick H. *Sent:* Monday, November 08, 2010 4:31 PM *To:* Syslog-ng users' and developers' mailing list *Subject:* Re: [syslog-ng] memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347
You seriously have a config thats 5000+ lines??? May I ask why?
-Patrick
Sent: Mon Nov 08 2010 16:54:45 GMT-0700 (Mountain Standard Time) From: Mukarram Syed <msyed@walmart.com> <mailto:msyed@walmart.com> To: 'syslog-ng@lists.balabit.hu <mailto:syslog-ng@lists.balabit.hu>' <syslog-ng@lists.balabit.hu> <mailto:syslog-ng@lists.balabit.hu> Subject: [syslog-ng] memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347
Hi ,
I am getting the following error when starting syslog-ng.conf file which has more than 5000 lines in it and getting the following error:
# service syslog-ng start
memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347.
syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng
[FAILED]
When I truncate the file to < 5000 lines it starts up fine.
This is running version syslog-ng-3.0.8-1.rhel5.x86_64
Mailing list does not have any clues to it.
Anything in the syslog-ng.conf file that needs to be done to accept more lines?
Any ideas would be appreciated.
Thanks
# mukarram
*Mukarram Syed*
------------------------------------------------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
------------------------------------------------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
I imagine a config that large has got to make syslog-ng really slow, and I cant imagine why it would need to be that large in the first place. -its pulling syslog info from about 1500 servers. Do you have any sort of ulimit in place controlling memory? (ulimit -m on most shells) # ulimit -m Unlimited Are you actually running out of memory (watch top while the process is starting up)? -I haven't tried that...it's a good idea though. The only other case I can think that would cause this is an internal limit in the code that prevents it from using more than a certain amount, and that I have no clue about. -possibly...but how come the older versions of syslog-ng work? Maybe a new addition to syslog-ng-3.0. Thanks # mukarram From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Patrick H. Sent: Monday, November 08, 2010 4:41 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347 I imagine a config that large has got to make syslog-ng really slow, and I cant imagine why it would need to be that large in the first place. Do you have any sort of ulimit in place controlling memory? (ulimit -m on most shells) Are you actually running out of memory (watch top while the process is starting up)? The only other case I can think that would cause this is an internal limit in the code that prevents it from using more than a certain amount, and that I have no clue about. -Patrick Sent: Mon Nov 08 2010 17:33:35 GMT-0700 (Mountain Standard Time) From: Mukarram Syed <msyed@walmart.com><mailto:msyed@walmart.com> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu><mailto:syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347 I was migrating from syslog-ng-1.6 to a new version of syslog-ng in a veritas cluster. The original config file has >5000 lines and we want to keep it that way since it's Production. I got around it using syslog-ng-2.0 but 3.0 did not work due to this error. Thanks for the response. # mukarram From: syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu> [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Patrick H. Sent: Monday, November 08, 2010 4:31 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347 You seriously have a config thats 5000+ lines??? May I ask why? -Patrick Sent: Mon Nov 08 2010 16:54:45 GMT-0700 (Mountain Standard Time) From: Mukarram Syed <msyed@walmart.com><mailto:msyed@walmart.com> To: 'syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>' <syslog-ng@lists.balabit.hu><mailto:syslog-ng@lists.balabit.hu> Subject: [syslog-ng] memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347 Hi , I am getting the following error when starting syslog-ng.conf file which has more than 5000 lines in it and getting the following error: # service syslog-ng start memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347. syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng [FAILED] When I truncate the file to < 5000 lines it starts up fine. This is running version syslog-ng-3.0.8-1.rhel5.x86_64 Mailing list does not have any clues to it. Anything in the syslog-ng.conf file that needs to be done to accept more lines? Any ideas would be appreciated. Thanks # mukarram Mukarram Syed ________________________________ ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html ________________________________ ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
The only other case I can think that would cause this is an internal limit in the code that prevents it from using more than a certain amount, and that I have no clue about. -possibly...but how come the older versions of syslog-ng work? Maybe a new addition to syslog-ng-3.0. A likely case is that whatever structures syslog-ng uses for parsing out config options grew in size. Like if the structure for a 'filter' config directive grew from 30 bytes to 36 bytes, and you have that in your config hundreds of times, then its making it use a lot more memory. But thats just a theory. -Patrick Sent: Mon Nov 08 2010 17:57:04 GMT-0700 (Mountain Standard Time) From: Mukarram Syed <msyed@walmart.com> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347
I imagine a config that large has got to make syslog-ng really slow, and I cant imagine why it would need to be that large in the first place. -its pulling syslog info from about 1500 servers.
Do you have any sort of ulimit in place controlling memory? (ulimit -m on most shells)
# ulimit -m
Unlimited
Are you actually running out of memory (watch top while the process is starting up)? -I haven't tried that...it's a good idea though.
The only other case I can think that would cause this is an internal limit in the code that prevents it from using more than a certain amount, and that I have no clue about.
-possibly...but how come the older versions of syslog-ng work? Maybe a new addition to syslog-ng-3.0.
Thanks
# mukarram
*From:* syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] *On Behalf Of *Patrick H. *Sent:* Monday, November 08, 2010 4:41 PM *To:* Syslog-ng users' and developers' mailing list *Subject:* Re: [syslog-ng] memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347
I imagine a config that large has got to make syslog-ng really slow, and I cant imagine why it would need to be that large in the first place. Do you have any sort of ulimit in place controlling memory? (ulimit -m on most shells) Are you actually running out of memory (watch top while the process is starting up)? The only other case I can think that would cause this is an internal limit in the code that prevents it from using more than a certain amount, and that I have no clue about.
-Patrick
Sent: Mon Nov 08 2010 17:33:35 GMT-0700 (Mountain Standard Time) From: Mukarram Syed <msyed@walmart.com> <mailto:msyed@walmart.com> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> <mailto:syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347
I was migrating from syslog-ng-1.6 to a new version of syslog-ng in a veritas cluster. The original config file has >5000 lines and we want to keep it that way since it's Production.
I got around it using syslog-ng-2.0 but 3.0 did not work due to this error.
Thanks for the response.
# mukarram
*From:* syslog-ng-bounces@lists.balabit.hu <mailto:syslog-ng-bounces@lists.balabit.hu> [mailto:syslog-ng-bounces@lists.balabit.hu] *On Behalf Of *Patrick H. *Sent:* Monday, November 08, 2010 4:31 PM *To:* Syslog-ng users' and developers' mailing list *Subject:* Re: [syslog-ng] memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347
You seriously have a config thats 5000+ lines??? May I ask why?
-Patrick
Sent: Mon Nov 08 2010 16:54:45 GMT-0700 (Mountain Standard Time) From: Mukarram Syed <msyed@walmart.com> <mailto:msyed@walmart.com> To: 'syslog-ng@lists.balabit.hu <mailto:syslog-ng@lists.balabit.hu>' <syslog-ng@lists.balabit.hu> <mailto:syslog-ng@lists.balabit.hu> Subject: [syslog-ng] memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347
Hi ,
I am getting the following error when starting syslog-ng.conf file which has more than 5000 lines in it and getting the following error:
# service syslog-ng start
memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347.
syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng
[FAILED]
When I truncate the file to < 5000 lines it starts up fine.
This is running version syslog-ng-3.0.8-1.rhel5.x86_64
Mailing list does not have any clues to it.
Anything in the syslog-ng.conf file that needs to be done to accept more lines?
Any ideas would be appreciated.
Thanks
# mukarram
*Mukarram Syed*
------------------------------------------------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
------------------------------------------------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
------------------------------------------------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
One for every store? :) On Mon, 2010-11-08 at 17:30 -0700, Patrick H. wrote:
You seriously have a config thats 5000+ lines??? May I ask why?
-Patrick
Sent: Mon Nov 08 2010 16:54:45 GMT-0700 (Mountain Standard Time) From: Mukarram Syed <msyed@walmart.com> To: 'syslog-ng@lists.balabit.hu' <syslog-ng@lists.balabit.hu> Subject: [syslog-ng] memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347
Hi ,
I am getting the following error when starting syslog-ng.conf file which has more than 5000 lines in it and getting the following error:
# service syslog-ng start
memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347.
syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng
[FAILED]
When I truncate the file to < 5000 lines it starts up fine.
This is running version syslog-ng-3.0.8-1.rhel5.x86_64
Mailing list does not have any clues to it.
Anything in the syslog-ng.conf file that needs to be done to accept more lines?
Any ideas would be appreciated.
Thanks
# mukarram
Mukarram Syed
____________________________________________________________________
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Chris, One for every server...is there a better way? # mukarram -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Christopher Barry Sent: Monday, November 08, 2010 5:46 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347 One for every store? :) On Mon, 2010-11-08 at 17:30 -0700, Patrick H. wrote:
You seriously have a config thats 5000+ lines??? May I ask why?
-Patrick
Sent: Mon Nov 08 2010 16:54:45 GMT-0700 (Mountain Standard Time) From: Mukarram Syed <msyed@walmart.com> To: 'syslog-ng@lists.balabit.hu' <syslog-ng@lists.balabit.hu> Subject: [syslog-ng] memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347
Hi ,
I am getting the following error when starting syslog-ng.conf file which has more than 5000 lines in it and getting the following error:
# service syslog-ng start
memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347.
syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng
[FAILED]
When I truncate the file to < 5000 lines it starts up fine.
This is running version syslog-ng-3.0.8-1.rhel5.x86_64
Mailing list does not have any clues to it.
Anything in the syslog-ng.conf file that needs to be done to accept more lines?
Any ideas would be appreciated.
Thanks
# mukarram
Mukarram Syed
____________________________________________________________________
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Hi, On Tue, Nov 9, 2010 at 12:54 AM, Mukarram Syed <msyed@walmart.com> wrote:
Hi ,
I am getting the following error when starting syslog-ng.conf file which has more than 5000 lines in it and getting the following error:
# service syslog-ng start
memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347.
See https://lists.balabit.hu/pipermail/syslog-ng/2009-January/012342.html https://lists.balabit.hu/pipermail/syslog-ng/2009-December/013744.html hth, Sandor
On Mon, 2010-11-08 at 15:54 -0800, Mukarram Syed wrote:
Hi ,
I am getting the following error when starting syslog-ng.conf file which has more than 5000 lines in it and getting the following error:
# service syslog-ng start
memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347.
This is a lex/yacc error message. Since flex/bison (the lex/yacc implementation syslog-ng is using by default) is quite dynamic, I'm guessing you are using traditional lex and/or yacc. Since the release tarballs I'm pushing out has the flex/bison code, I guess you are regenerating those during compilation. Make sure that you have bison/flex in your compile environment and make sure that cfg-grammar.c and cfg-lex.c are regenerated using the proper tools. (or make sure that your build env doesn't overwrite the ones shipped in the tarball). Although your config seems quite large, syslog-ng should be able to handle them just fine. -- Bazsi
Thanks much for this response. I am installing syslog-ng 3 via the rpm for RHEL/CentOS...so you recommend the installation using the tarball. Regards # mukarram -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Sunday, November 14, 2010 2:04 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347 On Mon, 2010-11-08 at 15:54 -0800, Mukarram Syed wrote:
Hi ,
I am getting the following error when starting syslog-ng.conf file which has more than 5000 lines in it and getting the following error:
# service syslog-ng start
memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347.
This is a lex/yacc error message. Since flex/bison (the lex/yacc implementation syslog-ng is using by default) is quite dynamic, I'm guessing you are using traditional lex and/or yacc. Since the release tarballs I'm pushing out has the flex/bison code, I guess you are regenerating those during compilation. Make sure that you have bison/flex in your compile environment and make sure that cfg-grammar.c and cfg-lex.c are regenerated using the proper tools. (or make sure that your build env doesn't overwrite the ones shipped in the tarball). Although your config seems quite large, syslog-ng should be able to handle them just fine. -- Bazsi ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Mon, 2010-11-15 at 11:15 -0800, Mukarram Syed wrote:
Thanks much for this response. I am installing syslog-ng 3 via the rpm for RHEL/CentOS...so you recommend the installation using the tarball. Regards
you mean, you downloaded them from our website? -- Bazsi
Yes. # mukarram -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Monday, November 22, 2010 8:34 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347 On Mon, 2010-11-15 at 11:15 -0800, Mukarram Syed wrote:
Thanks much for this response. I am installing syslog-ng 3 via the rpm for RHEL/CentOS...so you recommend the installation using the tarball. Regards
you mean, you downloaded them from our website? -- Bazsi ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
I've now looked into this. Although the generated parser allocates the stack dynamically, it seems there's a builtin limit of 10k elements by default. This can be changed by adding the following line to cfg-grammar.y: (somewhere where other defines are listed): #define YYMAXDEPTH 20000 Assuming that 20k would be enough for your purpose. I really don't see why your configuration file needs so much space, but anyway, this is the way to solve this problem. After changing .y file you need to recompile (you'll also need bison, since you'd be changing the .y file). On Mon, 2010-11-22 at 12:50 -0800, Mukarram Syed wrote:
Yes.
# mukarram
-----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Monday, November 22, 2010 8:34 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347
On Mon, 2010-11-15 at 11:15 -0800, Mukarram Syed wrote:
Thanks much for this response. I am installing syslog-ng 3 via the rpm for RHEL/CentOS...so you recommend the installation using the tarball. Regards
you mean, you downloaded them from our website?
-- Bazsi
Hi, I have syslog-ng version syslog-ng-2.1.4-1.el5 running on RHEL5. When I run the top, syslog-ng process runs on only one CPU, but this is a 24 Core server. How do I make syslog-ng share the load across several CPUs or cores rather than have it stuck in one CPU? Any suggestions are appreciated. TOP output for syslog-ng: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 11304 root 18 0 2023m 1.9g 1252 R 95.7 2.7 22658:29 syslog-ng MPSTAT output: 01:16:53 PM CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s 01:16:54 PM all 6.37 0.00 1.38 0.00 0.00 0.17 0.00 92.08 5073.74 01:16:54 PM 0 90.91 0.00 4.04 0.00 1.01 3.03 0.00 1.01 4942.42 - my guess is this is using syslog-ng 01:16:54 PM 1 0.00 0.00 1.00 0.00 0.00 0.00 0.00 99.00 0.00 01:16:54 PM 2 14.00 0.00 3.00 0.00 0.00 0.00 0.00 83.00 0.00 01:16:54 PM 3 0.00 0.00 1.00 0.00 0.00 0.00 0.00 99.00 0.00 01:16:54 PM 4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 01:16:54 PM 5 12.00 0.00 6.00 0.00 0.00 0.00 0.00 82.00 0.00 01:16:54 PM 6 11.11 0.00 4.04 0.00 0.00 0.00 0.00 84.85 0.00 01:16:54 PM 7 0.00 0.00 1.01 0.00 0.00 0.00 0.00 98.99 0.00 01:16:54 PM 8 0.00 0.00 1.00 0.00 0.00 0.00 0.00 99.00 0.00 01:16:54 PM 9 11.11 0.00 2.02 0.00 0.00 0.00 0.00 86.87 0.00 01:16:54 PM 10 1.00 0.00 6.00 1.00 0.00 1.00 0.00 91.00 122.22 01:16:54 PM 11 0.00 0.00 1.01 0.00 0.00 0.00 0.00 98.99 0.00 01:16:54 PM 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 01:16:54 PM 13 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 01:16:54 PM 14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 01:16:54 PM 15 0.00 0.00 1.01 0.00 0.00 0.00 0.00 98.99 0.00 01:16:54 PM 16 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 01:16:54 PM 17 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 01:16:54 PM 18 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 01:16:54 PM 19 10.20 0.00 1.02 0.00 0.00 0.00 0.00 88.78 0.00 01:16:54 PM 20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 5.05 01:16:54 PM 21 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 01:16:54 PM 22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 01:16:54 PM 23 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 5.05 Thanks # mukarram Mukarram Syed
I (and others) got around this problem by launching multiple syslog-ng processes and having a master process do very basic filtering and routing to the child processes. I think I remember hearing that syslog-ng 3.2 was going to use threads (at least for destination drivers) but dont quote me on that. -Patrick Sent: Wed Dec 01 2010 14:57:28 GMT-0700 (Mountain Standard Time) From: Mukarram Syed <msyed@walmart.com> To: 'syslog-ng@lists.balabit.hu' <syslog-ng@lists.balabit.hu> Subject: [syslog-ng] syslog-ng stuck on one CPU but need to share it with multiple CPUs.
Hi,
I have syslog-ng version syslog-ng-2.1.4-1.el5 running on RHEL5.
When I run the top, syslog-ng process runs on only one CPU, but this is a 24 Core server.
How do I make syslog-ng share the load across several CPUs or cores rather than have it stuck in one CPU?
Any suggestions are appreciated.
* *
*TOP output for syslog-ng:*
* PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND *
*11304 root 18 0 2023m 1.9g 1252 R 95.7 2.7 22658:29 syslog-ng *
*MPSTAT output:*
01:16:53 PM CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s
01:16:54 PM all 6.37 0.00 1.38 0.00 0.00 0.17 0.00 92.08 5073.74
*01:16:54 PM 0 90.91 0.00 4.04 0.00 1.01 3.03 0.00 1.01 4942.42 -- **my guess is this is using syslog-ng*
01:16:54 PM 1 0.00 0.00 1.00 0.00 0.00 0.00 0.00 99.00 0.00
01:16:54 PM 2 14.00 0.00 3.00 0.00 0.00 0.00 0.00 83.00 0.00
01:16:54 PM 3 0.00 0.00 1.00 0.00 0.00 0.00 0.00 99.00 0.00
01:16:54 PM 4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00
01:16:54 PM 5 12.00 0.00 6.00 0.00 0.00 0.00 0.00 82.00 0.00
01:16:54 PM 6 11.11 0.00 4.04 0.00 0.00 0.00 0.00 84.85 0.00
01:16:54 PM 7 0.00 0.00 1.01 0.00 0.00 0.00 0.00 98.99 0.00
01:16:54 PM 8 0.00 0.00 1.00 0.00 0.00 0.00 0.00 99.00 0.00
01:16:54 PM 9 11.11 0.00 2.02 0.00 0.00 0.00 0.00 86.87 0.00
01:16:54 PM 10 1.00 0.00 6.00 1.00 0.00 1.00 0.00 91.00 122.22
01:16:54 PM 11 0.00 0.00 1.01 0.00 0.00 0.00 0.00 98.99 0.00
01:16:54 PM 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00
01:16:54 PM 13 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00
01:16:54 PM 14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00
01:16:54 PM 15 0.00 0.00 1.01 0.00 0.00 0.00 0.00 98.99 0.00
01:16:54 PM 16 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00
01:16:54 PM 17 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00
01:16:54 PM 18 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00
01:16:54 PM 19 10.20 0.00 1.02 0.00 0.00 0.00 0.00 88.78 0.00
01:16:54 PM 20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 5.05
01:16:54 PM 21 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00
01:16:54 PM 22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00
01:16:54 PM 23 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 5.05
Thanks
# mukarram
*Mukarram Syed*
------------------------------------------------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Thanks Patrick. Any leads as to how you would do this? Thanks # mukarram From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Patrick H. Sent: Wednesday, December 01, 2010 2:32 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng stuck on one CPU but need to share it with multiple CPUs. I (and others) got around this problem by launching multiple syslog-ng processes and having a master process do very basic filtering and routing to the child processes. I think I remember hearing that syslog-ng 3.2 was going to use threads (at least for destination drivers) but dont quote me on that. -Patrick Sent: Wed Dec 01 2010 14:57:28 GMT-0700 (Mountain Standard Time) From: Mukarram Syed <msyed@walmart.com><mailto:msyed@walmart.com> To: 'syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>' <syslog-ng@lists.balabit.hu><mailto:syslog-ng@lists.balabit.hu> Subject: [syslog-ng] syslog-ng stuck on one CPU but need to share it with multiple CPUs. Hi, I have syslog-ng version syslog-ng-2.1.4-1.el5 running on RHEL5. When I run the top, syslog-ng process runs on only one CPU, but this is a 24 Core server. How do I make syslog-ng share the load across several CPUs or cores rather than have it stuck in one CPU? Any suggestions are appreciated. TOP output for syslog-ng: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 11304 root 18 0 2023m 1.9g 1252 R 95.7 2.7 22658:29 syslog-ng MPSTAT output: 01:16:53 PM CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s 01:16:54 PM all 6.37 0.00 1.38 0.00 0.00 0.17 0.00 92.08 5073.74 01:16:54 PM 0 90.91 0.00 4.04 0.00 1.01 3.03 0.00 1.01 4942.42 - my guess is this is using syslog-ng 01:16:54 PM 1 0.00 0.00 1.00 0.00 0.00 0.00 0.00 99.00 0.00 01:16:54 PM 2 14.00 0.00 3.00 0.00 0.00 0.00 0.00 83.00 0.00 01:16:54 PM 3 0.00 0.00 1.00 0.00 0.00 0.00 0.00 99.00 0.00 01:16:54 PM 4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 01:16:54 PM 5 12.00 0.00 6.00 0.00 0.00 0.00 0.00 82.00 0.00 01:16:54 PM 6 11.11 0.00 4.04 0.00 0.00 0.00 0.00 84.85 0.00 01:16:54 PM 7 0.00 0.00 1.01 0.00 0.00 0.00 0.00 98.99 0.00 01:16:54 PM 8 0.00 0.00 1.00 0.00 0.00 0.00 0.00 99.00 0.00 01:16:54 PM 9 11.11 0.00 2.02 0.00 0.00 0.00 0.00 86.87 0.00 01:16:54 PM 10 1.00 0.00 6.00 1.00 0.00 1.00 0.00 91.00 122.22 01:16:54 PM 11 0.00 0.00 1.01 0.00 0.00 0.00 0.00 98.99 0.00 01:16:54 PM 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 01:16:54 PM 13 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 01:16:54 PM 14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 01:16:54 PM 15 0.00 0.00 1.01 0.00 0.00 0.00 0.00 98.99 0.00 01:16:54 PM 16 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 01:16:54 PM 17 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 01:16:54 PM 18 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 01:16:54 PM 19 10.20 0.00 1.02 0.00 0.00 0.00 0.00 88.78 0.00 01:16:54 PM 20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 5.05 01:16:54 PM 21 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 01:16:54 PM 22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 01:16:54 PM 23 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 5.05 Thanks # mukarram Mukarram Syed ________________________________ ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
The way I do it is with separate init scripts and separate configs. The scripts launch the additional processes with args like '-f /etc/syslog-ng/syslog-ng.515.conf -R /var/lib/syslog-ng/syslog-ng.515.persist -p /var/lib/syslog-ng/syslog-ng.515.pid'. That config then has a tcp source driver listening on port 515, and the master process just sends traffic to localhost:515. You could use pipes or whatever transport you want, I just chose TCP. -Patrick Sent: Wed Dec 01 2010 15:35:36 GMT-0700 (Mountain Standard Time) From: Mukarram Syed <msyed@walmart.com> To: 'Syslog-ng users' and developers' mailing list' <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng stuck on one CPU but need to share it with multiple CPUs.
Thanks Patrick.
Any leads as to how you would do this?
Thanks
# mukarram
*From:* syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] *On Behalf Of *Patrick H. *Sent:* Wednesday, December 01, 2010 2:32 PM *To:* Syslog-ng users' and developers' mailing list *Subject:* Re: [syslog-ng] syslog-ng stuck on one CPU but need to share it with multiple CPUs.
I (and others) got around this problem by launching multiple syslog-ng processes and having a master process do very basic filtering and routing to the child processes. I think I remember hearing that syslog-ng 3.2 was going to use threads (at least for destination drivers) but dont quote me on that.
-Patrick
Sent: Wed Dec 01 2010 14:57:28 GMT-0700 (Mountain Standard Time) From: Mukarram Syed <msyed@walmart.com> <mailto:msyed@walmart.com> To: 'syslog-ng@lists.balabit.hu <mailto:syslog-ng@lists.balabit.hu>' <syslog-ng@lists.balabit.hu> <mailto:syslog-ng@lists.balabit.hu> Subject: [syslog-ng] syslog-ng stuck on one CPU but need to share it with multiple CPUs.
Hi,
I have syslog-ng version syslog-ng-2.1.4-1.el5 running on RHEL5.
When I run the top, syslog-ng process runs on only one CPU, but this is a 24 Core server.
How do I make syslog-ng share the load across several CPUs or cores rather than have it stuck in one CPU?
Any suggestions are appreciated.
* *
*TOP output for syslog-ng:*
* PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND *
*11304 root 18 0 2023m 1.9g 1252 R 95.7 2.7 22658:29 syslog-ng *
*MPSTAT output:*
01:16:53 PM CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s
01:16:54 PM all 6.37 0.00 1.38 0.00 0.00 0.17 0.00 92.08 5073.74
*01:16:54 PM 0 90.91 0.00 4.04 0.00 1.01 3.03 0.00 1.01 4942.42 -- **my guess is this is using syslog-ng*
01:16:54 PM 1 0.00 0.00 1.00 0.00 0.00 0.00 0.00 99.00 0.00
01:16:54 PM 2 14.00 0.00 3.00 0.00 0.00 0.00 0.00 83.00 0.00
01:16:54 PM 3 0.00 0.00 1.00 0.00 0.00 0.00 0.00 99.00 0.00
01:16:54 PM 4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00
01:16:54 PM 5 12.00 0.00 6.00 0.00 0.00 0.00 0.00 82.00 0.00
01:16:54 PM 6 11.11 0.00 4.04 0.00 0.00 0.00 0.00 84.85 0.00
01:16:54 PM 7 0.00 0.00 1.01 0.00 0.00 0.00 0.00 98.99 0.00
01:16:54 PM 8 0.00 0.00 1.00 0.00 0.00 0.00 0.00 99.00 0.00
01:16:54 PM 9 11.11 0.00 2.02 0.00 0.00 0.00 0.00 86.87 0.00
01:16:54 PM 10 1.00 0.00 6.00 1.00 0.00 1.00 0.00 91.00 122.22
01:16:54 PM 11 0.00 0.00 1.01 0.00 0.00 0.00 0.00 98.99 0.00
01:16:54 PM 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00
01:16:54 PM 13 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00
01:16:54 PM 14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00
01:16:54 PM 15 0.00 0.00 1.01 0.00 0.00 0.00 0.00 98.99 0.00
01:16:54 PM 16 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00
01:16:54 PM 17 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00
01:16:54 PM 18 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00
01:16:54 PM 19 10.20 0.00 1.02 0.00 0.00 0.00 0.00 88.78 0.00
01:16:54 PM 20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 5.05
01:16:54 PM 21 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00
01:16:54 PM 22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00
01:16:54 PM 23 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 5.05
Thanks
# mukarram
*Mukarram Syed*
------------------------------------------------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
------------------------------------------------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Interesting...I think I get the idea. That way you could split the syslog-ng.conf file into multiple files, running multiple syslog-ng processes and not run into the 5000 line limit problem in the syslog-ng.conf that I faced a few months ago. Thanks # mukarram From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Patrick H. Sent: Wednesday, December 01, 2010 3:00 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng stuck on one CPU but need to share it with multiple CPUs. The way I do it is with separate init scripts and separate configs. The scripts launch the additional processes with args like '-f /etc/syslog-ng/syslog-ng.515.conf -R /var/lib/syslog-ng/syslog-ng.515.persist -p /var/lib/syslog-ng/syslog-ng.515.pid'. That config then has a tcp source driver listening on port 515, and the master process just sends traffic to localhost:515. You could use pipes or whatever transport you want, I just chose TCP. -Patrick Sent: Wed Dec 01 2010 15:35:36 GMT-0700 (Mountain Standard Time) From: Mukarram Syed <msyed@walmart.com><mailto:msyed@walmart.com> To: 'Syslog-ng users' and developers' mailing list' <syslog-ng@lists.balabit.hu><mailto:syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng stuck on one CPU but need to share it with multiple CPUs. Thanks Patrick. Any leads as to how you would do this? Thanks # mukarram From: syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu> [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Patrick H. Sent: Wednesday, December 01, 2010 2:32 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng stuck on one CPU but need to share it with multiple CPUs. I (and others) got around this problem by launching multiple syslog-ng processes and having a master process do very basic filtering and routing to the child processes. I think I remember hearing that syslog-ng 3.2 was going to use threads (at least for destination drivers) but dont quote me on that. -Patrick Sent: Wed Dec 01 2010 14:57:28 GMT-0700 (Mountain Standard Time) From: Mukarram Syed <msyed@walmart.com><mailto:msyed@walmart.com> To: 'syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>' <syslog-ng@lists.balabit.hu><mailto:syslog-ng@lists.balabit.hu> Subject: [syslog-ng] syslog-ng stuck on one CPU but need to share it with multiple CPUs. Hi, I have syslog-ng version syslog-ng-2.1.4-1.el5 running on RHEL5. When I run the top, syslog-ng process runs on only one CPU, but this is a 24 Core server. How do I make syslog-ng share the load across several CPUs or cores rather than have it stuck in one CPU? Any suggestions are appreciated. TOP output for syslog-ng: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 11304 root 18 0 2023m 1.9g 1252 R 95.7 2.7 22658:29 syslog-ng MPSTAT output: 01:16:53 PM CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s 01:16:54 PM all 6.37 0.00 1.38 0.00 0.00 0.17 0.00 92.08 5073.74 01:16:54 PM 0 90.91 0.00 4.04 0.00 1.01 3.03 0.00 1.01 4942.42 - my guess is this is using syslog-ng 01:16:54 PM 1 0.00 0.00 1.00 0.00 0.00 0.00 0.00 99.00 0.00 01:16:54 PM 2 14.00 0.00 3.00 0.00 0.00 0.00 0.00 83.00 0.00 01:16:54 PM 3 0.00 0.00 1.00 0.00 0.00 0.00 0.00 99.00 0.00 01:16:54 PM 4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 01:16:54 PM 5 12.00 0.00 6.00 0.00 0.00 0.00 0.00 82.00 0.00 01:16:54 PM 6 11.11 0.00 4.04 0.00 0.00 0.00 0.00 84.85 0.00 01:16:54 PM 7 0.00 0.00 1.01 0.00 0.00 0.00 0.00 98.99 0.00 01:16:54 PM 8 0.00 0.00 1.00 0.00 0.00 0.00 0.00 99.00 0.00 01:16:54 PM 9 11.11 0.00 2.02 0.00 0.00 0.00 0.00 86.87 0.00 01:16:54 PM 10 1.00 0.00 6.00 1.00 0.00 1.00 0.00 91.00 122.22 01:16:54 PM 11 0.00 0.00 1.01 0.00 0.00 0.00 0.00 98.99 0.00 01:16:54 PM 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 01:16:54 PM 13 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 01:16:54 PM 14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 01:16:54 PM 15 0.00 0.00 1.01 0.00 0.00 0.00 0.00 98.99 0.00 01:16:54 PM 16 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 01:16:54 PM 17 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 01:16:54 PM 18 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 01:16:54 PM 19 10.20 0.00 1.02 0.00 0.00 0.00 0.00 88.78 0.00 01:16:54 PM 20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 5.05 01:16:54 PM 21 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 01:16:54 PM 22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 01:16:54 PM 23 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 5.05 Thanks # mukarram Mukarram Syed ________________________________ ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html ________________________________ ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Wed, 2010-12-01 at 15:31 -0700, Patrick H. wrote:
I (and others) got around this problem by launching multiple syslog-ng processes and having a master process do very basic filtering and routing to the child processes. I think I remember hearing that syslog-ng 3.2 was going to use threads (at least for destination drivers) but dont quote me on that.
It is actually syslog-ng 3.3 that's going to use threads. The code which I've completed so far is a work-in-progress, published at: http://git.balabit.hu/?p=bazsi/syslog-ng-3.3.git;a=shortlog;h=wip/epoll Work-in-progress means that I'm rebasing and restructuring the code quote often. Since this was mentioned, I thought I'd mention some numbers how it goes. * I was measuring everything on my development laptop (Core2 Duo, 2.2Ghz, 4GB RAM) * I was running a full GNOME desktop, running the tests in a gnome-terminal, I wasn't doing anything, but some jitter can be associated to background tasks. * I was running 10 second bursts, with each message containing 256 bytes of data (loggen defaults) * I was using the tcp() source * I was playing with various tuning parameters (window-size, fetch-limit, flush_lines, etc. but I didn't really record their value). * loggen was running on the same box as syslog-ng itself. With these in mind: 1) single connection, trying to send messages as fast as possible syslog-ng 3.2: 58000 msg/sec syslog-ng 3.3: 180000 msg/sec 2) also tested how it scales with 1000 clients syslog-ng 3.2 degraded with about 25% if 1 out of 1000 clients were sending messages syslog-ng 3.3 degrated with about 7% if 1 out of 1000 clients were sending messages (so using epoll instead of GLib's main loop was definitely worth the effort) Of course my development machine has only two cores, and with more cores and with more clients, it'll probably reach higher msg/sec. I would estimate that with real server hardware, it'll scale into the 800k-1000k msg/sec range assuming the disk isn't the bottleneck. (that's 200-250MB/sec, which is not impossible with a proper disk array) Anyway, the code isn't ready yet and even crashes (at exit for example), so definitely not production quality yet :) -- Bazsi
Thank You for the update. -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Saturday, December 04, 2010 5:19 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng stuck on one CPU but need to share it with multiple CPUs. On Wed, 2010-12-01 at 15:31 -0700, Patrick H. wrote:
I (and others) got around this problem by launching multiple syslog-ng processes and having a master process do very basic filtering and routing to the child processes. I think I remember hearing that syslog-ng 3.2 was going to use threads (at least for destination drivers) but dont quote me on that.
It is actually syslog-ng 3.3 that's going to use threads. The code which I've completed so far is a work-in-progress, published at: http://git.balabit.hu/?p=bazsi/syslog-ng-3.3.git;a=shortlog;h=wip/epoll Work-in-progress means that I'm rebasing and restructuring the code quote often. Since this was mentioned, I thought I'd mention some numbers how it goes. * I was measuring everything on my development laptop (Core2 Duo, 2.2Ghz, 4GB RAM) * I was running a full GNOME desktop, running the tests in a gnome-terminal, I wasn't doing anything, but some jitter can be associated to background tasks. * I was running 10 second bursts, with each message containing 256 bytes of data (loggen defaults) * I was using the tcp() source * I was playing with various tuning parameters (window-size, fetch-limit, flush_lines, etc. but I didn't really record their value). * loggen was running on the same box as syslog-ng itself. With these in mind: 1) single connection, trying to send messages as fast as possible syslog-ng 3.2: 58000 msg/sec syslog-ng 3.3: 180000 msg/sec 2) also tested how it scales with 1000 clients syslog-ng 3.2 degraded with about 25% if 1 out of 1000 clients were sending messages syslog-ng 3.3 degrated with about 7% if 1 out of 1000 clients were sending messages (so using epoll instead of GLib's main loop was definitely worth the effort) Of course my development machine has only two cores, and with more cores and with more clients, it'll probably reach higher msg/sec. I would estimate that with real server hardware, it'll scale into the 800k-1000k msg/sec range assuming the disk isn't the bottleneck. (that's 200-250MB/sec, which is not impossible with a proper disk array) Anyway, the code isn't ready yet and even crashes (at exit for example), so definitely not production quality yet :) -- Bazsi ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Also, I hope syslog-ng 3.3 will fix the issue with my syslog-ng.conf having greater than 5000 lines. Syslog-ng 3.0.8 chokes on this (error: memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347). I had to revert back to syslog-ng-2.1.4 which works fine. ...or do I have to still do this which you suggested... From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Friday, November 26, 2010 6:58 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347 I've now looked into this. Although the generated parser allocates the stack dynamically, it seems there's a builtin limit of 10k elements by default. This can be changed by adding the following line to cfg-grammar.y: (somewhere where other defines are listed): #define YYMAXDEPTH 20000 Assuming that 20k would be enough for your purpose. I really don't see why your configuration file needs so much space, but anyway, this is the way to solve this problem. After changing .y file you need to recompile (you'll also need bison, since you'd be changing the .y file). Thanks # mukarram -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Mukarram Syed Sent: Monday, December 06, 2010 6:18 AM To: 'Syslog-ng users' and developers' mailing list' Subject: Re: [syslog-ng] syslog-ng stuck on one CPU but need to share it with multiple CPUs. Thank You for the update. -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Saturday, December 04, 2010 5:19 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng stuck on one CPU but need to share it with multiple CPUs. On Wed, 2010-12-01 at 15:31 -0700, Patrick H. wrote:
I (and others) got around this problem by launching multiple syslog-ng processes and having a master process do very basic filtering and routing to the child processes. I think I remember hearing that syslog-ng 3.2 was going to use threads (at least for destination drivers) but dont quote me on that.
It is actually syslog-ng 3.3 that's going to use threads. The code which I've completed so far is a work-in-progress, published at: http://git.balabit.hu/?p=bazsi/syslog-ng-3.3.git;a=shortlog;h=wip/epoll Work-in-progress means that I'm rebasing and restructuring the code quote often. Since this was mentioned, I thought I'd mention some numbers how it goes. * I was measuring everything on my development laptop (Core2 Duo, 2.2Ghz, 4GB RAM) * I was running a full GNOME desktop, running the tests in a gnome-terminal, I wasn't doing anything, but some jitter can be associated to background tasks. * I was running 10 second bursts, with each message containing 256 bytes of data (loggen defaults) * I was using the tcp() source * I was playing with various tuning parameters (window-size, fetch-limit, flush_lines, etc. but I didn't really record their value). * loggen was running on the same box as syslog-ng itself. With these in mind: 1) single connection, trying to send messages as fast as possible syslog-ng 3.2: 58000 msg/sec syslog-ng 3.3: 180000 msg/sec 2) also tested how it scales with 1000 clients syslog-ng 3.2 degraded with about 25% if 1 out of 1000 clients were sending messages syslog-ng 3.3 degrated with about 7% if 1 out of 1000 clients were sending messages (so using epoll instead of GLib's main loop was definitely worth the effort) Of course my development machine has only two cores, and with more cores and with more clients, it'll probably reach higher msg/sec. I would estimate that with real server hardware, it'll scale into the 800k-1000k msg/sec range assuming the disk isn't the bottleneck. (that's 200-250MB/sec, which is not impossible with a proper disk array) Anyway, the code isn't ready yet and even crashes (at exit for example), so definitely not production quality yet :) -- Bazsi ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Hi Mukarram, If you posted an example of some of these 5000 lines to this list I think there's a good chance that some of the clever people here would be able to replace them with a much smaller equivalent (i.e. one that dynamically caters for the different source host names, etc.). Then it would not only be much shorter and more efficient, but you probably wouldn't ever have to change it again. Regards, Scott Also, I hope syslog-ng 3.3 will fix the issue with my syslog-ng.conf having greater than 5000 lines. Syslog-ng 3.0.8 chokes on this (error: memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347). I had to revert back to syslog-ng-2.1.4 which works fine. ...or do I have to still do this which you suggested...
On Tue, 2010-12-07 at 09:04 -0800, Mukarram Syed wrote:
Also, I hope syslog-ng 3.3 will fix the issue with my syslog-ng.conf having greater than 5000 lines. Syslog-ng 3.0.8 chokes on this (error: memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347). I had to revert back to syslog-ng-2.1.4 which works fine. ...or do I have to still do this which you suggested...
I've pushed a commit to 3.2, thus it should be fixed by the next release. -- Bazsi
participants (6)
-
Balazs Scheidler
-
Christopher Barry
-
Mukarram Syed
-
Patrick H.
-
Sandor Geller
-
Scott Rochford