pdbtool patternize update and my syslog-ng 3.2 branch
Hello, As the patterndb project is starting to gain some momentum I thought it'd be the right time to port my patternize tool to the new, plugin-based 3.2 codebase as the first step towards getting it integrated --- and to be able to use the fancy new pdbtool features along with patternize. To those who are unfamiliar with it, patternize is an addition to pdbtool that makes it possible to automatically generate a pattern database from raw logs using statistical data clustering methods: you can read more about it in this blog post: http://gyp.blogs.balabit.com/2010/01/introducing-pdbtool-patternize/ Besides the port to the new codebase, it's received some fixes and new features since my original post: * multiple small internal bugfixes to get rid of weird errors * added the option "/--named-parsers/" that names the found @ESTRING@s like "/.dict.string0,1,2,3.../" * Balint Kovacs has sent three contributions: added support for reading the logfile from the standard input, escaping special characters in the output and putting examples in the XML that can be used for self-testing. It can be found in my public syslog-ng 3.2 tree: http://git.balabit.hu/?p=gyp/syslog-ng-3.2.git;a=summary If you're already using it (I've received some feedback so I guess some of you do), please note that most probably this 3.2-based branch will get the fixes and new features from now on. It's only received a basic sanity check and the unit tests do pass, so as usual, handle it with care and all feedback is welcome. greets, Peter ps.: the branch also contains a patch that fixes a wrong section name in pdbtool's man page and I'll try to update the whole manpage a bit when adding a section for patternize soon -- Bazsi, you might want to pull those to the mainline.
Hello Peter, Thanks for the update on the status of patterndb patternize. I wondered if the memory leaks you said existed in the old version had been fixed, you did not say one way or the other in your mail. I also wonder if anybody at Balabit could tell me how to build a copy of your Git tree on RHEL 4 or RHEL 5. I get problems because the PCRE is too old but when I switch to new PCRE, PCRE will not build because the autotools and pkg-config are too old. It's a problem for me because unfortunately my company only supports RHEL here and otherwise I have to run it in an Ubuntu 10.04 or Debian VM with way too little memory for the tool to run right. Would it be possible to build a version of your tree for RHEL 4 or 5? Matthew. On Fri, Sep 24, 2010 at 11:27:48AM +0200, Peter Gyongyosi wrote:
Hello,
As the patterndb project is starting to gain some momentum I thought it'd be the right time to port my patternize tool to the new, plugin-based 3.2 codebase as the first step towards getting it integrated --- and to be able to use the fancy new pdbtool features along with patternize. To those who are unfamiliar with it, patternize is an addition to pdbtool that makes it possible to automatically generate a pattern database from raw logs using statistical data clustering methods: you can read more about it in this blog post: http://gyp.blogs.balabit.com/2010/01/introducing-pdbtool-patternize/
Besides the port to the new codebase, it's received some fixes and new features since my original post:
* multiple small internal bugfixes to get rid of weird errors * added the option "/--named-parsers/" that names the found @ESTRING@s like "/.dict.string0,1,2,3.../" * Balint Kovacs has sent three contributions: added support for reading the logfile from the standard input, escaping special characters in the output and putting examples in the XML that can be used for self-testing.
It can be found in my public syslog-ng 3.2 tree: http://git.balabit.hu/?p=gyp/syslog-ng-3.2.git;a=summary
If you're already using it (I've received some feedback so I guess some of you do), please note that most probably this 3.2-based branch will get the fixes and new features from now on.
It's only received a basic sanity check and the unit tests do pass, so as usual, handle it with care and all feedback is welcome.
greets, Peter
ps.: the branch also contains a patch that fixes a wrong section name in pdbtool's man page and I'll try to update the whole manpage a bit when adding a section for patternize soon -- Bazsi, you might want to pull those to the mainline.
Further investigation... The Balabit RPMs must be built specially because the daemon looks mostly statically linked. Is there some secret to building code for old Linuxes like the BalaBit RPMs? Matthew. On Fri, Sep 24, 2010 at 10:57:38AM -0700, Matthew Hall wrote:
Hello Peter,
Thanks for the update on the status of patterndb patternize.
I wondered if the memory leaks you said existed in the old version had been fixed, you did not say one way or the other in your mail.
I also wonder if anybody at Balabit could tell me how to build a copy of your Git tree on RHEL 4 or RHEL 5. I get problems because the PCRE is too old but when I switch to new PCRE, PCRE will not build because the autotools and pkg-config are too old.
It's a problem for me because unfortunately my company only supports RHEL here and otherwise I have to run it in an Ubuntu 10.04 or Debian VM with way too little memory for the tool to run right.
Would it be possible to build a version of your tree for RHEL 4 or 5?
Matthew.
On Fri, Sep 24, 2010 at 11:27:48AM +0200, Peter Gyongyosi wrote:
Hello,
As the patterndb project is starting to gain some momentum I thought it'd be the right time to port my patternize tool to the new, plugin-based 3.2 codebase as the first step towards getting it integrated --- and to be able to use the fancy new pdbtool features along with patternize. To those who are unfamiliar with it, patternize is an addition to pdbtool that makes it possible to automatically generate a pattern database from raw logs using statistical data clustering methods: you can read more about it in this blog post: http://gyp.blogs.balabit.com/2010/01/introducing-pdbtool-patternize/
Besides the port to the new codebase, it's received some fixes and new features since my original post:
* multiple small internal bugfixes to get rid of weird errors * added the option "/--named-parsers/" that names the found @ESTRING@s like "/.dict.string0,1,2,3.../" * Balint Kovacs has sent three contributions: added support for reading the logfile from the standard input, escaping special characters in the output and putting examples in the XML that can be used for self-testing.
It can be found in my public syslog-ng 3.2 tree: http://git.balabit.hu/?p=gyp/syslog-ng-3.2.git;a=summary
If you're already using it (I've received some feedback so I guess some of you do), please note that most probably this 3.2-based branch will get the fixes and new features from now on.
It's only received a basic sanity check and the unit tests do pass, so as usual, handle it with care and all feedback is welcome.
greets, Peter
ps.: the branch also contains a patch that fixes a wrong section name in pdbtool's man page and I'll try to update the whole manpage a bit when adding a section for patternize soon -- Bazsi, you might want to pull those to the mainline.
Ya, I imagine theyre static linked because its easier to release for more distro's that way. I build the RPMs for our environment because there are a few extra patches I throw in (and I want pcre). For our older boxes (RHEL4), I build both glib and pcre in a separate dir (not installed in the system), and then when compiling syslog-ng I static link just pcre and glib, and leave everything else linked against the normal system libs. Unless you have some reason to not use the binaries provided by balabit, I'd say its would be much easier to just use those. -Patrick Sent: Fri Sep 24 2010 14:53:25 GMT-0600 (Mountain Daylight Time) From: Matthew Hall <mhall@mhcomputing.net> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] pdbtool patternize update and my syslog-ng 3.2 branch
Further investigation...
The Balabit RPMs must be built specially because the daemon looks mostly statically linked. Is there some secret to building code for old Linuxes like the BalaBit RPMs?
Matthew.
On Fri, Sep 24, 2010 at 10:57:38AM -0700, Matthew Hall wrote:
Hello Peter,
Thanks for the update on the status of patterndb patternize.
I wondered if the memory leaks you said existed in the old version had been fixed, you did not say one way or the other in your mail.
I also wonder if anybody at Balabit could tell me how to build a copy of your Git tree on RHEL 4 or RHEL 5. I get problems because the PCRE is too old but when I switch to new PCRE, PCRE will not build because the autotools and pkg-config are too old.
It's a problem for me because unfortunately my company only supports RHEL here and otherwise I have to run it in an Ubuntu 10.04 or Debian VM with way too little memory for the tool to run right.
Would it be possible to build a version of your tree for RHEL 4 or 5?
Matthew.
On Fri, Sep 24, 2010 at 11:27:48AM +0200, Peter Gyongyosi wrote:
Hello,
As the patterndb project is starting to gain some momentum I thought it'd be the right time to port my patternize tool to the new, plugin-based 3.2 codebase as the first step towards getting it integrated --- and to be able to use the fancy new pdbtool features along with patternize. To those who are unfamiliar with it, patternize is an addition to pdbtool that makes it possible to automatically generate a pattern database from raw logs using statistical data clustering methods: you can read more about it in this blog post: http://gyp.blogs.balabit.com/2010/01/introducing-pdbtool-patternize/
Besides the port to the new codebase, it's received some fixes and new features since my original post:
* multiple small internal bugfixes to get rid of weird errors * added the option "/--named-parsers/" that names the found @ESTRING@s like "/.dict.string0,1,2,3.../" * Balint Kovacs has sent three contributions: added support for reading the logfile from the standard input, escaping special characters in the output and putting examples in the XML that can be used for self-testing.
It can be found in my public syslog-ng 3.2 tree: http://git.balabit.hu/?p=gyp/syslog-ng-3.2.git;a=summary
If you're already using it (I've received some feedback so I guess some of you do), please note that most probably this 3.2-based branch will get the fixes and new features from now on.
It's only received a basic sanity check and the unit tests do pass, so as usual, handle it with care and all feedback is welcome.
greets, Peter
ps.: the branch also contains a patch that fixes a wrong section name in pdbtool's man page and I'll try to update the whole manpage a bit when adding a section for patternize soon -- Bazsi, you might want to pull those to the mainline.
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
The problem is I really seriously need GYP's pdbtool patternize 3.2 Git tree. I am unaware of any prebuilt RPMs for it or I would have installed them long ago so I would not need to deal with this. :-) Matthew On Fri, Sep 24, 2010 at 03:44:37PM -0600, PATRICK HEMMER wrote:
Ya, I imagine theyre static linked because its easier to release for more distro's that way. I build the RPMs for our environment because there are a few extra patches I throw in (and I want pcre). For our older boxes (RHEL4), I build both glib and pcre in a separate dir (not installed in the system), and then when compiling syslog-ng I static link just pcre and glib, and leave everything else linked against the normal system libs.
Unless you have some reason to not use the binaries provided by balabit, I'd say its would be much easier to just use those.
-Patrick
Sent: Fri Sep 24 2010 14:53:25 GMT-0600 (Mountain Daylight Time) From: Matthew Hall <mhall@mhcomputing.net> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] pdbtool patternize update and my syslog-ng 3.2 branch
Further investigation...
The Balabit RPMs must be built specially because the daemon looks mostly statically linked. Is there some secret to building code for old Linuxes like the BalaBit RPMs?
Matthew.
On Fri, Sep 24, 2010 at 10:57:38AM -0700, Matthew Hall wrote:
Hello Peter,
Thanks for the update on the status of patterndb patternize.
I wondered if the memory leaks you said existed in the old version had been fixed, you did not say one way or the other in your mail.
I also wonder if anybody at Balabit could tell me how to build a copy of your Git tree on RHEL 4 or RHEL 5. I get problems because the PCRE is too old but when I switch to new PCRE, PCRE will not build because the autotools and pkg-config are too old.
It's a problem for me because unfortunately my company only supports RHEL here and otherwise I have to run it in an Ubuntu 10.04 or Debian VM with way too little memory for the tool to run right.
Would it be possible to build a version of your tree for RHEL 4 or 5?
Matthew.
On Fri, Sep 24, 2010 at 11:27:48AM +0200, Peter Gyongyosi wrote:
Hello,
As the patterndb project is starting to gain some momentum I thought it'd be the right time to port my patternize tool to the new, plugin-based 3.2 codebase as the first step towards getting it integrated --- and to be able to use the fancy new pdbtool features along with patternize. To those who are unfamiliar with it, patternize is an addition to pdbtool that makes it possible to automatically generate a pattern database from raw logs using statistical data clustering methods: you can read more about it in this blog post: http://gyp.blogs.balabit.com/2010/01/introducing-pdbtool-patternize/
Besides the port to the new codebase, it's received some fixes and new features since my original post:
* multiple small internal bugfixes to get rid of weird errors * added the option "/--named-parsers/" that names the found @ESTRING@s like "/.dict.string0,1,2,3.../" * Balint Kovacs has sent three contributions: added support for reading the logfile from the standard input, escaping special characters in the output and putting examples in the XML that can be used for self-testing.
It can be found in my public syslog-ng 3.2 tree: http://git.balabit.hu/?p=gyp/syslog-ng-3.2.git;a=summary
If you're already using it (I've received some feedback so I guess some of you do), please note that most probably this 3.2-based branch will get the fixes and new features from now on.
It's only received a basic sanity check and the unit tests do pass, so as usual, handle it with care and all feedback is welcome.
greets, Peter
ps.: the branch also contains a patch that fixes a wrong section name in pdbtool's man page and I'll try to update the whole manpage a bit when adding a section for patternize soon -- Bazsi, you might want to pull those to the mainline.
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Can you compile on Ubuntu or some other distro with --enable-static-linking and then just port the binaries over? On Fri, Sep 24, 2010 at 7:38 PM, Matthew Hall <mhall@mhcomputing.net> wrote:
The problem is I really seriously need GYP's pdbtool patternize 3.2 Git tree.
I am unaware of any prebuilt RPMs for it or I would have installed them long ago so I would not need to deal with this. :-)
Matthew
On Fri, Sep 24, 2010 at 03:44:37PM -0600, PATRICK HEMMER wrote:
Ya, I imagine theyre static linked because its easier to release for more distro's that way. I build the RPMs for our environment because there are a few extra patches I throw in (and I want pcre). For our older boxes (RHEL4), I build both glib and pcre in a separate dir (not installed in the system), and then when compiling syslog-ng I static link just pcre and glib, and leave everything else linked against the normal system libs.
Unless you have some reason to not use the binaries provided by balabit, I'd say its would be much easier to just use those.
-Patrick
Sent: Fri Sep 24 2010 14:53:25 GMT-0600 (Mountain Daylight Time) From: Matthew Hall <mhall@mhcomputing.net> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] pdbtool patternize update and my syslog-ng 3.2 branch
Further investigation...
The Balabit RPMs must be built specially because the daemon looks mostly statically linked. Is there some secret to building code for old Linuxes like the BalaBit RPMs?
Matthew.
On Fri, Sep 24, 2010 at 10:57:38AM -0700, Matthew Hall wrote:
Hello Peter,
Thanks for the update on the status of patterndb patternize.
I wondered if the memory leaks you said existed in the old version had been fixed, you did not say one way or the other in your mail.
I also wonder if anybody at Balabit could tell me how to build a copy of your Git tree on RHEL 4 or RHEL 5. I get problems because the PCRE is too old but when I switch to new PCRE, PCRE will not build because the autotools and pkg-config are too old.
It's a problem for me because unfortunately my company only supports RHEL here and otherwise I have to run it in an Ubuntu 10.04 or Debian VM with way too little memory for the tool to run right.
Would it be possible to build a version of your tree for RHEL 4 or 5?
Matthew.
On Fri, Sep 24, 2010 at 11:27:48AM +0200, Peter Gyongyosi wrote:
Hello,
As the patterndb project is starting to gain some momentum I thought it'd be the right time to port my patternize tool to the new, plugin-based 3.2 codebase as the first step towards getting it integrated --- and to be able to use the fancy new pdbtool features along with patternize. To those who are unfamiliar with it, patternize is an addition to pdbtool that makes it possible to automatically generate a pattern database from raw logs using statistical data clustering methods: you can read more about it in this blog post: http://gyp.blogs.balabit.com/2010/01/introducing-pdbtool-patternize/
Besides the port to the new codebase, it's received some fixes and new features since my original post:
* multiple small internal bugfixes to get rid of weird errors * added the option "/--named-parsers/" that names the found @ESTRING@s like "/.dict.string0,1,2,3.../" * Balint Kovacs has sent three contributions: added support for reading the logfile from the standard input, escaping special characters in the output and putting examples in the XML that can be used for self-testing.
It can be found in my public syslog-ng 3.2 tree: http://git.balabit.hu/?p=gyp/syslog-ng-3.2.git;a=summary
If you're already using it (I've received some feedback so I guess some of you do), please note that most probably this 3.2-based branch will get the fixes and new features from now on.
It's only received a basic sanity check and the unit tests do pass, so as usual, handle it with care and all feedback is welcome.
greets, Peter
ps.: the branch also contains a patch that fixes a wrong section name in pdbtool's man page and I'll try to update the whole manpage a bit when adding a section for patternize soon -- Bazsi, you might want to pull those to the mainline.
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
I guess I missed that build option. Has anybody else used it successfully? On Fri, Sep 24, 2010 at 08:55:06PM -0500, Martin Holste wrote:
Can you compile on Ubuntu or some other distro with --enable-static-linking and then just port the binaries over?
On Fri, Sep 24, 2010 at 7:38 PM, Matthew Hall <mhall@mhcomputing.net> wrote:
The problem is I really seriously need GYP's pdbtool patternize 3.2 Git tree.
I am unaware of any prebuilt RPMs for it or I would have installed them long ago so I would not need to deal with this. :-)
Matthew
On Fri, Sep 24, 2010 at 03:44:37PM -0600, PATRICK HEMMER wrote:
Ya, I imagine theyre static linked because its easier to release for more distro's that way. I build the RPMs for our environment because there are a few extra patches I throw in (and I want pcre). For our older boxes (RHEL4), I build both glib and pcre in a separate dir (not installed in the system), and then when compiling syslog-ng I static link just pcre and glib, and leave everything else linked against the normal system libs.
Unless you have some reason to not use the binaries provided by balabit, I'd say its would be much easier to just use those.
-Patrick
Sent: Fri Sep 24 2010 14:53:25 GMT-0600 (Mountain Daylight Time) From: Matthew Hall <mhall@mhcomputing.net> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] pdbtool patternize update and my syslog-ng 3.2 branch
Further investigation...
The Balabit RPMs must be built specially because the daemon looks mostly statically linked. Is there some secret to building code for old Linuxes like the BalaBit RPMs?
Matthew.
On Fri, Sep 24, 2010 at 10:57:38AM -0700, Matthew Hall wrote:
Hello Peter,
Thanks for the update on the status of patterndb patternize.
I wondered if the memory leaks you said existed in the old version had been fixed, you did not say one way or the other in your mail.
I also wonder if anybody at Balabit could tell me how to build a copy of your Git tree on RHEL 4 or RHEL 5. I get problems because the PCRE is too old but when I switch to new PCRE, PCRE will not build because the autotools and pkg-config are too old.
It's a problem for me because unfortunately my company only supports RHEL here and otherwise I have to run it in an Ubuntu 10.04 or Debian VM with way too little memory for the tool to run right.
Would it be possible to build a version of your tree for RHEL 4 or 5?
Matthew.
On Fri, Sep 24, 2010 at 11:27:48AM +0200, Peter Gyongyosi wrote:
Hello,
As the patterndb project is starting to gain some momentum I thought it'd be the right time to port my patternize tool to the new, plugin-based 3.2 codebase as the first step towards getting it integrated --- and to be able to use the fancy new pdbtool features along with patternize. To those who are unfamiliar with it, patternize is an addition to pdbtool that makes it possible to automatically generate a pattern database from raw logs using statistical data clustering methods: you can read more about it in this blog post: http://gyp.blogs.balabit.com/2010/01/introducing-pdbtool-patternize/
Besides the port to the new codebase, it's received some fixes and new features since my original post:
* multiple small internal bugfixes to get rid of weird errors * added the option "/--named-parsers/" that names the found @ESTRING@s like "/.dict.string0,1,2,3.../" * Balint Kovacs has sent three contributions: added support for reading the logfile from the standard input, escaping special characters in the output and putting examples in the XML that can be used for self-testing.
It can be found in my public syslog-ng 3.2 tree: http://git.balabit.hu/?p=gyp/syslog-ng-3.2.git;a=summary
If you're already using it (I've received some feedback so I guess some of you do), please note that most probably this 3.2-based branch will get the fixes and new features from now on.
It's only received a basic sanity check and the unit tests do pass, so as usual, handle it with care and all feedback is welcome.
greets, Peter
ps.: the branch also contains a patch that fixes a wrong section name in pdbtool's man page and I'll try to update the whole manpage a bit when adding a section for patternize soon -- Bazsi, you might want to pull those to the mainline.
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Fri, 2010-09-24 at 18:58 -0700, Matthew Hall wrote:
I guess I missed that build option.
Has anybody else used it successfully?
It does require some expertise with ld's internals, but it did work earlier. With 3.2 though it becomes different, static-linking is dropped (syslog-ng itself is dynamically linked because of the plugins), however --enable-mixed-linking still works. -- Bazsi
I'm on Ubuntu lucid trying to compile the latest git version and I'm getting "./.libs/libsyslog-ng.so: undefined reference to `block_ref_debug'" etc. when it tries to do the final ld command unless --enable-debug is on. --enable-mixed-linking works fine with --enable-debug, though I see a ton of dependencies with ldd, so that must be pretty far from a static link. On Mon, Sep 27, 2010 at 8:20 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Fri, 2010-09-24 at 18:58 -0700, Matthew Hall wrote:
I guess I missed that build option.
Has anybody else used it successfully?
It does require some expertise with ld's internals, but it did work earlier. With 3.2 though it becomes different, static-linking is dropped (syslog-ng itself is dynamically linked because of the plugins), however --enable-mixed-linking still works.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Sorry for delayed replies. I've gotten a bad liver infection. I also used the same OS as Martin with GYP's 3.2 patternize branch. I get a zillion library dependencies in pdbtool which will render it nonportable. There must be some Debian specific secrets or some other trick to this. Or maybe the 3.2 binaries will have way more shlibdeps than the 3.1 binaries. I don't know at all. Matthew. ./configure --prefix=/home/megahall/sng-3.2 --enable-debug --enable-ssl --enable-mixed-linking --enable-ipv6 --enable-tcp-wrapper --enable-spoof-source --enable-sql=false --enable-linux-caps --enable-pcre linux-vdso.so.1 => (0x00007fffd0f39000) libsyslog-ng.so.0 => /home/megahall/sng-3.2/lib/libsyslog-ng.so.0 (0x00007f3a96010000) libdbparser.so => /home/megahall/sng-3.2/lib/syslog-ng/libdbparser.so (0x00007f3a95dfc000) librt.so.1 => /lib/librt.so.1 (0x00007f3a95be3000) libnsl.so.1 => /lib/libnsl.so.1 (0x00007f3a959c9000) libuuid.so.1 => /lib/libuuid.so.1 (0x00007f3a957c4000) libgmodule-2.0.so.0 => /usr/lib/libgmodule-2.0.so.0 (0x00007f3a955bf000) libglib-2.0.so.0 => /lib/libglib-2.0.so.0 (0x00007f3a952e1000) libevtlog.so.0 => /usr/lib/libevtlog.so.0 (0x00007f3a950dc000) libpcre.so.3 => /lib/libpcre.so.3 (0x00007f3a94ead000) libcap.so.2 => /lib/libcap.so.2 (0x00007f3a94ca8000) libdl.so.2 => /lib/libdl.so.2 (0x00007f3a94aa4000) libpthread.so.0 => /lib/libpthread.so.0 (0x00007f3a94886000) libc.so.6 => /lib/libc.so.6 (0x00007f3a94503000) /lib64/ld-linux-x86-64.so.2 (0x00007f3a96295000) libattr.so.1 => /lib/libattr.so.1 (0x00007f3a942fd000) On Mon, Sep 27, 2010 at 09:32:20AM -0500, Martin Holste wrote:
I'm on Ubuntu lucid trying to compile the latest git version and I'm getting "./.libs/libsyslog-ng.so: undefined reference to `block_ref_debug'" etc. when it tries to do the final ld command unless --enable-debug is on. --enable-mixed-linking works fine with --enable-debug, though I see a ton of dependencies with ldd, so that must be pretty far from a static link.
On Mon, Sep 27, 2010 at 8:20 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Fri, 2010-09-24 at 18:58 -0700, Matthew Hall wrote:
I guess I missed that build option.
Has anybody else used it successfully?
It does require some expertise with ld's internals, but it did work earlier. With 3.2 though it becomes different, static-linking is dropped (syslog-ng itself is dynamically linked because of the plugins), however --enable-mixed-linking still works.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Mon, 2010-09-27 at 09:57 -0700, Matthew Hall wrote:
Sorry for delayed replies. I've gotten a bad liver infection.
I also used the same OS as Martin with GYP's 3.2 patternize branch.
I get a zillion library dependencies in pdbtool which will render it nonportable. There must be some Debian specific secrets or some other trick to this. Or maybe the 3.2 binaries will have way more shlibdeps than the 3.1 binaries. I don't know at all.
Matthew.
./configure --prefix=/home/megahall/sng-3.2 --enable-debug --enable-ssl --enable-mixed-linking --enable-ipv6 --enable-tcp-wrapper --enable-spoof-source --enable-sql=false --enable-linux-caps --enable-pcre
linux-vdso.so.1 => (0x00007fffd0f39000) libsyslog-ng.so.0 => /home/megahall/sng-3.2/lib/libsyslog-ng.so.0 (0x00007f3a96010000) libdbparser.so => /home/megahall/sng-3.2/lib/syslog-ng/libdbparser.so (0x00007f3a95dfc000) librt.so.1 => /lib/librt.so.1 (0x00007f3a95be3000) libnsl.so.1 => /lib/libnsl.so.1 (0x00007f3a959c9000) libuuid.so.1 => /lib/libuuid.so.1 (0x00007f3a957c4000) libgmodule-2.0.so.0 => /usr/lib/libgmodule-2.0.so.0 (0x00007f3a955bf000) libglib-2.0.so.0 => /lib/libglib-2.0.so.0 (0x00007f3a952e1000) libevtlog.so.0 => /usr/lib/libevtlog.so.0 (0x00007f3a950dc000) libpcre.so.3 => /lib/libpcre.so.3 (0x00007f3a94ead000) libcap.so.2 => /lib/libcap.so.2 (0x00007f3a94ca8000) libdl.so.2 => /lib/libdl.so.2 (0x00007f3a94aa4000) libpthread.so.0 => /lib/libpthread.so.0 (0x00007f3a94886000) libc.so.6 => /lib/libc.so.6 (0x00007f3a94503000) /lib64/ld-linux-x86-64.so.2 (0x00007f3a96295000) libattr.so.1 => /lib/libattr.so.1 (0x00007f3a942fd000)
If you can install the static versions of these libraries: $GLIB_LIBS $EVTLOG_LIBS $PCRE_LIBS $REGEX_LIBS Then those dependencies will be statically linked into the syslog-ng/pdbtool binary, but libsyslog-ng.so will always be linked dynamically. Not to mention that even pdbtool tries to load plugins. So as of right now, you can't move the binaries to different boxes. But I have good news as well, I've integrated gyp's patches to mainline, cleaned them up, removed memory leaks and also decreased memory usage a lot. So you don't need gyp's tree, it is enough to use the mainline. -- Bazsi
On Wed, Sep 29, 2010 at 11:06:18AM +0200, Balazs Scheidler wrote:
If you can install the static versions of these libraries:
$GLIB_LIBS $EVTLOG_LIBS $PCRE_LIBS $REGEX_LIBS
Then those dependencies will be statically linked into the syslog-ng/pdbtool binary, but libsyslog-ng.so will always be linked dynamically. Not to mention that even pdbtool tries to load plugins.
OK Good to know. It might help if you were comfortable to publish a copy of the dpkg --get-selections output from one of your working build systems or a list of the dependencies which are needed to build this thing the right way. It's not exactly very easy to figure out when you want portable binaries. Especially since the build succeeds and produces unexpected output when certain libraries are missing.
So as of right now, you can't move the binaries to different boxes.
Then how is Balabit able to produce packages which work on different boxes? Would it make a difference if I compiled on one box, copied the compiled tree, and then ran make install? That was what came to my mind as an ugly but possible effective technique.
But I have good news as well, I've integrated gyp's patches to mainline, cleaned them up, removed memory leaks and also decreased memory usage a lot. So you don't need gyp's tree, it is enough to use the mainline.
This is wonderful news. How would I get a nightly build of this if it's not possible for me to move the binaries from a system where it compiles (Ubuntu 10.04 LTS for example) to a system where it doesn't (RHEL 4 and 5 for example)?
Bazsi
Matthew.
Any updates on this one? I would really like to know: 1) how / where to get Bazsi's 3.2 tree 2) how to compile it for RHEL 4/5 using a newer OS like Ubuntu 10.04 LTS 3) how to install it once it has been compiled 4) i.e. can binaries be compiled, tree copied, then make install, or what??? I am dead in the water until I can figure this out, because I need patternize working before I can start to process logs and get any results. Matthew. On Wed, Sep 29, 2010 at 09:15:59AM -0700, Matthew Hall wrote:
On Wed, Sep 29, 2010 at 11:06:18AM +0200, Balazs Scheidler wrote:
If you can install the static versions of these libraries:
$GLIB_LIBS $EVTLOG_LIBS $PCRE_LIBS $REGEX_LIBS
Then those dependencies will be statically linked into the syslog-ng/pdbtool binary, but libsyslog-ng.so will always be linked dynamically. Not to mention that even pdbtool tries to load plugins.
OK Good to know. It might help if you were comfortable to publish a copy of the dpkg --get-selections output from one of your working build systems or a list of the dependencies which are needed to build this thing the right way. It's not exactly very easy to figure out when you want portable binaries. Especially since the build succeeds and produces unexpected output when certain libraries are missing.
So as of right now, you can't move the binaries to different boxes.
Then how is Balabit able to produce packages which work on different boxes? Would it make a difference if I compiled on one box, copied the compiled tree, and then ran make install? That was what came to my mind as an ugly but possible effective technique.
But I have good news as well, I've integrated gyp's patches to mainline, cleaned them up, removed memory leaks and also decreased memory usage a lot. So you don't need gyp's tree, it is enough to use the mainline.
This is wonderful news. How would I get a nightly build of this if it's not possible for me to move the binaries from a system where it compiles (Ubuntu 10.04 LTS for example) to a system where it doesn't (RHEL 4 and 5 for example)?
Bazsi
Matthew.
Further investigation reveals that the syslog-ng daemon and libsyslog-ng.so appear to be linking only against libc and its friends. However pdbtool's linking against a truckload of external libraries as shown before. It seems that the link logic is not consistent across different binaries. On Fri, Oct 01, 2010 at 10:02:45AM -0700, Matthew Hall wrote:
Any updates on this one? I would really like to know:
1) how / where to get Bazsi's 3.2 tree 2) how to compile it for RHEL 4/5 using a newer OS like Ubuntu 10.04 LTS 3) how to install it once it has been compiled 4) i.e. can binaries be compiled, tree copied, then make install, or what???
I am dead in the water until I can figure this out, because I need patternize working before I can start to process logs and get any results.
Matthew.
On Wed, Sep 29, 2010 at 09:15:59AM -0700, Matthew Hall wrote:
On Wed, Sep 29, 2010 at 11:06:18AM +0200, Balazs Scheidler wrote:
If you can install the static versions of these libraries:
$GLIB_LIBS $EVTLOG_LIBS $PCRE_LIBS $REGEX_LIBS
Then those dependencies will be statically linked into the syslog-ng/pdbtool binary, but libsyslog-ng.so will always be linked dynamically. Not to mention that even pdbtool tries to load plugins.
OK Good to know. It might help if you were comfortable to publish a copy of the dpkg --get-selections output from one of your working build systems or a list of the dependencies which are needed to build this thing the right way. It's not exactly very easy to figure out when you want portable binaries. Especially since the build succeeds and produces unexpected output when certain libraries are missing.
So as of right now, you can't move the binaries to different boxes.
Then how is Balabit able to produce packages which work on different boxes? Would it make a difference if I compiled on one box, copied the compiled tree, and then ran make install? That was what came to my mind as an ugly but possible effective technique.
But I have good news as well, I've integrated gyp's patches to mainline, cleaned them up, removed memory leaks and also decreased memory usage a lot. So you don't need gyp's tree, it is enough to use the mainline.
This is wonderful news. How would I get a nightly build of this if it's not possible for me to move the binaries from a system where it compiles (Ubuntu 10.04 LTS for example) to a system where it doesn't (RHEL 4 and 5 for example)?
Bazsi
Matthew.
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Fri, 2010-10-01 at 11:38 -0700, Matthew Hall wrote:
Further investigation reveals that the syslog-ng daemon and libsyslog-ng.so appear to be linking only against libc and its friends. However pdbtool's linking against a truckload of external libraries as shown before. It seems that the link logic is not consistent across different binaries.
pdbtool immediately links against the libdbparser plugin, as it uses some symbols in addition to the standard syslog-ng plugin interface. Also pdbtool depends on libssl because it uses the random generator to generate uuids. Also, syslog-ng starts early in the boot process and it was meant to be loaded without libssl for instance. With pdbtool only used when the system is up and running no such effort was made. But anyway the easiest way to distribute syslog-ng binaries between different boxes is to do something similar to what we do, e.g. compile everything into the same prefix and copy the whole structure. One thing caught my attention, you couldn't compile it on RHEL4/5? Why? I'd love to investigate if there's a problem there. -- Bazsi
On Sat, Oct 16, 2010 at 01:20:23PM +0200, Balazs Scheidler wrote:
On Fri, 2010-10-01 at 11:38 -0700, Matthew Hall wrote:
Further investigation reveals that the syslog-ng daemon and libsyslog-ng.so appear to be linking only against libc and its friends. However pdbtool's linking against a truckload of external libraries as shown before. It seems that the link logic is not consistent across different binaries.
pdbtool immediately links against the libdbparser plugin, as it uses some symbols in addition to the standard syslog-ng plugin interface. Also pdbtool depends on libssl because it uses the random generator to generate uuids.
I suspected something like this was the case.
Also, syslog-ng starts early in the boot process and it was meant to be loaded without libssl for instance. With pdbtool only used when the system is up and running no such effort was made.
Yes, this makes sense, unless you are trying to run pdbtool on a distro on which it cannot be compiled such as RHEL 4/5.
But anyway the easiest way to distribute syslog-ng binaries between different boxes is to do something similar to what we do, e.g. compile everything into the same prefix and copy the whole structure.
Makes sense. If I can figure out how to get mixed linking to work for other code besides the daemon itself. Otherwise it will not move over due to version conflicts in the other libs like libevent, libssl, etc. etc. etc.
One thing caught my attention, you couldn't compile it on RHEL4/5? Why? I'd love to investigate if there's a problem there.
On RHEL 4 and 5 it requires too new of a PCRE. If you install the PCRE, you can break the system because other things could get angry that PCRE was changed. But even when you try to compile PCRE that fails because the autotools are too old and a ton of macros you need are missing. So I think there are some tricks to compiling it for RHEL that I am not doing right, hence my questions about how Balabit managed to compile the code because I couldn't figure it out at all.
Bazsi
Matthew.
On Sat, 2010-10-16 at 10:44 -0700, Matthew Hall wrote:
On Sat, Oct 16, 2010 at 01:20:23PM +0200, Balazs Scheidler wrote:
On Fri, 2010-10-01 at 11:38 -0700, Matthew Hall wrote:
Further investigation reveals that the syslog-ng daemon and libsyslog-ng.so appear to be linking only against libc and its friends. However pdbtool's linking against a truckload of external libraries as shown before. It seems that the link logic is not consistent across different binaries.
pdbtool immediately links against the libdbparser plugin, as it uses some symbols in addition to the standard syslog-ng plugin interface. Also pdbtool depends on libssl because it uses the random generator to generate uuids.
I suspected something like this was the case.
Also, syslog-ng starts early in the boot process and it was meant to be loaded without libssl for instance. With pdbtool only used when the system is up and running no such effort was made.
Yes, this makes sense, unless you are trying to run pdbtool on a distro on which it cannot be compiled such as RHEL 4/5.
But anyway the easiest way to distribute syslog-ng binaries between different boxes is to do something similar to what we do, e.g. compile everything into the same prefix and copy the whole structure.
Makes sense. If I can figure out how to get mixed linking to work for other code besides the daemon itself. Otherwise it will not move over due to version conflicts in the other libs like libevent, libssl, etc. etc. etc.
you can always compile the libraries to a custom prefix so that it doesn't break the core system.
One thing caught my attention, you couldn't compile it on RHEL4/5? Why? I'd love to investigate if there's a problem there.
On RHEL 4 and 5 it requires too new of a PCRE.
Hmm.. I'm willing to change this, IIRC it is only the PCRE_NEWLINE_ANYCRLF flag is missing from old PCRE, which can be compiled conditionally. Can you try the attached patch?
If you install the PCRE, you can break the system because other things could get angry that PCRE was changed. But even when you try to compile PCRE that fails because the autotools are too old and a ton of macros you need are missing.
autotools is not that important issue since I'm generating the files when I release a tarball, so the generated files are there. of course if you want to change the autotools input files, then that's a problem.
So I think there are some tricks to compiling it for RHEL that I am not doing right, hence my questions about how Balabit managed to compile the code because I couldn't figure it out at all.
And the patch: $ git diff diff --git a/configure.in b/configure.in index 573ff42..775bbe5 100644 --- a/configure.in +++ b/configure.in @@ -24,7 +24,7 @@ GLIB_MIN_VERSION="2.10.1" EVTLOG_MIN_VERSION="0.2" OPENSSL_MIN_VERSION="0.9.8" LIBDBI_MIN_VERSION="0.8.0" -PCRE_MIN_VERSION="7.3" +PCRE_MIN_VERSION="6.1" dnl *************************************************************************** dnl Initial setup diff --git a/lib/logmatcher.c b/lib/logmatcher.c index f0a822c..44a84ed 100644 --- a/lib/logmatcher.c +++ b/lib/logmatcher.c @@ -520,8 +520,12 @@ log_matcher_pcre_re_compile(LogMatcher *s, const gchar *re) if (self->super.flags & LMF_ICASE) flags |= PCRE_CASELESS; + +#ifdef PCRE_NEWLINE_ANYCRLF if (self->super.flags & LMF_NEWLINE) flags |= PCRE_NEWLINE_ANYCRLF; +#endif + if (self->super.flags & LMF_UTF8) { gint support; -- Bazsi
On Mon, Oct 18, 2010 at 10:59:58AM +0200, Balazs Scheidler wrote:
you can always compile the libraries to a custom prefix so that it doesn't break the core system.
Yes I tried this method but PCRE also wanted newer autotools.
One thing caught my attention, you couldn't compile it on RHEL4/5? Why? I'd love to investigate if there's a problem there.
On RHEL 4 and 5 it requires too new of a PCRE.
Hmm.. I'm willing to change this, IIRC it is only the PCRE_NEWLINE_ANYCRLF flag is missing from old PCRE, which can be compiled conditionally.
Can you try the attached patch?
Thanks. I will certainly try out the patch if I can work around the next issue.
If you install the PCRE, you can break the system because other things could get angry that PCRE was changed. But even when you try to compile PCRE that fails because the autotools are too old and a ton of macros you need are missing.
autotools is not that important issue since I'm generating the files when I release a tarball, so the generated files are there. of course if you want to change the autotools input files, then that's a problem.
That wouldn't necessarily help in my case because I was using GYP's code from Git, then I switched to yours when you said you mainlined GYP's patternize code. The Git code does not include built-out autotools scripts. If the patternize is available in your recent 3.2 beta tarballs with built-out scripts, I can try the build on there. I will work on this today and report back, since your timezone is 9 hours ahead of mine. Matthew.
On Mon, Oct 18, 2010 at 09:55:28AM -0700, Matthew Hall wrote:
On Mon, Oct 18, 2010 at 10:59:58AM +0200, Balazs Scheidler wrote:
you can always compile the libraries to a custom prefix so that it doesn't break the core system.
Yes I tried this method but PCRE also wanted newer autotools.
One thing caught my attention, you couldn't compile it on RHEL4/5? Why? I'd love to investigate if there's a problem there.
On RHEL 4 and 5 it requires too new of a PCRE.
Hmm.. I'm willing to change this, IIRC it is only the PCRE_NEWLINE_ANYCRLF flag is missing from old PCRE, which can be compiled conditionally.
Can you try the attached patch?
Thanks. I will certainly try out the patch if I can work around the next issue.
If you install the PCRE, you can break the system because other things could get angry that PCRE was changed. But even when you try to compile PCRE that fails because the autotools are too old and a ton of macros you need are missing.
autotools is not that important issue since I'm generating the files when I release a tarball, so the generated files are there. of course if you want to change the autotools input files, then that's a problem.
That wouldn't necessarily help in my case because I was using GYP's code from Git, then I switched to yours when you said you mainlined GYP's patternize code. The Git code does not include built-out autotools scripts.
If the patternize is available in your recent 3.2 beta tarballs with built-out scripts, I can try the build on there. I will work on this today and report back, since your timezone is 9 hours ahead of mine.
Matthew.
Your patch modified configure.in. So I had to copy the patched 3.2 beta 1 over onto Ubuntu 10.04 LTS to be able to rerun the autotools and copy it back. I have another problem. RHEL 5 is missing this file: /usr/lib/pkgconfig/glib-2.0.pc So I created a faked out version based on Ubuntu's file with the version number modified to match the RHEL 5 glib RPM's version. prefix=/usr exec_prefix=${prefix} libdir=${exec_prefix}/lib includedir=${prefix}/include glib_genmarshal=glib-genmarshal gobject_query=gobject-query glib_mkenums=glib-mkenums Name: GLib Description: C Utility Library Version: 2.24.1 Libs: -L${libdir} -lglib-2.0 Libs.private: Cflags: -I${includedir}/glib-2.0 -I${libdir}/glib-2.0/include After adding that, and running configure as follows: ./configure --prefix=/home/megahall/sng --enable-debug --enable-ssl --enable-mixed-linking --enable-tcp-wrapper --enable-spoof-source --disable-sql --enable-pacct --enable-pcre Then I get the following errors: checking for LIBDBI... Package dbi was not found in the pkg-config search path. Perhaps you should add the directory containing `dbi.pc' to the PKG_CONFIG_PATH environment variable No package 'dbi' found no checking for dbi_initialize in -ldbi... no checking for GLIB... Requested 'glib-2.0 >= 2.10.1' but version of GLib is 2.4.7-1 configure: error: Package requirements (glib-2.0 >= 2.10.1 gmodule-2.0) were not met: Consider adjusting the PKG_CONFIG_PATH environment variable if you installed software in a non-standard prefix. Alternatively, you may set the environment variables GLIB_CFLAGS and GLIB_LIBS to avoid the need to call pkg-config. So apparently RHEL 5 has too old of a glib too. I assume I must have missed it last time because it failed on PCRE before it got to glib and the script stopped running. Matthew.
On Mon, 18 Oct 2010 12:47:35 -0700, Matthew Hall wrote:
I have another problem. RHEL 5 is missing this file:
/usr/lib/pkgconfig/glib-2.0.pc
So I created a faked out version based on Ubuntu's file with the version number modified to match the RHEL 5 glib RPM's version. [...] checking for GLIB... Requested 'glib-2.0 >= 2.10.1' but version of GLib is 2.4.7-1
configure: error: Package requirements (glib-2.0 >= 2.10.1 gmodule-2.0) were not met: [...] So apparently RHEL 5 has too old of a glib too. I assume I must have missed it last time because it failed on PCRE before it got to glib and the script stopped running.
Are you sure you're having correct packages installed? Namely, glib2-devel. [root@dev32 ~]# rpm -qf /usr/lib/pkgconfig/glib-2.0.pc glib2-devel-2.12.3-4.el5_3.1 [root@dev32 ~]# HTH -- Jakub Jankowski|shasta@toxcorp.com|http://toxcorp.com/ GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D
On Tue, 2010-10-19 at 10:41 +0200, Jakub Jankowski wrote:
On Mon, 18 Oct 2010 12:47:35 -0700, Matthew Hall wrote:
I have another problem. RHEL 5 is missing this file:
/usr/lib/pkgconfig/glib-2.0.pc
So I created a faked out version based on Ubuntu's file with the version number modified to match the RHEL 5 glib RPM's version. [...] checking for GLIB... Requested 'glib-2.0 >= 2.10.1' but version of GLib is 2.4.7-1
configure: error: Package requirements (glib-2.0 >= 2.10.1 gmodule-2.0) were not met: [...] So apparently RHEL 5 has too old of a glib too. I assume I must have missed it last time because it failed on PCRE before it got to glib and the script stopped running.
Are you sure you're having correct packages installed? Namely, glib2-devel.
[root@dev32 ~]# rpm -qf /usr/lib/pkgconfig/glib-2.0.pc glib2-devel-2.12.3-4.el5_3.1 [root@dev32 ~]#
this should be enough, at least in the configure script syslog-ng is looking for GLIB_MIN_VERSION="2.10.1" this may or may not be correct. -- Bazsi
Sorry, I didn't notice this e-mail until now. On Fri, 2010-10-01 at 10:02 -0700, Matthew Hall wrote:
Any updates on this one? I would really like to know:
1) how / where to get Bazsi's 3.2 tree
http://git.balabit.hu/ you can get a "snapshot" of any of the commits. But I'm compiling 3.2beta1 right now, and I plan to publish binaries this time.
2) how to compile it for RHEL 4/5 using a newer OS like Ubuntu 10.04 LTS
well, it is usually the other way around. compile it on RHEL5 to get a working binary on Ubuntu 10.04.
3) how to install it once it has been compiled 4) i.e. can binaries be compiled, tree copied, then make install, or what???
What we do is that we compile _all_ external dependencies into /opt/syslog-ng with an rpath setting that points to /opt/syslog-ng/lib Then you only need to copy the /opt/syslog-ng/ tree to move it to another computer.
I am dead in the water until I can figure this out, because I need patternize working before I can start to process logs and get any results.
-- Bazsi
On Mon, 2010-09-27 at 09:32 -0500, Martin Holste wrote:
I'm on Ubuntu lucid trying to compile the latest git version and I'm getting "./.libs/libsyslog-ng.so: undefined reference to `block_ref_debug'" etc. when it tries to do the final ld command unless --enable-debug is on. --enable-mixed-linking works fine with --enable-debug, though I see a ton of dependencies with ldd, so that must be pretty far from a static link.
hmmm... it shouldn't do that, block_ref_debug is only referenced if enable-debug is defined. let me try a non-debug build... it built successfully, and block_ref_debug is not an undefined symbol: $ nm .libs/block-ref-parser.o U _GLOBAL_OFFSET_TABLE_ 0000000000000000 D block_def_keywords 0000000000000000 T block_ref_error 0000000000000020 d block_ref_keywords 0000000000000050 T block_ref_lex U block_ref_parse 0000000000000000 D block_ref_parser U cfg_lexer_get_context_description U cfg_lexer_lex U report_syntax_error Isn't it possible that you switched to non-debug build and the makefiles failed to recompile block-ref-parser.c ? Can you try a make clean first? -- Bazsi
On Fri, 2010-09-24 at 13:53 -0700, Matthew Hall wrote:
Further investigation...
The Balabit RPMs must be built specially because the daemon looks mostly statically linked. Is there some secret to building code for old Linuxes like the BalaBit RPMs?
it is not secret :) we build all code on Debian systems (even the code in the rpm). We're using --enable-mixed-linking configure option and it is built by the code in tgz2build/rules. -- Bazsi
Hi, On 09/24/2010 07:57 PM, Matthew Hall wrote:
I wondered if the memory leaks you said existed in the old version had been fixed, you did not say one way or the other in your mail.
Most of the major memleaks are fixed, yes. Valgrind still shows some problems I couldn't fix, but they're either minor (a couple of Ks compared to the ~1G test run I checked it with) or only occur at the end of the process: pretty much the whole struct containing the patterns is leaked when the program ends. As it only happens right before pdbtool exits, I didn't really care about it so far, but I might fix it in the future for the sake of general neatness, but it shouldn't affect the memory usage of the tool. The bigger problem is that the memory usage of patternize is, while being linear to the number of loglines, still huge. It could be optimized here&there, maybe even up to being 30-50% more effective and this is something I'm planning to do, but the main problem is that it needs to read everything into memory. I'm trying to figure out how to avoid this or at least how to make it degrade more gracefully when running out of physical RAM than start swapping which slows down things terribly. The core of the problem is that as it goes over the loglines, it needs to be able to look up the already collected words/patterns to find out which words/patterns are frequent to be able to create the final patterns. Maybe it'd be possible to use some disk-backed solution that writes out things when they couldn't fit into physical memory, but it woudn't perform much better than swapping as "frequent words" in loglines are really rare and we'd end up touching the written-to-disk part of the database all the time which would ruin the performance... Anyway, I'm just thinking loud :) What I'm really trying to say here is that unless some miracle happens, the memory usage won't improve drastically, and it's because of a conceptual problem, not memory leaks :(
I also wonder if anybody at Balabit could tell me how to build a copy of your Git tree on RHEL 4 or RHEL 5. I get problems because the PCRE is too old but when I switch to new PCRE, PCRE will not build because the autotools and pkg-config are too old.
It's a problem for me because unfortunately my company only supports RHEL here and otherwise I have to run it in an Ubuntu 10.04 or Debian VM with way too little memory for the tool to run right.
Would it be possible to build a version of your tree for RHEL 4 or 5?
Regarding this I'll have to refer you to other guys here -- I've personally never tried to compile syslog-ng on anything but Ubuntus. I've sent in the code to our internal buildsystem but because patternize introduces a new dependency (libuuid for generating the pattern ids) the compilation has failed and I did not want to mess with the builders without asking the guys managing them. I'll try to ask around tomorrow and get you an RPM or at least a more usable answer with some tips :) greets, Peter
participants (6)
-
Balazs Scheidler
-
Jakub Jankowski
-
Martin Holste
-
Matthew Hall
-
PATRICK HEMMER
-
Peter Gyongyosi