syslog-ng large NVTable patch
Hi, as requested by bazsi in the forums http://www.syslog.org/forum/index.php?topic=1218.0 here's a patch against syslog-ng enabling it to handle log messages larger than the current 256k limit. This is done by extending NVTable's data structures from 16 to 32 bit. The patch also contains simple code for automatic resizing of the receive buffer. 'log_msg_size' is still honoured, though. I have been using this to regularly receive multi-MB XML messages and process them into log files. I developed the patch against 3.3.1, but also tested it with 3.3.3 and on x64 as well. 'make check' runs fine, too -- but still, use at your own risk. Peter
On Tue, 2012-01-10 at 13:14 +0100, Peter Eisenlohr wrote:
Hi,
as requested by bazsi in the forums
http://www.syslog.org/forum/index.php?topic=1218.0
here's a patch against syslog-ng enabling it to handle log messages larger than the current 256k limit. This is done by extending NVTable's data structures from 16 to 32 bit. The patch also contains simple code for automatic resizing of the receive buffer. 'log_msg_size' is still honoured, though.
I have been using this to regularly receive multi-MB XML messages and process them into log files.
I developed the patch against 3.3.1, but also tested it with 3.3.3 and on x64 as well. 'make check' runs fine, too -- but still, use at your own risk.
Thanks for posting this here. I'll have to read this again when I have a little bit more thinking time and I'll check how the structures have grown and if there's padding or anything like that. Otherwise it looks nice. Thanks again. -- Bazsi
Hi, Thanks for your contribution. I still made some additional adjustments to your patch, eliminated the shifts used for size/len variables (which was used to stuff 256k into 16 bit variables). This became unneeded as 32 bit can be more than enough. I still added an artifical limit to make NVTable instances less than 256MB, hope that's still enough for your usecase. The patches are not yet on master, but are available in the "32bit-nvtable" branch in my github repo. Also, I hope you could sign off your patch, as that's needed for proper attribution. This is the same policy that the Linux kernel uses, nothing but something similar like: Signed-off-by: Peter Eisenlohr <peter.eisenlohr@inform-software.com> beside the patch, so that I can include it in the commit message. This change caused a slightly measurable drop in performance in one of my microbenchmark, but nothing really intolerable. The per-value overhead in NVTable became 12 bytes instead of the previous 6, but I didn't have any better idea, rather than imposing stricter limits like I did previously. Also, I'd welcome any feedback you might have on the NVTable code, it's a tricky one at best, and you really went ahead and did things right. Was it easy to understand/modify? Thanks again, On Tue, 2012-01-10 at 13:14 +0100, Peter Eisenlohr wrote:
Hi,
as requested by bazsi in the forums
http://www.syslog.org/forum/index.php?topic=1218.0
here's a patch against syslog-ng enabling it to handle log messages larger than the current 256k limit. This is done by extending NVTable's data structures from 16 to 32 bit. The patch also contains simple code for automatic resizing of the receive buffer. 'log_msg_size' is still honoured, though.
I have been using this to regularly receive multi-MB XML messages and process them into log files.
I developed the patch against 3.3.1, but also tested it with 3.3.3 and on x64 as well. 'make check' runs fine, too -- but still, use at your own risk.
Peter ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
Hi Balazs, thanks for taking this on. 31.01.2012 18:22, Balazs Scheidler:
The patches are not yet on master, but are available in the "32bit-nvtable" branch in my github repo.
I'm having trouble accessing github because I'm behind a proxy, but on the other hand, this might just be my lack of experience with git. # git clone https://github.com/bazsi/syslog-ng-3.4.git So I currently don't know how to get the updated version of the patch. If that is enough, I could 'sign' my original positing of the patch, though. Otherwise, I will try from home later.
Also, I'd welcome any feedback you might have on the NVTable code, it's a tricky one at best, and you really went ahead and did things right. Was it easy to understand/modify?
Frankly: No :) I understand that this was programmed with efficiency in mind, but this also results in a couple of constructs which would not be needed for a generic (but less efficient) key-value-storage. When making my changes, I ran into a couple of bugs which were very hard to debug: Putting the first value into the table seemingly succeeded, but silently overwrote some other part of the table -- which I would notice only much later. I ended up writing check_internal_structure() and dump_table() functions. Getting rid of the shifting arithemtics makes the code more simple. I did not do that because I wanted to keep the patch as small as possible. Peter
Peter Eisenlohr <peter.eisenlohr@inform-software.com> writes:
Hi Balazs,
thanks for taking this on.
31.01.2012 18:22, Balazs Scheidler:
The patches are not yet on master, but are available in the "32bit-nvtable" branch in my github repo.
I'm having trouble accessing github because I'm behind a proxy, but on the other hand, this might just be my lack of experience with git.
# git clone https://github.com/bazsi/syslog-ng-3.4.git
Is it the https that causes problems? There's also a read-only version of the repo available over the git protocol: git://github.com/bazsi/syslog-ng-3.4.git -- |8]
Am 06.02.2012 17:12, schrieb Gergely Nagy:
Peter Eisenlohr <peter.eisenlohr@inform-software.com> writes:
Hi Balazs,
thanks for taking this on.
31.01.2012 18:22, Balazs Scheidler:
The patches are not yet on master, but are available in the "32bit-nvtable" branch in my github repo.
I'm having trouble accessing github because I'm behind a proxy, but on the other hand, this might just be my lack of experience with git.
# git clone https://github.com/bazsi/syslog-ng-3.4.git
Is it the https that causes problems? There's also a read-only version of the repo available over the git protocol: git://github.com/bazsi/syslog-ng-3.4.git
It's a mandatory HTTP proxy, no other traffic to the internet is allowed. Still don't know what's wrong with the https, though. Anyway, I tried to amend/signoff the patches at home. Hopefully I did this right, the result is attached (just added Signed-Off to what was in the 32bit-nvtable branch). Still coming to grips with git... Peter
Thanks for the signoffs, merged this time to 3.4 master. On Sun, 2012-02-12 at 19:31 +0100, Peter Eisenlohr wrote:
Am 06.02.2012 17:12, schrieb Gergely Nagy:
Peter Eisenlohr <peter.eisenlohr@inform-software.com> writes:
Hi Balazs,
thanks for taking this on.
31.01.2012 18:22, Balazs Scheidler:
The patches are not yet on master, but are available in the "32bit-nvtable" branch in my github repo.
I'm having trouble accessing github because I'm behind a proxy, but on the other hand, this might just be my lack of experience with git.
# git clone https://github.com/bazsi/syslog-ng-3.4.git
Is it the https that causes problems? There's also a read-only version of the repo available over the git protocol: git://github.com/bazsi/syslog-ng-3.4.git
It's a mandatory HTTP proxy, no other traffic to the internet is allowed. Still don't know what's wrong with the https, though.
Anyway, I tried to amend/signoff the patches at home. Hopefully I did this right, the result is attached (just added Signed-Off to what was in the 32bit-nvtable branch). Still coming to grips with git...
Peter ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
participants (3)
-
Balazs Scheidler
-
Gergely Nagy
-
Peter Eisenlohr