[syslog-ng] syslog-ng large NVTable patch

[Balazs Scheidler]

Hi,

Thanks for your contribution.

I still made some additional adjustments to your patch, eliminated the
shifts used for size/len variables (which was used to stuff 256k into 16
bit variables). This became unneeded as 32 bit can be more than enough.

I still added an artifical limit to make NVTable instances less than
256MB, hope that's still enough for your usecase.

The patches are not yet on master, but are available in the
"32bit-nvtable" branch in my github repo.

Also, I hope you could sign off your patch, as that's needed for proper
attribution. This is the same policy that the Linux kernel uses, nothing
but something similar like:

Signed-off-by: Peter Eisenlohr <peter.eisenlohr at inform-software.com>

beside the patch, so that I can include it in the commit message.

This change caused a slightly measurable drop in performance in one of
my microbenchmark, but nothing really intolerable. The per-value
overhead in NVTable became 12 bytes instead of the previous 6, but I
didn't have any better idea, rather than imposing stricter limits like I
did previously.

Also, I'd welcome any feedback you might have on the NVTable code, it's
a tricky one at best, and you really went ahead and did things right.
Was it easy to understand/modify?

Thanks again,


On Tue, 2012-01-10 at 13:14 +0100, Peter Eisenlohr wrote:
> Hi,
> 
> as requested by bazsi in the forums
> 
>   http://www.syslog.org/forum/index.php?topic=1218.0
> 
> here's a patch against syslog-ng enabling it to handle log messages
> larger than the current 256k limit. This is done by extending NVTable's
> data structures from 16 to 32 bit. The patch also contains simple code
> for automatic resizing of the receive buffer. 'log_msg_size' is still
> honoured, though.
> 
> I have been using this to regularly receive multi-MB XML messages and
> process them into log files.
> 
> I developed the patch against 3.3.1, but also tested it with 3.3.3 and
> on x64 as well. 'make check' runs fine, too -- but still, use at your
> own risk.
> 
>   Peter
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> 

-- 
Bazsi