[syslog-ng] syslog 1024 byte limit work around

[Corey Davelaar]

I'm referring to syslog messages over the wire.  Even at default MTU sizes of 1500 bytes an 8K syslog message would need to be split up, unless jumbo frames are enabled.  And even then, if a 64K syslog message size was enabled, a jumbo frame would be cut off at 9K.  By default in my testing of rsyslog sending remotely to a server running syslog-ng, there seems to be a network message size option of 1024 bytes, sticking to the RFC standard.


I'm OK with leaving that as is, I'm mostly looking for an option to enable a receive buffer for the syslog-ng server that can combine the messages if it sees a continuation of a specific 8K message over a tcp connection.  The local rsyslog server seems to honor the log_msg_size() option of 8K as those messages are all on one line, but the network pcaps show it in two different packets, the first one maxed out at 1024 bytes.


It's probably more of an rsyslog server setting on the remote server, but I thought I'd run it by this group.

________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Sandor Geller <sandor.geller at ericsson.com>
Sent: Wednesday, May 31, 2017 10:30:36 AM
To: syslog-ng at lists.balabit.hu
Subject: Re: [syslog-ng] syslog 1024 byte limit work around

Hi,

Could happen however the default of log_msg_size() was 8192 for a decade and it has been raised to 64k recently so maybe multiline logs are involved too.

The OP should provide more details.

Regards,

Sandor

On 05/31/2017 04:57 PM, Evan Rempel wrote:
This might be resolved by setting the syslog-ng configuration option of log_msg_size to a larger value. We use

log_msg_size(32768);

Evan.

On 05/31/2017 05:32 AM, Corey Davelaar wrote:

Hello,


I'm new to the group and hopefully it's not a repeat question.  I was guided to this mailing list by a balabit tech support rep.


We use splunk on the back end of syslog-ng and we have some syslog sources that utilize very long syslog messages, that end up being over 1024 bytes.  We switched from udp to tcp as the transport and now we are seeing the tail end of the syslog messages, but splunk gets confused by the inputs and treats each line as a separate syslog message and completely unrelated to the previous one, because that�s how syslog-ng is treating it.

I was wondering if there is a way to tell syslog-ng that if a new packet doesn�t start with a standard syslog timestamp to treat it as a continuation of the previous message or somehow combine it with a previous message, utilizing a buffer type of setting.

Thanks,

Corey




______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170531/b7677a42/attachment-0001.html>