[syslog-ng] syslog 1024 byte limit work around

[Evan Rempel]

You are confusing a lot of things.

Packet sizes (MTU) her 100% separate from syslog message sizes. The 
packet sizes can be left out of this discussion completely.
Any message that is split in order to meet the MTU size is reassembled 
at the receiving end BEFORE it is passed to the application. This means 
that syslog-ng would be unaware of any MTU size split or reassembly that 
took place.

If you are observing syslog messages that are split, then it is one of 
two things:

1. The source application is logging messages with newline characters 
(multi-line messages).
2. The sending syslog application is splitting the messages when it 
sends them on the wire.


#1 is only a problem in when it reaches the network AND when TCP is 
being used. Technically it is a difference between datagram and stream 
based protocols. UDP is a datagram protocol and TCP is a stream 
protocol. When a datagram protocol is used each datagram contains one 
message thus providing an inherent end of syslog message identifier. 
When stream protocols are used there is no implicit end of the syslog 
message, so a newline character is used to terminate a syslog message. 
When the message itself contains a newline, the receiving end of the 
stream protocol has no way to differentiate between the newline in the 
middle of the message and the newline at the end of the message, so it 
treats the data as multiple syslog messages.

I hope that explains it.

Evan.

On 05/31/2017 11:57 AM, Corey Davelaar wrote:
>
> I'm referring to syslog messages over the wire.  Even at default MTU 
> sizes of 1500 bytes an 8K syslog message would need to be split up, 
> unless jumbo frames are enabled.  And even then, if a 64K syslog 
> message size was enabled, a jumbo frame would be cut off at 9K.  By 
> default in my testing of rsyslog sending remotely to a server running 
> syslog-ng, there seems to be a network message size option of 1024 
> bytes, sticking to the RFC standard.
>
>
> I'm OK with leaving that as is, I'm mostly looking for an option to 
> enable a receive buffer for the syslog-ng server that can combine the 
> messages if it sees a continuation of a specific 8K message over a tcp 
> connection.  The local rsyslog server seems to honor the 
> log_msg_size() option of 8K as those messages are all on one line, but 
> the network pcaps show it in two different packets, the first one 
> maxed out at 1024 bytes.
>
>
> It's probably more of an rsyslog server setting on the remote server, 
> but I thought I'd run it by this group.
>
> ------------------------------------------------------------------------
> *From:* syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of 
> Sandor Geller <sandor.geller at ericsson.com>
> *Sent:* Wednesday, May 31, 2017 10:30:36 AM
> *To:* syslog-ng at lists.balabit.hu
> *Subject:* Re: [syslog-ng] syslog 1024 byte limit work around
> Hi,
>
> Could happen however the default of log_msg_size() was 8192 for a 
> decade and it has been raised to 64k recently so maybe multiline logs 
> are involved too.
>
> The OP should provide more details.
>
> Regards,
>
> Sandor
>
> On 05/31/2017 04:57 PM, Evan Rempel wrote:
>> This might be resolved by setting the syslog-ng configuration option 
>> of log_msg_size to a larger value. We use
>>
>> log_msg_size(32768);
>>
>> Evan.
>>
>> On 05/31/2017 05:32 AM, Corey Davelaar wrote:
>>>
>>> Hello,
>>>
>>>
>>> I'm new to the group and hopefully it's not a repeat question.  I 
>>> was guided to this mailing list by a balabit tech support rep.
>>>
>>>
>>> We use splunk on the back end of syslog-ng and we have some syslog 
>>> sources that utilize very long syslog messages, that end up being 
>>> over 1024 bytes.  We switched from udp to tcp as the transport and 
>>> now we are seeing the tail end of the syslog messages, but splunk 
>>> gets confused by the inputs and treats each line as a separate 
>>> syslog message and completely unrelated to the previous one, because 
>>> that�s how syslog-ng is treating it.
>>>
>>> I was wondering if there is a way to tell syslog-ng that if a new 
>>> packet doesn�t start with a standard syslog timestamp to treat it as 
>>> a continuation of the previous message or somehow combine it with a 
>>> previous message, utilizing a buffer type of setting.
>>>
>>> Thanks,
>>>
>>> Corey

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170531/86c25f5c/attachment.html>