[syslog-ng] syslog 1024 byte limit work around

[Corey Davelaar]

As I said, it could be something that I'd need to set with rsyslog on the remote side, but I was hoping for something to be done on the syslog-ng side.  You seem to be aware of the limitations, and I'm seeing the results of scenario #2.  So the sending rsyslog is splitting it up and sending it as a separate syslog message, thus the field parsers in splunk get confused because the syslog messages are split up.


I'll take a closer look at the sending box, it doesn't sound like there will be a way around this on syslog-ng.

________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Evan Rempel <erempel at uvic.ca>
Sent: Wednesday, May 31, 2017 2:18:46 PM
To: syslog-ng at lists.balabit.hu
Subject: Re: [syslog-ng] syslog 1024 byte limit work around

You are confusing a lot of things.

Packet sizes (MTU) her 100% separate from syslog message sizes. The packet sizes can be left out of this discussion completely.
Any message that is split in order to meet the MTU size is reassembled at the receiving end BEFORE it is passed to the application. This means that syslog-ng would be unaware of any MTU size split or reassembly that took place.

If you are observing syslog messages that are split, then it is one of two things:

1. The source application is logging messages with newline characters (multi-line messages).
2. The sending syslog application is splitting the messages when it sends them on the wire.


#1 is only a problem in when it reaches the network AND when TCP is being used. Technically it is a difference between datagram and stream based protocols. UDP is a datagram protocol and TCP is a stream protocol. When a datagram protocol is used each datagram contains one message thus providing an inherent end of syslog message identifier. When stream protocols are used there is no implicit end of the syslog message, so a newline character is used to terminate a syslog message. When the message itself contains a newline, the receiving end of the stream protocol has no way to differentiate between the newline in the middle of the message and the newline at the end of the message, so it treats the data as multiple syslog messages.

I hope that explains it.

Evan.

On 05/31/2017 11:57 AM, Corey Davelaar wrote:

I'm referring to syslog messages over the wire.  Even at default MTU sizes of 1500 bytes an 8K syslog message would need to be split up, unless jumbo frames are enabled.  And even then, if a 64K syslog message size was enabled, a jumbo frame would be cut off at 9K.  By default in my testing of rsyslog sending remotely to a server running syslog-ng, there seems to be a network message size option of 1024 bytes, sticking to the RFC standard.


I'm OK with leaving that as is, I'm mostly looking for an option to enable a receive buffer for the syslog-ng server that can combine the messages if it sees a continuation of a specific 8K message over a tcp connection.  The local rsyslog server seems to honor the log_msg_size() option of 8K as those messages are all on one line, but the network pcaps show it in two different packets, the first one maxed out at 1024 bytes.


It's probably more of an rsyslog server setting on the remote server, but I thought I'd run it by this group.

________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu><mailto:syslog-ng-bounces at lists.balabit.hu> on behalf of Sandor Geller <sandor.geller at ericsson.com><mailto:sandor.geller at ericsson.com>
Sent: Wednesday, May 31, 2017 10:30:36 AM
To: syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>
Subject: Re: [syslog-ng] syslog 1024 byte limit work around

Hi,

Could happen however the default of log_msg_size() was 8192 for a decade and it has been raised to 64k recently so maybe multiline logs are involved too.

The OP should provide more details.

Regards,

Sandor

On 05/31/2017 04:57 PM, Evan Rempel wrote:
This might be resolved by setting the syslog-ng configuration option of log_msg_size to a larger value. We use

log_msg_size(32768);

Evan.

On 05/31/2017 05:32 AM, Corey Davelaar wrote:

Hello,


I'm new to the group and hopefully it's not a repeat question.  I was guided to this mailing list by a balabit tech support rep.


We use splunk on the back end of syslog-ng and we have some syslog sources that utilize very long syslog messages, that end up being over 1024 bytes.  We switched from udp to tcp as the transport and now we are seeing the tail end of the syslog messages, but splunk gets confused by the inputs and treats each line as a separate syslog message and completely unrelated to the previous one, because that�s how syslog-ng is treating it.

I was wondering if there is a way to tell syslog-ng that if a new packet doesn�t start with a standard syslog timestamp to treat it as a continuation of the previous message or somehow combine it with a previous message, utilizing a buffer type of setting.

Thanks,

Corey

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170531/1ed7fba2/attachment-0001.html>