[syslog-ng] query on using throttle in syslog-ng.conf file

Tue Feb 9 07:54:15 UTC 2021

Hi,

file() destinations operate in a mode called "soft-flow-control". This is
almost the same as flow control, except that we release the flow controlled
nature of these desetinations if the file cannot be created or the disk is
full. This is to avoid a completely locked up system in case of file
related failures.

On the other hand, as long as the file exists and is writable, it is
automatically in flow-control mode, which means that backpressure is
provided to the source, which should slow down to the rate the file is able
to accept messages.

The question is how the source handles this backpressure, this depends on
the transport protocol that specific source is using:

   - udp() would simply stop fetching datagrams from the socket, meaning
   that once the kernel's receive buffer is full, UDP messages will start
   dropping, even if the datagrams themselves are delivered to the host.
   - tcp() would also stop fetching, but this means that the TCP protocol
   would propagate this back to the actual sender, which should slow down too.
   - any other combination (tls, rfc5424 with tcp based transports work
   like tcp above)
   - local transports (unix-dgram, unix-stream) would _both_ propagate this
   backpressure back
   - file sources on the local host: would stop reading at a specific
   position and restart when the backpressure is released.

If the source is on a remote host (e.g. another syslog-ng instance or
another syslog implementation), the settings of that instance control how
it reacts to the back-pressure provided by syslog-ng and the transport.

With syslog-ng, if the destination is a tcp based transport:

   - you need to specify flags(flow-control) on the log path leading to the
   tcp() destination, this would enable this back-pressure propagation to the
   source of the sender syslog-ng instance (soft-flow-control is not effective
   here, as this is not a file destination)
   - the backpressure to TCP would propagate back to the source on the
   sender syslog-ng instance.

And so on and so on.

Once the flow-control feedback loop establishes, we still need to have some
buffer space to store the in-flight messages, thus each hop on the path
between the actual source and the final file destination uses a memory or
disk based buffer.

On Mon, Feb 8, 2021 at 10:17 PM SIMON BABY <simonkbaby at gmail.com> wrote:

> Hello Balaza,
>
> Can you please suggest if I could use flow control flags with file
> destination .
>
> Thank you for helping
>
> Regards
> Simon
> On Sunday, February 7, 2021, SIMON BABY <simonkbaby at gmail.com> wrote:
>
>> Hi Balaza,
>>
>> Thank you so much for helping.
>> My end target is a boot CF card which is a SATA device operating in AHCI
>> mode (ATA device). The log file target in the syslog configuration is
>> writing's on this device.
>>
>> What is the best option  in my case to control the sending rate of
>> writing to the file?
>> During heavy writing into the log, the device driver on the CF card could
>> not handle it and eventually the ATA link was broken. So I am trying to
>> control the rate of message writing into the file.
>> Can you please suggest the best way to limit the rate of messages
>> writing into the file with syslog-ng.
>>
>>
>> Thank you once again for your time and helping.
>>
>> Regards
>> Simon
>>
>>
>>
>> On Sun, Feb 7, 2021 at 1:46 AM Balazs Scheidler <bazsi77 at gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> On Sun, Feb 7, 2021, 07:18 SIMON BABY <simonkbaby at gmail.com> wrote:
>>>
>>>> Hello Team,
>>>>
>>>> I am new to this group and I have a query on adding the throttle
>>>> attribute in syslog-ng.conf file. My target is little slow to process all
>>>> the messages sent by the sender and sometimes  the link connected to the
>>>> target device is broken. I am thinking of slowing down the sender by adding
>>>> the throttle attribute.
>>>> I have the below queries:
>>>>
>>>> 1) What exactly the throttle confoguration does?
>>>>
>>>
>>> It limits the number of syslog messages to be sent to the device per
>>> second.
>>>
>>> The implementation allows short spikes of traffic where the short term
>>> rate is higher, but over a few second the average stabilizes to the value
>>> specified.
>>>
>>> It uses a tbf like algorithm.
>>>
>>>
>>> 2) What does throttle(500) mean ? will it send 500 Bytes per second or
>>>> 500 messages per second? What does the message here mean ? Can it be the
>>>> entire message sent by the application ? Is there an upper limit and lower
>>>> limit ?
>>>>
>>>
>>> 500 messages, not bytes. The maximum message size can be controlled
>>> using the log-msg-size() option.
>>>
>>>
>>> 3) Any side effect of my system if I am going to use throttle().
>>>>
>>>
>>> If your input rate is higher than the output, syslog-ng would either
>>> need to store the incoming messages (memory or disk), backpressure to the
>>> source if possible (using flow control and a tcp based transport) or drop
>>> them.
>>>
>>> There are a number of options that control this behavior.
>>>
>>> flags(flow-control) to turn on flow control on a log path
>>>
>>> log-fifo-size() for controlling the memory buffer size at the destination
>>>
>>> disk-buffer() for allowing the excess to overflow to disk
>>>
>>> transport(tcp) or transport(tls) on the source to select the transport
>>> protocol
>>>
>>>
>>> 4) Any other method  in syslog-ng to delay the logs sending at the
>>>> sender?
>>>>
>>>
>>> Depending on your use-case a flow controlled path end-to-end
>>> (application, client-syslog-ng, server syslog-ng, final destination) could
>>> work too. In that case, syslog-ng would automatically converge to the
>>> amount of messages the destination is able to consume.
>>>
>>> 5) My destination configuration is below. Is it a valid configuration ?
>>>>
>>>>
>>>> destination logFiler { file("/var/log/wq.log"
>>>>
>>>>     template("${FULLDATE}${TZ} ${HOST} ${PROGRAM} [$LEVEL] ${MSG}\n")
>>>>
>>>>     template_escape(yes)
>>>>
>>>>     throttle(500));};
>>>>
>>>
>>> This is, however this is a file() destination, where the throttle option
>>> may have limited use.
>>>
>>>
>>>>
>>>> Thank you for your time.
>>>>
>>>> Regards
>>>> Simon
>>>>
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>

-- 
Bazsi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20210209/94247796/attachment-0001.html>