[syslog-ng] query on using throttle in syslog-ng.conf file

[SIMON BABY]

Hello Balaza,

Can you please suggest if I could use flow control flags with file
destination .

Thank you for helping

Regards
Simon
On Sunday, February 7, 2021, SIMON BABY <simonkbaby at gmail.com> wrote:

> Hi Balaza,
>
> Thank you so much for helping.
> My end target is a boot CF card which is a SATA device operating in AHCI
> mode (ATA device). The log file target in the syslog configuration is
> writing's on this device.
>
> What is the best option  in my case to control the sending rate of writing
> to the file?
> During heavy writing into the log, the device driver on the CF card could
> not handle it and eventually the ATA link was broken. So I am trying to
> control the rate of message writing into the file.
> Can you please suggest the best way to limit the rate of messages
> writing into the file with syslog-ng.
>
>
> Thank you once again for your time and helping.
>
> Regards
> Simon
>
>
>
> On Sun, Feb 7, 2021 at 1:46 AM Balazs Scheidler <bazsi77 at gmail.com> wrote:
>
>> Hi,
>>
>> On Sun, Feb 7, 2021, 07:18 SIMON BABY <simonkbaby at gmail.com> wrote:
>>
>>> Hello Team,
>>>
>>> I am new to this group and I have a query on adding the throttle
>>> attribute in syslog-ng.conf file. My target is little slow to process all
>>> the messages sent by the sender and sometimes  the link connected to the
>>> target device is broken. I am thinking of slowing down the sender by adding
>>> the throttle attribute.
>>> I have the below queries:
>>>
>>> 1) What exactly the throttle confoguration does?
>>>
>>
>> It limits the number of syslog messages to be sent to the device per
>> second.
>>
>> The implementation allows short spikes of traffic where the short term
>> rate is higher, but over a few second the average stabilizes to the value
>> specified.
>>
>> It uses a tbf like algorithm.
>>
>>
>> 2) What does throttle(500) mean ? will it send 500 Bytes per second or
>>> 500 messages per second? What does the message here mean ? Can it be the
>>> entire message sent by the application ? Is there an upper limit and lower
>>> limit ?
>>>
>>
>> 500 messages, not bytes. The maximum message size can be controlled using
>> the log-msg-size() option.
>>
>>
>> 3) Any side effect of my system if I am going to use throttle().
>>>
>>
>> If your input rate is higher than the output, syslog-ng would either need
>> to store the incoming messages (memory or disk), backpressure to the source
>> if possible (using flow control and a tcp based transport) or drop them.
>>
>> There are a number of options that control this behavior.
>>
>> flags(flow-control) to turn on flow control on a log path
>>
>> log-fifo-size() for controlling the memory buffer size at the destination
>>
>> disk-buffer() for allowing the excess to overflow to disk
>>
>> transport(tcp) or transport(tls) on the source to select the transport
>> protocol
>>
>>
>> 4) Any other method  in syslog-ng to delay the logs sending at the sender?
>>>
>>
>> Depending on your use-case a flow controlled path end-to-end
>> (application, client-syslog-ng, server syslog-ng, final destination) could
>> work too. In that case, syslog-ng would automatically converge to the
>> amount of messages the destination is able to consume.
>>
>> 5) My destination configuration is below. Is it a valid configuration ?
>>>
>>>
>>> destination logFiler { file("/var/log/wq.log"
>>>
>>>     template("${FULLDATE}${TZ} ${HOST} ${PROGRAM} [$LEVEL] ${MSG}\n")
>>>
>>>     template_escape(yes)
>>>
>>>     throttle(500));};
>>>
>>
>> This is, however this is a file() destination, where the throttle option
>> may have limited use.
>>
>>
>>>
>>> Thank you for your time.
>>>
>>> Regards
>>> Simon
>>>
>>> ____________________________________________________________
>>> __________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support/documentation/?
>>> product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?
>> product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20210208/e7e866af/attachment.html>