[syslog-ng] query on using throttle in syslog-ng.conf file

Wed Feb 10 03:19:47 UTC 2021

Thank you so much . I will test with throttle and disk buffer . Thank you
for your time

On Tuesday, February 9, 2021, Balazs Scheidler <bazsi77 at gmail.com> wrote:

> Yes, throttle could work. You just have to make sure that your file
> destination has enough buffer space to avoid message loss if the apps emit
> more messages per second than the throttle value. In this case the question
> could arise how much ram you allocate to this purpose.
>
> Disk-buffer() could be of use, as long as the queue files reside on a
> normal, high performance disk, I am not 100% sure that disk buffer works
> with file destinations though (they should but I am pretty sure no one uses
> them that way).
>
> On Tue, Feb 9, 2021, 17:35 SIMON BABY <simonkbaby at gmail.com> wrote:
>
>> Thank you so much . I understand that file destinations are operated on
>> default soft flow control . My destination and source are running on the
>> same system. I understand the hard flow control will not help here. Do you
>> suggest any other method like throttle()  or any other method help here .
>>
>> My sources are all the application processes running in my Linux based
>> system and destination is a file on a boot CF card mounted on the same
>> Linux system. I see during heavy logging, the CF card could not handle the
>> high writing rate and eventually it fails to respond and the link connected
>> to the ata driver(kernel) is broken. So I am looking in syslog if I could
>> decrease the logging rate .
>>
>> Thank you so much
>>
>> Regards
>> Simon
>>
>>
>>
>> On Monday, February 8, 2021, Balazs Scheidler <bazsi77 at gmail.com> wrote:
>>
>>> Hi,
>>>
>>> file() destinations operate in a mode called "soft-flow-control". This
>>> is almost the same as flow control, except that we release the flow
>>> controlled nature of these desetinations if the file cannot be created or
>>> the disk is full. This is to avoid a completely locked up system in case of
>>> file related failures.
>>>
>>> On the other hand, as long as the file exists and is writable, it is
>>> automatically in flow-control mode, which means that backpressure is
>>> provided to the source, which should slow down to the rate the file is able
>>> to accept messages.
>>>
>>> The question is how the source handles this backpressure, this depends
>>> on the transport protocol that specific source is using:
>>>
>>>
>>>    - udp() would simply stop fetching datagrams from the socket,
>>>    meaning that once the kernel's receive buffer is full, UDP messages will
>>>    start dropping, even if the datagrams themselves are delivered to the host.
>>>    - tcp() would also stop fetching, but this means that the TCP
>>>    protocol would propagate this back to the actual sender, which should slow
>>>    down too.
>>>    - any other combination (tls, rfc5424 with tcp based transports work
>>>    like tcp above)
>>>    - local transports (unix-dgram, unix-stream) would _both_ propagate
>>>    this backpressure back
>>>    - file sources on the local host: would stop reading at a specific
>>>    position and restart when the backpressure is released.
>>>
>>> If the source is on a remote host (e.g. another syslog-ng instance or
>>> another syslog implementation), the settings of that instance control how
>>> it reacts to the back-pressure provided by syslog-ng and the transport.
>>>
>>> With syslog-ng, if the destination is a tcp based transport:
>>>
>>>    - you need to specify flags(flow-control) on the log path leading to
>>>    the tcp() destination, this would enable this back-pressure propagation to
>>>    the source of the sender syslog-ng instance (soft-flow-control is not
>>>    effective here, as this is not a file destination)
>>>    - the backpressure to TCP would propagate back to the source on the
>>>    sender syslog-ng instance.
>>>
>>> And so on and so on.
>>>
>>> Once the flow-control feedback loop establishes, we still need to have
>>> some buffer space to store the in-flight messages, thus each hop on the
>>> path between the actual source and the final file destination uses a memory
>>> or disk based buffer.
>>>
>>>
>>>
>>> On Mon, Feb 8, 2021 at 10:17 PM SIMON BABY <simonkbaby at gmail.com> wrote:
>>>
>>>> Hello Balaza,
>>>>
>>>> Can you please suggest if I could use flow control flags with file
>>>> destination .
>>>>
>>>> Thank you for helping
>>>>
>>>> Regards
>>>> Simon
>>>> On Sunday, February 7, 2021, SIMON BABY <simonkbaby at gmail.com> wrote:
>>>>
>>>>> Hi Balaza,
>>>>>
>>>>> Thank you so much for helping.
>>>>> My end target is a boot CF card which is a SATA device operating in
>>>>> AHCI mode (ATA device). The log file target in the syslog configuration is
>>>>> writing's on this device.
>>>>>
>>>>> What is the best option  in my case to control the sending rate of
>>>>> writing to the file?
>>>>> During heavy writing into the log, the device driver on the CF card
>>>>> could not handle it and eventually the ATA link was broken. So I am trying
>>>>> to control the rate of message writing into the file.
>>>>> Can you please suggest the best way to limit the rate of messages
>>>>> writing into the file with syslog-ng.
>>>>>
>>>>>
>>>>> Thank you once again for your time and helping.
>>>>>
>>>>> Regards
>>>>> Simon
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Feb 7, 2021 at 1:46 AM Balazs Scheidler <bazsi77 at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> On Sun, Feb 7, 2021, 07:18 SIMON BABY <simonkbaby at gmail.com> wrote:
>>>>>>
>>>>>>> Hello Team,
>>>>>>>
>>>>>>> I am new to this group and I have a query on adding the throttle
>>>>>>> attribute in syslog-ng.conf file. My target is little slow to process all
>>>>>>> the messages sent by the sender and sometimes  the link connected to the
>>>>>>> target device is broken. I am thinking of slowing down the sender by adding
>>>>>>> the throttle attribute.
>>>>>>> I have the below queries:
>>>>>>>
>>>>>>> 1) What exactly the throttle confoguration does?
>>>>>>>
>>>>>>
>>>>>> It limits the number of syslog messages to be sent to the device per
>>>>>> second.
>>>>>>
>>>>>> The implementation allows short spikes of traffic where the short
>>>>>> term rate is higher, but over a few second the average stabilizes to the
>>>>>> value specified.
>>>>>>
>>>>>> It uses a tbf like algorithm.
>>>>>>
>>>>>>
>>>>>> 2) What does throttle(500) mean ? will it send 500 Bytes per second
>>>>>>> or 500 messages per second? What does the message here mean ? Can it be the
>>>>>>> entire message sent by the application ? Is there an upper limit and lower
>>>>>>> limit ?
>>>>>>>
>>>>>>
>>>>>> 500 messages, not bytes. The maximum message size can be controlled
>>>>>> using the log-msg-size() option.
>>>>>>
>>>>>>
>>>>>> 3) Any side effect of my system if I am going to use throttle().
>>>>>>>
>>>>>>
>>>>>> If your input rate is higher than the output, syslog-ng would either
>>>>>> need to store the incoming messages (memory or disk), backpressure to the
>>>>>> source if possible (using flow control and a tcp based transport) or drop
>>>>>> them.
>>>>>>
>>>>>> There are a number of options that control this behavior.
>>>>>>
>>>>>> flags(flow-control) to turn on flow control on a log path
>>>>>>
>>>>>> log-fifo-size() for controlling the memory buffer size at the
>>>>>> destination
>>>>>>
>>>>>> disk-buffer() for allowing the excess to overflow to disk
>>>>>>
>>>>>> transport(tcp) or transport(tls) on the source to select the
>>>>>> transport protocol
>>>>>>
>>>>>>
>>>>>> 4) Any other method  in syslog-ng to delay the logs sending at the
>>>>>>> sender?
>>>>>>>
>>>>>>
>>>>>> Depending on your use-case a flow controlled path end-to-end
>>>>>> (application, client-syslog-ng, server syslog-ng, final destination) could
>>>>>> work too. In that case, syslog-ng would automatically converge to the
>>>>>> amount of messages the destination is able to consume.
>>>>>>
>>>>>> 5) My destination configuration is below. Is it a valid configuration
>>>>>>> ?
>>>>>>>
>>>>>>>
>>>>>>> destination logFiler { file("/var/log/wq.log"
>>>>>>>
>>>>>>>     template("${FULLDATE}${TZ} ${HOST} ${PROGRAM} [$LEVEL] ${MSG}\n")
>>>>>>>
>>>>>>>     template_escape(yes)
>>>>>>>
>>>>>>>     throttle(500));};
>>>>>>>
>>>>>>
>>>>>> This is, however this is a file() destination, where the throttle
>>>>>> option may have limited use.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Thank you for your time.
>>>>>>>
>>>>>>> Regards
>>>>>>> Simon
>>>>>>>
>>>>>>> ____________________________________________________________
>>>>>>> __________________
>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>> Documentation: http://www.balabit.com/support/documentation/?
>>>>>>> product=syslog-ng
>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>
>>>>>>> ____________________________________________________________
>>>>>> __________________
>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>> Documentation: http://www.balabit.com/support/documentation/?
>>>>>> product=syslog-ng
>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>
>>>>>> ____________________________________________________________
>>>> __________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation: http://www.balabit.com/support/documentation/?
>>>> product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>
>>> --
>>> Bazsi
>>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?
>> product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20210209/264a897b/attachment.html>