[syslog-ng] query on using throttle in syslog-ng.conf file

Balazs Scheidler bazsi77 at gmail.com
Sat Feb 20 22:25:21 UTC 2021 

On Fri, Feb 19, 2021, 23:35 SIMON BABY <simonkbaby at gmail.com> wrote:

> Hi Balaza,
> I have some more queries below:
>
> 1. You mentioned to make sure the file destination has enough buffer space
> to avoid message loss if the application emits more messages per second
> than the throttle value. So the remaining messages (after the throttle
> value) are stored in system RAM ? Is this memory allocated by syslog-ng
> process internally? or do we have any syslog-ng configuration parameter to
> allocate this much memory?
>

Yes. It is controlled by the log-fifo-size () option, can be specified at
the destination or globally.


> 2. Can I know from which version the disk-buffer feature was added in
> syslog-ng ? I am using version 3.0.8.
>

Can't remember, but more recent than that. 3.0.8 is ancient.


> 3. With a disk-buffer, the extra messages can be stored temporarily to any
> path (for example: /tmp/message.log). Will this temporary file get deleted
> after the complete transfer?
>

Disk buffer files are in a dedicated directory (/var/lib/syslog-ng), and
the files themselves are kept until the disk buffer is used, but the size
of the file is related to the amount of data being in the buffer. It's a
ring buffer on disk.

Once the buffer is empty the file size would be minimal, few kb.


> 4. Do we have any internal syslog-ng delay timer settings between messages
> other than throttle feature?
>

But really, what would you expect?


> Thank you for helping.
>
> Regards
> Simon
>
>
>
> On Tue, Feb 9, 2021 at 9:42 AM Balazs Scheidler <bazsi77 at gmail.com> wrote:
>
>> Yes, throttle could work. You just have to make sure that your file
>> destination has enough buffer space to avoid message loss if the apps emit
>> more messages per second than the throttle value. In this case the question
>> could arise how much ram you allocate to this purpose.
>>
>> Disk-buffer() could be of use, as long as the queue files reside on a
>> normal, high performance disk, I am not 100% sure that disk buffer works
>> with file destinations though (they should but I am pretty sure no one uses
>> them that way).
>>
>> On Tue, Feb 9, 2021, 17:35 SIMON BABY <simonkbaby at gmail.com> wrote:
>>
>>> Thank you so much . I understand that file destinations are operated on
>>> default soft flow control . My destination and source are running on the
>>> same system. I understand the hard flow control will not help here. Do you
>>> suggest any other method like throttle()  or any other method help here .
>>>
>>> My sources are all the application processes running in my Linux based
>>> system and destination is a file on a boot CF card mounted on the same
>>> Linux system. I see during heavy logging, the CF card could not handle the
>>> high writing rate and eventually it fails to respond and the link connected
>>> to the ata driver(kernel) is broken. So I am looking in syslog if I could
>>> decrease the logging rate .
>>>
>>> Thank you so much
>>>
>>> Regards
>>> Simon
>>>
>>>
>>>
>>> On Monday, February 8, 2021, Balazs Scheidler <bazsi77 at gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> file() destinations operate in a mode called "soft-flow-control". This
>>>> is almost the same as flow control, except that we release the flow
>>>> controlled nature of these desetinations if the file cannot be created or
>>>> the disk is full. This is to avoid a completely locked up system in case of
>>>> file related failures.
>>>>
>>>> On the other hand, as long as the file exists and is writable, it is
>>>> automatically in flow-control mode, which means that backpressure is
>>>> provided to the source, which should slow down to the rate the file is able
>>>> to accept messages.
>>>>
>>>> The question is how the source handles this backpressure, this depends
>>>> on the transport protocol that specific source is using:
>>>>
>>>>
>>>>    - udp() would simply stop fetching datagrams from the socket,
>>>>    meaning that once the kernel's receive buffer is full, UDP messages will
>>>>    start dropping, even if the datagrams themselves are delivered to the host.
>>>>    - tcp() would also stop fetching, but this means that the TCP
>>>>    protocol would propagate this back to the actual sender, which should slow
>>>>    down too.
>>>>    - any other combination (tls, rfc5424 with tcp based transports
>>>>    work like tcp above)
>>>>    - local transports (unix-dgram, unix-stream) would _both_ propagate
>>>>    this backpressure back
>>>>    - file sources on the local host: would stop reading at a specific
>>>>    position and restart when the backpressure is released.
>>>>
>>>> If the source is on a remote host (e.g. another syslog-ng instance or
>>>> another syslog implementation), the settings of that instance control how
>>>> it reacts to the back-pressure provided by syslog-ng and the transport.
>>>>
>>>> With syslog-ng, if the destination is a tcp based transport:
>>>>
>>>>    - you need to specify flags(flow-control) on the log path leading
>>>>    to the tcp() destination, this would enable this back-pressure propagation
>>>>    to the source of the sender syslog-ng instance (soft-flow-control is not
>>>>    effective here, as this is not a file destination)
>>>>    - the backpressure to TCP would propagate back to the source on the
>>>>    sender syslog-ng instance.
>>>>
>>>> And so on and so on.
>>>>
>>>> Once the flow-control feedback loop establishes, we still need to have
>>>> some buffer space to store the in-flight messages, thus each hop on the
>>>> path between the actual source and the final file destination uses a memory
>>>> or disk based buffer.
>>>>
>>>>
>>>>
>>>> On Mon, Feb 8, 2021 at 10:17 PM SIMON BABY <simonkbaby at gmail.com>
>>>> wrote:
>>>>
>>>>> Hello Balaza,
>>>>>
>>>>> Can you please suggest if I could use flow control flags with file
>>>>> destination .
>>>>>
>>>>> Thank you for helping
>>>>>
>>>>> Regards
>>>>> Simon
>>>>> On Sunday, February 7, 2021, SIMON BABY <simonkbaby at gmail.com> wrote:
>>>>>
>>>>>> Hi Balaza,
>>>>>>
>>>>>> Thank you so much for helping.
>>>>>> My end target is a boot CF card which is a SATA device operating in
>>>>>> AHCI mode (ATA device). The log file target in the syslog configuration is
>>>>>> writing's on this device.
>>>>>>
>>>>>> What is the best option  in my case to control the sending rate of
>>>>>> writing to the file?
>>>>>> During heavy writing into the log, the device driver on the CF card
>>>>>> could not handle it and eventually the ATA link was broken. So I am trying
>>>>>> to control the rate of message writing into the file.
>>>>>> Can you please suggest the best way to limit the rate of messages
>>>>>> writing into the file with syslog-ng.
>>>>>>
>>>>>>
>>>>>> Thank you once again for your time and helping.
>>>>>>
>>>>>> Regards
>>>>>> Simon
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sun, Feb 7, 2021 at 1:46 AM Balazs Scheidler <bazsi77 at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> On Sun, Feb 7, 2021, 07:18 SIMON BABY <simonkbaby at gmail.com> wrote:
>>>>>>>
>>>>>>>> Hello Team,
>>>>>>>>
>>>>>>>> I am new to this group and I have a query on adding the throttle
>>>>>>>> attribute in syslog-ng.conf file. My target is little slow to process all
>>>>>>>> the messages sent by the sender and sometimes  the link connected to the
>>>>>>>> target device is broken. I am thinking of slowing down the sender by adding
>>>>>>>> the throttle attribute.
>>>>>>>> I have the below queries:
>>>>>>>>
>>>>>>>> 1) What exactly the throttle confoguration does?
>>>>>>>>
>>>>>>>
>>>>>>> It limits the number of syslog messages to be sent to the device per
>>>>>>> second.
>>>>>>>
>>>>>>> The implementation allows short spikes of traffic where the short
>>>>>>> term rate is higher, but over a few second the average stabilizes to the
>>>>>>> value specified.
>>>>>>>
>>>>>>> It uses a tbf like algorithm.
>>>>>>>
>>>>>>>
>>>>>>> 2) What does throttle(500) mean ? will it send 500 Bytes per second
>>>>>>>> or 500 messages per second? What does the message here mean ? Can it be the
>>>>>>>> entire message sent by the application ? Is there an upper limit and lower
>>>>>>>> limit ?
>>>>>>>>
>>>>>>>
>>>>>>> 500 messages, not bytes. The maximum message size can be controlled
>>>>>>> using the log-msg-size() option.
>>>>>>>
>>>>>>>
>>>>>>> 3) Any side effect of my system if I am going to use throttle().
>>>>>>>>
>>>>>>>
>>>>>>> If your input rate is higher than the output, syslog-ng would either
>>>>>>> need to store the incoming messages (memory or disk), backpressure to the
>>>>>>> source if possible (using flow control and a tcp based transport) or drop
>>>>>>> them.
>>>>>>>
>>>>>>> There are a number of options that control this behavior.
>>>>>>>
>>>>>>> flags(flow-control) to turn on flow control on a log path
>>>>>>>
>>>>>>> log-fifo-size() for controlling the memory buffer size at the
>>>>>>> destination
>>>>>>>
>>>>>>> disk-buffer() for allowing the excess to overflow to disk
>>>>>>>
>>>>>>> transport(tcp) or transport(tls) on the source to select the
>>>>>>> transport protocol
>>>>>>>
>>>>>>>
>>>>>>> 4) Any other method  in syslog-ng to delay the logs sending at the
>>>>>>>> sender?
>>>>>>>>
>>>>>>>
>>>>>>> Depending on your use-case a flow controlled path end-to-end
>>>>>>> (application, client-syslog-ng, server syslog-ng, final destination) could
>>>>>>> work too. In that case, syslog-ng would automatically converge to the
>>>>>>> amount of messages the destination is able to consume.
>>>>>>>
>>>>>>> 5) My destination configuration is below. Is it a valid
>>>>>>>> configuration ?
>>>>>>>>
>>>>>>>>
>>>>>>>> destination logFiler { file("/var/log/wq.log"
>>>>>>>>
>>>>>>>>     template("${FULLDATE}${TZ} ${HOST} ${PROGRAM} [$LEVEL]
>>>>>>>> ${MSG}\n")
>>>>>>>>
>>>>>>>>     template_escape(yes)
>>>>>>>>
>>>>>>>>     throttle(500));};
>>>>>>>>
>>>>>>>
>>>>>>> This is, however this is a file() destination, where the throttle
>>>>>>> option may have limited use.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Thank you for your time.
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Simon
>>>>>>>>
>>>>>>>>
>>>>>>>> ______________________________________________________________________________
>>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>>> Documentation:
>>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>>
>>>>>>>>
>>>>>>> ______________________________________________________________________________
>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>> Documentation:
>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>
>>>>>>>
>>>>> ______________________________________________________________________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation:
>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>>>
>>>> --
>>>> Bazsi
>>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20210220/80f2e2da/attachment-0001.html>

More information about the syslog-ng mailing list