<div dir="ltr">Hi Balaza,<div>I have some more queries below:</div><div><br></div><div>1. You mentioned to make sure the file destination has enough buffer space to avoid message loss if the application emits more messages per second than the throttle value. So the remaining messages (after the throttle value) are stored in system RAM ? Is this memory allocated by syslog-ng process internally? or do we have any syslog-ng configuration parameter to allocate this much memory?</div><div><br></div><div>2. Can I know from which version the disk-buffer feature was added in syslog-ng ? I am using version 3.0.8. </div><div><br></div><div>3. With a disk-buffer, the extra messages can be stored temporarily to any path (for example: /tmp/message.log). Will this temporary file get deleted after the complete transfer?</div><div><br></div><div>4. Do we have any internal syslog-ng delay timer settings between messages other than throttle feature? </div><div><br></div><div>Thank you for helping.</div><div><br></div><div>Regards</div><div>Simon</div><div> <br><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Feb 9, 2021 at 9:42 AM Balazs Scheidler <<a href="mailto:bazsi77@gmail.com">bazsi77@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Yes, throttle could work. You just have to make sure that your file destination has enough buffer space to avoid message loss if the apps emit more messages per second than the throttle value. In this case the question could arise how much ram you allocate to this purpose.<div dir="auto"><br></div><div dir="auto">Disk-buffer() could be of use, as long as the queue files reside on a normal, high performance disk, I am not 100% sure that disk buffer works with file destinations though (they should but I am pretty sure no one uses them that way).</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Feb 9, 2021, 17:35 SIMON BABY <<a href="mailto:simonkbaby@gmail.com" target="_blank">simonkbaby@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Thank you so much . I understand that file destinations are operated on default soft flow control . My destination and source are running on the same system. I understand the hard flow control will not help here. Do you suggest any other method like throttle() or any other method help here .<div> <div>My sources are all the application processes running in my Linux based system and destination is a file on a boot CF card mounted on the same Linux system. I see during heavy logging, the CF card could not handle the high writing rate and eventually it fails to respond and the link connected to the ata driver(kernel) is broken. So I am looking in syslog if I could decrease the logging rate .</div><div><br></div><div>Thank you so much </div><div><br></div><div>Regards</div><div>Simon<br><div><br></div><div><br><div><br>On Monday, February 8, 2021, Balazs Scheidler <<a href="mailto:bazsi77@gmail.com" rel="noreferrer" target="_blank">bazsi77@gmail.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi,</div><div><br></div><div>file() destinations operate in a mode called "soft-flow-control". This is almost the same as flow control, except that we release the flow controlled nature of these desetinations if the file cannot be created or the disk is full. This is to avoid a completely locked up system in case of file related failures.</div><div><br></div><div>On the other hand, as long as the file exists and is writable, it is automatically in flow-control mode, which means that backpressure is provided to the source, which should slow down to the rate the file is able to accept messages.</div><div><br></div><div>The question is how the source handles this backpressure, this depends on the transport protocol that specific source is using:</div><div><br></div><div><ul><li>udp() would simply stop fetching datagrams from the socket, meaning that once the kernel's receive buffer is full, UDP messages will start dropping, even if the datagrams themselves are delivered to the host.</li><li>tcp() would also stop fetching, but this means that the TCP protocol would propagate this back to the actual sender, which should slow down too.</li><li>any other combination (tls, rfc5424 with tcp based transports work like tcp above)</li><li>local transports (unix-dgram, unix-stream) would _both_ propagate this backpressure back</li><li>file sources on the local host: would stop reading at a specific position and restart when the backpressure is released.<br></li></ul><div>If the source is on a remote host (e.g. another syslog-ng instance or another syslog implementation), the settings of that instance control how it reacts to the back-pressure provided by syslog-ng and the transport.</div><div><br></div><div>With syslog-ng, if the destination is a tcp based transport:</div><div><ul><li>you need to specify flags(flow-control) on the log path leading to the tcp() destination, this would enable this back-pressure propagation to the source of the sender syslog-ng instance (soft-flow-control is not effective here, as this is not a file destination)<br></li><li>the backpressure to TCP would propagate back to the source on the sender syslog-ng instance.</li></ul><div>And so on and so on.</div><div><br></div><div>Once the flow-control feedback loop establishes, we still need to have some buffer space to store the in-flight messages, thus each hop on the path between the actual source and the final file destination uses a memory or disk based buffer.<br></div></div></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 8, 2021 at 10:17 PM SIMON BABY <<a href="mailto:simonkbaby@gmail.com" rel="noreferrer" target="_blank">simonkbaby@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello Balaza,<div><br><div>Can you please suggest if I could use flow control flags with file destination . </div><div><br></div><div>Thank you for helping <br><br>Regards</div><div>Simon<br>On Sunday, February 7, 2021, SIMON BABY <<a href="mailto:simonkbaby@gmail.com" rel="noreferrer" target="_blank">simonkbaby@gmail.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Balaza,<div><br></div><div>Thank you so much for helping. </div><div>My end target is a boot CF card which is a SATA device operating in AHCI mode (ATA device). The log file target in the syslog configuration is writing's on this device.</div><div><br></div><div>What is the best option in my case to control the sending rate of writing to the file? </div><div>During heavy writing into the log, the device driver on the CF card could not handle it and eventually the ATA link was broken. So I am trying to control the rate of message writing into the file. </div><div>Can you please suggest the best way to limit the rate of messages writing into the file with syslog-ng.</div><div><br></div><div><br></div><div>Thank you once again for your time and helping.</div><div><br></div><div>Regards</div><div>Simon</div><div><br><h3 style="overflow:hidden;white-space:nowrap;font-size:0.75rem;font-weight:inherit;margin:inherit;text-overflow:ellipsis;font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;letter-spacing:0.3px;color:rgb(95,99,104);line-height:20px"><br></h3></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Feb 7, 2021 at 1:46 AM Balazs Scheidler <<a href="mailto:bazsi77@gmail.com" rel="noreferrer" target="_blank">bazsi77@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div>Hi,<br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Feb 7, 2021, 07:18 SIMON BABY <<a href="mailto:simonkbaby@gmail.com" rel="noreferrer" target="_blank">simonkbaby@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hello Team,<div><br><div>I am new to this group and I have a query on adding the throttle attribute in syslog-ng.conf file. My target is little slow to process all the messages sent by the sender and sometimes the link connected to the target device is broken. I am thinking of slowing down the sender by adding the throttle attribute.</div><div>I have the below queries:</div><div><br></div><div>1) What exactly the throttle confoguration does?</div></div></div></blockquote></div></div><div dir="auto"><br></div><div dir="auto">It limits the number of syslog messages to be sent to the device per second.</div><div dir="auto"><br></div><div dir="auto">The implementation allows short spikes of traffic where the short term rate is higher, but over a few second the average stabilizes to the value specified.</div><div dir="auto"><br></div><div dir="auto">It uses a tbf like algorithm.</div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>2) What does throttle(500) mean ? will it send 500 Bytes per second or 500 messages per second? What does the message here mean ? Can it be the entire message sent by the application ? Is there an upper limit and lower limit ?</div></div></div></blockquote></div></div><div dir="auto"><br></div><div dir="auto">500 messages, not bytes. The maximum message size can be controlled using the log-msg-size() option.</div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>3) Any side effect of my system if I am going to use throttle().</div></div></div></blockquote></div></div><div dir="auto"><br></div><div dir="auto">If your input rate is higher than the output, syslog-ng would either need to store the incoming messages (memory or disk), backpressure to the source if possible (using flow control and a tcp based transport) or drop them.</div><div dir="auto"><br></div><div dir="auto">There are a number of options that control this behavior.</div><div dir="auto"><br></div><div dir="auto">flags(flow-control) to turn on flow control on a log path</div><div dir="auto"><br></div><div dir="auto">log-fifo-size() for controlling the memory buffer size at the destination</div><div dir="auto"><br></div><div dir="auto">disk-buffer() for allowing the excess to overflow to disk</div><div dir="auto"><br></div><div dir="auto">transport(tcp) or transport(tls) on the source to select the transport protocol</div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>4) Any other method in syslog-ng to delay the logs sending at the sender?</div></div></div></blockquote></div></div><div dir="auto"><br></div><div dir="auto">Depending on your use-case a flow controlled path end-to-end (application, client-syslog-ng, server syslog-ng, final destination) could work too. In that case, syslog-ng would automatically converge to the amount of messages the destination is able to consume.</div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>5) My destination configuration is below. Is it a valid configuration ?</div><div><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><br></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">destination logFiler { file("/var/log/wq.log"</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> template("${FULLDATE}${TZ} ${HOST}
${PROGRAM} [$LEVEL] ${MSG}\n")</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> template_escape(yes)</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> throttle(500));};</p></div><div></div></div></div></blockquote></div></div><div dir="auto"><br></div><div dir="auto">This is, however this is a file() destination, where the throttle option may have limited use.</div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><br></div><div><br></div><div>Thank you for your time.</div><div><br></div><div>Regards</div><div>Simon</div><div><br></div></div></div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer noreferrer noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer noreferrer noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer noreferrer noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div></div></div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div>
</blockquote></div></div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr">Bazsi</div>
</blockquote></div></div></div></div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div>