<div dir="ltr">Hi Balaza,<div><br></div><div>Thank you so much for helping. </div><div>My end target is a boot CF card which is a SATA device operating in AHCI mode (ATA device). The log file target in the syslog configuration is writing's on this device.</div><div><br></div><div>What is the best option in my case to control the sending rate of writing to the file? </div><div>During heavy writing into the log, the device driver on the CF card could not handle it and eventually the ATA link was broken. So I am trying to control the rate of message writing into the file. </div><div>Can you please suggest the best way to limit the rate of messages writing into the file with syslog-ng.</div><div><br></div><div><br></div><div>Thank you once again for your time and helping.</div><div><br></div><div>Regards</div><div>Simon</div><div><br><h3 class="gmail-iw" style="overflow:hidden;white-space:nowrap;font-size:0.75rem;font-weight:inherit;margin:inherit;text-overflow:ellipsis;font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;letter-spacing:0.3px;color:rgb(95,99,104);line-height:20px"><br></h3></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Feb 7, 2021 at 1:46 AM Balazs Scheidler <<a href="mailto:bazsi77@gmail.com">bazsi77@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div>Hi,<br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Feb 7, 2021, 07:18 SIMON BABY <<a href="mailto:simonkbaby@gmail.com" target="_blank">simonkbaby@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hello Team,<div><br><div>I am new to this group and I have a query on adding the throttle attribute in syslog-ng.conf file. My target is little slow to process all the messages sent by the sender and sometimes the link connected to the target device is broken. I am thinking of slowing down the sender by adding the throttle attribute.</div><div>I have the below queries:</div><div><br></div><div>1) What exactly the throttle confoguration does?</div></div></div></blockquote></div></div><div dir="auto"><br></div><div dir="auto">It limits the number of syslog messages to be sent to the device per second.</div><div dir="auto"><br></div><div dir="auto">The implementation allows short spikes of traffic where the short term rate is higher, but over a few second the average stabilizes to the value specified.</div><div dir="auto"><br></div><div dir="auto">It uses a tbf like algorithm.</div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>2) What does throttle(500) mean ? will it send 500 Bytes per second or 500 messages per second? What does the message here mean ? Can it be the entire message sent by the application ? Is there an upper limit and lower limit ?</div></div></div></blockquote></div></div><div dir="auto"><br></div><div dir="auto">500 messages, not bytes. The maximum message size can be controlled using the log-msg-size() option.</div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>3) Any side effect of my system if I am going to use throttle().</div></div></div></blockquote></div></div><div dir="auto"><br></div><div dir="auto">If your input rate is higher than the output, syslog-ng would either need to store the incoming messages (memory or disk), backpressure to the source if possible (using flow control and a tcp based transport) or drop them.</div><div dir="auto"><br></div><div dir="auto">There are a number of options that control this behavior.</div><div dir="auto"><br></div><div dir="auto">flags(flow-control) to turn on flow control on a log path</div><div dir="auto"><br></div><div dir="auto">log-fifo-size() for controlling the memory buffer size at the destination</div><div dir="auto"><br></div><div dir="auto">disk-buffer() for allowing the excess to overflow to disk</div><div dir="auto"><br></div><div dir="auto">transport(tcp) or transport(tls) on the source to select the transport protocol</div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>4) Any other method in syslog-ng to delay the logs sending at the sender?</div></div></div></blockquote></div></div><div dir="auto"><br></div><div dir="auto">Depending on your use-case a flow controlled path end-to-end (application, client-syslog-ng, server syslog-ng, final destination) could work too. In that case, syslog-ng would automatically converge to the amount of messages the destination is able to consume.</div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>5) My destination configuration is below. Is it a valid configuration ?</div><div><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><br></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">destination logFiler { file("/var/log/wq.log"</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> template("${FULLDATE}${TZ} ${HOST}
${PROGRAM} [$LEVEL] ${MSG}\n")</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> template_escape(yes)</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> throttle(500));};</p></div><div></div></div></div></blockquote></div></div><div dir="auto"><br></div><div dir="auto">This is, however this is a file() destination, where the throttle option may have limited use.</div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><br></div><div><br></div><div>Thank you for your time.</div><div><br></div><div>Regards</div><div>Simon</div><div><br></div></div></div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div></div></div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div>