Hi Balazs, Sorry, couldnt get back to you earlier. Below is a sample of config destination d_msg_04 { tcp( "172.49.20.136" port(2193) # log-fifo-size should be atleast max-connections * log-fetch-limit # 500 * 10 (default value of log-fetch-limit) log-fifo-size(5000) # throttle to max 5k logs throttle(5000) disk-buffer( # number of bytes to store in memory mem-buf-size(10000) # number of bytes to store on disk disk-buf-size(134217728000) # 100GB reliable(yes) # directory location to persist messages dir("/data/store") ) persist-name(d_msg_04) ); }; The throttle value as you can see is set to 5000. Throttling is an important part of using syslog-ng for us here and it does a good job of it. The exception is when there is a backlog built up on syslog-ng (it received a lot more logs than it could push out honoring the throttle limit), and we stop or restart syslog-ng. At that time, it appears to be "flushing out" whatever messages it has in its buffer and disregard the throttle limit at this time. This is causing issues for us. So, wanted to understand if there is a way to prevent this. Tried tinkering with the log-fifo-size thinking it may prevent too many messages in the buffer and there by help with this problem but it didnt help. It would be great if you could help us with this. Thanks ! On Wed, Apr 21, 2021 at 10:29 AM Balazs Scheidler <bazsi77@gmail.com> wrote:

Hi,

I am not sure I understand your problem. Can you elaborate it a bit? Configuration, the problem as you see it and your expectations would help a lot to respond to your question.

On Wed, Apr 21, 2021, 02:50 Pradeep Gotaparthi <pradeep.gp@gmail.com> wrote:

...

Hi,

Could anybody help us out with the below query?

Thanks.

On Fri, Mar 26, 2021 at 12:06 AM Pradeep Gotaparthi <pradeep.gp@gmail.com> wrote:

...

Hi,

We have been using syslog-ng for over a year now and it is working great, thanks!

One issue we face is that when we stop the process when there is some buffer, the threshold is ignored and a bunch of logs get pushed out (as much as 10 times the threshold value). This is causing problems on the receiving end as it can't handle the load. We thought the log-fifo-size parameter can help as syslog uses this parameter to store logs in memory but it is not helping.

It would be great if you can guide us.

Thanks.

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Hello Daniel, If I understand you correctly: you have a problem that one of your Syslog-ng server is not receiving a log message. (Not that you do not receive a message FROM one of your Syslog-ng server.) From my experience: In those cases when someone tries to diagnose an issue where Syslog-ng do not receive a message, it is always a good first step to determine if the message actually reaches Syslog-ng or not. i.e.: in case of a network source, try to receive the message with a simple netcat command. (Maybe the message was dropped by a firewall before reaching your machine, and Syslog-ng has nothing to do with it.) At this point we do not really care about the format of the message. If you made sure that those messages are reaching the application, we should try to diagnose Syslog-ng itself. For that we will need some information about your setup. i.e.: - your platform - version of Syslog-ng (Where it is obtained from? i.e. local build) - Related config parts. (including the source driver which is expected to receive the logs) - How do you start Syslog-ng? (i.e.: as a service) Best regards, Laci ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Daniel Ehrlich <Daniel.Ehrlich@usq.edu.au> Sent: Wednesday, June 9, 2021 07:24 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: [syslog-ng] Tips to diagnose missing syslog messages? CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Hi Everyone, Does anyone have some diagnostic tips to offer to diagnose why syslog messages are not being received? I have syslog-ng on a few servers but one is losing messages (others syslog-ng servers might be dropping that I am unaware off). Not sure if there is some options I should add to the conf or diag commands or load specifications I should check? Options: options { chain_hostnames(no); create_dirs (yes); dir_perm(0755); dns_cache(yes); keep_hostname(yes); log_fifo_size(2048); log_msg_size(8192); perm(0644); time_reopen (10); use_dns(yes); use_fqdn(yes); flush_lines(100); }; Also adding the flag-control flag to the log stanza. Thanks Daniel __________________________________________________________________ This email (including any attached files) is confidential and is for the intended recipient(s) only. If you received this email by mistake, please, as a courtesy, tell the sender, then delete this email. The views and opinions are the originator's and do not necessarily reflect those of the University of Southern Queensland. Although all reasonable precautions were taken to ensure that this email contained no viruses at the time it was sent we accept no liability for any losses arising from its receipt. The University of Southern Queensland is a registered provider of education with the Australian Government. (CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081)

Hello Daniel, 3 to 500 events per second is definitely in the manageable range for Syslog-ng. However, in case of UDP there is no guarantee for delivery and it also lacks the traffic shaping mechanism of TCP. Thus, while your events/sec seems to be low, there could be peaks in your traffic, which can cause packet drops on several levels. At a first glance I would check the output of the ifconfig command to see if there is any packet drop on the interface. If there is no loss on the interface, you can check the statistics of Syslog-ng. https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edit... Regarding to the "multi-threaded" topic. I would recommend Peter Czanik's blog post for a start. https://www.syslog-ng.com/community/b/blog/posts/improved-log-collection-ove... However tuning and optimizing your system is a trial-and-error process, and there is no "one fit for all" solution. Note: From the information you provided, at this point those logs can be missing because of a simple filter rule, which drops them. This is not necessarily a transmission issue. This is the reason why we need much more precise information about your setup. Br, Laci ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Daniel Ehrlich <Daniel.Ehrlich@usq.edu.au> Sent: Tuesday, June 15, 2021 07:48 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Tips to diagnose missing syslog messages? CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Thanks Laci, Yes this is syslog-ng as a receiver, from a network source of UDP 514. The destination is files on the local drive. The network path is allowed by firewall as we receive several messages, though some go missing. For example, some hosts send every minute and only 5 of the 15 might make it to the dest file. I may be overloading the listener with 3-500 events per second? I think I am running syslog-ng 3.5 from RedHat repos. I was sure if the multi-threaded option would help? Thanks Daniel ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Laszlo Szemere (lszemere) <Laszlo.Szemere@oneidentity.com> Sent: Tuesday, 15 June 2021 12:34 AM To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Tips to diagnose missing syslog messages? Hello Daniel, If I understand you correctly: you have a problem that one of your Syslog-ng server is not receiving a log message. (Not that you do not receive a message FROM one of your Syslog-ng server.) From my experience: In those cases when someone tries to diagnose an issue where Syslog-ng do not receive a message, it is always a good first step to determine if the message actually reaches Syslog-ng or not. i.e.: in case of a network source, try to receive the message with a simple netcat command. (Maybe the message was dropped by a firewall before reaching your machine, and Syslog-ng has nothing to do with it.) At this point we do not really care about the format of the message. If you made sure that those messages are reaching the application, we should try to diagnose Syslog-ng itself. For that we will need some information about your setup. i.e.: - your platform - version of Syslog-ng (Where it is obtained from? i.e. local build) - Related config parts. (including the source driver which is expected to receive the logs) - How do you start Syslog-ng? (i.e.: as a service) Best regards, Laci ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Daniel Ehrlich <Daniel.Ehrlich@usq.edu.au> Sent: Wednesday, June 9, 2021 07:24 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: [syslog-ng] Tips to diagnose missing syslog messages? CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Hi Everyone, Does anyone have some diagnostic tips to offer to diagnose why syslog messages are not being received? I have syslog-ng on a few servers but one is losing messages (others syslog-ng servers might be dropping that I am unaware off). Not sure if there is some options I should add to the conf or diag commands or load specifications I should check? Options: options { chain_hostnames(no); create_dirs (yes); dir_perm(0755); dns_cache(yes); keep_hostname(yes); log_fifo_size(2048); log_msg_size(8192); perm(0644); time_reopen (10); use_dns(yes); use_fqdn(yes); flush_lines(100); }; Also adding the flag-control flag to the log stanza. Thanks Daniel __________________________________________________________________ This email (including any attached files) is confidential and is for the intended recipient(s) only. If you received this email by mistake, please, as a courtesy, tell the sender, then delete this email. The views and opinions are the originator's and do not necessarily reflect those of the University of Southern Queensland. Although all reasonable precautions were taken to ensure that this email contained no viruses at the time it was sent we accept no liability for any losses arising from its receipt. The University of Southern Queensland is a registered provider of education with the Australian Government. (CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081) __________________________________________________________________ This email (including any attached files) is confidential and is for the intended recipient(s) only. If you received this email by mistake, please, as a courtesy, tell the sender, then delete this email. The views and opinions are the originator's and do not necessarily reflect those of the University of Southern Queensland. Although all reasonable precautions were taken to ensure that this email contained no viruses at the time it was sent we accept no liability for any losses arising from its receipt. The University of Southern Queensland is a registered provider of education with the Australian Government. (CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081)

Hi folks, It would be really helpful if any of you are able to help with the below query. Thanks! On Sat, Jun 5, 2021 at 12:56 AM Pradeep Gotaparthi <pradeep.gp@gmail.com> wrote:

Hi Balazs,

Sorry, couldnt get back to you earlier. Below is a sample of config

destination d_msg_04 { tcp( "172.49.20.136" port(2193) # log-fifo-size should be atleast max-connections * log-fetch-limit # 500 * 10 (default value of log-fetch-limit) log-fifo-size(5000) # throttle to max 5k logs throttle(5000) disk-buffer( # number of bytes to store in memory mem-buf-size(10000) # number of bytes to store on disk disk-buf-size(134217728000) # 100GB

reliable(yes) # directory location to persist messages dir("/data/store") ) persist-name(d_msg_04) ); };

The throttle value as you can see is set to 5000. Throttling is an important part of using syslog-ng for us here and it does a good job of it. The exception is when there is a backlog built up on syslog-ng (it received a lot more logs than it could push out honoring the throttle limit), and we stop or restart syslog-ng. At that time, it appears to be "flushing out" whatever messages it has in its buffer and disregard the throttle limit at this time. This is causing issues for us.

So, wanted to understand if there is a way to prevent this. Tried tinkering with the log-fifo-size thinking it may prevent too many messages in the buffer and there by help with this problem but it didnt help.

It would be great if you could help us with this.

Thanks !

On Wed, Apr 21, 2021 at 10:29 AM Balazs Scheidler <bazsi77@gmail.com> wrote:

...

Hi,

I am not sure I understand your problem. Can you elaborate it a bit? Configuration, the problem as you see it and your expectations would help a lot to respond to your question.

On Wed, Apr 21, 2021, 02:50 Pradeep Gotaparthi <pradeep.gp@gmail.com> wrote:

...

Hi,

Could anybody help us out with the below query?

Thanks.

On Fri, Mar 26, 2021 at 12:06 AM Pradeep Gotaparthi < pradeep.gp@gmail.com> wrote:

...

Hi,

We have been using syslog-ng for over a year now and it is working great, thanks!

One issue we face is that when we stop the process when there is some buffer, the threshold is ignored and a bunch of logs get pushed out (as much as 10 times the threshold value). This is causing problems on the receiving end as it can't handle the load. We thought the log-fifo-size parameter can help as syslog uses this parameter to store logs in memory but it is not helping.

It would be great if you can guide us.

Thanks.

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq