[syslog-ng] Request for config help

[Pradeep Gotaparthi]

Hi Balazs,

Sorry, couldnt get back to you earlier.  Below is a sample of config

destination d_msg_04 {
    tcp(
        "172.49.20.136"
        port(2193)
        # log-fifo-size should be atleast max-connections * log-fetch-limit
        # 500 * 10 (default value of log-fetch-limit)
        log-fifo-size(5000)
        # throttle to max 5k logs
        throttle(5000)
        disk-buffer(
            # number of bytes to store in memory
            mem-buf-size(10000)
            # number of bytes to store on disk
            disk-buf-size(134217728000) # 100GB

            reliable(yes)
            # directory location to persist messages
            dir("/data/store")
        )
        persist-name(d_msg_04)
    );
};

The throttle value as you can see is set to 5000.  Throttling is an
important part of using syslog-ng for us here and it does a good job of
it.   The exception is when there is a backlog built up on syslog-ng (it
received a lot more logs than it could push out honoring the throttle
limit),  and we stop or restart syslog-ng.   At that time, it appears to be
"flushing out" whatever messages it has in its buffer and disregard the
throttle limit at this time.     This is causing issues for us.

So, wanted to understand if there is a way to prevent this.  Tried
tinkering with the log-fifo-size thinking it may prevent too many messages
in the buffer and there by help with this problem but it didnt help.

It would be great if you could help us with this.


Thanks !




On Wed, Apr 21, 2021 at 10:29 AM Balazs Scheidler <bazsi77 at gmail.com> wrote:

> Hi,
>
> I am not sure I understand your problem. Can you elaborate it a bit?
> Configuration, the problem as you see it and your expectations would help a
> lot to respond to your question.
>
> On Wed, Apr 21, 2021, 02:50 Pradeep Gotaparthi <pradeep.gp at gmail.com>
> wrote:
>
>> Hi,
>>
>> Could anybody help us out with the below query?
>>
>>
>> Thanks.
>>
>> On Fri, Mar 26, 2021 at 12:06 AM Pradeep Gotaparthi <pradeep.gp at gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> We have been using syslog-ng for over a year now and it is working
>>> great, thanks!
>>>
>>> One issue we face is that when we stop the process when there is some
>>> buffer, the threshold is ignored and a bunch of logs get pushed out (as
>>> much as 10 times the threshold value).   This is causing problems on the
>>> receiving end as it can't handle the load.  We thought the log-fifo-size
>>> parameter can help as syslog uses this parameter to store logs in memory
>>> but it is not helping.
>>>
>>> It would be great if you can guide us.
>>>
>>>
>>> Thanks.
>>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20210605/4a72ad13/attachment.html>