[syslog-ng] Disk buffer file truncation issue

[Raghunath Adhyapak]

Hi, I don't have an exact way to reproduce it. However, this is what we
observe:
- The buffer file keeps increasing in size.
- We keep getting syslog messages and there doesn't seem to be any issue
with the service.
- When we check the buffer size it grows into gigabytes.
- At this point, in one case we saw the buffer file size was around 4GB. We
restarted the service and after the restart, boom, the buffer files got
cleared immediately (size of buffer file was 4k)
- Next time when this happened, we checked the contents of the buffer file
and investigated whether the events were received or not. We found that we
had received all the events.

Therefore we believe that there was a file truncation issue and restart
helped to clear it.

Let me know if you need any further details and also on how to prevent this
from happening

Thanks
Raghu



On Tue, Jun 9, 2020, 12:58 Nagy Gábor <gabor.hl at gmail.com> wrote:

> Hi,
>
> Can you describe the issue in detail, please? Could you share the
> reproduction steps?
>
> You say there is an issue with disk-buffer truncating after it's emptied,
> and that is resolved when syslog-ng restart?
> How do you check if the queue is empty?
>
> Regards,
> Gabor
>
>
>
>
>
> Raghunath Adhyapak <funduraghu at gmail.com> ezt írta (időpont: 2020. jún.
> 9., K, 0:08):
>
>> Correcting subject
>>
>> We did some more troubleshooting on this and we found that all logs in
>> buffer were indeed sent out and that syslog-ng was facing issues in
>> truncating the file.
>> This issue got fixed with restart.
>> However, we are observing that this issue is happening too often.
>> Would anyone help me understand why this could be happening?
>>
>> Thanks
>> Raghu
>>
>> On Mon, May 11, 2020, 19:34 Fabien Wernli <wernli at in2p3.fr> wrote:
>>
>>> Hi,
>>>
>>> On Mon, May 11, 2020 at 12:23:27PM +0000, László Várady (lvarady) wrote:
>>> > > 2. Why couldn't syslog-ng resume operations after partition was
>>> freed up and destination was available?
>>> >
>>> > That might be a bug. Once a destination becomes available (set
>>> time-reopen() to a lower value to check more frequently), syslog-ng should
>>> send messages out from the disk buffer.
>>> > Could you reproduce this issue and share the reproduction steps?
>>>
>>> I remember having a lot of corrupt disk buffers when disk was full.
>>> In my case, they caused a segfault on startup, maybe that got "fixed" by
>>> a
>>> deletion instead?
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20200613/74ca8861/attachment.html>