[syslog-ng] disk-buffer in elasticsearch2 destination loses messages if docker container is killed

Thu Sep 13 16:07:03 UTC 2018

Hello,

It would not make sense to replace the persist file after restart, so it is
not something that syslog-ng does. Only if that file is corrupted, in that
case at startup there should be a log about it, have you checked the
syslog-ng logs ? (it would be better to enable debug and/or verbose logs,
and if possible share it with us.)

Could you reproduce the same behavior without docker (if possible) ?

Best Regards,
Peter Kokai

On Thu, Sep 13, 2018 at 4:44 PM Jose Angel Santiago <jasantiago at stratio.com>
wrote:

> Hi,
>
> I guess I know what is happening, when I start from scratch the docker
> container, even I provide a persist file and a buffer file within the
> mapped volume, syslog-ng recreates them so all messages in buffer file
> which were not processed by the relay are lost.
>
> Is there any way to tell syslog-ng to use an already existing .persist
> file so it doesn't recreate the .rqf file?
>
> Regards.
>
>
>
> 2018-09-13 16:23 GMT+02:00 Budai, László <laszlo.budai at oneidentity.com>:
>
>> Hi,
>>
>> one problem could be if the flush-limit would be greater than 1... in
>> that case syslog-ng would use a HttpBulkMessageProcessor.
>> In this case syslog-ng pass the message to the HttpBulkMessageProcessor
>> and sends back a positive ACK to the LogSource (so the message is removed
>> from the diskbuffer), and if the dockerimage is killed, all the messages
>> stored in the HttpBulkMessageProcessor are lost.
>> But in your case syslog-ng should use the HttpSingleMessageProcessor...
>> which means that the messages are sent one-by-one...
>> Could you check the diskbuffer with the dqtool?
>>
>>
>> L.
>>
>> On Thu, Sep 13, 2018 at 3:50 PM, Jose Angel Santiago <
>> jasantiago at stratio.com> wrote:
>>
>>> Hi,
>>>
>>> I'm running syslog-ng (with an elasticsearch2 destination configured)
>>> within a docker container, and I'm trying to avoid loss of messages if I
>>> kill the docker container and I start it again.
>>>
>>> This is my scenary:
>>>
>>> - A service which produces 20 lines of log per second
>>> - A sislog-ng instance reading from a wildcard-file source (but actually
>>> it only reads logs from the above service, let's call it syslog-agent),
>>> which sends all logs to another syslog-ng instance (the one running in a
>>> docker container, let's call it syslog-relay) though a network destination.
>>> - The syslog-relay sends messages to an elasticsearch instance, with
>>> following configuration:
>>>
>>> options {
>>>     chain-hostnames(no);
>>>     use-dns(no);
>>>     keep-hostname(yes);
>>>     owner("syslog-ng");
>>>     group("stratio");
>>>     perm(0640);
>>>     time-reap(30);
>>>     mark-freq(10);
>>>     stats-freq(0);
>>>     bad-hostname("^gconfd$");
>>>     flush-lines(100);
>>>     log-fifo-size(1000);
>>>     };
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *destination d_elastic_default_0 {    elasticsearch2(
>>> cluster("myelastic")        cluster-url("https://myelastic.logs:9200
>>> <https://myelastic.logs:9200>")        client_mode("https")
>>> index("default")        type("log")        flush-limit(1)
>>> disk-buffer(            mem-buf-size(16M)
>>> disk-buf-size(16M)            reliable(yes)
>>> dir("/syslog-ng/log")        )        http-auth-type("clientcert")
>>> java-keystore-filepath("/etc/syslog-ng/certificates/syslog-relay.jks")
>>> java-keystore-password("XXXXXX")
>>> java-truststore-filepath("/etc/syslog-ng/certificates/ca-bundle.jks")
>>> java-truststore-password("XXXXXXXXXX")    );};*
>>>
>>> - The dir "/syslog-ng/log" is mapped to a path "/tmp/buffer" from the
>>> host where the docker container is running, so when I kill the docker
>>> container, the buffer file is not lost.
>>> - I've set flush-limit to 1 because I thought that I may lost 1 message
>>> only as much.
>>>
>>> This architecture is working fine (flush-limit=1 makes very slow, but
>>> for this test is ok), but if I kill the syslog-relay docker container, wait
>>> 5 to 10 seconds and start it again from scratch, I can see that several
>>> hundreds of logs are missing in elasticsearch. I check it by stopping the
>>> logger service and letting syslog-ng agent & relay to finish the process
>>> enqueued messages.
>>>
>>> I can see in the syslog-agent stats that all logs messages have been
>>> processed, so it seems the problem is on the syslog-relay.
>>>
>>> Is this behaviour expected? If so, how can I protect against loss of
>>> messages in case of a syslog-relay docker container unexpected kill?
>>>
>>> Thanks in advance.
>>>
>>>
>>> --
>>>
>>> | Jose Angel Santiago
>>>
>>> [image: Logo_signature2.png] <http://www.stratio.com/>
>>>
>>> Vía de las dos Castillas, 33, Ática 4, 3ª Planta
>>>
>>> 28224 Pozuelo de Alarcón, Madrid, Spain
>>>
>>> +34 918 286 473 <+34%20918%2028%2064%2073> | www.stratio.com
>>> <https://twitter.com/stratiobd>
>>> <https://www.linkedin.com/company/stratiobd>
>>> <https://www.youtube.com/c/StratioBD>
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
>
> --
>
> | Jose Angel Santiago
>
> [image: Logo_signature2.png] <http://www.stratio.com/>
>
> Vía de las dos Castillas, 33, Ática 4, 3ª Planta
>
> 28224 Pozuelo de Alarcón, Madrid, Spain
>
> +34 918 286 473 <+34%20918%2028%2064%2073> | www.stratio.com
> <https://twitter.com/stratiobd>
> <https://www.linkedin.com/company/stratiobd>
> <https://www.youtube.com/c/StratioBD>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180913/441d7580/attachment-0001.html>