[syslog-ng] disk-buffer in elasticsearch2 destination loses messages if docker container is killed

Thu Sep 13 14:43:35 UTC 2018

Hi,

I guess I know what is happening, when I start from scratch the docker
container, even I provide a persist file and a buffer file within the
mapped volume, syslog-ng recreates them so all messages in buffer file
which were not processed by the relay are lost.

Is there any way to tell syslog-ng to use an already existing .persist file
so it doesn't recreate the .rqf file?

Regards.

2018-09-13 16:23 GMT+02:00 Budai, László <laszlo.budai at oneidentity.com>:

> Hi,
>
> one problem could be if the flush-limit would be greater than 1... in that
> case syslog-ng would use a HttpBulkMessageProcessor.
> In this case syslog-ng pass the message to the HttpBulkMessageProcessor
> and sends back a positive ACK to the LogSource (so the message is removed
> from the diskbuffer), and if the dockerimage is killed, all the messages
> stored in the HttpBulkMessageProcessor are lost.
> But in your case syslog-ng should use the HttpSingleMessageProcessor...
> which means that the messages are sent one-by-one...
> Could you check the diskbuffer with the dqtool?
>
>
> L.
>
> On Thu, Sep 13, 2018 at 3:50 PM, Jose Angel Santiago <
> jasantiago at stratio.com> wrote:
>
>> Hi,
>>
>> I'm running syslog-ng (with an elasticsearch2 destination configured)
>> within a docker container, and I'm trying to avoid loss of messages if I
>> kill the docker container and I start it again.
>>
>> This is my scenary:
>>
>> - A service which produces 20 lines of log per second
>> - A sislog-ng instance reading from a wildcard-file source (but actually
>> it only reads logs from the above service, let's call it syslog-agent),
>> which sends all logs to another syslog-ng instance (the one running in a
>> docker container, let's call it syslog-relay) though a network destination.
>> - The syslog-relay sends messages to an elasticsearch instance, with
>> following configuration:
>>
>> options {
>>     chain-hostnames(no);
>>     use-dns(no);
>>     keep-hostname(yes);
>>     owner("syslog-ng");
>>     group("stratio");
>>     perm(0640);
>>     time-reap(30);
>>     mark-freq(10);
>>     stats-freq(0);
>>     bad-hostname("^gconfd$");
>>     flush-lines(100);
>>     log-fifo-size(1000);
>>     };
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *destination d_elastic_default_0 {    elasticsearch2(
>> cluster("myelastic")        cluster-url("https://myelastic.logs:9200
>> <https://myelastic.logs:9200>")        client_mode("https")
>> index("default")        type("log")        flush-limit(1)
>> disk-buffer(            mem-buf-size(16M)
>> disk-buf-size(16M)            reliable(yes)
>> dir("/syslog-ng/log")        )        http-auth-type("clientcert")
>> java-keystore-filepath("/etc/syslog-ng/certificates/syslog-relay.jks")
>> java-keystore-password("XXXXXX")
>> java-truststore-filepath("/etc/syslog-ng/certificates/ca-bundle.jks")
>> java-truststore-password("XXXXXXXXXX")    );};*
>>
>> - The dir "/syslog-ng/log" is mapped to a path "/tmp/buffer" from the
>> host where the docker container is running, so when I kill the docker
>> container, the buffer file is not lost.
>> - I've set flush-limit to 1 because I thought that I may lost 1 message
>> only as much.
>>
>> This architecture is working fine (flush-limit=1 makes very slow, but for
>> this test is ok), but if I kill the syslog-relay docker container, wait 5
>> to 10 seconds and start it again from scratch, I can see that several
>> hundreds of logs are missing in elasticsearch. I check it by stopping the
>> logger service and letting syslog-ng agent & relay to finish the process
>> enqueued messages.
>>
>> I can see in the syslog-agent stats that all logs messages have been
>> processed, so it seems the problem is on the syslog-relay.
>>
>> Is this behaviour expected? If so, how can I protect against loss of
>> messages in case of a syslog-relay docker container unexpected kill?
>>
>> Thanks in advance.
>>
>>
>> --
>>
>> | Jose Angel Santiago
>>
>> [image: Logo_signature2.png] <http://www.stratio.com/>
>>
>> Vía de las dos Castillas, 33, Ática 4, 3ª Planta
>>
>> 28224 Pozuelo de Alarcón, Madrid, Spain
>>
>> +34 918 286 473 | www.stratio.com
>> <https://twitter.com/stratiobd>
>> <https://www.linkedin.com/company/stratiobd>
>> <https://www.youtube.com/c/StratioBD>
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support
>> /documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>

-- 

| Jose Angel Santiago

[image: Logo_signature2.png] <http://www.stratio.com/>

Vía de las dos Castillas, 33, Ática 4, 3ª Planta

28224 Pozuelo de Alarcón, Madrid, Spain

+34 918 286 473 | www.stratio.com
<https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd>
<https://www.youtube.com/c/StratioBD>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180913/80ee7075/attachment.html>