[syslog-ng] email alert on timeout

Thu Oct 31 13:40:28 UTC 2019

 Hi Gregg,

I tried your configuration, and it works for me.

However, while experimenting, I noticed that mark message timer only starts when there is at least one message received. This is probably due that without any message, syslog-ng does not create the file writer. File writers are created on demand, because filename can be template, so syslog-ng might not know the file name without the message.

This is what might interfere with your test? To workaround that, you might use example-msg-generator. From 3.20, there is a num() option that limits the number of messages generated. I see you are using 3.19, so  unfortunately that means an upgrade.

@version: 3.24

log {
#    source { example-msg-generator(freq(1) template("I am here\n") num(1)); };
    source { network(port(5555)); };
    destination { file(/tmp/heartbeat.txt mark-freq(5) mark-mode(dst-idle)); };
};

log {
    source { file(/tmp/heartbeat.txt); };
    filter {  match ("MARK --" value("MESSAGE")); };
    destination { file(/dev/stdout); };
};

results in

$ bash -c "sleep 2; ../bin/loggen -S localhost -n 1 5555; sleep 1; ../bin/loggen -S localhost -n 1 5555"&
[1] 21679
$ ./syslog-ng -Fe -f ../etc/tmp.conf
[2019-10-31T14:30:20.457194] Accepting connections; addr='AF_INET(0.0.0.0:5555)'
[2019-10-31T14:30:20.457610] syslog-ng starting up; version='3.24.1.59.g2ab166d'
[2019-10-31T14:30:21.162964] Syslog connection accepted; fd='15', client='AF_INET(127.0.0.1:53860)', local='AF_INET(0.0.0.0:5555)'
[2019-10-31T14:30:21.163379] Syslog connection closed; fd='15', client='AF_INET(127.0.0.1:53860)', local='AF_INET(0.0.0.0:5555)'
average rate = 2.00 msg/sec, count=1, time=0.500232, (average) msg size=256, bandwidth=0.50 kB/sec
[2019-10-31T14:30:22.672276] Syslog connection accepted; fd='15', client='AF_INET(127.0.0.1:53862)', local='AF_INET(0.0.0.0:5555)'
[2019-10-31T14:30:22.672501] Syslog connection closed; fd='15', client='AF_INET(127.0.0.1:53862)', local='AF_INET(0.0.0.0:5555)'
average rate = 2.00 msg/sec, count=1, time=0.500194, (average) msg size=256, bandwidth=0.50 kB/sec
Oct 31 14:30:27 furiel -- MARK --
Oct 31 14:30:32 furiel -- MARK --
Oct 31 14:30:37 furiel -- MARK --
Oct 31 14:30:42 furiel -- MARK --

Br,
  Antal

________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Laszlo Szemere (lszemere) <Laszlo.Szemere at oneidentity.com>
Sent: Thursday, October 31, 2019 14:30
To: syslog-ng at lists.balabit.hu <syslog-ng at lists.balabit.hu>
Subject: Re: [syslog-ng] email alert on timeout

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

Hello Gregg,

 I think you are almost on the right track. A little addition to MARK messages:
 Syslog-ng's destinations will ONLY emit a mark message IF otherwise there will be no message at all from that destination, during a "mark-freq" time period.
 So if there is a message on the Destination, it will reset the "mark-freq" timer, and the interval starts again without sending any mark message. So during a normal work of a busy log path there should be no mark messages at all.

 One more thing: I don't know if it is intentional from you, but you can spare the whole "mark" file logic from your configuration in certain cases, if you use the "internal" mark-mode. Unfortunately I can not give you a direct link, but in the "global options" section of the administration guide: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.syslog-ng.com%2Ftechnical-documents%2Fdoc%2Fsyslog-ng-open-source-edition%2F3.24%2Fadministration-guide%2F59%23TOPIC-1298095&data=02%7C01%7Cantal.nemes%40oneidentity.com%7C49a647056f144ad1387e08d75e068066%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637081254344960778&sdata=pXsr8zVit8T7SqYm9QRftxv1iRYYzj4V8le17r2FPqM%3D&reserved=0 there is a chapter about "mark-mode"s.

Best regards,
Laci

________________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Gregg Nicholas <gnichola at berriencounty.org>
Sent: Thursday, October 31, 2019 13:56
To: syslog-ng at lists.balabit.hu
Subject: [syslog-ng] email alert on timeout

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

I'm new to syslog-ng OSE. I've been able to follow the samples, and configured an email alert when a syslog message is received.

Now I'm trying to configure an email alert when it has been too long since I've received a particular syslog message.

First, I configured a destination:

                destination d_heartbeat {

                                file("/var/log/heartbeat.log" mark-freq(60) mark-mode(dst-idle) );

                };

Then, I configured a filter and log:

                filter f_heartbeat {

                                match("I am still here" value("MESSAGE) );

                };

                Log { source(s_network); filter(f_heartbeat); destination(d_heartbeat); };

I was pleasantly surprised that this worked. The heartbeat.log file received the message, and 60 seconds later it received the "syslog -- MARK --"

Since I wanted an email alert, I defined a new source, to grab the output of my heartbeat.log as input:

                source s_heartbeat { file(""/var/log/heartbeat.log"); };

                destination d_smtp { smtp(

                                host("10.10.10.25") port(25)

                                from("syslog-ng" "noreply at mydomain.com<mailto:noreply at mydomain.com>")

                                to("me" "myemail at mydomain.com<mailto:myemail at mydomain.com>")

                                subject("host is down")

                                body("no heartbeat received from the program on host\n")

                ); };

                filter f_timeout { match ("MARK --" value("MESSAGE")); };

                log { source(s_heartbeat); filter(f_timeout); destination(d_smtp); };

This seemed to work when I tested it with a single heartbeat followed by timeout, but I’m still misunderstanding something. If it gets more than one message written to heartbeat.log, then the timeout MARK doesn’t happen.

What am I missing? Is there a better way to accomplish this alert?

Thanks,

Gregg

(running syslog-ng v3.19.1-5 on debian 10)
______________________________________________________________________________
Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cantal.nemes%40oneidentity.com%7C49a647056f144ad1387e08d75e068066%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637081254344960778&sdata=KZ%2BQryJh8SevPwe3V2dup%2Bgig43DQjDv9vyIGMM%2FzrQ%3D&reserved=0
Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cantal.nemes%40oneidentity.com%7C49a647056f144ad1387e08d75e068066%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637081254344960778&sdata=TlzOnS39Oy6hEnKT8PzskVbVLW4E2FFeC%2FXhYxOfTaA%3D&reserved=0
FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cantal.nemes%40oneidentity.com%7C49a647056f144ad1387e08d75e068066%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637081254344960778&sdata=DIH1%2F789kvu0zotC778v6%2B%2BRv70nN1R7%2B7PvjBUCbvM%3D&reserved=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20191031/d7d18592/attachment-0001.html>