[syslog-ng] snmptrap handler parses SNMP traps correctly, but the file it reads from is not emptying

[Scheidler, Balázs]

In Unix you can rename an opened file, and any processes having that file
open will continue to write to that file.

So this is what should happen:
* Logrotate moves the current log file to a different name
* Logrotate alerts snmptrapd that it needs to write to the new file
(usually via SIGHUP)
* syslog-ng continues to read the old file (keeping it opened during the
process above, e.g. still referencing the old file even if it has been
renamed
* When eof is reached, syslog-ng checks if there's a different file with
the original name.
* If there's it switches to it.
* The renamed "old" logfile can be compressed/moved away/etc.

This handshake ensures that no data is lost.

On Tue, May 21, 2019, 17:41 Allen Pouratian <Allen.Pouratian at alticeusa.com>
wrote:

> Balázs –
>
>
>
> What guarantee is there that I won’t lose traps?
>
>
>
>    1. Does logrotate make sure snmptrapd has finished writing to *"/var/log/snmptrapd.log"?
>    *Is the write by snmptrapd 5.7.3+ always atomic for one or more traps?
>    2. Does syslog-ng 3.20.1 make sure it instantly reads every trap
>    written by snmptrapd to *"/var/log/snmptrapd.log" *before it’s
>    switched out by logrotate?
>
>
>
> I apologize if this is basic knowledge I’m not aware of.
>
>
>
> Best Regards,
>
>
>
> -Allen
>
>
>
> *From:* syslog-ng <syslog-ng-bounces at lists.balabit.hu> * On Behalf Of *Scheidler,
> Balázs
> *Sent:* Monday, May 20, 2019 11:55 AM
> *To:* Syslog-ng users' and developers' mailing list <
> syslog-ng at lists.balabit.hu>
> *Subject:* Re: [syslog-ng] snmptrap handler parses SNMP traps correctly,
> but the file it reads from is not emptying
>
>
>
> *[External Email]*
>
> Just set up logrotate on that file, syslog-ng will read till the end and
> start the new one if the rotation happens.
>
>
>
> So instead of truncating, rename it to a new name, and let snmptrapd write
> to a new file.
>
>
>
> On Mon, May 20, 2019, 15:55 Allen Pouratian <Allen.Pouratian at alticeusa.com>
> wrote:
>
> Hello –
>
>
>
> I set up snmptrapd to write to */var/log/snmptrapd.log* and syslog-ng is
> configured to read from it like this …
>
>
>
> *source { *
>
> *snmptrap(filename("/var/log/snmptrapd.log")); *
>
> *};*
>
>
>
> … but */var/log/snmptrapd.log* is not getting emptied, so it needs to be
> truncated every so often to avoid filling up the filesystem.
>
>
>
> But the problem with truncating */var/log/snmptrapd.log* is that we’re
> going to lose traps, since we get a lot of them.
>
>
>
> When I was testing this syslog-ng snmptrap facility with a few traps a
> minute, I saw */var/log/snmptrapd.log* empty, but with hundreds of traps
> incoming per second, it does not empty any more.
>
>
>
> I have what looks like a work-around to the syslog-ng snmptrap
> handler/parser where …
>
>
>
>    1. 1) Snmptrapd 5.7.3 writes to syslog with -Lsd
>    2. 2) Syslog-ng 3.20 reads from syslog with system-journal()
>    3. 3) Rewrite the trap $MESSAGE with a series of “substitutions”
>    (subst) into space separated key=value pairs
>    4. 4) point kv-parser() at $MESSAGE and specify “ “ as a separator
>    5. 5) delete the original $MESSAGE block
>
>
>
> … but perhaps I didn’t have to do that, since perhaps I’m mis-using the
> snmptrap facility built into syslog-ng, and thus causing
> */var/log/snmptrapd.log* to not empty.
>
>
>
> Your comments and insights would be appreciated.
>
>
>
> - Allen
>
>
>
>
>
> --------------------------------------------------------
> The information transmitted in this email and any of its attachments is
> intended only for the person or entity to which it is addressed and may
> contain information concerning Altice USA and/or its affiliates and
> subsidiaries that is proprietary, privileged, confidential and/or subject
> to copyright. Any review, retransmission, dissemination or other use of, or
> taking of any action in reliance upon, this information by persons or
> entities other than the intended recipient(s) is prohibited and may be
> unlawful. If you received this in error, please contact the sender
> immediately and delete and destroy the communication and all of the
> attachments you have received and all copies thereof.
> --------------------------------------------------------
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.balabit.hu_mailman_listinfo_syslog-2Dng&d=DwMFaQ&c=j5qhuXVFDG9qAPVzJdE5nsirLGtagR5tRVwNBd9MNeo&r=BBDIJ7y03FqO0tp_wGoZn961RS3d3lkq0jHFc3Knid4&m=uVtE5BY0NtoZSHcV7M9rVxkmtCUx34u93UPnbMDlgco&s=-wSgv6P6Tu67nZwvsDpZ8FSGWw3vRdbr9k2k8lOsYN8&e=>
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.balabit.com_support_documentation_-3Fproduct-3Dsyslog-2Dng&d=DwMFaQ&c=j5qhuXVFDG9qAPVzJdE5nsirLGtagR5tRVwNBd9MNeo&r=BBDIJ7y03FqO0tp_wGoZn961RS3d3lkq0jHFc3Knid4&m=uVtE5BY0NtoZSHcV7M9rVxkmtCUx34u93UPnbMDlgco&s=qEw0ZLUxEifTpc9_6CVz8tXvfTCvtmcMX3nvPyriky8&e=>
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.balabit.com_wiki_syslog-2Dng-2Dfaq&d=DwMFaQ&c=j5qhuXVFDG9qAPVzJdE5nsirLGtagR5tRVwNBd9MNeo&r=BBDIJ7y03FqO0tp_wGoZn961RS3d3lkq0jHFc3Knid4&m=uVtE5BY0NtoZSHcV7M9rVxkmtCUx34u93UPnbMDlgco&s=1ihsxbF1hC0FOjy9_eIGny6fXBoHYk5XPxzJxAB9-Gg&e=>
>
>
>
> *Caution: This email originated outside of Altice USA. Please do not click
> links or attachments unless you recognize the sender and know the content
> is safe.*
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20190521/91527108/attachment.html>