[syslog-ng] SOLVED: syslog-ng and python program destination constantly being restarted again and again
Andrew Bell
abell at factset.com
Fri Sep 26 15:41:38 CEST 2014
Hi,.
Thanks for the feedback. I actually figured out the reason as to why this was happening, and it had to do with syslog-ng referencing a the default python path on the system instead of the custom python path I've installed in my home directory ("python" alias pointed to "/usr/bin/python" instead of my "/home/bin/python" path). The custom python path had all the third-party libraries that my script depended on for execution, and so that was why syslog-ng kept sending all these /var/log/messages because it couldn't import the libraries and was using the wrong python instance (Used an strace command against syslog-ng in order to confirm this). So I specified the fully qualified path to my custom home python instance and now my script is working and staying up indefinitely as expected. I also fully qualified the one error exception path as well.
~Andrew
-----Original Message-----
From: syslog-ng-bounces at lists.balabit.hu [mailto:syslog-ng-bounces at lists.balabit.hu] On Behalf Of Sandor Geller
Sent: Friday, September 26, 2014 3:49 AM
To: syslog-ng at lists.balabit.hu
Subject: Re: [syslog-ng] syslog-ng and python program destination constantly being restarted again and again
Hi,
The logs you quoted imply that your custom script keeps dying therefore syslog-ng keeps respawning it. Without seeing the whole script I'd assume that the original problem occurs in the code path where the script tries to forward logs and an expection gets raised (BTW not all exceptions are derived from the Exception class so the exception handler is a bit incomplete but let's put this aside for now).
In the exception handler you're trying to open a file containing '~'
without using os.path.expanduser() so you end up with an invalid filename which can't get opened. As this exception isn't handled the interpreter aborts the script. So you should eliminate the root cause why the script can't forward logs and also improve the exception handling.
hth,
Sandor
On 09/26/2014 01:12 AM, Andrew Bell wrote:
> Hello all,
>
> Wondering if someone could help me with an issue I've run into
> recently regarding the program destination in syslog-ng and a python
> script I wrote to automate parsing log events sent over from our Web
> Application firewall. For some months now, the script and syslog-ng
> configuration were working swimmingly but then things recently started
> to stop working and now the script refuses to remain up and forwarding as expected.
>
> First, here is what and destination looks like in my syslog-ng conf.
> The general flow here being ASM events are generated on the firewall
> (for whatever purpose), they get sent over to syslog-ng and my python
> script then parses these logs and sends them over to another server
> running ElasticSearch using the python requests library. This takes
> place within the context of an infinite loop in my script - wait for a
> log event to come in, execute, go back to waiting, and so forth.
>
> /destination asm_post {/
>
> / program("python -u /home/data/asm_logs/asmlogPost.py"
> flush_lines(1) flags(no_multi_line));/
>
> /};/
>
> I've specified the following flags based off other help forums that
> suggested this would help to address the stdin buffer flushing issue
> sometimes encountered with syslog-ng and python scripts. There are no
> filters being applied here.
>
> //
>
> /source(sn_asm);/
>
> /destination(asm_post);/
>
> Now, my asmLogPost.py script is setup to always read in standard input
> through an infinite loop as various best practice syslog-ng sources
> have recommended. Here's an excerpt of where the reading in happens -
> encapsulated within the context of a try/exception clause:
>
> /try:/
>
> / while 1:/
>
> / line = sys.stdin.readline()/
>
> / #...read in line log event, log event parsing and POSTing to
> other box executes in context of infinite loop...after each POST, it
> should go back and wait to read in the next line/
>
> /Exception, e:/
>
> / # ...exception handling block to catch any errors, write to a file.../
>
> / f = open('~/ error.txt','ab')/
>
> / f.write('Error happened, here are the details - %s\n'% str(e))/
>
> / f.close()/
>
> As was said earlier, this has all worked out fine and dandy for quite
> some time (and I haven't made any alterations to my script for about a
> month or so). But just recently, syslog-ng now appears to be caught in
> a loop where it constantly stops, starts, and restarts my script
> continuously such that it can't log anything anymore sufficiently.
> Also the PID is constantly changing for my script whenever I do a ps
> -aef command, which would indicate that a new instance is constantly
> being spun up by syslog-ng.
>
> Not only that, but there apparently appear to be the following
> messages sprouting up in /var/log/messages constantly as well which I
> think is due to syslog-ng always restarting my script
>
> /Sep 25 16:28:57 sysa03 abrt: can't communicate with ABRT daemon, is
> it running? [Errno 2] No such file or directory/
>
> /Sep 25 16:28:57 sysa03 abrt: detected unhandled Python exception in
> '/home/data/asm_logs/asmlogPost.py'/
>
> /Sep 25 16:28:57 sysa03 abrt: can't communicate with ABRT daemon, is
> it running? [Errno 2] No such file or directory/
>
> /Sep 25 16:28:57 sysa03 abrt: detected unhandled Python exception in
> '/home/data/asm_logs/asmlogPost.py'/
>
> /Sep 25 16:28:57 sysa03 abrt: can't communicate with ABRT daemon, is
> it running? [Errno 2] No such file or directory/
>
> /Sep 25 16:28:57 sysa03 abrt: detected unhandled Python exception in
> '/home/data/asm_logs/asmlogPost.py'/
>
> /Sep 25 16:28:57 sysa03 abrt: can't communicate with ABRT daemon, is
> it running? [Errno 2] No such file or directory/
>
> /Sep 25 16:28:57 sysa03 abrt: detected unhandled Python exception in
> '/home/data/asm_logs/asmlogPost.py'/
>
> /Sep 25 16:28:57 sysa03 abrt: can't communicate with ABRT daemon, is
> it running? [Errno 2] No such file or directory/
>
> /Sep 25 16:28:57 sysa03 abrt: detected unhandled Python exception in
> '/home/data/asm_logs/asmlogPost.py'/
>
> /Sep 25 16:28:57 sysa03 abrt: can't communicate with ABRT daemon, is
> it running? [Errno 2] No such file or directory/
>
> /Sep 25 16:28:58 sysa03 abrt: detected unhandled Python exception in
> '/home/data/asm_logs/asmlogPost.py'/
>
> /Sep 25 16:28:58 sysa03 abrt: can't communicate with ABRT daemon, is
> it running? [Errno 2] No such file or directory/
>
> /Sep 25 16:28:58 sysa03 abrt: detected unhandled Python exception in
> '/home/data/asm_logs/asmlogPost.py'/
>
> /Sep 25 16:28:58 sysa03 abrt: can't communicate with ABRT daemon, is
> it running? [Errno 2] No such file or directory/
>
> But the thing is, there are no exceptions or error files being
> generated by my script even though these messages seem to say this
> happening...if there were, my script should catch this and write it
> down. Besides, it did used to do this initially as I was developing my
> script but it would never halt overall execution or cause syslog-ng to
> tear it down, it would just write out to an error.txt file within the same directory.
>
> //
>
> Can anyone offer any insight here? I'll admit I'm not familiar with
> how the ABRT daemon functions and if it and syslog-ng are somehow
> related here or not. And I'm pretty confident that there's nothing
> wrong with my script as I am able to execute its current version just
> fine from a shell prompt - it can send over the log event and
> everything parsed as expected with no errors. I'm kind of at a dead
> end, going through my script and these log messages, trying to restart
> syslog-ng to refresh, trying to place debug log statements before the
> "while 1:" loop, trying to move the while 1: loop to be outside the
> try/exception clause, but not much of this seems to be helping so far.
>
> Many Thanks,
>
> Andrew
>
>
>
> ______________________________________________________________________
> ________ Member info:
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq
More information about the syslog-ng
mailing list