[syslog-ng] advice/assistance with parsing attempt requested

[Bill Anderson]

On Dec 8, 2010, at 8:42 AM, <syslog-ng2010 at hushmail.com> <syslog-ng2010 at hushmail.com> wrote:

> MANY thanks for all of the great answers guys!
> 
> as the pattern permutations would be less than desirable to 
> maintain, i've gone ahead and given the program destination another 
> attempt. this time leveraging python as opposed to perl (even 
> though i too believe perl ought be able to handle the volume 
> (roughly 300k per 5 minutes). i didn't write the original non-
> functional script and happen to favor python. by setting two 
> destinations (one writing the raw stream to file and the other 
> sending to a program which does parsing) i can see that this 
> external parsing does indeed keep a one to one ratio!
Interesting!


> destination d_parser {
> program("/opt/bin/python /opt/syslog-ng/config/parser.py >> 
> /opt/syslog-ng/logs/logfile");
> };
> 
> when i run "/opt/bin/python /opt/syslog-ng/config/parser.py" from 
> the cmd line, and pipe input into it (i'm reading stdin); it runs 
> as expected and the desired output is sent to screen/stdout
> 
> when i start the syslog-ng service (which uses the conf file which 
> in turn calls the python program in destination d_parser), unlike 
> the perl script which does the same thing (print to screen/stdout) 
> and does in fact add entries to /opt/syslog-ng/logs/logfile; no 
> output is ever sent to the /opt/syslog-ng/logs/logfile ... UNTIL, i 
> stop the syslog service and then, all the expected output comes 
> flushing into /opt/syslog-ng/logs/logfile
> 
> the issue has me really stumped. how/why would the output from perl 
> make it to /opt/syslog-ng/logs/logfile in "real-time" yet, the 
> output from python only make it to /opt/syslog-ng/logs/logfile as a 
> batch upon stopping the service!?!?


The perl script may have had output buffering disabled. You could do the same w/Python though I'd recommend just writing to the file directly in Python over relying on pipe redirection.

If you want to stick w/output redirection, try modifying adding the unbuffered option to the shebang line:
#/path/to/python -u

Cheers,
Bill