On Dec 8, 2010, at 8:42 AM, <syslog-ng2010@hushmail.com> <syslog-ng2010@hushmail.com> wrote:
MANY thanks for all of the great answers guys!
as the pattern permutations would be less than desirable to maintain, i've gone ahead and given the program destination another attempt. this time leveraging python as opposed to perl (even though i too believe perl ought be able to handle the volume (roughly 300k per 5 minutes). i didn't write the original non- functional script and happen to favor python. by setting two destinations (one writing the raw stream to file and the other sending to a program which does parsing) i can see that this external parsing does indeed keep a one to one ratio! Interesting!
destination d_parser { program("/opt/bin/python /opt/syslog-ng/config/parser.py >> /opt/syslog-ng/logs/logfile"); };
when i run "/opt/bin/python /opt/syslog-ng/config/parser.py" from the cmd line, and pipe input into it (i'm reading stdin); it runs as expected and the desired output is sent to screen/stdout
when i start the syslog-ng service (which uses the conf file which in turn calls the python program in destination d_parser), unlike the perl script which does the same thing (print to screen/stdout) and does in fact add entries to /opt/syslog-ng/logs/logfile; no output is ever sent to the /opt/syslog-ng/logs/logfile ... UNTIL, i stop the syslog service and then, all the expected output comes flushing into /opt/syslog-ng/logs/logfile
the issue has me really stumped. how/why would the output from perl make it to /opt/syslog-ng/logs/logfile in "real-time" yet, the output from python only make it to /opt/syslog-ng/logs/logfile as a batch upon stopping the service!?!?
The perl script may have had output buffering disabled. You could do the same w/Python though I'd recommend just writing to the file directly in Python over relying on pipe redirection. If you want to stick w/output redirection, try modifying adding the unbuffered option to the shebang line: #/path/to/python -u Cheers, Bill