[syslog-ng] advice/assistance with parsing attempt requested

[syslog-ng2010 at hushmail.com]

MANY thanks for all of the great answers guys!

as the pattern permutations would be less than desirable to 
maintain, i've gone ahead and given the program destination another 
attempt. this time leveraging python as opposed to perl (even 
though i too believe perl ought be able to handle the volume 
(roughly 300k per 5 minutes). i didn't write the original non-
functional script and happen to favor python. by setting two 
destinations (one writing the raw stream to file and the other 
sending to a program which does parsing) i can see that this 
external parsing does indeed keep a one to one ratio!

now, i'm now faced with a new challenge which for the life of me 
can't begin to get my head around ...

original script had:

destination d_parser {
program("/opt/bin/perl /opt/syslog-ng/config/parser.pl >> 
/opt/syslog-ng/logs/logfile");
};

and, for what it was able to output (albeit anemic), worked fine. 
however ...

current script has:

destination d_parser {
program("/opt/bin/python /opt/syslog-ng/config/parser.py >> 
/opt/syslog-ng/logs/logfile");
};

when i run "/opt/bin/python /opt/syslog-ng/config/parser.py" from 
the cmd line, and pipe input into it (i'm reading stdin); it runs 
as expected and the desired output is sent to screen/stdout

when i start the syslog-ng service (which uses the conf file which 
in turn calls the python program in destination d_parser), unlike 
the perl script which does the same thing (print to screen/stdout) 
and does in fact add entries to /opt/syslog-ng/logs/logfile; no 
output is ever sent to the /opt/syslog-ng/logs/logfile ... UNTIL, i 
stop the syslog service and then, all the expected output comes 
flushing into /opt/syslog-ng/logs/logfile

the issue has me really stumped. how/why would the output from perl 
make it to /opt/syslog-ng/logs/logfile in "real-time" yet, the 
output from python only make it to /opt/syslog-ng/logs/logfile as a 
batch upon stopping the service!?!? to clarify, i'm only sending to 
file currently to test, eventually this will be handed off to 
another syslog server via:

destination d_logdevice {
tcp("10.0.0.1" port(11235));
};

i hope this is something simple because as soon as i can get the 
output from the python like the perl, this project is almost done! 
lastly, this all being done with syslog-ng ose 3.1.1

thanks in advance!

On Mon, 06 Dec 2010 21:01:37 +0100 Bill Anderson 
<Bill.Anderson at bodybuilding.com> wrote:
>On Dec 6, 2010, at 12:37 PM, Martin Holste wrote:
>
>>> Agreed, Perl is plenty quick, hence my wondering about the 
>actual volume. If it is too much for Perl I'd go w/C++.
>> 
>> From what I can tell, PCRE in Perl (or Python or whatever) is 
>really
>> close to C/C++ speeds because they're essentially using the same
>> library and therefore mostly the same syscalls.  I'd be really
>> interested if anyone has benchmarks.  I'd expect something like 
>10%
>> better performance in C, but not much more, assuming that the 
>vast
>> majority of CPU time is spent on PCRE.
>
>Yeah I was thinking the overhead might be in what is done, as 
>opposed to just the RE portion. Of course, the OP script might be 
>implemented rather differently. ;)
>
>
>> 
>>> Personally, I'd make the last step routing back into syslog-ng 
>with a source on a custom port and letting syslog handle the 
>writing to disk. That way you can still use macros such as 
>timestamps, etc.. Then again, that may be because I do that all 
>the time. ;) A log statement that takes everything from the custom 
>source and logs to a file should work beautifully; no need for 
>filters though you could still do additional processing if needed. 
>That said I'd also consider running a daemon that accepted all the 
>input, formatted it, and then sent it to syslog-ng, pointing the 
>clients at the custom daemon if that was possible.
>>> 
>>> One advantage to the daemon route is that it wouldn't *have* to 
>reside on the same system.
>> 
>> Yep, you could definitely let Syslog-NG handle the last mile as 
>well.
>> I was trying to keep the scope as narrow as possible in my 
>example.
>> 
>> I wonder if you could build an NFA state machine by 
>conditionally
>> looping output from a pattern-db parsed message into a source in
>> Syslog-NG with a different pattern-db, depending on the previous
>> output.  Something like a token parser pdb that does an ESTRING 
>up
>> until " " and another one that only expects the key/val pair to 
>be
>> sent to it as the message.  So it comes in as k1=v1 k2=v2 and 
>the
>> first kv gets gobbled up and then sent to another pdb source 
>with a
>> pdb which only matches if the message starts with certain terms. 
> Then
>> the rest of the original message is looped back to itself using
>> @ANYSTRING@ to capture the remainder, that is, minus the kv 
>which was
>> sent to the kv pdb.  It would keep recursively looping like that 
>until
>> there's no message left.  If that all worked, your pattern db 
>would be
>> extremely simple as it would just be a pattern per key you were
>> looking for, and order would no longer be an issue.  
>
>Maybe I'm nuts, but that sounds awesome to me. :D
>
>> Of course there's
>> still the problem of demuxing the whole thing back into a 
>coherent
>> message, but I think that could be done a number of ways by 
>passing
>> the MSGID token with each part and using the new conditionals 
>present
>> in OSE 3.2.  
>
>Well, there is message correlation in 3.2.1 right?  muahahaha
>
>> If OSE 3.3 can really do close to 1 million msgs/sec,
>> then the overhead of resubmitting the same log many times may be
>> bearable, especially with the threading.
>
>True the rate might be the downside to that mechanism. However, 
>the terseness of the messages might make up for some of it.
>
>
>___________________________________________________________________
>___________
>Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>Documentation: 
>http://www.balabit.com/support/documentation/?product=syslog-ng
>FAQ: http://www.campin.net/syslog-ng/faq.html