[syslog-ng]Urgent: Mangled messages in pipe() destinations in versions >1.5.8

[Andreas Schulze]

>>>>Balazs Scheidler wrote:
>>>>
>>>Thanks for the detailed report. I think I have found the problem this time.
>>>Can you check this patch?
>>
>>I try to explain it. Look at the following truss output.
>>(Remember, that we use several log path's pointing to the same pipe(),
>>so all fd's (13,18,24,41,54) points to /tmp/pipe e.g.
>>fd 41 is the heavy loaded path and caused the EAGAIN's.)
> 
> Hmm... I see.. You should not point multiple destinations to the same pipe.
> As each destination has its own buffer.
> Is it not possible to separate the pipe to a single destination?

Unfortunately no.
One of the great features of syslog-ng we need, are the mighty
filter capabilities.
We use filters, to let syslog-ng classify some types of messages,
add a 'class' tag (based on a special filter) to the message and
send it to a pipe.
Imagine the following example config.
I hope it illustrates, what we are doing.

destination d_a { pipe("/tmp/pipe" template("@TAG_A: | $MSG\n")); };
destination d_b { pipe("/tmp/pipe" template("@TAG_B: | $MSG\n")); };
destination d_c { pipe("/tmp/pipe" template("@TAG_C: | $MSG\n")); };
destination d_d { pipe("/tmp/pipe" template("@TAG_D: | $MSG\n")); };
log { source(..); filter(fa); destination(d_a); };
log { source(..); filter(fb); destination(d_b); };
log { source(..); filter(fc); destination(d_c); };
log { source(..); filter(fd); destination(d_d); };

We have to do this tagging/filtering via syslog-ng
for the following reasons:
- syslog-ng does this very fast and excellent
- the applications that reads the pipe is commercial software and
doesn't have such great filter caps.
- a 'middle ware' filter (a special script/program/etc.) isn't
fast enough for us, because we are handling millions of messages
per hour/day by this way via syslog-ng
- and last but not least, this filtering is very easy and handy
to configure and to implement using syslog-ng

Maybe, you have some ideas, for a better way, to do this.

> I'm thinking about the possibility of destination references, would that
> solve your problem?

Sorry. I don't understand. I'm not sure, that this solves our problem.
Can you explain 'destination references' a little bit, please.


Thanks a lot for your feedback.
-- 
Best regards --Andreas Schulze
                [phone: +49.5246.80.1275, fax: +49.5246.80.2275]

| I believe, it was Dennis Ritchie who said something like:
|   "C is rarely the best language for a given task,
|    but it's often the second-best".
| The implication being that: "[...]"
|
| sh# cat>$$.c<<EOT
| main(l,a,n,d)char**a;{for(d=atoi(a[1])/10*80-atoi(a[2])/5-596;n="@NK\
| ACLCCGZAAQBEAADAFaISADJABBA^SNLGAQABDAXIMBAACTBATAHDBANZcEMMCCCCAAhE\
| IJFAEAAABAfHJETBdFLDAANEfDNBPHdBcBBBEA_AL H E L L O,    W O R L D! "
| [l++-3];)for(;n-->64;)putchar(!d+++33^l&1);}
| EOT
| gcc -o$$ $$.c;clear;./$$ 52 8;rm -f $$*