[syslog-ng] log path with two destinations - flow control problem

[Tomáš Novosad]

Hello to all

i'm trying to set syslog to store sendmaill messages into PostgreSQL
tables.
One simple table (tableA) which stores all messages, and another table
(tableB) which stores only some relevant iformation.

All messages are parsed in my parser, then logged into tableA, then
checked against the filter, an those messages which pass are logged into
tableB.

it all looks like this:
log {
  source(pipe_load);
  filter(f_sendmail); #-- basic filter for sendmail program
  parser(maillog); #-- parse message details
  destination(maillog_base); #-- store message in tableA - basic info only

  filter(f_sendmail_from_or_to); #-- filter only messages which have "from" or "to" address defined
  destination(maillog_detail); #-- strore the deatiled info into tableB
  flags(flow-control);
}

filterinhg itself works fine, but i have problem with flow-control

Messages are send from file (old archived maillog) into syslog pipe
source.
Problemm is, that alhough flow-control is enabled, messages get droped
on the maillog_base destination.

Im doing "cat old_maillog > syslog-pile" where old_maillog has approx 1 800 000 records,
but only about 400 000 records gets logged into tableA.
Flow control does not seem to work in this scenario.

In case when i comment-out lines with filter(f_sendmail_from_or_to) and
destination(maillog_detail)
effectively disabling logging into the 2nd table,
all messages from input file are processed and stored into tableA.
The flow control works fine in this case.

Is there something wrong with such filtering and storing only some
messages in the 2nd destination?

I also thought about having two different log path for each table, but
in that case, the parsing of the message should be done twice, which i
want to avoid.

My question is, if is it OK to use filtering like in my example,

- filter
- than destination
- then filter again
- then another destination
+ flow control

Many thanks in advance
-- 
Tomáš Novosad
LinuxBox.cz, s.r.o.
28. října 168, 709 00 Ostrava

tel.:   +420 591 166 221
mobil:  +420 737 238 655
email:  tomas.novosad at linuxbox.cz
jabber: novosad at linuxbox.cz
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis at linuxbox.cz