[syslog-ng] flow-control and pipes

[Steen Thomsen]

Hello

 

I am currently exploring the possibilitys with flowcontrol from
syslog-ng 2.0 on a freeBSD system. 

 

I have logs coming in from different sources to a centralized logserver,
where I want to save the logs in an archive, and also read the logs into
a database. What I seek to accomplish is that in case of database
breakdown, logs should still be written to the archive, and when the
database is up again, the missing logs should then automatically be
written to the database, all in chronological order.

 

The way I plan to do it, with the new possibilities flow-control offer,
is that incoming logs are piped to a script that do 2 things. One is to
keep count of the linenumbers in the archivefiles, and add to each log
its own linenumber, before writting the log to its archivefile. The
second task is to pipe a copy of the log (with the line number added)
back to syslog-ng. Syslog-ng then pipes the log to another script that
reads the linenumber from the log, and compare it to the linenumber from
the previous log this script read. If the linenumbers are not
consecutive, it means that some logs where lost on the way from the
archivescript to this accounterscript. It therefore fetches the missing
logs from the archive, identified by the linenumbers, before reading
from the pipe again. 

 

If I use flow-control on the logs coming in from the sources and piped
to the archive script, and no flow-control on the logs that are piped
back to syslog-ng and further on to the accounterscript, I accomplish
that logs are always written to the archive, even when the database
fails, and all logs are later written to database when it comes up
again, since there is account for all the logs through their linenumber.
The basic idea being that everything is piped under normal conditions,
and I only read from the archive in case of databaseproblems.

 

The flow-control works fine for the first part of the process. I can
have the archivescript write to the archive with no loss of logs. I can
also let the archive script write the same log both to the archive and a
pipe, when I place a script at the other end of the pipe to read the
pipe, and still not loose logs.. But when I try to write to a pipe that
syslog-ng shall read, the flowcontrol doesn't seem to work. Log are lost
on the way to the archive script, even though its flowcontrolled. 

 

My question is, if there is something I have overlooked that might cause
this behaviour. It works when I read the pipe from a script of my own,
but not when syslog-ng reads it. The reason I want syslog-ng to read the
pipe, and not simply a script of my own, is that syslog-ng  will read
from the pipe, even if the database is down, where my own script would
block on the pipe, and consequently block for incoming logs, since that
part ( the other end of the pipe where the archivescript is)  is
flow-controlled.

 

I read somewhere that syslog-ng opens a pipe for both reading and
writing, but I don't quite understand how this comply with the nature of
pipes. Can anyone elaborate on this? And could that be a problem in my
setup?

 

 

Kind Regards 

Steen Thomsen

softwaredeveloper

 

  

 


 

<http://www.bridicum.com/> 

<http://www.bridicum.com/> 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20060726/309c76de/attachment.html