using pipe results in lost messages?
Hi, this is my first posting to syslog-ng@lists Syslog-NG is very nice. This is the future of network wide logging. I am using syslog-ng-1.5.16 with no modifications as downloaded from balabit.hu with libol-0.3.2 also with no modifications as downloaded from balabit.hu I am running on RedHat v7.2 with all the latest patches. I compiled syslog-ng with "rpm -tb" to build i386.rpms. I got it set up with a minimum of hassle. Here is my /etc/syslog-ng/syslog-ng.conf file (ln -s /etc/syslog-ng /usr/etc/syslog-ng) ========================================================BEGIN syslog-ng.conf options { sync (0); time_reopen (10); log_fifo_size (100); long_hostnames (off); use_dns (yes); use_fqdn (yes); create_dirs (yes); keep_hostname (yes); time_reap(65536); }; source s_sys { unix-stream ("/dev/log"); internal(); }; source s_udp { udp(); }; destination d_files { file("/var/log/syslog-ng/$HOST/$FACILITY.$LEVEL" template("$ISODATE $HOST $FACILITY $PRIORITY $PROCESS $MSG\n")); }; destination d_pipe { pipe("/var/lib/pgsql/syslog-ng.pipe" template("INSERT INTO testtable VALUES \( '$R_ISODATE', '$S_ISODATE', '$HOST', '$FACILITY', '$PRIORITY', '$MSG'\)\;\n") template_escape(yes)); }; log { source(s_sys); source(s_udp); destination(d_files); }; log { source(s_sys); source(s_udp); destination(d_pipe); }; ==================================================================END OF FILE Then I have a script that runs: while true do read INSERT < /var/lib/pgsql/syslog-ng.pipe echo $INSERT > /tmp/outfile done And I have a testfile /tmp/wholelog with 1000 lines in it. I run "logger -p user.info -f /tmp/wholelog" at the end of the run I have 942 messages in /var/log/syslog-ng/testbox.tnsi.com/user.info and I have 332 lines in /tmp/outfile Can anyone tell me how I can tune this up so that I don't loose so many messages? Eventually I will have 100 hosts in my data center redirecting logs to this box over a 100Mb/s full duplex switched network and I will replace the script that does reads with one that redirects input to psql BTW, I have already tested a script that does: cat /var/lib/pgsql/syslog-ng.pipe | psql -U dbauser -d test And it works as long as I don't send more than about 20 messages per second. Although there are some "template_escape" problems I have seen that cause me to loose a few messages here and there... I'm not that worried about it. But losing 5% of the messages just on the localhost seems a little odd? It should be closer to 0% right? One other question I have (after reading as much of src/affile.c as I could understand) Is there a way to output the priority and facility as seperate integer values rather than as a bitshifted combined 2 digit hex value? -Ben.
On Tue, Apr 23, 2002 at 04:53:10PM -0400, Russo, Ben wrote:
Hi, this is my first posting to syslog-ng@lists Syslog-NG is very nice. This is the future of network wide logging.
The processing within syslog-ng is as follows: main poll() loop: check each destination whether they are writable and flush output queues check each source, and fetch messages. Fetched messages are stored in output queues of destinations. If output queues are full, a message gets dropped for that destination This means that there's no flow control of messages. So if the reading end of your pipe destination is not read fast enough, messages get dropped. You might rewrite your pipe() program to use perl or python, or even C. syslog-ng 2 will have the capability of using flow control on its pipes. -- Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1
participants (2)
-
Balazs Scheidler
-
Russo, Ben