April 2002 - syslog-ng - lists.balabit.hu

template() usage with non-file() output
by Tycho Fruru 25 Apr '02

25 Apr '02

Hello all, Apparently, templates are not (yet ?) supported for non-file output methods, which is too bad for me (wanting to use templates with an external program() or tcp forwarding) :( I've looked rapidly at the code but it seems that template functionality is part of the file affile.c ... so perhaps it will not be easy to extract this and make a generic template mechanism. I would like to know whether anyone of you guys already have written or are planning a patch to include this functionality (which would of course make syslog-ng even better than it is today :-) Cheers, Tycho -- Tycho Fruru tycho.fruru(a)conostix.com "Prediction is extremely difficult. Especially about the future." - Niels Bohr

2 1

Flush message buffer
by Jerome Peducasse 25 Apr '02

25 Apr '02

When a program sends (unfortunaly) a line with '\n' such as "part1\npart2", syslog-ng writes part1 in appropriate destination but when the next message is received, it is "polluted" by part2 of previous message and it losts its facility/priority. For sample: [root@peducasse log]# logger -psyslog.info "Test #1 > Test #2" [root@peducasse log]# logger -psyslog.info "Test #3" condensed strace output: read(3, "<46>Apr 25 17:02:53 jerome: Test #1\nTest #2", 2048) = 43 write(4, "Apr 25 17:02:53 peducasse [syslog.info] jerome jerome: Test #1\n", 63) = 63 read(3, "<46>Apr 25 17:02:56 jerome: Test #3", 2041) = 35 write(14, "Apr 25 17:02:56 peducasse Test #2<46>Apr 25 17:02:56 jerome: Test #3\n", 69) = 69 Thanks for any help. -- Jerome Peducasse

1 0

syslog-ng 1.5.16 released
by Balazs Scheidler 25 Apr '02

25 Apr '02

Hi, I have released syslog-ng 1.5.16 with the following NEWS entry: News for the 1.5.16 release Fri, 12 Apr 2002 09:42:28 +0200 Changes: * added log_prefix support for file sources (for adding 'kernel: ' prefix to messages from /proc/kmsg) * fixed the internal DNS cache, so now it really works * fixed localip() and localport() directives for afinet destinations (they erroneously changed the destination address) * Detect O_LARGEFILE on the system and use it if available * Never close fds 0, 1, 2 and, use /dev/null for 0 and 1, and keep 2 open for messages during reinitialization * Added UNIXTIME, R_UNIXTIME, S_UNIXTIME macros to templates and filenames * Fixed TCP wrapper support. Our webserver is currently down, but as soon as it comes back up, I'll upload this release to the usual place: http://www.balabit.hu/en/downloads/syslog-ng/ -- Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1

2 1

RE: [syslog-ng]AIX 4.3.3 and pipe problem?
by Michael Earls 25 Apr '02

25 Apr '02

i am not sure how to do that, can u explain how michael -----Original Message----- From: bazsi(a)balabit.hu Sent: Thursday, April 25, 2002 5:04 AM To: <syslog-ng(a)lists.balabit.hu> Cc: <AGehring(a)bis.adp.com> Subject: Re: [syslog-ng]AIX 4.3.3 and pipe problem? On Wed, Apr 24, 2002 at 09:07:07AM 0400, Michael Earls wrote: > i am running the pkg from debian /testing, 1.5.15. and i am trying to > run this command. i created my fifo pipe and i am not seeing anything > being posted there. > > what am i doing wrong? > > destination d_mysql { > pipe("/tmp/mysql.pipe" > template("INSERT INTO logs (host, facility, priority, level, tag, > date, > time, program, msg) VALUES ('$HOST', > '$FACILILITY','$PRIORITY',$LEVEL','$TAG','$YEAR $MONTH $DAY','$HOUR:$MIN:$SEC','$PROGRAM','$MSG');\n") > template escpae(yes)); > }; > log { source(net); destination(d_mysql); > }; Can you strace the process as it receives a single message? Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1 _______________________________________________ syslog ng maillist syslog ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog ng

1 0

RE: [syslog-ng]AIX 4.3.3 and pipe problem?
by Michael Earls 25 Apr '02

25 Apr '02

i am running the pkg from debian /testing, 1.5.15. and i am trying to run this command. i created my fifo pipe and i am not seeing anything being posted there. what am i doing wrong? destination d_mysql { pipe("/tmp/mysql.pipe" template("INSERT INTO logs (host, facility, priority, level, tag, date, time, program, msg) VALUES ('$HOST', '$FACILILITY','$PRIORITY',$LEVEL','$TAG','$YEAR-$MONTH-$DAY','$HOUR:$MIN:$SEC','$PROGRAM','$MSG');\n") template-escpae(yes)); }; log { source(net); destination(d_mysql); }; thanks michael -----Original Message----- From: bazsi(a)balabit.hu Sent: Wednesday, April 24, 2002 3:33 AM To: <syslog-ng(a)lists.balabit.hu> Cc: <AGehring(a)bis.adp.com> Subject: Re: [syslog-ng]AIX 4.3.3 and pipe problem? On Tue, Apr 23, 2002 at 12:43:56PM 0400, Michael Earls wrote: > i have not seen that on the AIX but i have seen that on linux ver > 1.5.15, the pipe does not work. I've just tested my CVS version and it works. (it has a few changes against 1.5.16) Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1 _______________________________________________ syslog ng maillist syslog ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog ng

2 1

syslog-ng dying after kill -HUP
by Robert Berlinger 25 Apr '02

25 Apr '02

Now experimenting with 1.5.16. Works OK except it consistently dies with a Segmentation Fault after a kill -HUP. This is on a RH 7.2 system. I ran it with the -d option, and the fault occurs after printing io.c: listening on fd 3 io.c: Preparing fd 4 for reading io.c: listening on fd 5 Segmentation fault Any pointers would be greatly appreciated.

3 2

RE: [syslog-ng]Random Directories
by Tom Horan 25 Apr '02

25 Apr '02

I had a similar problem on my host creating a 'last' directory. So I changed keep_hostname to (no), afaik this means that syslog-ng adds the hostname into the message incase the sending server hasnt done it. It started logging these type of messages to the correct server. Hope it helps, Tom -----Original Message----- From: Nate Campi [mailto:nate@campin.net] Sent: 24 April 2002 18:47 To: syslog-ng(a)lists.balabit.hu Subject: Re: [syslog-ng]Random Directories On Wed, Apr 24, 2002 at 10:32:20AM -0700, Painter, Jennifer wrote: > Directories with names like " Error", " SCSI", "," > > Are showing up in the directory that holds the syslogs for the different hosts that we monitor. > > Has anyone seen these random directories. Any suggestions on how to deal with them. I archive by host, but syslog-ng can only do so much with certain messages. [root@loghost last]# cat /var/log/HOSTS/last/local2/2001/09/14/local220010914 Sep 14 12:21:58 last message repeated 17 times Sep 14 12:21:58 last message repeated 17 times Sep 14 12:21:58 last message repeated 17 times Sep 14 12:21:58 last message repeated 17 times Obviously I have no host named "last", syslog-ng can't second guess the hostname sent when it could be valid. I want a macro taken from gethostbyaddr() instead of from the message contents that could be used to force a valid hostname no matter what. Baszi, could this be done please? It would help all of us who use syslog-ng for log archiving in the filesystem. -- Linux is like an indian's tent: No gates, no windows, and apache inside. _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng ************************************************************************ This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. http://www.esatbusiness.com Subscribe to the Esat Business Online Magazine: http://www.esatbusiness.com/news/subscribe.asp Subscribe to REALISE - the online magazine from BT Ignite: http://www.btignite.com/realise ************************************************************************

2 1

largefile-problem with linux 2.4.18 / glibc 2.2.4
by Jussi Silvennoinen 24 Apr '02

24 Apr '02

Hello, I'm having a problem with largefile-support with syslog-ng 1.5.16 on redhat 7.2 with all all updates as of April 19 installed. Log-partition is a 300gb hardware raid5 through an AMI Megaraid Elite 1600, filesystem is XFS dated 2002-03-04 with latest userspace-utils installed. Other software installed on the box have no problems with largefile-support. Output of ./configure and make [root@gull syslog-ng-1.5.16]# ./configure creating cache ./config.cache checking for a BSD compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking whether make sets ${MAKE}... yes checking for working aclocal... found checking for working autoconf... found checking for working automake... found checking for working autoheader... found checking for working makeinfo... found checking whether build environment is sane... yes checking for gcc... gcc checking whether the C compiler (gcc ) works... yes checking whether the C compiler (gcc ) is a cross-compiler... no checking whether we are using GNU C... yes checking whether gcc accepts -g... yes checking for gcc option to accept ANSI C... none needed checking for bison... bison -y checking how to run the C preprocessor... gcc -E checking for flex... flex checking for flex... (cached) flex checking for yywrap in -lfl... yes checking lex output file root... lex.yy checking whether yytext is a pointer... yes checking whether make sets ${MAKE}... (cached) yes checking for ANSI C header files... yes checking for malloc.h... yes checking for unistd.h... yes checking for door.h... no checking for stropts.h... yes checking for sys/strlog.h... no checking for stdarg.h... yes checking for sys/klog.h... yes checking for working const... yes checking whether time.h and sys/time.h may both be included... yes checking for modern utmp... yes checking whether to compile klogctl... yes checking size of short... 2 checking size of int... 4 checking size of long... 4 checking for I_CONSLOG... no checking for O_LARGEFILE... yes checking for working alloca.h... yes checking for alloca... yes checking for vprintf... yes checking for door_create in -ldoor... no checking for select... yes checking for snprintf... yes checking for vsnprintf... yes checking for strerror... yes checking for inet_aton... yes checking for strncpy... yes checking for getutent... yes checking for getopt_long... yes checking for strcasecmp... yes checking for strptime... yes checking whether to enable Sun STREAMS support... no checking whether to enable Sun door support... no checking whether to enable TCP wrapper support... no checking libol version >= 0.3.2... ok updating cache ./config.cache creating ./config.status creating Makefile creating src/Makefile creating src/tests/Makefile creating doc/Makefile creating doc/sgml/Makefile creating contrib/Makefile creating utils/Makefile sed: can't read /usr/local/src/testifoo/syslog-ng-1.5.16/utils/Makefile.in: No such file or directory creating syslog-ng.spec creating src/config.h [root@gull syslog-ng-1.5.16]# make Making all in src make[1]: Entering directory `/usr/local/src/testifoo/syslog-ng-1.5.16/src' cd /usr/local/src/testifoo/syslog-ng-1.5.16 && autoheader make all-recursive make[2]: Entering directory `/usr/local/src/testifoo/syslog-ng-1.5.16/src' Making all in . make[3]: Entering directory `/usr/local/src/testifoo/syslog-ng-1.5.16/src' gcc -DHAVE_CONFIG_H -I. -I/usr/local/src/testifoo/syslog-ng-1.5.16/src -I. -g -O2 -Wall -I/usr/local/include/libol -D_GNU_SOURCE -c main.c gcc -DHAVE_CONFIG_H -I. -I/usr/local/src/testifoo/syslog-ng-1.5.16/src -I. -g -O2 -Wall -I/usr/local/include/libol -D_GNU_SOURCE -c sources.c gcc -DHAVE_CONFIG_H -I. -I/usr/local/src/testifoo/syslog-ng-1.5.16/src -I. -g -O2 -Wall -I/usr/local/include/libol -D_GNU_SOURCE -c center.c gcc -DHAVE_CONFIG_H -I. -I/usr/local/src/testifoo/syslog-ng-1.5.16/src -I. -g -O2 -Wall -I/usr/local/include/libol -D_GNU_SOURCE -c filters.c filters.c: In function `do_filter_call': filters.c:342: warning: implicit declaration of function `memcmp' gcc -DHAVE_CONFIG_H -I. -I/usr/local/src/testifoo/syslog-ng-1.5.16/src -I. -g -O2 -Wall -I/usr/local/include/libol -D_GNU_SOURCE -c destinations.c gcc -DHAVE_CONFIG_H -I. -I/usr/local/src/testifoo/syslog-ng-1.5.16/src -I. -g -O2 -Wall -I/usr/local/include/libol -D_GNU_SOURCE -c log.c gcc -DHAVE_CONFIG_H -I. -I/usr/local/src/testifoo/syslog-ng-1.5.16/src -I. -g -O2 -Wall -I/usr/local/include/libol -D_GNU_SOURCE -c cfgfile.c gcc -DHAVE_CONFIG_H -I. -I/usr/local/src/testifoo/syslog-ng-1.5.16/src -I. -g -O2 -Wall -I/usr/local/include/libol -D_GNU_SOURCE -c cfg-grammar.c gcc -DHAVE_CONFIG_H -I. -I/usr/local/src/testifoo/syslog-ng-1.5.16/src -I. -g -O2 -Wall -I/usr/local/include/libol -D_GNU_SOURCE -c cfg-lex.c gcc -DHAVE_CONFIG_H -I. -I/usr/local/src/testifoo/syslog-ng-1.5.16/src -I. -g -O2 -Wall -I/usr/local/include/libol -D_GNU_SOURCE -c affile.c gcc -DHAVE_CONFIG_H -I. -I/usr/local/src/testifoo/syslog-ng-1.5.16/src -I. -g -O2 -Wall -I/usr/local/include/libol -D_GNU_SOURCE -c afsocket.c gcc -DHAVE_CONFIG_H -I. -I/usr/local/src/testifoo/syslog-ng-1.5.16/src -I. -g -O2 -Wall -I/usr/local/include/libol -D_GNU_SOURCE -c afunix.c gcc -DHAVE_CONFIG_H -I. -I/usr/local/src/testifoo/syslog-ng-1.5.16/src -I. -g -O2 -Wall -I/usr/local/include/libol -D_GNU_SOURCE -c afinet.c gcc -DHAVE_CONFIG_H -I. -I/usr/local/src/testifoo/syslog-ng-1.5.16/src -I. -g -O2 -Wall -I/usr/local/include/libol -D_GNU_SOURCE -c afinter.c gcc -DHAVE_CONFIG_H -I. -I/usr/local/src/testifoo/syslog-ng-1.5.16/src -I. -g -O2 -Wall -I/usr/local/include/libol -D_GNU_SOURCE -c afuser.c gcc -DHAVE_CONFIG_H -I. -I/usr/local/src/testifoo/syslog-ng-1.5.16/src -I. -g -O2 -Wall -I/usr/local/include/libol -D_GNU_SOURCE -c afstreams.c gcc -DHAVE_CONFIG_H -I. -I/usr/local/src/testifoo/syslog-ng-1.5.16/src -I. -g -O2 -Wall -I/usr/local/include/libol -D_GNU_SOURCE -c afprogram.c gcc -DHAVE_CONFIG_H -I. -I/usr/local/src/testifoo/syslog-ng-1.5.16/src -I. -g -O2 -Wall -I/usr/local/include/libol -D_GNU_SOURCE -c afremctrl.c afremctrl.c: In function `do_handle_afremctrl_dest': afremctrl.c:56: warning: implicit declaration of function `strcmp' gcc -DHAVE_CONFIG_H -I. -I/usr/local/src/testifoo/syslog-ng-1.5.16/src -I. -g -O2 -Wall -I/usr/local/include/libol -D_GNU_SOURCE -c nscache.c gcc -DHAVE_CONFIG_H -I. -I/usr/local/src/testifoo/syslog-ng-1.5.16/src -I. -g -O2 -Wall -I/usr/local/include/libol -D_GNU_SOURCE -c utils.c gcc -DHAVE_CONFIG_H -I. -I/usr/local/src/testifoo/syslog-ng-1.5.16/src -I. -g -O2 -Wall -I/usr/local/include/libol -D_GNU_SOURCE -c syslog-names.c gcc -g -O2 -Wall -I/usr/local/include/libol -D_GNU_SOURCE -o syslog-ng main.o sources.o center.o filters.o destinations.o log.o cfgfile.o cfg-grammar.o cfg-lex.o affile.o afsocket.o afunix.o afinet.o afinter.o afuser.o afstreams.o afprogram.o afremctrl.o nscache.o utils.o syslog-names.o -lfl /usr/local/lib/libol.a -lnsl make[3]: Leaving directory `/usr/local/src/testifoo/syslog-ng-1.5.16/src' Making all in tests make[3]: Entering directory `/usr/local/src/testifoo/syslog-ng-1.5.16/src/tests' gcc -DHAVE_CONFIG_H -I. -I/usr/local/src/testifoo/syslog-ng-1.5.16/src/tests -I../../src -g -O2 -Wall -I/usr/local/include/libol -D_GNU_SOURCE -c test_nscache.c gcc -g -O2 -Wall -I/usr/local/include/libol -D_GNU_SOURCE -o test_nscache test_nscache.o ../nscache.o -lfl /usr/local/lib/libol.a -lnsl make[3]: Leaving directory `/usr/local/src/testifoo/syslog-ng-1.5.16/src/tests' make[2]: Leaving directory `/usr/local/src/testifoo/syslog-ng-1.5.16/src' make[1]: Leaving directory `/usr/local/src/testifoo/syslog-ng-1.5.16/src' Making all in doc make[1]: Entering directory `/usr/local/src/testifoo/syslog-ng-1.5.16/doc' Making all in sgml make[2]: Entering directory `/usr/local/src/testifoo/syslog-ng-1.5.16/doc/sgml' make[2]: Nothing to be done for `all'. make[2]: Leaving directory `/usr/local/src/testifoo/syslog-ng-1.5.16/doc/sgml' make[2]: Entering directory `/usr/local/src/testifoo/syslog-ng-1.5.16/doc' make[2]: Nothing to be done for `all-am'. make[2]: Leaving directory `/usr/local/src/testifoo/syslog-ng-1.5.16/doc' make[1]: Leaving directory `/usr/local/src/testifoo/syslog-ng-1.5.16/doc' Making all in contrib make[1]: Entering directory `/usr/local/src/testifoo/syslog-ng-1.5.16/contrib' make[1]: Nothing to be done for `all'. make[1]: Leaving directory `/usr/local/src/testifoo/syslog-ng-1.5.16/contrib' make[1]: Entering directory `/usr/local/src/testifoo/syslog-ng-1.5.16' make[1]: Nothing to be done for `all-am'. make[1]: Leaving directory `/usr/local/src/testifoo/syslog-ng-1.5.16' From this I would assume that it would support largefiles, however it appears it does not. [root@gull syslog-ng-1.5.16]# src/syslog-ng -d io.c: listening on fd 3 io.c: Preparing fd 4 for reading syslog-ng version 1.5.16 starting io.c: Preparing fd 5 for writing io.c: Preparing fd 6 for writing io.c: Preparing fd 7 for writing io.c: Preparing fd 8 for writing io.c: Preparing fd 9 for writing File size limit exceeded [root@gull syslog-ng-1.5.16]# ls -l /data/syslog/mail.log -rw-rw---- 1 root root 2147483647 Apr 18 21:54 /data/syslog/mail.log Any ideas how to proceed solving this problem? I am currently running syslog-ng 1.4.14 on Solaris 8 without any problems regarding largefile-support. -- Jussi

3 3

Re: [syslog-ng]rotation log for syslog-ng
by klaas.vangend＠philips.com 24 Apr '02

24 Apr '02

> Hello, > i am trying to use syslog-ng. > Could anyone brievely explain me how to have rotation log ? > I can find any document about that. Is it possible ? > Haw can i fixe the size of the files, and how can i specifie > the number of files i want ? > Thanks > -- > Jérôme Jerome (sorry, no accents on my keyboard), Syslog-ng in the latest 1.5 versions can handle log rotation in the following way: * use macros to create logs per weekday or monthday * remove a log file if it is older (i.e. not written to) than one week minus 1 day . Example: destination warn { file("/var/log/warn.$WEEKDAY" remove_if_older(432000)); }; filter f_allwarnings { level(warn..emerg); }; log { source(src); source(kernel); filter(f_allwarnings); destination(warn); }; For more advanced log rotation, you will need to use the usual log rotation scripts/daemons. Groetjes, Klaas van Gend

1 0

using pipe results in lost messages?
by Russo, Ben 24 Apr '02

24 Apr '02

Hi, this is my first posting to syslog-ng@lists Syslog-NG is very nice. This is the future of network wide logging. I am using syslog-ng-1.5.16 with no modifications as downloaded from balabit.hu with libol-0.3.2 also with no modifications as downloaded from balabit.hu I am running on RedHat v7.2 with all the latest patches. I compiled syslog-ng with "rpm -tb" to build i386.rpms. I got it set up with a minimum of hassle. Here is my /etc/syslog-ng/syslog-ng.conf file (ln -s /etc/syslog-ng /usr/etc/syslog-ng) ========================================================BEGIN syslog-ng.conf options { sync (0); time_reopen (10); log_fifo_size (100); long_hostnames (off); use_dns (yes); use_fqdn (yes); create_dirs (yes); keep_hostname (yes); time_reap(65536); }; source s_sys { unix-stream ("/dev/log"); internal(); }; source s_udp { udp(); }; destination d_files { file("/var/log/syslog-ng/$HOST/$FACILITY.$LEVEL" template("$ISODATE $HOST $FACILITY $PRIORITY $PROCESS $MSG\n")); }; destination d_pipe { pipe("/var/lib/pgsql/syslog-ng.pipe" template("INSERT INTO testtable VALUES $ '$R_ISODATE', '$S_ISODATE', '$HOST', '$FACILITY', '$PRIORITY', '$MSG'$\;\n") template_escape(yes)); }; log { source(s_sys); source(s_udp); destination(d_files); }; log { source(s_sys); source(s_udp); destination(d_pipe); }; ==================================================================END OF FILE Then I have a script that runs: while true do read INSERT < /var/lib/pgsql/syslog-ng.pipe echo $INSERT > /tmp/outfile done And I have a testfile /tmp/wholelog with 1000 lines in it. I run "logger -p user.info -f /tmp/wholelog" at the end of the run I have 942 messages in /var/log/syslog-ng/testbox.tnsi.com/user.info and I have 332 lines in /tmp/outfile Can anyone tell me how I can tune this up so that I don't loose so many messages? Eventually I will have 100 hosts in my data center redirecting logs to this box over a 100Mb/s full duplex switched network and I will replace the script that does reads with one that redirects input to psql BTW, I have already tested a script that does: cat /var/lib/pgsql/syslog-ng.pipe | psql -U dbauser -d test And it works as long as I don't send more than about 20 messages per second. Although there are some "template_escape" problems I have seen that cause me to loose a few messages here and there... I'm not that worried about it. But losing 5% of the messages just on the localhost seems a little odd? It should be closer to 0% right? One other question I have (after reading as much of src/affile.c as I could understand) Is there a way to output the priority and facility as seperate integer values rather than as a bitshifted combined 2 digit hex value? -Ben.

2 1