March 2006 - syslog-ng - lists.balabit.hu

starting syslog with unreachable loghost
by luksedj luksedj 23 Mar '06

23 Mar '06

Hello I'm trying some stuff with syslog-ng 1.9.9. Is it possible to start the syslog-ng daemon when one of the remote hosts to which messages have to be send is not reachable? it seems not to be the case, but my config might be a little bit too thin... I have two destinations for my logs, one is on the local disk, the other is a host in the network. It seems that when the host is not available syslog-ng won't start, although it should better start logging to the local disk. What happens when the remote host connection gets broken while syslog-ng is running? thx for any pointers ... pieter

3 4

Creating a named pipe (FIFO)
by Cary, Kim 22 Mar '06

22 Mar '06

We create our pipes by placing the appropriate commands in the syslog-ng startup file (/etc/init.d/syslog-ng). [Actually, we test to see if the pipe already exists; if not, we create it. We do this creation in /var/run and here the pipe needs to be recreated after every boot. We don't want to recreate it when we restart the process (syslog-ng) that is feeding it, since recreating it requires us to bounce the process that is reading from it (fisq.pl)]. Kim Cary Infrastructure Security Administrator M-F 7-4 ~ 310 506 6655

2 1

Solaris syslog-ng tuning
by Cary, Kim 22 Mar '06

22 Mar '06

Vincent! Thanks much for confirming the issue and repeating the link to me. Well, the only _intended_ udp traffic to the system is syslog. Currently, the system is logging from a PIX on one GigE interface, and from a few servers plus a less active PIX on another GigE. We send the PIX logs, separately, to pipes. And log everything to file. # vmstat 5 4 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr s0 s1 s3 -- in sy cs us sy id 0 0 4 2793888 737144 34 5 232 2 1 0 0 0 3 0 0 124 18 107 11 9 80 0 0 36 2680456 673440 5 6 0 0 0 0 0 3 5 0 0 1897 15728 2516 9 15 76 0 0 36 2680456 673440 5 5 0 0 0 0 0 0 2 0 0 1612 13349 2268 10 10 80 0 0 36 2680456 673440 5 5 0 0 0 0 0 0 3 0 0 1854 15740 2520 13 15 73 # iostat 5 4 tty sd0 sd1 sd30 nfs1 cpu tin tout kps tps serv kps tps serv kps tps serv kps tps serv us sy wt id 0 37 5 0 11 421 3 22 0 0 0 0 0 0 11 9 0 80 0 47 0 0 0 259 2 29 0 0 0 0 0 0 11 14 0 75 0 16 28 4 17 334 5 25 0 0 0 0 0 0 15 16 1 69 0 16 2 0 10 293 2 40 0 0 0 0 0 0 11 11 0 78 At: ndd /dev/udp udp_max_buf 33554432 (32Mb!) We have these time/counter readings for udpInOverflows: 00 - 645628929 33 - 645630391 96 - 645632008 Or, about 1924 packets/minute lost. At udp_max_buf 64Mb (!!!), 2713 packets/minute lost. I am FAR from out of memory 700Mb free. 1) Am I reading that loss right?? 2) Any tips from Solaris/syslog-ng tuners would be appreciated! Kim On Mar 6, 2006, at 8:49 AM, syslog-ng-request(a)lists.balabit.hu wrote: > Le Mon Mar 6 07:45:39 2006, Cary, Kim a ecrit: > | Syslog-ng 1.6.4 on Solaris 9: > | > | IPv4 > | udpInOverflows =640473547 > | > | UDP > | udpInDatagrams =409687632 udpInErrors = 0 > | udpOutDatagrams =466811 udpOutErrors = 0 > | > | Does the udpInOverflows indicate I'm losing packets? > > Yes, as mentioned in this link > http://www.29west.com/docs/THPM/udp-buffer-sizing.html given today by > Mike, it means that some udp packets could not be inserted in the > sockets buffers. > > Be careful, it means you are losing udp packets, not only syslog > packets... > > Vincent.

4 5

"MSG_DONTWAIT undeclared"-error during compiling libol-0.3.17 on HP-UX 11.00/11.11
by Tob_Sch＠gmx.de 22 Mar '06

22 Mar '06

Hi, compiling libol-0.3.16 (and syslog-ng-1.6.8) on HP-UX 11.00/11.11 worked without any problems. Today, I tried to compile libol-0.3.17 (and syslog-ng-1.6.9). I got following errors during 'make': Making all in src make all-am source='io.c' object='io.lo' libtool=yes \ depfile='.deps/io.Plo' tmpdepfile='.deps/io.TPlo' \ depmode=hp /bin/sh /tmp/libol-0.3.17/depcomp \ /bin/sh ../libtool --mode=compile cc -DHAVE_CONFIG_H -I. -I/tmp/libol-0.3.17/src -I. -D_GNU_SOURCE -D_XOPEN_SOURCE_EXTENDED -g -Wall -O2 -c -o io.lo `test -f 'io.c' || echo '/tmp/libol-0.3.17/src/'`io.c cc -DHAVE_CONFIG_H -I. -I/tmp/libol-0.3.17/src -I. -D_GNU_SOURCE -D_XOPEN_SOURCE_EXTENDED -g -Wall -O2 -c io.c -Wp,-M.deps/io.TPlo +Z -DPIC -o .libs/io.o cc: "io.c", line 493: error 1588: "MSG_DONTWAIT" undefined. cc: "io.c", line 493: warning 563: Argument #4 is not the correct type. *** Error exit code 1 Stop. *** Error exit code 1 Stop. *** Error exit code 1 Stop. It seems, that there is no MSG_DONTWAIT variable defined in the socket.h file on HP-UX like for example on SUSE 10.0. It seems, that the reference to MSG_DONTWAIT came new with libol-0.3.17. -- Lust, ein paar Euro nebenbei zu verdienen? Ohne Kosten, ohne Risiko! Satte Provisionen f�r GMX Partner: http://www.gmx.net/de/go/partner

4 8

RE: [syslog-ng] Creating a named pipe (FIFO)
by Timothy A. Holmes 22 Mar '06

22 Mar '06

Michael - -I created it and pointed syslog-ng at it and its working great -- thank you very much TIM Timothy A. Holmes IT Manager / Network Admin / Web Master / Computer Teacher Medina Christian Academy A Higher Standard... Jeremiah 33:3 Jeremiah 29:11 Esther 4:14 > -----Original Message----- > From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng- > bounces(a)lists.balabit.hu] On Behalf Of Michael Clark > Sent: Wednesday, March 22, 2006 11:36 AM > To: 'Syslog-ng users' and developers' mailing list' > Subject: RE: [syslog-ng] Creating a named pipe (FIFO) > > Timothy, > > Technically, it's not a "file" but a named pipe. That means it doesn't > take > up space on the disk. It's a stream that allows you to connect to and > read > information that is being presented on it. Once you have the > /var/syslog-ng/syslog_fifo named pipe created, you can refer to it by name > in the configuration files. I've never used a named pipe in syslog-ng (I > have in other programs), so I don't know if you use the same command that > you would to reference a file name. In the Micromuse Syslog Probe, you > have > to use a different parameter name to specify a named pipe than you would > for > a file. > > Check the syslog-ng docs to see what the command would be to specify a > named > pipe. It's possible it's the same command. > > -----Original Message----- > From: Timothy A. Holmes [mailto:tholmes@mcaschool.net] > Sent: Wednesday, March 22, 2006 9:15 AM > To: holtzl.peter(a)balabit.hu; Syslog-ng users' and developers' mailing list > Subject: RE: [syslog-ng] Creating a named pipe (FIFO) > > Does that just create the file, which is then used as the destination? > > Is that how it works? > > > Timothy A. Holmes > IT Manager / Network Admin / Web Master / Computer Teacher > > Medina Christian Academy > A Higher Standard... > > Jeremiah 33:3 > Jeremiah 29:11 > Esther 4:14 > > > -----Original Message----- > > From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng- > > bounces(a)lists.balabit.hu] On Behalf Of HÖLTZL Péter > > Sent: Wednesday, March 22, 2006 9:35 AM > > To: Syslog-ng users' and developers' mailing list > > Subject: RE: [syslog-ng] Creating a named pipe (FIFO) > > > > Hi Timothy, > > > > Insted of putting mkfifo into the syslog-ng.conf, add the command > > 'mkfifo /var/syslog-ng/syslog_fifo' from a command prompt when you are > > root. > > > > regards, > > > > Péter > > > > > > > Does the name of the named pipe need to be the same as the destination > > line > > > mkfifo splunk; > > > > > > -- > > BalaBit IT Bizt. Kft | Tel: +36 1 371-0540 | GnuPG Fingerprint: > > holtzl.peter(a)balabit.hu | Mobil: +36 20 366-9667 | 796B C9D3 E492 B006 > > C8B2 > > http://www.balabit.hu/ | Fax: +36 1 208-0875 | 4D1F 5320 28E3 9A1B > > 3FC6 > > _______________________________________________ > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > https://lists.balabit.hu/mailman/listinfo/syslog-ng > Frequently asked questions at http://www.campin.net/syslog-ng/faq.html > > > _______________________________________________ > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > https://lists.balabit.hu/mailman/listinfo/syslog-ng > Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

1 0

RE: [syslog-ng] Creating a named pipe (FIFO)
by Timothy A. Holmes 22 Mar '06

22 Mar '06

Does that just create the file, which is then used as the destination? Is that how it works? Timothy A. Holmes IT Manager / Network Admin / Web Master / Computer Teacher Medina Christian Academy A Higher Standard... Jeremiah 33:3 Jeremiah 29:11 Esther 4:14 > -----Original Message----- > From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng- > bounces(a)lists.balabit.hu] On Behalf Of HÖLTZL Péter > Sent: Wednesday, March 22, 2006 9:35 AM > To: Syslog-ng users' and developers' mailing list > Subject: RE: [syslog-ng] Creating a named pipe (FIFO) > > Hi Timothy, > > Insted of putting mkfifo into the syslog-ng.conf, add the command > 'mkfifo /var/syslog-ng/syslog_fifo' from a command prompt when you are > root. > > regards, > > Péter > > > > Does the name of the named pipe need to be the same as the destination > line > > mkfifo splunk; > > > -- > BalaBit IT Bizt. Kft | Tel: +36 1 371-0540 | GnuPG Fingerprint: > holtzl.peter(a)balabit.hu | Mobil: +36 20 366-9667 | 796B C9D3 E492 B006 > C8B2 > http://www.balabit.hu/ | Fax: +36 1 208-0875 | 4D1F 5320 28E3 9A1B > 3FC6

2 1

RE: 1.9.9 segmentation fault
by Balazs Scheidler 22 Mar '06

22 Mar '06

On Wed, 2006-03-22 at 07:43 -0600, Martin, David M wrote: > here you go. let me know if you need additional info. And here's the patch which should fix it: --- orig/src/logwriter.c +++ mod/src/logwriter.c @@ -327,6 +327,7 @@ log_writer_flush_log(LogWriter *self, FD line = NULL; if (errno != EAGAIN) goto write_error; + return TRUE; } else { -- Bazsi

1 0

Creating a named pipe (FIFO)
by Timothy A. Holmes 22 Mar '06

22 Mar '06

Good Morning All: I have hit a bit of a problem with the set up of my Splunk system. I am trying to set it up for using a named pipe to get my data into splunk. I have the logging server receiving data and logging it to files with no problem, but, someplace, I have gotten the named pipe configuration messed up. Here is my syslog-ng.conf file # $Header: /var/cvsroot/gentoo-x86/app-admin/syslog-ng/files/syslog-ng.conf.gentoo, v 1.5 2005/05/12 05:46:10 mr_bones_ Exp $ # # Syslog-ng default configuration file for Gentoo Linux # contributed by Michael Sterrett options { chain_hostnames(off); sync(0); # The default action of syslog-ng 1.6.0 is to log a STATS line # to the file every 10 minutes. That's pretty ugly after a while. # Change it to every 12 hours so you get a nice daily update of # how many messages syslog-ng missed (0). stats(43200); }; source src { unix-stream("/dev/log"); internal(); pipe("/proc/kmsg"); }; destination messages { file("/var/log/messages"); }; source remote { udp(); }; destination splunk { pipe("/var/syslog-ng/syslog_fifo"); }; log{ source(remote); destination(splunk); }; destination hosts { file("/var/log/hosts/$HOST/messages" owner(root) group(logs) perm(0640) dir_perm(0750) create_dirs(yes) ); }; log { source(remote); destination(hosts); }; # By default messages are logged to tty12... destination console_all { file("/dev/tty12"); }; # ...if you intend to use /dev/console for programs like xconsole # you can comment out the destination line above that references /dev/tty12 # and uncomment the line below. #destination console_all { file("/dev/console"); }; log { source(src); destination(messages); }; log { source(src); destination(console_all); }; srvnms-01 ~ # As the file is currently configured, syslog-ng starts with no problems and runs logging to the files as specified in the destination hosts directive. The data never appears in the named pipe. I did some investigation and asked a few questions on IRC, and someone told me that I needed to use the mkpipe command, however I have been unable to locate (using google) the proper syntax for the mkpipe command. Another person told me that once I got it set up, I would need to configure logrotate as well. Any and all pointers that you can provide would be most welcome. I have gone over the FAQ and also the users manual. As well as googling about for the proper answers. If you prefer to just point me to resources and let me figure it out myself, That is also quite cool, I like to learn. Thank you Timothy A. Holmes IT Manager / Network Admin / Web Master / Computer Teacher Medina Christian Academy A Higher Standard... Jeremiah 33:3 Jeremiah 29:11 Esther 4:14

2 1

1.9.9 segmentation fault
by Martin, David M 22 Mar '06

22 Mar '06

testing with syslog-ng 1.9.9 recently configured a pipe destination (config below) log processing to the pipe occurs as expected - then get a segmentation fault have seen it run perfectly for days, other times hours/minutes Linux distribution is FC4 (also RHEL 3 and 4) server has handled 500k-2M transactions daily over the past week (works fine when logging to file). below if the info I've collected so far. Let me know if you need additional. any ideas? strace -p {pid} output: time([1142745045]) = 1142745045 time([1142745045]) = 1142745045 gettimeofday({1142745045, 307328}, NULL) = 0 poll([{fd=6, events=POLLOUT, revents=POLLOUT}, {fd=4, events=POLLIN}, {fd=3, events=POLLIN}], 3, 550317) = 1 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1279, ...}) =0 write(6, "INSERT INTO logs (\n host,\n "..., 625) = 625 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1279, ...}) =0 write(6, "INSERT INTO logs (\n host,\n "..., 625) = 625 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1279, ...}) =0 write(6, "INSERT INTO logs (\n host,\n "..., 625) = 625 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1279, ...}) =0 write(6, "INSERT INTO logs (\n host,\n "..., 625) = 625 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1279, ...}) =0 write(6, "INSERT INTO logs (\n host,\n "..., 625) = 625 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1279, ...}) =0 write(6, "INSERT INTO logs (\n host,\n "..., 585) = 585 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1279, ...}) =0 write(6, "INSERT INTO logs (\n host,\n "..., 746) = 746 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1279, ...}) =0 write(6, "INSERT INTO logs (\n host,\n "..., 644) = -1 EAGAIN (Resource temporarily unavailable) gettimeofday({1142745045, 310203}, NULL) = 0 time(NULL) = 1142745045 time(NULL) = 1142745045 gettimeofday({1142745045, 310286}, NULL) = 0 time(NULL) = 1142745045 time(NULL) = 1142745045 --- SIGSEGV (Segmentation fault) @ 0 (0) --- kill(12488, SIGSEGV) = 0 sigreturn() = ? (mask now []) --- SIGSEGV (Segmentation fault) @ 0 (0) --- internal shows nothing out of the ordinary. daemon in foreground w/debug: Reducing stack by rule 2 (line 188), stmt ';' stmts -> stmts Stack now 0 9 25 Entering state 74 Reducing stack by rule 2 (line 188), stmt ';' stmts -> stmts Stack now 0 Entering state 8 Reducing stack by rule 1 (line 184), stmts -> start Stack now 0 Entering state 7 Now at end of input. Segmentation fault Here is the config: #------------------------------------------------------------------------- options { chain_hostnames(no); #default yes keep_hostname(yes); #default no create_dirs(no); #default use_dns(no); #default yes use_fqdn(no); #default owner(root); group(root); perm(0600); }; ############################################################################### # sources source abs_udp514 { udp(ip(0.0.0.0) port(514)); }; source abs_localhost { internal(); }; ############################################################################### # destinations destination abd_localhost { file( "/var/log/localhost.log" ); }; destination abd_mysql { pipe( "/tmp/mysql.pipe" template("INSERT INTO logs ( host, facility, priority, datetime, date, time, year, month, day, hour, min, sec, week, unixtime, tzoffset, tz, program, msg ) VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC', '$YEAR-$MONTH-$DAY', '$HOUR:$MIN:$SEC', '$YEAR', '$MONTH', '$DAY', '$HOUR', '$MIN', '$SEC', '$WEEK', '$UNIXTIME', '$TZOFFSET', '$TZ', '$PROGRAM', '$MSG');\n" ) template_escape(yes) ); }; ############################################################################### # logs log { source(abs_udp514); destination(abd_mysql); }; log { source(abs_localhost); destination(abd_localhost); }; # #------------------------------------------------------------------------- Thanks, The information transmitted (including attachments) is covered by the Electronic Communications Privacy Act, 18 U.S.C. 2510-2521, is intended only for the person(s) or entity/entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited. If you received this in error, please contact the sender and delete the material from any computer.

2 1

netmask
by Olleg Samoylov 18 Mar '06

18 Mar '06

After upgrade from 1.6.x to 1.9.9 netmask filter rule don't work. Write "syntax error at 188" on "netmask(149.49.32.121/255.255.255.255);". -- Olleg Samoylov

2 2