- syslog-ng - lists.balabit.hu

Logrotate ?
by Brian Ipsen 19 Feb '05

19 Feb '05

Hi, The re-occurring question... How com that logrotate isn't working on my logs ?? I have a file in /etc/logrotate.d like this: /var/log/snmp/*.log { weekly notifempty compress sharedscripts missingok prerotate /bin/kill -HUP `cat /var/run/syslog-ng.pid 2> /dev/null` 2> /dev/null || true /etc/init.d/syslog-ng restart endscript postrotate /bin/kill -HUP `cat /var/run/syslog-ng.pid 2> /dev/null` 2> /dev/null || true /etc/init.d/syslog-ng restart touch /var/log/snmp/network.log /var/log/snmp/snmptrapd.log /var/log/snmp/ups.log /var/log/snmp/vpn.log /var/log/snmp/witraz.log endscript } I did the kill -HUP both as pre- and post-rotate because there seems to a problem, I've even tried to restart the daemon - nothing helps... Checking my /var/log/snmp dir gives me: -rw-r--r-- 1 root root 2935158 Feb 18 20:12 network.log -rw------- 1 root root 1106 Nov 18 23:44 network.log.1.gz -rw------- 1 root root 7981 Aug 24 21:19 network.log.2.gz -rw------- 1 root root 1664 Jul 31 2004 network.log.3.gz -rw------- 1 root root 7012 Jul 23 2004 network.log.4.gz -rw-r--r-- 1 root root 70908362 Feb 18 20:14 snmptrapd.log -rw------- 1 root root 28929 Nov 19 04:00 snmptrapd.log.1.gz -rw------- 1 root root 4591 Aug 25 02:58 snmptrapd.log.2.gz -rw------- 1 root root 3366 Aug 1 2004 snmptrapd.log.3.gz -rw------- 1 root root 4729 Jul 25 2004 snmptrapd.log.4.gz -rw-r--r-- 1 root root 767 Feb 3 09:43 ups.log -rw------- 1 root root 467 Aug 24 14:36 ups.log.1.gz -rw------- 1 root root 220 Jul 15 2004 ups.log.2.gz -rw-r--r-- 1 root root 36442477 Feb 18 20:10 vpn.log -rw-r--r-- 1 root root 1045305 Dec 30 10:35 vpn.log.1.gz -rw------- 1 root root 36723 Nov 21 03:56 vpn.log.2.gz -rw-r--r-- 1 root root 104 Nov 4 09:40 vpn-trap.log.1.gz -rw-r--r-- 1 root root 1533667 Feb 17 17:17 witraz.log -rw------- 1 root root 553 Nov 19 17:49 witraz.log.1.gz -rw------- 1 root root 187 Aug 24 15:56 witraz.log.2.gz -rw------- 1 root root 620 Jul 31 2004 witraz.log.3.gz -rw------- 1 root root 703 Jul 24 2004 witraz.log.4.gz I've rotated some of the logfiles manually - but check e.g. network.log - it's been 3 months since last rotate ?? I'm running on a RHEL 3.0 Any idea why this logrotate isn't working ?? /Brian

1 0

FAQ entries wanted for performance tuning
by Nate Campi 18 Feb '05

18 Feb '05

Everyone please send me your syslog-ng performance tuning tips from the real world. I added something to the end of the FAQ, but I know there's much much more we can add from our collective experience. Email directly to me is appropriate for this. I'm happy to include OS-level stuff, since the focus here is to build a fast loghost. This includes everything along the path from the NIC through the filesystem and onto some kind of final storage. Of course the focus is on syslog-ng, though. :) -- Nate "It is good sportsmanship to not pick up lost golf balls while they are still rolling." - Samuel Clemens

1 0

new to syslog-ng
by Mike Pepe 17 Feb '05

17 Feb '05

Hello all, I'm new to syslog-ng. I have the program ready to go, but I was wondering if there's a faq or guide somewhere so I can just replace regular syslog functionality on Fedora Core 3. Once I get that working normally, I want to then log different events to different files. Any pointers would be appreciated. thanks! -Mike

5 7

Message Templates and Syslog ID
by Rhugga 16 Feb '05

16 Feb '05

Im trying to normalize the logs from my various systems I support (Solaris, IRIX, Tru64, SLES 8 & 9, RHAS 3, Cisco, Brocade, etc...) These logs are processed and stored into oracle. On Solaris, syslogd adds a field containing the syslog message ID and the facility and priority: EX: Jan 20 22:07:26 back-0202 tldd[3184]: [ID 817171 daemon.notice] Calling the drive_mount_notify ^^^^^^^^^^^^^^^^^^^^^^^^^ This behavior can only be changed by a reboot and sun also does not recommend setting msgid=0 in /kernerl/drv/log.conf. So, using the following template I can come close to normalizing this behavior: template("$DATE $HOST $PROGRAM [ID $FACILITY.$LEVEL] $MSG\n") The information that is lost is the internal syslog ID and the pid of the sending process. Here is a log entry processed thru this template: Feb 14 07:07:52 db-0201 su [ID auth.info] su: [ID 366847 auth.info] 'su oracle' succeeded for root on /dev/??? Another problem is that field in question now appears twice. Anyone know of any possible way to configure syslog-ng to solves this roadblock? Thx, CC

2 2

FIFO / pipe destination problem
by Howell, Matt 15 Feb '05

15 Feb '05

This may end up being something really easy that I am missing, but I have been struggling with it for a couple days. Googling the problem has yet to review any fruitful answers. I am trying to pass messages to MySQL via the pipe() function. I am working with syslog-ng 1.6.5-6 compiled from source for Fedora Core 3. I have setup a FIFO in the filesystem as follows: [root@furw tmp]# mkfifo mysql.pipe [root@furw tmp]# chmod 0600 mysql.pipe [root@furw tmp]# ls -al ... drwxrwxrwt 24 root root 4096 Feb 14 17:38 . drwxr-xr-x 23 root root 4096 Feb 11 16:00 .. prw------- 1 root root 0 Feb 14 17:38 mysql.pipe ... I then created my syslog-ng.conf file: ... options { sync (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source src { unix-stream ("/dev/log"); internal(); udp(ip(0.0.0.0) port(514)); }; destination d_mysql { pipe("/tmp/mysql.pipe" template("INSERT INTO logs (host, facility, priority, level, tag, date, time, program, msg) VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY', '$HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n") template-escape(yes)); }; destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" sync(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_mlal { usertty("*"); }; filter f_filter1 { facility(kern); }; filter f_filter2 { level(info..emerg) and not facility(mail,authpriv,cron); }; filter f_filter3 { facility(authpriv); }; filter f_filter4 { facility(mail); }; filter f_filter5 { level(emerg); }; filter f_filter6 { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_filter7 { facility(local7); }; filter f_filter8 { facility(cron); }; #log { source(src); filter(f_filter1); destination(d_cons); }; log { source(src); filter(f_filter2); destination(d_mysql); }; log { source(src); filter(f_filter2); destination(d_mesg); }; log { source(src); filter(f_filter3); destination(d_auth); }; log { source(src); filter(f_filter4); destination(d_mail); }; log { source(src); filter(f_filter5); destination(d_mlal); }; log { source(src); filter(f_filter6); destination(d_spol); }; log { source(src); filter(f_filter7); destination(d_boot); }; log { source(src); filter(f_filter8); destination(d_cron); }; ... Then I started syslog-ng and received the following errors in /var/log/messages: ... Feb 14 16:21:59 furw syslog-ng[21895]: Cannot open file /var/run/mysql.pipe for writing (Permission denied) ... When I run an strace on the file, I get the following: ... uname({sys="Linux", node="furw", ...}) = 0 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=130, ...}) = 0 open("/tmp/mysql.pipe", O_RDWR|O_NONBLOCK|O_NOCTTY|O_LARGEFILE) = -1 EACCES (Permission denied) chown32("/tmp/mysql.pipe", 0, 0) = -1 EACCES (Permission denied) chmod("/tmp/mysql.pipe", 0600) = -1 EACCES (Permission denied) time(NULL) = 1108422581 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=130, ...}) = 0 ... Does anyone have an idea of why syslog-ng cannot read my pipe? I am open to ideas... -Matt

2 1

Update for syslog-ng FAQ
by Kevin 14 Feb '05

14 Feb '05

The FAQ (http://www.campin.net/syslog-ng/faq.html) talks about syslog volume and mentions "sites that exceed five gigabytes per day of log data". Our log volume has been growing slowly over time, I recently checked my primary logger and noticed that the raw log volume for this past Monday (the 24-hour period from midnight to midnight) was 10,982,118,488 bytes, or 10.22GiB. Peak hourly volume was 913MiB. Peak one-second volume in that hour was 2,626 messages totaling 440KiB, no duplicates. System specs: OpenBSD on a Dell 2650 (single 2.8Ghz P4) running syslog-ng 1.6.X. Logs are written to a Dell PERC 3/Di SCSI RAID-0, as a 2-drive stripe. Kevin Kadow

2 4

syslog-ng.conf.5 patch
by Jose Pedro Oliveira 14 Feb '05

14 Feb '05

Balasz, Two more patches: 1) doc/syslog-ng.conf.5 The syslog-ng.conf.5 manpage patch *only* updates the information regarding the Global Options. Problems: The global option "long_hostnames" is only documented in the contrib/syslog-ng.conf.doc file (not in the SGML file). The global option "check_hostname" is only documented in the SGML file (not in the syslog-ng.conf.doc). 2) contrib/syslog-ng.conf.doc The syslog-ng.conf.doc patch corrects the source type of /proc/kmsg (pipe -> file). Shouldn't this file be better located in the doc directory? (if this file is relocated, the contrib/README file will also need to be updated). Regards, jpo -- José Pedro Oliveira mailto: jpo(a)di.uminho.pt * http://gsd.di.uminho.pt/~jpo diff -ruN syslog-ng-1.6.6+20050208-orig/contrib/syslog-ng.conf.doc syslog-ng-1.6.6+20050208/contrib/syslog-ng.conf.doc --- syslog-ng-1.6.6+20050208-orig/contrib/syslog-ng.conf.doc 2003-04-10 11:51:00.000000000 +0100 +++ syslog-ng-1.6.6+20050208/contrib/syslog-ng.conf.doc 2005-02-09 03:33:53.370583904 +0000 @@ -283,7 +283,7 @@ # source src # { # internal(); -# pipe("/proc/kmsg" log_prefix("kernel: ")); +# file("/proc/kmsg" log_prefix("kernel: ")); # tcp(ip(127.0.0.1) port(4800) keep-alive(yes)); # udp(); # unix-stream("/dev/log"); @@ -304,7 +304,7 @@ { internal(); }; source s_kernel - { pipe("/proc/kmsg" log_prefix("kernel: ")); }; + { file("/proc/kmsg" log_prefix("kernel: ")); }; source s_tcp { tcp(port(4800) keep-alive(yes) max_connections(100)); }; diff -ruN syslog-ng-1.6.6+20050208-orig/doc/syslog-ng.conf.5 syslog-ng-1.6.6+20050208/doc/syslog-ng.conf.5 --- syslog-ng-1.6.6+20050208-orig/doc/syslog-ng.conf.5 1999-07-10 16:58:32.000000000 +0100 +++ syslog-ng-1.6.6+20050208/doc/syslog-ng.conf.5 2005-02-09 03:23:45.813946000 +0000 @@ -89,11 +89,130 @@ Where optn can be any of the following: -* mark(n) set mark frequency to n seconds - -* sync(n) set sync frequence to n lines (can be overridden locally) - -* long_hostnames(yes|no) set long hostname generation on or off +.TP +.B bad_hostname(regex) +A regexp which matches hostnames which should not be taken as such. + +.TP +.B chain_hostnames(yes|no) +Enable or disable the chained hostname format. + +.TP +.B check_hostname(yes|no) +Enable or disable whether the hostname contains valid characters. + +.TP +.B create_dirs(yes|no) +Enable or disable directory creation for destination files. + +.TP +.B dir_owner(uid) +User id. + +.TP +.B dir_group(gid) +Group id. + +.TP +.B dir_perm(perm) +Permission value (octal mask). + +.TP +.B dns_cache(yes|no) +Enable or disable DNS cache usage. + +.TP +.B dns_cache_expire(n) +Number of seconds while a successful lookup is cached. + +.TP +.B dns_cache_expire_failed(n) +Number of seconds while a failed lookup is cached. + +.TP +.B dns_cache_size(n) +Number of hostnames in the DNS cache. + +.TP +.B gc_busy_threshold(n) +Sets the threshold value for the garbage collector, when syslog-ng is +busy. GC phase starts when the number of allocated objects reach this +number. Default: 3000. + +.TP +.B gc_idle_threshold(n) +Sets the threshold value for the garbage collector, when syslog-ng is +idle. GC phase starts when the number of allocated objects reach this +number. Default: 100. + +.TP +.B group(gid) +Group id. + +.TP +.B keep_hostname(yes|no) +Enable or disable hostname rewriting. + +.TP +.B log_fifo_size(n) +The number of lines fitting to the output queue. + +.TP +.B log_msg_size(n) +Maximum length of message in bytes. + +.TP +.B long_hostnames(yes|no) +Set long hostname generation on or off. This options appears to only +really have an affect on the local system, which removes the source of +the log. As an example, normally the local logs will state +src@hostname, but with this feature off, the source is not reported. + +.TP +.B mark(n) +The number of seconds between two MARK lines. +NOTE: not implemented yet. + +.TP +.B owner(uid) +User id. + +.TP +.B perm(perm) +Permission value (octal mask). + +.TP +.B stats(n) +The number of seconds between two STATS. + +.TP +.B sync(n) +The number of lines buffered before written to file +(can be overridden locally). + +.TP +.B time_reap(n) +The time to wait before an idle destination file is closed. + +.TP +.B time_reopen(n) +The time to wait before a died connection is reestablished. + +.TP +.B use_dns(yes|no) +Enable or disable DNS usage. syslog-ng blocks on DNS queries, so +enabling DNS may lead to a Denial of Service attack. To prevent DoS, +protect your syslog-ng network endpoint with firewall rules, and make +sure that all hosts, which may get to syslog-ng is resolvable. + +.TP +.B use_fqdn(yes|no) +Add Fully Qualified Domain Name instead of short hostname. + +.TP +.B use_time_recvd(yes|no) +Use the time a message is received instead of the one specified in the +message. .SH FILES /etc/syslog-ng/syslog-ng.conf

2 4

Compiling syslog-ng-1.9.3 on OpenBSD 3.6?
by Kevin 12 Feb '05

12 Feb '05

I encounter a strange error related to the variable 'timezone' when attempting to compile syslog-ng-1.9.3 on OpenBSD 3.6. Suggestions appreciated. First problem I found was easily addressed: main.c main.c:39: wait.h: No such file or directory Replacing line 39 (#include <wait.h>) with these three lines solved that issue: #include <sys/time.h> #include <sys/resource.h> #include <sys/wait.h> After resolving that issue, I encounter the following: echo './'`logmsg.c logmsg.c: In function `log_stamp_format': logmsg.c:68: warning: assignment makes integer from pointer without a cast logmsg.c: In function `log_msg_parse': logmsg.c:234: warning: assignment makes integer from pointer without a cast logmsg.c:237: invalid operands to binary - logmsg.c:265: warning: assignment makes integer from pointer without a cast logmsg.c: In function `log_msg_init': logmsg.c:470: warning: assignment makes integer from pointer without a cast *** Error code 1 These lines are as follows: 68: target_zone_offset = timezone; 234: self->stamp.zone_offset = timezone; /* assume local timezone */ 237: self->stamp.time.tv_sec = mktime(&tm) - timezone + self->stamp.zone_offset; 265: self->stamp.zone_offset = timezone; /* assume local timezone */ 470: self->recvd.zone_offset = timezone; In OpenBSD, timezone is a function, char *timezone(int, int), not a variable. it appears that older versions of syslog-ng included options in configure for "checking for global timezone variable", but not in 1.9.3? Thanks, Kevin Kadow

3 2

Error compiling syslog-ng1.6.6 on HP-UX 11.23 IPF
by Jenny-Lou Sequeira 11 Feb '05

11 Feb '05

Hi, I get the following error when running "make" for syslog-ng1.6.6 on HP-UX 11.23 IPF. I do not get the error for 1.6.5. gcc -g -O2 -Wall -I/usr/local/include/libol -D_GNU_SOURCE -o syslog-ng main.o sources.o center.o filters.o destinations.o log.o cfgfile.o cfg-grammar.o cfg-lex.o affile.o afsocket.o afunix.o afinet.o afinter.o afuser.o afstreams.o afprogram.o afremctrl.o nscache.o utils.o syslog-names.o macros.o getopt.o getopt1.o -lnsl /usr/local/lib/libol.a -lnsl -lxnet -Wl,-Bstatic -lfl -lwrap -Wl,-Bdynamic ld: Bad keyword argument: static Fatal error. collect2: ld returned 1 exit status *** Error exit code 1 Stop. Thanks, Jenny-Lou

2 1

Am I an idiot? (tcp destinations)
by Jay 'Whip' Grizzard 10 Feb '05

10 Feb '05

Okay, perhaps I'm an idiot, but with a freshly downloaded and compiled syslog-ng 1.6.6, I get the following: syslogng.conf (one line long): destination d_tcp { tcp("10.1.2.3") port(1999); localport(999)); }; (which is, you'll notice, copied directly out of the docs) But I get: : [elfchief@bluetail /tmp]% /sbin/syslog-ng --cfgfile=/tmp/syslogng.conf --syntax-only : syntax error at 1 : Parse error reading configuration file, exiting. (line 1) ... and no matter what I put in there (I tried a lot of permutations, and even cut&pasted bits of other people's configs I saw in the mailing list archives), I can't seem to get the option to parse. Am I missing something fundamental here? Given that this is the one feature I'd really like to use syslog-ng for, it'd be nice if I could get it to work! I appreciate any thoughts anyone cares to share... -jay

4 6