- syslog-ng - lists.balabit.hu

using the program() destination to send mail
by John_Delisle＠ceridian.ca 22 Feb '01

22 Feb '01

Has anyone got some code handy that can send email when a specific error is logged? Ideally I'd like to mail an admin with the hostname, error, etc. John Delisle Corporate Technology Ceridian Canada Ltd 204-975-5909

1 0

RE: [syslog-ng]Trying to send log over network
by Hamilton, Andrew Mr RAYTHEON 5 SIG CMD 21 Feb '01

21 Feb '01

1. No syslog-ng runs as its own daemon. Works much better this way. 2. If the answer to 1 was yes it would make a difference. But since it isn't then no. 3. Could you post a sample of your config file? That might be helpful. Also could you post part of your strace results? That would at least give us info on where to look for your problem. Random thoughts...do you have some sort of port monitor that could be blocking your default ports? I have seen that before. Regards. Drew > -----Original Message----- > From: Simeon Johnston [SMTP:simeonuj@eetc.com] > Sent: Tuesday, February 20, 2001 11:26 PM > To: syslog-ng(a)lists.balabit.hu > Subject: Re: [syslog-ng]Trying to send log over network > > I used strace although I didn't get much out of it. I have never used it > before. > I think that it is not being allowed to initiate a connection or it is > trying but > is not allowed to connect to the logserver so it stops. I have a few more > questions. > 1. Does syslog-ng need an entry in the inetd/xinetd configuration? I > didn't see > anything about it in the manual. > 2. Would this disallow any information from getting through if there > wasn't an > entry? > 3. Any other thoughts? > > Any help would be appreciated. > > sim > > > Balazs Scheidler wrote: > > > On Wed, Feb 14, 2001 at 10:07:51AM -0600, Simeon Johnston wrote: > > > I put the new rule in and it didn't give any errors when I started it. > But > > > there seems to be another problem. I don't know why but it looks like > the > > > daemon doesn't even start. I restart it with the bootup script and > everything > > > appears to run OK but then I do a "ps ax" and the process isn't there. > I then > > > start it manually and it gives me this error. > > > > > > Error creating AF_INET socket (Success) > > > Error initializing configuration, exiting. > > > > > > What is this? > > > > a truss or strace output should help you out here. > > > > -- > > Bazsi > > PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C > 8EB1 > > > > _______________________________________________ > > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > > https://lists.balabit.hu/mailman/listinfo/syslog-ng > > > _______________________________________________ > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > https://lists.balabit.hu/mailman/listinfo/syslog-ng

6 6

DNS Caching
by Gaël Roualland 21 Feb '01

21 Feb '01

Hello, We've been using syslog-ng here for a while as a remote loghost for several high-volume servers and the loghost would stay at a high load and often lose some log lines. We have name resolution enabled and it turned out to be one of the big performance hit. Using a local DNS cache together with nscd and name service files tuning made things a lot better but syslog-ng was still spending lots of time doing reverse name resolution. So I tried to add a simple DNS caching code to syslog-ng itself, and it really improved the overall performance : the syslog-ng process now uses between 10 to 20% of cpu time, whereas it used to be around 100%. As this is an issue that could affect many users, it might be a good idea to have this feature in syslog-ng (maybe as an option ?). The attached patch is my implementation of it. The algorithm used is quite simple (flat fixed-size hash, with refresh and aging) and should be sufficient regarding the fact that a loghost will probably always lookup the same set of hosts. I'd be glad to know what you think of it and if it can be intregated in the main release (it might need some work to fit nicer than now). Gaël. -- Gaël Roualland -+- gael.roualland(a)oleane.net diff -ruN syslog-ng-1.4.10/src/Makefile.am syslog-ng-1.4.10.nscache/src/Makefile.am --- syslog-ng-1.4.10/src/Makefile.am Wed Nov 29 12:04:29 2000 +++ syslog-ng-1.4.10.nscache/src/Makefile.am Mon Feb 19 19:11:59 2001 @@ -65,6 +65,7 @@ afuser.c \ afstreams.c \ afprogram.c \ + nscache.c \ utils.c \ syslog-names.c diff -ruN syslog-ng-1.4.10/src/main.c syslog-ng-1.4.10.nscache/src/main.c --- syslog-ng-1.4.10/src/main.c Wed Nov 29 12:04:29 2000 +++ syslog-ng-1.4.10.nscache/src/main.c Mon Feb 19 19:17:56 2001 @@ -29,6 +29,7 @@ #include "format.h" #include "werror.h" #include "queue.h" +#include "nscache.h" #include <sys/types.h> #include <sys/wait.h> @@ -144,7 +145,8 @@ signal(SIGHUP, sig_hup); signal(SIGTERM, sig_term); signal(SIGCHLD, sig_child); - + nscache_init(); + while (io_iter(&backend->super)) { if (exit_main_loop) break; @@ -156,6 +158,10 @@ } notice("SIGHUP received, restarting syslog-ng\n"); + + nscache_flush(); + nscache_init(); + restarting = 1; backend->newconfig = make_syslog_config(cfgfilename, &backend->super); diff -ruN syslog-ng-1.4.10/src/nscache.c syslog-ng-1.4.10.nscache/src/nscache.c --- syslog-ng-1.4.10/src/nscache.c Thu Jan 1 01:00:00 1970 +++ syslog-ng-1.4.10.nscache/src/nscache.c Mon Feb 19 19:12:09 2001 @@ -0,0 +1,134 @@ +#include <netdb.h> +#include <netinet/in.h> +#include <arpa/inet.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <time.h> +#include "nscache.h" + +/* Simple NS cache, using a flat hash table (NOT thread safe). + * This is designed to be used with a fixed set of ip adresses. + * Inspired from similar caches in ippl, iplog... + * Gaël Roualland <gael.roualland(a)iname.com>, 2001. */ + +#define NSCACHE_SIZE 1000 +#define NSCACHE_EXPIRE 3600 +#undef NSCACHE_DEBUG + +struct nsentry { + struct in_addr addr; + char * name; + time_t used; + time_t created; +}; + +static struct nsentry * nscache = NULL; + +void nscache_init() { + int i; + + if (nscache != NULL) + nscache_flush(); + + if ((nscache = malloc(NSCACHE_SIZE * sizeof(struct nsentry))) == NULL) + return; + + for (i = 0; i < NSCACHE_SIZE; i++) { + nscache[i].addr.s_addr = INADDR_NONE; + nscache[i].name = NULL; + nscache[i].created = -1; + nscache[i].used = -1; + } +} + +void nscache_flush() { + int i; + + if (nscache == NULL) + return; + + for (i = 0; i < NSCACHE_SIZE; i++) { + if (nscache[i].name != NULL) + free(nscache[i].name); + } + + free(nscache); + nscache = NULL; +} + +static char * nsresolve(struct in_addr addr) { + struct hostent *host; + + if ((host = gethostbyaddr((char *) &addr, + sizeof(struct in_addr), + AF_INET)) != NULL && + host->h_name != NULL) + return strdup(host->h_name); + return NULL; +} + +char *nscache_lookup(struct in_addr in) { + int h, k, old, new; +#ifdef NSCACHE_DEBUG + int iter = 0; +#endif + time_t now = time(NULL); + time_t oldest = now; + + k = h = ntohl(in.s_addr) % (NSCACHE_SIZE - 1) + 1; + old = new = -1; + + do { + if (nscache[k].created > 0 && nscache[k].addr.s_addr == in.s_addr) { /* found it! */ + if (now - nscache[k].created >= NSCACHE_EXPIRE) { + /* entry has expired, update it */ +#ifdef NSCACHE_DEBUG + fprintf(stderr, "updating entry for %s...\n", inet_ntoa(in)); +#endif + if (nscache[k].name != NULL) + free (nscache[k].name); + nscache[k].name = nsresolve(in); + nscache[k].created = now; + } + nscache[k].used = now; +#ifdef NSCACHE_DEBUG + fprintf(stderr, "FOUND! %s <=> %s (%d iters)...\n", inet_ntoa(in), + nscache[k].name, iter); +#endif + return nscache[k].name; + } + if (nscache[k].created < 0 && new < 0) + new = k; /* use this to save new entry if not found */ + else { + if (nscache[k].used <= oldest) { + old = k; + oldest = nscache[k].used; + } + } + k = (h + k) % NSCACHE_SIZE; +#ifdef NSCACHE_DEBUG + iter++; +#endif + } while (k > 0); + + /* not found, add to cache */ + if (new < 0) { + /* no new entries ? use oldest. */ + if (old < 0) + old = 0; /* shouldn't happen.. */ +#ifdef NSCACHE_DEBUG + fprintf(stderr, "NOT FOUND! Replacing %s..\n", inet_ntoa(nscache[old].addr)); +#endif + if (nscache[old].name != NULL) + free(nscache[old].name); + new = old; + } +#ifdef NSCACHE_DEBUG + fprintf(stderr, "NOT FOUND! %s... (adding as %d)\n", inet_ntoa(in), new); +#endif + nscache[new].name = nsresolve(in); + nscache[new].created = nscache[new].used = now; + nscache[new].addr.s_addr = in.s_addr; + return nscache[new].name; +} diff -ruN syslog-ng-1.4.10/src/nscache.h syslog-ng-1.4.10.nscache/src/nscache.h --- syslog-ng-1.4.10/src/nscache.h Thu Jan 1 01:00:00 1970 +++ syslog-ng-1.4.10.nscache/src/nscache.h Mon Feb 19 19:12:11 2001 @@ -0,0 +1,8 @@ +#ifndef NSCACHE_H +#define NSCACHE_H + +void nscache_init(); +void nscache_flush(); +char * nscache_lookup(struct in_addr in); + +#endif diff -ruN syslog-ng-1.4.10/src/sources.c syslog-ng-1.4.10.nscache/src/sources.c --- syslog-ng-1.4.10/src/sources.c Wed Nov 29 12:04:29 2000 +++ syslog-ng-1.4.10.nscache/src/sources.c Mon Feb 19 19:14:20 2001 @@ -40,7 +40,7 @@ #undef CLASS_DEFINE #include "sources.c.x" - +#include "nscache.h" /* CLASS: (class (name log_reader) @@ -155,15 +155,13 @@ if (a && a->super.isa == &inet_address_info_class) { CAST(inet_address_info, inet_addr, a); - struct hostent *hp; char *hname, *p; - hp = usedns ? gethostbyaddr((char *) &(inet_addr->sa.sin_addr), sizeof(struct in_addr), AF_INET) : NULL; - if (!hp) { + hname = usedns ? nscache_lookup(inet_addr->sa.sin_addr) : NULL; + if (!hname) { return ol_string_use(inet_addr->ip); } else { - hname = hp->h_name; if (!usefqdn) { p = strchr(hname, '.'); if (p) *p = 0;

4 3

Trying to send log over network
by Simeon Johnston 20 Feb '01

20 Feb '01

I am trying to send logs over our network but am having trouble with the rules. I have set up a host to accept the logs and I am trying to set up the sender. This is the destination rule so far. destination d_bg { tcp(ip(192.168.1.19) port(514)); }; When I try to start syslog-ng it gives me this error. parse error at 31 Parse error reading configuration file, exiting. Line 31 is the above rule. This is the complete config of syslog-ng so far. options { sync (0); time_reopen (10); log_fifo_size (1000); long_hostnames (on); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (yes); }; source s_sys { unix-stream ("/dev/log"); internal(); }; destination d_cons { file("/var/log/ng/kern.log"); }; destination d_mesg { file("/var/log/ng/messages"); }; destination d_auth { file("/var/log/ng/secure"); }; destination d_mail { file("/var/log/ng/maillog"); }; destination d_spol { file("/var/log/ng/spooler"); }; destination d_boot { file("/var/log/ng/boot.log"); }; destination d_mlal { file("/var/log/ng/emerg.log"); }; destination d_dhcp { file("/var/log/ng/dhcpd.log"); }; destination d_ipchains { file("/var/log/ng/ipchains.log"); }; destination d_bg { tcp(ip(192.168.1.19) port(514)); }; filter f_filter1 { facility(kern); }; filter f_filter2 { level(info); }; filter f_filter3 { facility(authpriv); }; filter f_filter4 { facility(mail); }; filter f_filter5 { level(emerg); }; filter f_filter6 { facility(uucp) or (facility(news) and level(crit)); }; filter f_dhcp { match("dhcpd"); }; filter f_ipchains { match("ipchains"); }; filter f_tcplog { level(debug); }; log { source(s_sys); filter(f_filter1); destination(d_cons); }; log { source(s_sys); filter(f_filter2); destination(d_mesg); }; log { source(s_sys); filter(f_filter3); destination(d_auth); }; log { source(s_sys); filter(f_filter4); destination(d_mail); }; log { source(s_sys); filter(f_filter5); destination(d_mlal); }; log { source(s_sys); filter(f_filter6); destination(d_spol); }; log { source(s_sys); filter(f_dhcp); destination(d_dhcp); }; log { source(s_sys); filter(f_ipchains); destination(d_ipchains); }; log { source(s_sys); filter(f_tcplog); destination(d_bg); }; Any help would be appreciated. sim

3 5

syslog-ng duplicating messages
by Matt Studley 20 Feb '01

20 Feb '01

hello. I am having a slight problem with syslog-ng. It is running fine on all of our servers, however we did set up a logging host to keep an archive of other servers logs. It is using udp as the protocol to send the logger messages to the loghost. When the messages comes through to the loghost and is filtered to the appropriate destination, it always appears twice. For example the following is common to 99% of all the logs on the loghost: Feb 16 13:13:47 sun08 su: 'su root' succeeded for mjs on /dev/pts/24 Feb 16 13:13:47 sun08 su: 'su root' succeeded for mjs on /dev/pts/24 Feb 16 14:27:19 sun08 login: pam_authenticate: error Authentication failed Feb 16 14:27:19 sun08 login: pam_authenticate: error Authentication failed Feb 19 08:06:33 sun08 su: pam_authenticate: error Authentication failed Feb 19 08:06:33 sun08 su: pam_authenticate: error Authentication failed Feb 19 08:06:33 sun08 su: 'su root' failed for tvd on /dev/pts/12 Feb 19 08:06:33 sun08 su: 'su root' failed for tvd on /dev/pts/12 This is not happening on the local servers. Anyone have any suggestions? Thanks in advance for your help. Matt Studley American Mathematical Society Systems Administrator

1 0

sync within tcp
by Matthew Copeland 19 Feb '01

19 Feb '01

I am using the following for my config file. I have been told that you can sync a tcp connection to a certain number of lines, and I thought this was how you do it, but it dies on the first line that has the sync with a parse error. If I move it to its own line, outside of the tcp, it dies parsing there. Can someone please tell me where I am messing up here? Thanks, Matthew M. Copeland options { long_hostnames(off); log_fifo_size(1440000); }; source src { unix-stream("/dev/log"); internal(); }; destination normal_messages { tcp("129.235.66.4" port(999) sync(10)); }; destination immediate_messages { tcp("129.235.66.4" port(999) sync(0)); }; filter f_gs_alert { match("GS_ALERT"); }; log { source (src); filter(f_gs_alert); destination(immediate_messages); }; log { source(src); filter(DEFAULT); destination(normal_messages); };

1 0

syslogging with perl
by Bernd Jucknischke 14 Feb '01

14 Feb '01

Hi all! I've just been facing a strange problem. Attached you'll find a perl script. I've been running this script on different machines with different results (the path to perl has been adapted, of course): machinetype logging daemon OS Perl result pc syslogd Debian unstable 5.6 doesn't work Sun syslogd Solaris 2.6 5.004_4 works Sun syslog-ng Solaris 2.6 5.004_4 doesn't work Did anybody ever experience problems like that? I'd say it's perl related, but I'm not sure; any hint or advice is appreciated. TIA, Bernd

2 3

Re: [syslog-ng]Trying to send log over network
by Simeon Johnston 14 Feb '01

14 Feb '01

Same problem sim Don Armstrong wrote: > On Tue, 13 Feb 2001, Simeon Johnston wrote: > > I am trying to send logs over our network but am having trouble with the > > rules. I have set up a host to accept the logs and I am trying to set > > up the sender. This is the destination rule so far. > > > > destination d_bg { tcp(ip(192.168.1.19) port(514)); }; > > This should be > > destination d_bg {tcp(ip(192.168.1.19); port 514););}; > > Don Armstrong > > --------------------------------------------------- > UC Riverside | .naidnE elttil atah I > Residential Computing | > http://resnet.ucr.edu | http://www.donarmstrong.com > ---------------------------------------------------

2 1

Re: [syslog-ng]syslog-ng performance problem
by Balazs Scheidler 13 Feb '01

13 Feb '01

> I think the server has enough CPU and memory resources (Pentium III 650 > with 1Gb RAM) to handle the logs incoming from the web servers as it > works fine at the beginning without loosing messages and the performace > decreases progressively instead of from the beginning. I think I could > solve the problem with the appropiate gc_idle_threshold() and > gc_busy_threshold() settings (now I using the default values), but I'm > not sure if it would be the solution and in this case I don't know how > to calculate the right values for these para meters. > > Any advice? What about first upgrading to 1.4.10 ? -- Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1

1 0

RE: [syslog-ng]syslog-ng performance problem
by Hamilton, Andrew Mr RAYTHEON 5 SIG CMD 13 Feb '01

13 Feb '01

You need to upgrade to a new version. This was a well known problem on some of the earlier 1.4 versions of syslog-ng for some OS's. Subsequent releases have cleared this up. Look at http://www.balabit.hu/en/products/syslog-ng/ That is the syslog-ng home page. Regards Drew > -----Original Message----- > From: Santiago Gomez Cano [SMTP:santiago.gomez@tiscalinet.es] > Sent: Tuesday, February 13, 2001 12:16 PM > To: syslog-ng(a)lists.balabit.hu > Subject: [syslog-ng]syslog-ng performance problem > > Hi > > I'm using the syslog-ng 1.40rc3-3 deb package running on a Debian > GNU/Linux box collectin logs from 3 different web serves. > > When starting the syslgog-ng daemon it begins using a reasonable CPU > amount (about 2 or 4%), sleeping most of the time and running when > someone access our web pages and the logging server receives logs from > the web servers. The CPU usage increases to 5 or 6% after one hour > running and keeps on growing up to saturate the server with a 99.99% CPU > usage about two or three ours after the start. > > I think the server has enough CPU and memory resources (Pentium III 650 > with 1Gb RAM) to handle the logs incoming from the web servers as it > works fine at the beginning without loosing messages and the performace > decreases progressively instead of from the beginning. I think I could > solve the problem with the appropiate gc_idle_threshold() and > gc_busy_threshold() settings (now I using the default values), but I'm > not sure if it would be the solution and in this case I don't know how > to calculate the right values for these para meters. > > Any advice? > > -- > Santiago Gomez Cano > Systems Administrator / Administrador de Sistemas > santiago.gomez(a)tiscali.es > Tiscali Telecomunicaciones, S.A. > www.tiscali.es > T. +34 918370546 > T. +34 699459573 > > > _______________________________________________ > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > https://lists.balabit.hu/mailman/listinfo/syslog-ng

1 0