- syslog-ng - lists.balabit.hu

[patch] only stat dns_cache_hosts file once in a second
by ILLES, Marton 28 Feb '09

28 Feb '09

Hi, I was doing today some performance tests where use_dns(yes) and persistent dns cache (dns_cache_hosts) were configured and I got awful results. When I disabled dns_cache_hosts the message rate doubled. After a quick check I realized that for every message a dns cache lookup was performed which stat()-ed my dns_cache_hosts file. However it does not make much sense to check the file for modification so often, I guess once a second is enough most of the cases. In my test configuration I received msgs over syslog() source and sent to an empty destination. The server was a 2way quad-core 1.9GHz Opteron box running 32bit syslog-ng and messsages were generated with loggen. >From an average 80,000-90,000 msg/s using the fix I could do 220,000-230,000 msg/s. Interesting that on my laptop (dual-core 2.4GHz Intel 64bit) the performance gain was smaller from 90,000 to 100,000 msg/s. I am still wondering about this big difference, specially because my laptop has higher frequency CPU... Anyhow the patch could be found in my git repo, Bazsi please apply. git://git.balabit.hu/marci/syslog-ng-3.0.git M -- Key fingerprint = F78C 25CA 5F88 6FAF EA21 779D 3279 9F9E 1155 670D

2 1

Re: [syslog-ng] syslog-ng Digest, Vol 46, Issue 31
by 牛田　善信 28 Feb '09

28 Feb '09

Dear Bazsi-san, Thank you for the reply. >what kind of binaries are you using? PE/OSE binaries supplied by >BalaBit, or you are using binaries from another source? I am using what downloaded from rpmsearch. It is following URL. http://rpm.pbone.net/index.php3/stat/4/idpl/10055325/com/syslog-ng-2.1.1-5.… And, I used syslog-ng-2.1.3-1 supplied by balabit. I have done the question with indefinite purpose of me. My final purpose is to get rid of CPU use rate 100%. Could you direct the best possible method that I should do? Best Regards, Yoshinobu

2 1

Re: [syslog-ng] File descriptor leaks with 3.0.1
by chris packham 28 Feb '09

28 Feb '09

On Mon, 2009-02-16 at 15:52 +1300, chris packham wrote: > Also seems to be present in the latest version checked out from the git repo but I'll do some more digging. > I've confirmed that this does indeed happen on the latest version from the git repo. I've attached a patch that addresses this. Looks like the 'include' implementation assumes that yy_switch_to_buffer closes the old file (which flex 2.5.33 doesn't, I haven't tried other versions).

2 1

high memory usage (swapping)
by Otto Berger 27 Feb '09

27 Feb '09

Hi, today i got a weird cenario: i rebooted our syslog-ng server and forgot to start the syslog-ng daemon. So far so (not) good. One of our webservers which sends his logs via UDP to the syslog-server became suddenly very slow. It is reproducable: whenever i stop the central syslog-server the sender-servers syslog-ng-process gets a very high memory usage (4GB!) within seconds... The log-traffic on this server is about 30MB/day... UDP is a connection-less protocol isnt? ;) The configuration of the sender-server: ----- options { sync (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source s_sys { file ("/proc/kmsg" log_prefix("kernel: ")); unix-stream ("/dev/log"); internal(); # udp(ip(0.0.0.0) port(514)); }; destination a_udp { udp( "example.com" port(514) ); }; #log { source(s_sys); destination (a_udp); }; ----- sender: syslog-ng 2.1.3 reciever: syslog-ng 2.1.2 someone got an idea on this issue? Otto

2 2

Load balancing using $SEC
by Vincent Panel 27 Feb '09

27 Feb '09

Hi, I'm trying to do some kind of load-balancing using syslog-ng. I'm willing to write to 2 different files named "syslog1" and "syslog2". The first one would get all messages received by syslog whose timestamp have a second ending with 0, 1, 2, 3 or 4. Syslog 2 would get all messages whose second ends with 5, 6, 7, 8 or 9. For instance, if I get those messages : Feb 27 00:00:03 blabla1 Feb 27 00:00:04 blabla2 Feb 27 00:00:05 blabla3 Feb 27 00:00:06 blabla4 The first two messages would end up in syslog1 and the last 2 messages would end up in syslog2 However, I was not able to do this because : * filtering is not possible on any date-related macro (I don't know why, but only MSG and MSGHDR macros seem populated at that point in time) * I can't apply a rewrite rule on $SEC * I can't apply any function to the $SEC macro in the destination statement, or I don't know how to do it Do you think it's still possible to achieve what I want with syslog-ng ? Vincent

1 0

creating new custumised facility
by gatfi sami 26 Feb '09

26 Feb '09

Hi i am getting started with syslog can i add a new facility different from ( mail ... local0 to 7) what i did til now is sending every thing (src) to a syslog server via udp #########" client syslog > filter myfil1 { facility(local1); };# let only local1 = error log apache > > destination loghost { > udp("192.168.1.66" port(514)); > }; > log { > source(src); > > filter(myfil1); > destination(loghost); > }; > ###Server syslog source src { > # > # include internal syslog-ng messages > # note: the internal() soure is required! > # > # internal(); > > # > # the default log socket for local logging: > # > # unix-dgram("/dev/log"); > > # > # uncomment to process log messages from network: > # > #udp(ip("0.0.0.0") port(514)); > udp(port(514)); > > > }; > source src { > # > # include internal syslog-ng messages > # note: the internal() soure is required! > # > # internal(); > > # > # the default log socket for local logging: > # > # unix-dgram("/dev/log"); > > # > # uncomment to process log messages from network: > # > #udp(ip("0.0.0.0") port(514)); > udp(port(514)); > > > }; > and sent Apache erreor to syslog :local1 /mod_log_config.conf Errorlog syslog:local1 ************** what i tried and didnot work is adding /var/log/apache2/errorlog as sorce because of permession denial and using unix-stream unix-dgram pipe ( i created file with) mkififo between apache and syslog nothing so sorry if i am bothering but my questions are 1 can i add a new facility like local1 and collect on it apapche error log if no is Errorlog syslog: local1 the *best* way with a filter on local1 2 does local1in my case contain only appache error log 3 if there is another way be my gest i am hering Sorry am not good in eglish

1 0

Syslog-ng refuses to create directories on Linux
by John Chajecki 24 Feb '09

24 Feb '09

We have been running a syslog server on a Solaris server for some time. The server collects syslogs from a number of sources and places the log data into separate files for each system according to the host IP address and date. Recently, we have begun migrating the facility to a Linux server running Novell SuSE Linux 10. (SLES10) 64bit. We are running syslog-ng version 2.0.9-6.1 for the 64bit platform which I obtained from OpenSuSE and that is supposed to be compatible with SLES10. The problem is that although I am using essentially the same config file I am getting my log filled with: Feb 24 14:55:04 itssyslog syslog-ng[10090]: Error opening file for writing; filename='/media/syslog/HOSTS/192.168.1.146', error='Permission denied (13)' The directories /media, /media/syslog and /media/syslog/HOSTS have permissions of 0640. This is what happens in strace: open("/var/run/syslog-ng.pid", O_WRONLY|O_CREAT|O_TRUNC|O_NOCTTY, 0600) = 7 write(7, "10090", 5) = 5 close(7) = 0 open("/dev/null", O_RDONLY) = 7 dup2(7, 0) = 0 close(7) = 0 open("/dev/null", O_WRONLY) = 7 dup2(7, 1) = 1 dup2(7, 2) = 2 close(7) = 0 setsid() = 10090 poll([{fd=3, events=POLLIN}, {fd=4, events=POLLIN}], 2, 0) = 0 uname({sys="Linux", node="itssyslog", ...}) = 0 open("/var/log/messages", O_WRONLY|O_NONBLOCK|O_APPEND|O_CREAT|O_NOCTTY, 0640) = 7 fcntl(7, F_GETFD) = 0 fcntl(7, F_SETFD, FD_CLOEXEC) = 0 fchown(7, 0, 4294967295) = 0 fchown(7, 4294967295, 0) = 0 fchmod(7, 0640) = 0 poll([{fd=7, events=POLLOUT, revents=POLLOUT}, {fd=3, events=POLLIN}, {fd=4, eve nts=POLLIN}], 3, 29983) = 1 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0 write(7, "Feb 24 14:16:50 itssyslog syslog"..., 83) = 83 poll([{fd=3, events=POLLIN}, {fd=4, events=POLLIN, revents=POLLIN}], 2, 29983) = 1 recvfrom(4, "<22>Feb 24 14:12:24 192.168.193."..., 8192, 0, {sa_family=AF_INET, sin_port=htons(61643), sin_addr=inet_addr("192.168.1.146")}, [16]) = 155 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0 brk(0x5be000) = 0x5be000 brk(0x5df000) = 0x5df000 open("/media/syslog/HOSTS/192.168.1.146/2009/02/24/192.168.1.146.log", O_WRONLY| O_NONBLOCK|O_APPEND|O_CREAT|O_NOCTTY, 0640) = -1 ENOENT (No such file or directory) stat("/media", {st_mode=S_IFDIR|0640, st_size=4096, ...}) = 0 stat("/media/syslog", {st_mode=S_IFDIR|0640, st_size=4096, ...}) = 0 stat("/media/syslog/HOSTS", {st_mode=S_IFDIR|0640, st_size=4096, ...}) = 0 stat("/media/syslog/HOSTS/192.168.1.146", 0x7fff7eb5c400) = -1 ENOENT (No such file or directory) mkdir("/media/syslog/HOSTS/192.168.1.146", 0640) = -1 EACCES (Permission denied) recvfrom(4, 0x583fe0, 8192, 0, 0x7fff7eb5c780, 0x7fff7eb5c80c) = -1 EAGAIN (Resource temporarily unavailable) poll([{fd=3, events=POLLIN}, {fd=4, events=POLLIN}], 2, 0) = 0 ..... For some reason, the syslog-ng daemon tries to create the directory structure but fails with the Permission denied error. itssyslog:/var/log # The daemon is running as root: ps -ef |grep syslog-ng root 10077 1 0 14:15 ? 00:00:00 /sbin/syslog-ng I have tried creating a directory structure manually and I have no problem creating, writing to and removing the directories. Just to test, I created another directory stucture /syslog/HOSTS with permissions 0777. I restarted the syslog-ng service but I got the same error. I also ran the syslog-ng executable manually as root, but still got the same problem. What's stopping syslog-ng from creating the directory structure? Does it perhaps run in a chroot jail by default? As I mentioned, this worked fine on the Solaris system. John.

2 3

syslog-ng takes 100% CPU(RHEL 5)
by Yoshinobu Ushida 24 Feb '09

24 Feb '09

Hello, We are running syslog-ng-2.1.1-5.el5 on a RHEL5. The problem in HP-UX was heard fixed and I want to do the upgrade. but,libdbi8.so.0() is necessary for 2.1.3. This is not found. Where is libdbi8.so.0()? Please tell me. I am sorry in clumsy English. I hope help.

2 1

Kevin Gorski/IS/NEU is out of the office.
by K.Gorski＠neu.edu 23 Feb '09

23 Feb '09

I will be out of the office starting Mon 02/23/2009 and will not return until Mon 03/02/2009. I will respond to your message when I return Thursday morning.

1 0

3.01 Starting system logger: Error resolving user; user='root'
by Tim Boyer 17 Feb '09

17 Feb '09

I've compiled and installed 3.0.1. I'm seeing something surface that was a reported bug in the 2.1 era. Perhaps it's the same patch? [root@princeton syslog-ng-3.0.1]# service syslog-ng start Starting system logger: Error resolving user; user='root' syntax error in /etc/syslog-ng.conf at line 61. [FAILED] root exists, of course, and it doesn't matter which username I put there - good or bad, they all fail. -- Tim Boyer Denman Tire Corporation

2 2