August 2001 - syslog-ng - lists.balabit.hu

RE: [syslog-ng]Re: syslog-ng's memory footprint growing constantl y
by Hamilton, Andrew Mr RAYTHEON 5 SIG CMD 03 Aug '01

03 Aug '01

Actually, I ran it with nisplus for a while but it has been some time ago. I didn't have any problems with syslog-ng in connection to nisplus but we had enough problems with nisplus to eventually wind up going back to nis. I have used nis for over a year and don't have any problems using it with syslog-ng. Problems with nisplus would not be surprising. Regards, Drew -----Original Message----- From: Balazs Scheidler [mailto:bazsi@balabit.hu] Sent: Thursday, August 02, 2001 7:48 PM To: Craig J Constantine Cc: syslog-ng(a)lists.balabit.hu Subject: [syslog-ng]Re: syslog-ng's memory footprint growing constantly On Thu, Aug 02, 2001 at 12:05:29PM -0400, Craig J Constantine wrote: > ...on 8/2/01 11:19 AM, the following was received from bazsi(a)balabit.hu > > >ok, we have some kind of string leak. what does "hosts.org_dir" in your > >environment mean? is it a directory which syslog-ng creates, or a filename? > >or some part of some message? > > I have no idea. :) That string isn't in my syslog-ng config, nor > *anywhere* in the syslogs recorded by syslog-ng today, nor even anywhere > in the syslog-ng distribution. Might be just a lot over strings overlayed > in memory... very weird. I found hosts.org_dir, it's in the file /lib/libnss_nisplus.so, so I assume you are using nisplus. syslog-ng resolves a lot of hostnames, isn't it possible that nisplus is leaking somewhere? I've never ran syslog-ng with a computer using nisplus, that might be an explanation why others didn't encounter this bug. -- Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1 _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng

1 0

Re: syslog-ng's memory footprint growing constantly
by Balazs Scheidler 02 Aug '01

02 Aug '01

On Thu, Aug 02, 2001 at 12:05:29PM -0400, Craig J Constantine wrote: > ...on 8/2/01 11:19 AM, the following was received from bazsi(a)balabit.hu > > >ok, we have some kind of string leak. what does "hosts.org_dir" in your > >environment mean? is it a directory which syslog-ng creates, or a filename? > >or some part of some message? > > I have no idea. :) That string isn't in my syslog-ng config, nor > *anywhere* in the syslogs recorded by syslog-ng today, nor even anywhere > in the syslog-ng distribution. Might be just a lot over strings overlayed > in memory... very weird. I found hosts.org_dir, it's in the file /lib/libnss_nisplus.so, so I assume you are using nisplus. syslog-ng resolves a lot of hostnames, isn't it possible that nisplus is leaking somewhere? I've never ran syslog-ng with a computer using nisplus, that might be an explanation why others didn't encounter this bug. -- Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1

1 0

Re: syslog-ng's memory footprint growing constantly
by Craig J Constantine 02 Aug '01

02 Aug '01

...on 7/27/01 1:02 AM, the following was received from syslog-ng-admin(a)lists.balabit.hu >Date: Thu, 26 Jul 2001 10:25:08 +0200 >From: Balazs Scheidler <bazsi(a)balabit.hu> >To: syslog-ng(a)lists.balabit.hu >Subject: Re: syslog-ng's memory footprint growing constantly > >On Wed, Jul 25, 2001 at 04:17:06PM -0400, Craig J Constantine wrote: >> Hi, >> Basically, the memory size of the process simply grows and grows: >> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND >> root 13899 0.1 19.3 25412 24764 ? S Jul24 2:05 syslog-ng >> That was snapped after about 4 hours of run time. If I let it run for >> days... well, it once got to 90000 VSZ before the box choked for lack of >> RAM. For gauging how heavily worked syslog-ng is, the line counts for all >> the files it writes (this is one day of logs) are: [rest of my original, snipped] > >There have been memory leaks, but they were fixed in 1.4.8. To help me to >debug your problem, please compile syslog-ng _AND_ libol with --with-debug >enabled (configure time option), and run syslog-ng with --debug --verbose >switches, and save the resulting output. I'm mainly interested in lines >starting with "Objects alive ... garbage collected...". > >If the number after "objects alive" increases then there's some spurious >link within the data structure (the gc frees only unlinked objects), if it >doesn't it is some kind of string data (for example messages). I recompiled libol/syslog-ng from fresh tarballs using the configure flag. Then I fired it up with runtime debug arguments and captured everything to a file. I started the run around 9am, and left it go for a good half an hour. The resulting file was, well, truly huge. 8^) Let me know if you'd like to see it, I can drop it on a server where you can grab it via anon FTP. Likewise if you want to see my config file again. ...around 9:07 syslog-ng was: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 30271 0.3 0.7 1652 1000 pts/1 S 08:54 0:02 ./syslog-ng.debug (ie, tiny :) The "Objects alive" was at 293. ...around 9:37 syslog-ng was: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 30271 0.3 1.1 2104 1452 pts/1 S 08:54 0:09 ./syslog-ng.debug The "Objects alive" was at 299. ...around 10:07: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 30271 0.3 1.4 2552 1900 pts/1 S 08:54 0:16 ./syslog-ng.debug The "Objects alive" was at 299. ...around 10:37: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 30271 0.4 1.9 3124 2472 pts/1 S 08:54 0:24 ./syslog-ng.debug The "Objects alive" was still at 299. ...around 11:07: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 30271 0.4 2.3 3720 3068 pts/1 S 08:54 0:35 ./syslog-ng.debug The "Objects alive" was still at 299. ...looks like the objects count is stable, but the memory size is still growing. I sent it a SIGQUIT and scooped up the core file. It looks like a large portion (more than half) of the image is memory that held plain text messages at some point. Notably the string piece "hosts.org_dir" appears often enough that I visually noticed it. Thoughts? If you want to see the core file, I can make it available via anon FTP as well. -Craig

1 0

RE: [syslog-ng]Adding new facilities
by Hamilton, Andrew Mr RAYTHEON 5 SIG CMD 02 Aug '01

02 Aug '01

Brad, you are correct. It is basically an integer define. regards, Drew -----Original Message----- From: Brad Arlt [mailto:arlt@cpsc.ucalgary.ca] Sent: Wednesday, August 01, 2001 6:57 PM To: syslog-ng(a)lists.balabit.hu Subject: [syslog-ng]Adding new facilities Hello... I am compiling syslog-ng 1.4.12 for Solaris 8, and noticed the IRIX factilities of LOG_AUDIT and LOG_LFMT are not mentioned anywhere in syslog-names.c. Also the facilities of LOG_NTP, LOG_SECURITY, and LOG_CONSOLE from BSD are not mentioned anywhere. I have in the past (and will in this build) manually defined LOG_AUTHPRIV and LOG_FTP for receiving messages from our Linux/BSD machines (AUTHPRIV and FTP are a Linux and BSD thing). If I understand the source code correctly the names are only used to translate configuration file strings to number (e.g. "auth" -> (4<<3)). If I am right, this should mean I can add the above facilities without any problems. Can anyone tell me if this sounds right? I am going to give it a try anyway, but I wanted to see if anyone thought I was crazy, or if there was a better way to do this. ---------------------------------------------------------------------------- __o Bradley Arlt Email: arlt(a)cpsc.ucalgary.ca o__ _ \<_ WWW: www.acs.ucalgary.ca/~bdarlt _>/ _ (_)/(_) -Eat well, sleep peacefully, drink lots, and ride like hell. (_)\(_) _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng

1 0

Adding new facilities
by Brad Arlt 01 Aug '01

01 Aug '01

Hello... I am compiling syslog-ng 1.4.12 for Solaris 8, and noticed the IRIX factilities of LOG_AUDIT and LOG_LFMT are not mentioned anywhere in syslog-names.c. Also the facilities of LOG_NTP, LOG_SECURITY, and LOG_CONSOLE from BSD are not mentioned anywhere. I have in the past (and will in this build) manually defined LOG_AUTHPRIV and LOG_FTP for receiving messages from our Linux/BSD machines (AUTHPRIV and FTP are a Linux and BSD thing). If I understand the source code correctly the names are only used to translate configuration file strings to number (e.g. "auth" -> (4<<3)). If I am right, this should mean I can add the above facilities without any problems. Can anyone tell me if this sounds right? I am going to give it a try anyway, but I wanted to see if anyone thought I was crazy, or if there was a better way to do this. ---------------------------------------------------------------------------- __o Bradley Arlt Email: arlt(a)cpsc.ucalgary.ca o__ _ \<_ WWW: www.acs.ucalgary.ca/~bdarlt _>/ _ (_)/(_) -Eat well, sleep peacefully, drink lots, and ride like hell. (_)\(_)

1 0