[syslog-ng] Known memory leak issues in 3.3.4?

[Marvin Nipper]

After considerable effort, I managed to get 3.3.4 built (using gcc 4.4.2, along with glib 2.29.92 and the requisite eventlog 0.2.12).  The problem is that the resulting syslog-ng component grows continuously (at an alarming rate).  Whereas my 3.0.5 syslog-ng process starts out running in just under 3MB, and then stabilizes at about 3.3MB, the 3.3.4 version starts out small, and then at the point where I killed it (about 2 hours after starting it), it had already grown to 277MB.

I'm doing nothing fancy here (config shown below).  No data transformations, no use of time normalization features, no DB components.  In short, I'm not trying to use any of the many new/elaborate features that have been added over the past few years.  I'm just "eating log input", writing it to handful of files and pipes, just like I've been doing since the syslog-ng 2.x days.

As one of the regular problems that I see mentioned in various threads is "some new fix for a memory leak", I'm just trying to see if there's some known problem in the virgin 3.3.4 code, that might be causing my problems.  (I.E. should I wait for 3.3.5, and try again?)

FYI, my environment is Solaris 10 x86.  Aside from having moved to the newer gcc compiler (in order to even attempt to compile the latest syslog-ng version), the only other major component change was the newer version of glib, which as I recall, was also necessary to resolve compilation issues with the new syslog-ng versions.

Anyway... before I beat my head against the wall trying to figure out what to do next, I figured that I'd ask the easy question (about "known bugs") first, so that I don't spend any time chasing a problem that may already be fixed.  (I did dig thru the mailing list, and saw at least one mention of a memory leak in 3.3.4, but it seemed to be related to a specific bit of functionality, that I didn't think applied in my situation.)

(And just a side-comment, FWIW, I noticed that when I swapped the 3.3.4 version, for the 3.0.5 version, the host system immediately incurred what I estimate to be at least a 20% high CPU load that had been required by the 3.0.5 version.  And when I reverted back to 3.0.5 a few hours later, I saw that happen in reverse.  I'm not sure why the newest version requires more resources, again, when I'm not asking it to do anything new, but it definitely seems that it does.)

Thanks for your time and input.

Existing config file shown below.  Note that other than the version change, the only other modification (from the file that I was using in 3.0.5) relates to the threading changes, which I made based upon Bazsi's recommendations in an email on 3/18, about how to avoid potential data loss on inbound UDP packets.

Again, thanks for any help/input:
@version:3.3
options { dir_perm(0755); perm(0640); group(wheel); chain_hostnames(no);
          keep_hostname(yes); log_fifo_size(41000); threaded(no);
          dns_cache_size(5000); dns_cache_expire(86400);
          dns_cache_expire_failed(86400); };

source any_udp { udp(flags(store-legacy-msghdr)); };
source any_tcp { tcp(port(601) max-connections(40)
                 flags("store-legacy-msghdr", "threaded") use_dns(no)); };

destination SEC {pipe("/tmp/sec"); };
destination WIN-PIPE {pipe("/tmp/windows"); };
destination routers_log {
        file("/var/adm/log/routers.log" create_dirs(yes) flags("threaded")); };
destination ravlin_log {
        file("/var/adm/log/ravlin.log" create_dirs(yes) flags("threaded")); };
destination windows_log {
        file("/var/adm/log/windows.log" create_dirs(yes) flags("threaded")); };
destination workstation_log {
        file("/var/adm/log/workstation.log" create_dirs(yes)
                                            flags("threaded")); };
destination catch-all_log {
        file("/var/adm/log/catch-all.log" create_dirs(yes)
                                          flags("threaded")); };
destination test {
        file("/var/adm/log/test.log" create_dirs(yes) flags("threaded")); };

destination dev_null {};

destination secdevedc01 { udp("10.132.193.245" spoof_source(yes)
                          flags("threaded")); };
destination engedc01 { udp("10.132.192.199" spoof_source(yes)
                       flags("threaded")); };
destination egssiem { udp("10.10.36.157" spoof_source(yes)
                      flags("threaded")); };

log { source(any_udp); destination(SEC); };
log { source(any_udp); destination(egssiem); };

filter f_4 { facility(syslog) and level(info..emerg); };
log { source(any_udp); filter(f_4); destination(WIN-PIPE); };
log { source(any_udp); filter(f_4); destination(windows_log); flags(final); };

filter f_1_acl { facility(local5) and level(debug..emerg)
                 and match("SEC-6-IPACCESSLOG" value("MESSAGE")) ; };
filter f_1 { facility(local5) and level(debug..emerg); };
log { source(any_udp); filter(f_1_acl); destination(routers_log);
      flags(final); };
log { source(any_udp); filter(f_1); destination(engedc01); };
log { source(any_udp); filter(f_1); destination(routers_log); flags(final); };

filter SonicWallNoise {
        match("id=firewall" value("MESSAGE"))
               and filter(SonicWallMsgs); };
filter SonicWallMsgs {
        match("m=97" value("MESSAGE")) or match("m=98" value("MESSAGE"))
               or match("m=537" value("MESSAGE")); };
log { source(any_udp); filter(SonicWallNoise); destination(dev_null);
      flags(final); };

filter f_3 { facility(local0) and level(debug..emerg); };
log { source(any_udp); filter(f_3); destination(ravlin_log); flags(final); };

log { source(any_tcp); destination(workstation_log); flags(final); };

log { source(any_udp); destination(catch-all_log); flags(final); };



Marvin Nipper
Director of Security
Stream Global Services
mailto:marvin.nipper at stream.com
ph: (303) 670-2705
PGP Key ID: 0x8EE28551 (DSS/DH)
8C5D 403A D107 0A95 672B  B637 BCF1 919A 8EE2 8551




This e-mail may contain confidential and/or privileged information. If you are
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy this e-mail. Any unauthorized 
copying, disclosure or distribution of the material in this e-mail is strictly 
forbidden.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20120322/aa33e090/attachment.htm