<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#0050d0">
<font size="-1"><font face="Helvetica, Arial, sans-serif">I'm running
3.0.5 self-compiled with libdbi and the oracle driver as well, and
havent run across any memory issues. Its only running in our test
environment, so only handles a few hundred thousand entries a day, but
I've checked it from one day to the next, and memory increase would
only go up by a few hundred K at most (sometimes it wouldnt go up a
single bit).<br>
Its got to be some feature youre using in your config that were not.<br>
I'd try chopping your config to be fairly minimal and and add stuff
until you find whats doing it. (like remove all the netmask() filters
at the same time, etc)<br>
</font></font><br>
Sent: Monday, March 15, 2010 10:03:45 AM<br>
From: Andreas Sartori <a class="moz-txt-link-rfc2396E" href="mailto:andreas.sartori@fh-salzburg.ac.at">&lt;andreas.sartori@fh-salzburg.ac.at&gt;</a><br>
To: Syslog-ng users' and developers' mailing list
<a class="moz-txt-link-rfc2396E" href="mailto:syslog-ng@lists.balabit.hu">&lt;syslog-ng@lists.balabit.hu&gt;</a> <br>
Subject: Re: [syslog-ng] possible memleak or bad configuration?
<blockquote cite="mid:4B9E5A61.1030305@fh-salzburg.ac.at" type="cite">
  <pre wrap="">today i compiled a 3.1.beta2 and its the same issue with the memory.
after a reboot in the morning, we are currently at 2gb mem.

i hope we can get that fixed!

-andy

On 3/15/10 8:45 AM, Andreas Sartori wrote:
  </pre>
  <blockquote type="cite">
    <pre wrap="">we were running 3.0.4 (self compiled with libdbi for oracle) (same
problem) and then upgraded to 3.0.5 rhel5 from (directly from the website).

the box itself is a vm on esxi4u1 with centos 5.4 x86_84.

-andy



On 3/13/10 7:03 PM, Martin Holste wrote:
    </pre>
    <blockquote type="cite">
      <pre wrap="">The db parser code had a big memory leak in previous 3.1 versions but
was fixed a few months ago; what build are you running?  We process 2
billion logs per day through db parser with no leaks at all using the
build from git commit 9ef6062c1cf72a3f7da880ac245f9ee080bea992.

--Martin

On Sat, Mar 13, 2010 at 2:22 AM, Andreas Sartori
&lt;<a class="moz-txt-link-abbreviated" href="mailto:andreas.sartori@fh-salzburg.ac.at">andreas.sartori@fh-salzburg.ac.at</a>
<a class="moz-txt-link-rfc2396E" href="mailto:andreas.sartori@fh-salzburg.ac.at">&lt;mailto:andreas.sartori@fh-salzburg.ac.at&gt;</a>&gt;  wrote:

     hello,


     we have setup a central logging server. currently we are logging
     firewalls
     and  some webserver / mailserver for testing purpose. the memory
     usage on
     the logging server is badly increasing. after 2 days of operation we are
     at 6.8 gb ram usage.

     can someone help out, what information do you need to help?

     thanks in advance.

     -andy

     ------------

     @version:3.0
     #
     # configuration file for syslog-ng, customized for remote logging
     #

     options {
             owner("root");
             group("root");
             perm(0600);
             dir_perm(0750);
             create_dirs(yes);
             log_fifo_size(10000);
     };



     ################################################################################################
     #########################                SOURCES
     ##############################
     ################################################################################################

     # Syslog internal logging
     source s_internal { internal(); };
     destination d_syslognglog { file("/var/log/syslog-ng.log"); };
     log { source(s_internal); destination(d_syslognglog); };


     # Remote logging
     source s_remote {
             tcp(ip(0.0.0.0) max-connections(20) port(514)
     keep_hostname(yes));
             udp(ip(0.0.0.0) port(514) use_dns(no) log_fetch_limit(500)
     log_iw_size(1000));
     };


     ################################################################################################
     #########################                FILTER
     ##############################
     ################################################################################################

     filter http-official { netmask(xxx.xxx.xxx.47/255.255.255.255
     <a class="moz-txt-link-rfc2396E" href="http://255.255.255.255">&lt;http://255.255.255.255&gt;</a>) or
     netmask(xxx.xxx.xxx.48/255.255.255.255<a class="moz-txt-link-rfc2396E" href="http://255.255.255.255">&lt;http://255.255.255.255&gt;</a>) or
     netmask(xxx.xxx.xxx.167/255.255.255.255<a class="moz-txt-link-rfc2396E" href="http://255.255.255.255">&lt;http://255.255.255.255&gt;</a>) or
     netmask(xxx.xxx.xxx.46/255.255.255.255<a class="moz-txt-link-rfc2396E" href="http://255.255.255.255">&lt;http://255.255.255.255&gt;</a>) or
     netmask(xxx.xxx.xxx.52/255.255.255.255<a class="moz-txt-link-rfc2396E" href="http://255.255.255.255">&lt;http://255.255.255.255&gt;</a>) or
     netmask(xxx.xxx.xxx.25/255.255.255.255<a class="moz-txt-link-rfc2396E" href="http://255.255.255.255">&lt;http://255.255.255.255&gt;</a>) or
     netmask(xxx.xxx.xxx.26/255.255.255.255<a class="moz-txt-link-rfc2396E" href="http://255.255.255.255">&lt;http://255.255.255.255&gt;</a>); };

     filter mail-proxy-internal { netmask(10.10.9.20/255.255.255.255
     <a class="moz-txt-link-rfc2396E" href="http://10.10.9.20/255.255.255.255">&lt;http://10.10.9.20/255.255.255.255&gt;</a>) and not
     program("perdition"); };
     filter mail-relay-internal { netmask(10.10.9.30/255.255.255.255
     <a class="moz-txt-link-rfc2396E" href="http://10.10.9.30/255.255.255.255">&lt;http://10.10.9.30/255.255.255.255&gt;</a>); };

     filter mail-relay-alpha-external-out {
     netmask(xxx.xxx.xxx.59/255.255.255.255<a class="moz-txt-link-rfc2396E" href="http://255.255.255.255">&lt;http://255.255.255.255&gt;</a>) and
     facility(local1); };
     filter mail-relay-beta-external-out {
     netmask(xxx.xxx.xxx.60/255.255.255.255<a class="moz-txt-link-rfc2396E" href="http://255.255.255.255">&lt;http://255.255.255.255&gt;</a>) and
     facility(local1); };
     filter mail-relay-alpha-external-in {
     netmask(xxx.xxx.xxx.59/255.255.255.255<a class="moz-txt-link-rfc2396E" href="http://255.255.255.255">&lt;http://255.255.255.255&gt;</a>) and
     facility(mail); };
     filter mail-relay-beta-external-in {
     netmask(xxx.xxx.xxx.60/255.255.255.255<a class="moz-txt-link-rfc2396E" href="http://255.255.255.255">&lt;http://255.255.255.255&gt;</a>) and
     facility(mail); };

     filter mail-proxy-node1-external {
     netmask(xxx.xxx.xxx.18/255.255.255.255<a class="moz-txt-link-rfc2396E" href="http://255.255.255.255">&lt;http://255.255.255.255&gt;</a>)
     and not program("perdition"); };
     filter mail-proxy-node2-external {
     netmask(xxx.xxx.xxx.22/255.255.255.255<a class="moz-txt-link-rfc2396E" href="http://255.255.255.255">&lt;http://255.255.255.255&gt;</a>)
     and not program("perdition"); };

     filter vpn { netmask(10.20.40.0/255.255.255.0
     <a class="moz-txt-link-rfc2396E" href="http://10.20.40.0/255.255.255.0">&lt;http://10.20.40.0/255.255.255.0&gt;</a>); };
     filter fw-intern-all { netmask(10.10.20.1/255.255.255.255
     <a class="moz-txt-link-rfc2396E" href="http://10.10.20.1/255.255.255.255">&lt;http://10.10.20.1/255.255.255.255&gt;</a>); };

     filter fw-intern-security {
                     netmask(10.10.20.1/255.255.255.255
     <a class="moz-txt-link-rfc2396E" href="http://10.10.20.1/255.255.255.255">&lt;http://10.10.20.1/255.255.255.255&gt;</a>) and
                     match("security" value(".classifier.class")
     type("string"));
     };

     filter fw-intern-info {
                     netmask(10.10.20.1/255.255.255.255
     <a class="moz-txt-link-rfc2396E" href="http://10.10.20.1/255.255.255.255">&lt;http://10.10.20.1/255.255.255.255&gt;</a>) and
                     match("informational" value(".classifier.class")
     type("string"));
     };

     filter fw-intern-rest {
                     netmask(10.10.20.1/255.255.255.255
     <a class="moz-txt-link-rfc2396E" href="http://10.10.20.1/255.255.255.255">&lt;http://10.10.20.1/255.255.255.255&gt;</a>) and not
                     match("security" value(".classifier.class")
     type("string")) and not
                     match("informational" value(".classifier.class")
     type("string"));
     };


     filter fw-extern-all { netmask(10.80.11.20/255.255.255.255
     <a class="moz-txt-link-rfc2396E" href="http://10.80.11.20/255.255.255.255">&lt;http://10.80.11.20/255.255.255.255&gt;</a>); };

     filter fw-extern-security {
                     netmask(10.80.11.20/255.255.255.255
     <a class="moz-txt-link-rfc2396E" href="http://10.80.11.20/255.255.255.255">&lt;http://10.80.11.20/255.255.255.255&gt;</a>) and
                     match("security" value(".classifier.class")
     type("string"));
     };

     filter fw-extern-info {
                     netmask(10.80.11.20/255.255.255.255
     <a class="moz-txt-link-rfc2396E" href="http://10.80.11.20/255.255.255.255">&lt;http://10.80.11.20/255.255.255.255&gt;</a>) and
                     match("informational" value(".classifier.class")
     type("string"));
     };

     filter fw-extern-rest {
                     netmask(10.80.11.20/255.255.255.255
     <a class="moz-txt-link-rfc2396E" href="http://10.80.11.20/255.255.255.255">&lt;http://10.80.11.20/255.255.255.255&gt;</a>) and not
                     match("security" value(".classifier.class")
     type("string")) and not
                     match("informational" value(".classifier.class")
     type("string"));
     };

     filter fw-extern-new { netmask(10.80.11.30/255.255.255.255
     <a class="moz-txt-link-rfc2396E" href="http://10.80.11.30/255.255.255.255">&lt;http://10.80.11.30/255.255.255.255&gt;</a>); };

     ################################################################################################
     #########################                PARSER
     ##############################
     ################################################################################################

     parser pattern_db_fwint {
             db_parser(
             file("/etc/syslog-ng/fw-int_patterndb.xml")
             );
     };

     parser pattern_db_fwext {
             db_parser(
             file("/etc/syslog-ng/fw-ext_patterndb.xml")
             );
     };

     ################################################################################################
     #########################             DESTINATIONS
     ##############################
     ################################################################################################

     destination http-log { file("/logging/server/web/$HOST"
     template("$MSGONLY\n") template-escape(no) owner("root") group("root")
     perm(0644));  };

     destination mail-out {
     file("/logging/server/mail/mail-out_$MONTH.log"); };
     destination mail-in {
     file("/logging/server/mail/mail-in_$MONTH.log"); };

     destination vpn {
             file("/logging/network/vpn_$MONTH.log" flush_lines(10));
     };

     destination fw-intern-all {
             file("/logging/network/fw-intern_$MONTH.log" flush_lines(10));
     };

     destination fw-extern-all {
             file("/logging/network/fw-extern_$MONTH.log" flush_lines(10));
     };


     destination fw-extern-new {
             file("/logging/network/fw-new_$MONTH.log" flush_lines(10));
     };


     destination dump {
             file("/logging/network/dump.log" template
     ("$R_YEAR-$R_MONTH-$R_DAY
     $R_HOUR:$R_MIN:$R_SEC, $HOST, $FIREWALL_SEQ, $MSGHDR, 0, $FIREWALL_IO,
     $FIREWALL_PROTO, $FIREWALL_SCR_LAN, $FIREWALL_SRC_IP,
     $FIREWALL_SRC_PORT,
     $FIREWALL_DST_LAN, $FIREWALL_DST_IP, $FIREWALL_DST_PORT,
     $FIREWALL_NAT_SRC_IP, $FIREWALL_NAT_DST_IP, $FIREWALL_RULE,
     $FIREWALL_REASON, $FIREWALL_DURATION\n"));
     #       file("/logging/network/dump.log" template ("$MSGHDR\n")
     flush_lines(5));
     };



     ################################################################################################
     #########################              FINAL-LOGS
     ##############################
     ################################################################################################

     ##### TO FILE

     log { source(s_remote); filter(http-official); destination(http-log); };
     log { source(s_remote); filter(mail-proxy-internal);
     destination(mail-out); };
     log { source(s_remote); filter(mail-relay-internal);
     destination(mail-out); };
     log { source(s_remote); filter(mail-relay-alpha-external-out);
     destination(mail-out); };
     log { source(s_remote); filter(mail-relay-beta-external-out);
     destination(mail-out); };
     log { source(s_remote); filter(mail-proxy-node1-external);
     destination(mail-out); };
     log { source(s_remote); filter(mail-proxy-node2-external);
     destination(mail-out); };
     log { source(s_remote); filter(mail-relay-alpha-external-in);
     destination(mail-in); };
     log { source(s_remote); filter(mail-relay-beta-external-in);
     destination(mail-in); };
     log { source(s_remote); filter(vpn); destination(vpn); };
     log { source(s_remote); filter(fw-intern-all);
     destination(fw-intern-all); };
     log { source(s_remote); filter(fw-extern-new);
     destination(fw-extern-new); };
     log { source(s_remote); filter(fw-extern-all);
     destination(fw-extern-all);
     flags(final); };



     ______________________________________________________________________________
     Member info: <a class="moz-txt-link-freetext" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
     Documentation:
     <a class="moz-txt-link-freetext" href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
     FAQ: <a class="moz-txt-link-freetext" href="http://www.campin.net/syslog-ng/faq.html">http://www.campin.net/syslog-ng/faq.html</a>




______________________________________________________________________________
Member info: <a class="moz-txt-link-freetext" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a class="moz-txt-link-freetext" href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a class="moz-txt-link-freetext" href="http://www.campin.net/syslog-ng/faq.html">http://www.campin.net/syslog-ng/faq.html</a>

      </pre>
    </blockquote>
  </blockquote>
  <pre wrap=""><!---->
  </pre>
</blockquote>
</body>
</html>