<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#0050d0">
<font size="-1"><font face="Helvetica, Arial, sans-serif">I'm running
3.0.5 self-compiled with libdbi and the oracle driver as well, and
havent run across any memory issues. Its only running in our test
environment, so only handles a few hundred thousand entries a day, but
I've checked it from one day to the next, and memory increase would
only go up by a few hundred K at most (sometimes it wouldnt go up a
single bit).<br>
Its got to be some feature youre using in your config that were not.<br>
I'd try chopping your config to be fairly minimal and and add stuff
until you find whats doing it. (like remove all the netmask() filters
at the same time, etc)<br>
</font></font><br>
Sent: Monday, March 15, 2010 10:03:45 AM<br>
From: Andreas Sartori <a class="moz-txt-link-rfc2396E" href="mailto:andreas.sartori@fh-salzburg.ac.at"><andreas.sartori@fh-salzburg.ac.at></a><br>
To: Syslog-ng users' and developers' mailing list
<a class="moz-txt-link-rfc2396E" href="mailto:syslog-ng@lists.balabit.hu"><syslog-ng@lists.balabit.hu></a> <br>
Subject: Re: [syslog-ng] possible memleak or bad configuration?
<blockquote cite="mid:4B9E5A61.1030305@fh-salzburg.ac.at" type="cite">
<pre wrap="">today i compiled a 3.1.beta2 and its the same issue with the memory.
after a reboot in the morning, we are currently at 2gb mem.
i hope we can get that fixed!
-andy
On 3/15/10 8:45 AM, Andreas Sartori wrote:
</pre>
<blockquote type="cite">
<pre wrap="">we were running 3.0.4 (self compiled with libdbi for oracle) (same
problem) and then upgraded to 3.0.5 rhel5 from (directly from the website).
the box itself is a vm on esxi4u1 with centos 5.4 x86_84.
-andy
On 3/13/10 7:03 PM, Martin Holste wrote:
</pre>
<blockquote type="cite">
<pre wrap="">The db parser code had a big memory leak in previous 3.1 versions but
was fixed a few months ago; what build are you running? We process 2
billion logs per day through db parser with no leaks at all using the
build from git commit 9ef6062c1cf72a3f7da880ac245f9ee080bea992.
--Martin
On Sat, Mar 13, 2010 at 2:22 AM, Andreas Sartori
<<a class="moz-txt-link-abbreviated" href="mailto:andreas.sartori@fh-salzburg.ac.at">andreas.sartori@fh-salzburg.ac.at</a>
<a class="moz-txt-link-rfc2396E" href="mailto:andreas.sartori@fh-salzburg.ac.at"><mailto:andreas.sartori@fh-salzburg.ac.at></a>> wrote:
hello,
we have setup a central logging server. currently we are logging
firewalls
and some webserver / mailserver for testing purpose. the memory
usage on
the logging server is badly increasing. after 2 days of operation we are
at 6.8 gb ram usage.
can someone help out, what information do you need to help?
thanks in advance.
-andy
------------
@version:3.0
#
# configuration file for syslog-ng, customized for remote logging
#
options {
owner("root");
group("root");
perm(0600);
dir_perm(0750);
create_dirs(yes);
log_fifo_size(10000);
};
################################################################################################
######################### SOURCES
##############################
################################################################################################
# Syslog internal logging
source s_internal { internal(); };
destination d_syslognglog { file("/var/log/syslog-ng.log"); };
log { source(s_internal); destination(d_syslognglog); };
# Remote logging
source s_remote {
tcp(ip(0.0.0.0) max-connections(20) port(514)
keep_hostname(yes));
udp(ip(0.0.0.0) port(514) use_dns(no) log_fetch_limit(500)
log_iw_size(1000));
};
################################################################################################
######################### FILTER
##############################
################################################################################################
filter http-official { netmask(xxx.xxx.xxx.47/255.255.255.255
<a class="moz-txt-link-rfc2396E" href="http://255.255.255.255"><http://255.255.255.255></a>) or
netmask(xxx.xxx.xxx.48/255.255.255.255<a class="moz-txt-link-rfc2396E" href="http://255.255.255.255"><http://255.255.255.255></a>) or
netmask(xxx.xxx.xxx.167/255.255.255.255<a class="moz-txt-link-rfc2396E" href="http://255.255.255.255"><http://255.255.255.255></a>) or
netmask(xxx.xxx.xxx.46/255.255.255.255<a class="moz-txt-link-rfc2396E" href="http://255.255.255.255"><http://255.255.255.255></a>) or
netmask(xxx.xxx.xxx.52/255.255.255.255<a class="moz-txt-link-rfc2396E" href="http://255.255.255.255"><http://255.255.255.255></a>) or
netmask(xxx.xxx.xxx.25/255.255.255.255<a class="moz-txt-link-rfc2396E" href="http://255.255.255.255"><http://255.255.255.255></a>) or
netmask(xxx.xxx.xxx.26/255.255.255.255<a class="moz-txt-link-rfc2396E" href="http://255.255.255.255"><http://255.255.255.255></a>); };
filter mail-proxy-internal { netmask(10.10.9.20/255.255.255.255
<a class="moz-txt-link-rfc2396E" href="http://10.10.9.20/255.255.255.255"><http://10.10.9.20/255.255.255.255></a>) and not
program("perdition"); };
filter mail-relay-internal { netmask(10.10.9.30/255.255.255.255
<a class="moz-txt-link-rfc2396E" href="http://10.10.9.30/255.255.255.255"><http://10.10.9.30/255.255.255.255></a>); };
filter mail-relay-alpha-external-out {
netmask(xxx.xxx.xxx.59/255.255.255.255<a class="moz-txt-link-rfc2396E" href="http://255.255.255.255"><http://255.255.255.255></a>) and
facility(local1); };
filter mail-relay-beta-external-out {
netmask(xxx.xxx.xxx.60/255.255.255.255<a class="moz-txt-link-rfc2396E" href="http://255.255.255.255"><http://255.255.255.255></a>) and
facility(local1); };
filter mail-relay-alpha-external-in {
netmask(xxx.xxx.xxx.59/255.255.255.255<a class="moz-txt-link-rfc2396E" href="http://255.255.255.255"><http://255.255.255.255></a>) and
facility(mail); };
filter mail-relay-beta-external-in {
netmask(xxx.xxx.xxx.60/255.255.255.255<a class="moz-txt-link-rfc2396E" href="http://255.255.255.255"><http://255.255.255.255></a>) and
facility(mail); };
filter mail-proxy-node1-external {
netmask(xxx.xxx.xxx.18/255.255.255.255<a class="moz-txt-link-rfc2396E" href="http://255.255.255.255"><http://255.255.255.255></a>)
and not program("perdition"); };
filter mail-proxy-node2-external {
netmask(xxx.xxx.xxx.22/255.255.255.255<a class="moz-txt-link-rfc2396E" href="http://255.255.255.255"><http://255.255.255.255></a>)
and not program("perdition"); };
filter vpn { netmask(10.20.40.0/255.255.255.0
<a class="moz-txt-link-rfc2396E" href="http://10.20.40.0/255.255.255.0"><http://10.20.40.0/255.255.255.0></a>); };
filter fw-intern-all { netmask(10.10.20.1/255.255.255.255
<a class="moz-txt-link-rfc2396E" href="http://10.10.20.1/255.255.255.255"><http://10.10.20.1/255.255.255.255></a>); };
filter fw-intern-security {
netmask(10.10.20.1/255.255.255.255
<a class="moz-txt-link-rfc2396E" href="http://10.10.20.1/255.255.255.255"><http://10.10.20.1/255.255.255.255></a>) and
match("security" value(".classifier.class")
type("string"));
};
filter fw-intern-info {
netmask(10.10.20.1/255.255.255.255
<a class="moz-txt-link-rfc2396E" href="http://10.10.20.1/255.255.255.255"><http://10.10.20.1/255.255.255.255></a>) and
match("informational" value(".classifier.class")
type("string"));
};
filter fw-intern-rest {
netmask(10.10.20.1/255.255.255.255
<a class="moz-txt-link-rfc2396E" href="http://10.10.20.1/255.255.255.255"><http://10.10.20.1/255.255.255.255></a>) and not
match("security" value(".classifier.class")
type("string")) and not
match("informational" value(".classifier.class")
type("string"));
};
filter fw-extern-all { netmask(10.80.11.20/255.255.255.255
<a class="moz-txt-link-rfc2396E" href="http://10.80.11.20/255.255.255.255"><http://10.80.11.20/255.255.255.255></a>); };
filter fw-extern-security {
netmask(10.80.11.20/255.255.255.255
<a class="moz-txt-link-rfc2396E" href="http://10.80.11.20/255.255.255.255"><http://10.80.11.20/255.255.255.255></a>) and
match("security" value(".classifier.class")
type("string"));
};
filter fw-extern-info {
netmask(10.80.11.20/255.255.255.255
<a class="moz-txt-link-rfc2396E" href="http://10.80.11.20/255.255.255.255"><http://10.80.11.20/255.255.255.255></a>) and
match("informational" value(".classifier.class")
type("string"));
};
filter fw-extern-rest {
netmask(10.80.11.20/255.255.255.255
<a class="moz-txt-link-rfc2396E" href="http://10.80.11.20/255.255.255.255"><http://10.80.11.20/255.255.255.255></a>) and not
match("security" value(".classifier.class")
type("string")) and not
match("informational" value(".classifier.class")
type("string"));
};
filter fw-extern-new { netmask(10.80.11.30/255.255.255.255
<a class="moz-txt-link-rfc2396E" href="http://10.80.11.30/255.255.255.255"><http://10.80.11.30/255.255.255.255></a>); };
################################################################################################
######################### PARSER
##############################
################################################################################################
parser pattern_db_fwint {
db_parser(
file("/etc/syslog-ng/fw-int_patterndb.xml")
);
};
parser pattern_db_fwext {
db_parser(
file("/etc/syslog-ng/fw-ext_patterndb.xml")
);
};
################################################################################################
######################### DESTINATIONS
##############################
################################################################################################
destination http-log { file("/logging/server/web/$HOST"
template("$MSGONLY\n") template-escape(no) owner("root") group("root")
perm(0644)); };
destination mail-out {
file("/logging/server/mail/mail-out_$MONTH.log"); };
destination mail-in {
file("/logging/server/mail/mail-in_$MONTH.log"); };
destination vpn {
file("/logging/network/vpn_$MONTH.log" flush_lines(10));
};
destination fw-intern-all {
file("/logging/network/fw-intern_$MONTH.log" flush_lines(10));
};
destination fw-extern-all {
file("/logging/network/fw-extern_$MONTH.log" flush_lines(10));
};
destination fw-extern-new {
file("/logging/network/fw-new_$MONTH.log" flush_lines(10));
};
destination dump {
file("/logging/network/dump.log" template
("$R_YEAR-$R_MONTH-$R_DAY
$R_HOUR:$R_MIN:$R_SEC, $HOST, $FIREWALL_SEQ, $MSGHDR, 0, $FIREWALL_IO,
$FIREWALL_PROTO, $FIREWALL_SCR_LAN, $FIREWALL_SRC_IP,
$FIREWALL_SRC_PORT,
$FIREWALL_DST_LAN, $FIREWALL_DST_IP, $FIREWALL_DST_PORT,
$FIREWALL_NAT_SRC_IP, $FIREWALL_NAT_DST_IP, $FIREWALL_RULE,
$FIREWALL_REASON, $FIREWALL_DURATION\n"));
# file("/logging/network/dump.log" template ("$MSGHDR\n")
flush_lines(5));
};
################################################################################################
######################### FINAL-LOGS
##############################
################################################################################################
##### TO FILE
log { source(s_remote); filter(http-official); destination(http-log); };
log { source(s_remote); filter(mail-proxy-internal);
destination(mail-out); };
log { source(s_remote); filter(mail-relay-internal);
destination(mail-out); };
log { source(s_remote); filter(mail-relay-alpha-external-out);
destination(mail-out); };
log { source(s_remote); filter(mail-relay-beta-external-out);
destination(mail-out); };
log { source(s_remote); filter(mail-proxy-node1-external);
destination(mail-out); };
log { source(s_remote); filter(mail-proxy-node2-external);
destination(mail-out); };
log { source(s_remote); filter(mail-relay-alpha-external-in);
destination(mail-in); };
log { source(s_remote); filter(mail-relay-beta-external-in);
destination(mail-in); };
log { source(s_remote); filter(vpn); destination(vpn); };
log { source(s_remote); filter(fw-intern-all);
destination(fw-intern-all); };
log { source(s_remote); filter(fw-extern-new);
destination(fw-extern-new); };
log { source(s_remote); filter(fw-extern-all);
destination(fw-extern-all);
flags(final); };
______________________________________________________________________________
Member info: <a class="moz-txt-link-freetext" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation:
<a class="moz-txt-link-freetext" href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a class="moz-txt-link-freetext" href="http://www.campin.net/syslog-ng/faq.html">http://www.campin.net/syslog-ng/faq.html</a>
______________________________________________________________________________
Member info: <a class="moz-txt-link-freetext" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a class="moz-txt-link-freetext" href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a class="moz-txt-link-freetext" href="http://www.campin.net/syslog-ng/faq.html">http://www.campin.net/syslog-ng/faq.html</a>
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!---->
</pre>
</blockquote>
</body>
</html>