[syslog-ng] possible memleak or bad configuration?
Patrick H.
syslogng at feystorm.net
Mon Mar 15 17:15:56 CET 2010
I'm running 3.0.5 self-compiled with libdbi and the oracle driver as
well, and havent run across any memory issues. Its only running in our
test environment, so only handles a few hundred thousand entries a day,
but I've checked it from one day to the next, and memory increase would
only go up by a few hundred K at most (sometimes it wouldnt go up a
single bit).
Its got to be some feature youre using in your config that were not.
I'd try chopping your config to be fairly minimal and and add stuff
until you find whats doing it. (like remove all the netmask() filters at
the same time, etc)
Sent: Monday, March 15, 2010 10:03:45 AM
From: Andreas Sartori <andreas.sartori at fh-salzburg.ac.at>
To: Syslog-ng users' and developers' mailing list
<syslog-ng at lists.balabit.hu>
Subject: Re: [syslog-ng] possible memleak or bad configuration?
> today i compiled a 3.1.beta2 and its the same issue with the memory.
> after a reboot in the morning, we are currently at 2gb mem.
>
> i hope we can get that fixed!
>
> -andy
>
> On 3/15/10 8:45 AM, Andreas Sartori wrote:
>
>> we were running 3.0.4 (self compiled with libdbi for oracle) (same
>> problem) and then upgraded to 3.0.5 rhel5 from (directly from the website).
>>
>> the box itself is a vm on esxi4u1 with centos 5.4 x86_84.
>>
>> -andy
>>
>>
>>
>> On 3/13/10 7:03 PM, Martin Holste wrote:
>>
>>> The db parser code had a big memory leak in previous 3.1 versions but
>>> was fixed a few months ago; what build are you running? We process 2
>>> billion logs per day through db parser with no leaks at all using the
>>> build from git commit 9ef6062c1cf72a3f7da880ac245f9ee080bea992.
>>>
>>> --Martin
>>>
>>> On Sat, Mar 13, 2010 at 2:22 AM, Andreas Sartori
>>> <andreas.sartori at fh-salzburg.ac.at
>>> <mailto:andreas.sartori at fh-salzburg.ac.at>> wrote:
>>>
>>> hello,
>>>
>>>
>>> we have setup a central logging server. currently we are logging
>>> firewalls
>>> and some webserver / mailserver for testing purpose. the memory
>>> usage on
>>> the logging server is badly increasing. after 2 days of operation we are
>>> at 6.8 gb ram usage.
>>>
>>> can someone help out, what information do you need to help?
>>>
>>> thanks in advance.
>>>
>>> -andy
>>>
>>> ------------
>>>
>>> @version:3.0
>>> #
>>> # configuration file for syslog-ng, customized for remote logging
>>> #
>>>
>>> options {
>>> owner("root");
>>> group("root");
>>> perm(0600);
>>> dir_perm(0750);
>>> create_dirs(yes);
>>> log_fifo_size(10000);
>>> };
>>>
>>>
>>>
>>> ################################################################################################
>>> ######################### SOURCES
>>> ##############################
>>> ################################################################################################
>>>
>>> # Syslog internal logging
>>> source s_internal { internal(); };
>>> destination d_syslognglog { file("/var/log/syslog-ng.log"); };
>>> log { source(s_internal); destination(d_syslognglog); };
>>>
>>>
>>> # Remote logging
>>> source s_remote {
>>> tcp(ip(0.0.0.0) max-connections(20) port(514)
>>> keep_hostname(yes));
>>> udp(ip(0.0.0.0) port(514) use_dns(no) log_fetch_limit(500)
>>> log_iw_size(1000));
>>> };
>>>
>>>
>>> ################################################################################################
>>> ######################### FILTER
>>> ##############################
>>> ################################################################################################
>>>
>>> filter http-official { netmask(xxx.xxx.xxx.47/255.255.255.255
>>> <http://255.255.255.255>) or
>>> netmask(xxx.xxx.xxx.48/255.255.255.255<http://255.255.255.255>) or
>>> netmask(xxx.xxx.xxx.167/255.255.255.255<http://255.255.255.255>) or
>>> netmask(xxx.xxx.xxx.46/255.255.255.255<http://255.255.255.255>) or
>>> netmask(xxx.xxx.xxx.52/255.255.255.255<http://255.255.255.255>) or
>>> netmask(xxx.xxx.xxx.25/255.255.255.255<http://255.255.255.255>) or
>>> netmask(xxx.xxx.xxx.26/255.255.255.255<http://255.255.255.255>); };
>>>
>>> filter mail-proxy-internal { netmask(10.10.9.20/255.255.255.255
>>> <http://10.10.9.20/255.255.255.255>) and not
>>> program("perdition"); };
>>> filter mail-relay-internal { netmask(10.10.9.30/255.255.255.255
>>> <http://10.10.9.30/255.255.255.255>); };
>>>
>>> filter mail-relay-alpha-external-out {
>>> netmask(xxx.xxx.xxx.59/255.255.255.255<http://255.255.255.255>) and
>>> facility(local1); };
>>> filter mail-relay-beta-external-out {
>>> netmask(xxx.xxx.xxx.60/255.255.255.255<http://255.255.255.255>) and
>>> facility(local1); };
>>> filter mail-relay-alpha-external-in {
>>> netmask(xxx.xxx.xxx.59/255.255.255.255<http://255.255.255.255>) and
>>> facility(mail); };
>>> filter mail-relay-beta-external-in {
>>> netmask(xxx.xxx.xxx.60/255.255.255.255<http://255.255.255.255>) and
>>> facility(mail); };
>>>
>>> filter mail-proxy-node1-external {
>>> netmask(xxx.xxx.xxx.18/255.255.255.255<http://255.255.255.255>)
>>> and not program("perdition"); };
>>> filter mail-proxy-node2-external {
>>> netmask(xxx.xxx.xxx.22/255.255.255.255<http://255.255.255.255>)
>>> and not program("perdition"); };
>>>
>>> filter vpn { netmask(10.20.40.0/255.255.255.0
>>> <http://10.20.40.0/255.255.255.0>); };
>>> filter fw-intern-all { netmask(10.10.20.1/255.255.255.255
>>> <http://10.10.20.1/255.255.255.255>); };
>>>
>>> filter fw-intern-security {
>>> netmask(10.10.20.1/255.255.255.255
>>> <http://10.10.20.1/255.255.255.255>) and
>>> match("security" value(".classifier.class")
>>> type("string"));
>>> };
>>>
>>> filter fw-intern-info {
>>> netmask(10.10.20.1/255.255.255.255
>>> <http://10.10.20.1/255.255.255.255>) and
>>> match("informational" value(".classifier.class")
>>> type("string"));
>>> };
>>>
>>> filter fw-intern-rest {
>>> netmask(10.10.20.1/255.255.255.255
>>> <http://10.10.20.1/255.255.255.255>) and not
>>> match("security" value(".classifier.class")
>>> type("string")) and not
>>> match("informational" value(".classifier.class")
>>> type("string"));
>>> };
>>>
>>>
>>> filter fw-extern-all { netmask(10.80.11.20/255.255.255.255
>>> <http://10.80.11.20/255.255.255.255>); };
>>>
>>> filter fw-extern-security {
>>> netmask(10.80.11.20/255.255.255.255
>>> <http://10.80.11.20/255.255.255.255>) and
>>> match("security" value(".classifier.class")
>>> type("string"));
>>> };
>>>
>>> filter fw-extern-info {
>>> netmask(10.80.11.20/255.255.255.255
>>> <http://10.80.11.20/255.255.255.255>) and
>>> match("informational" value(".classifier.class")
>>> type("string"));
>>> };
>>>
>>> filter fw-extern-rest {
>>> netmask(10.80.11.20/255.255.255.255
>>> <http://10.80.11.20/255.255.255.255>) and not
>>> match("security" value(".classifier.class")
>>> type("string")) and not
>>> match("informational" value(".classifier.class")
>>> type("string"));
>>> };
>>>
>>> filter fw-extern-new { netmask(10.80.11.30/255.255.255.255
>>> <http://10.80.11.30/255.255.255.255>); };
>>>
>>> ################################################################################################
>>> ######################### PARSER
>>> ##############################
>>> ################################################################################################
>>>
>>> parser pattern_db_fwint {
>>> db_parser(
>>> file("/etc/syslog-ng/fw-int_patterndb.xml")
>>> );
>>> };
>>>
>>> parser pattern_db_fwext {
>>> db_parser(
>>> file("/etc/syslog-ng/fw-ext_patterndb.xml")
>>> );
>>> };
>>>
>>> ################################################################################################
>>> ######################### DESTINATIONS
>>> ##############################
>>> ################################################################################################
>>>
>>> destination http-log { file("/logging/server/web/$HOST"
>>> template("$MSGONLY\n") template-escape(no) owner("root") group("root")
>>> perm(0644)); };
>>>
>>> destination mail-out {
>>> file("/logging/server/mail/mail-out_$MONTH.log"); };
>>> destination mail-in {
>>> file("/logging/server/mail/mail-in_$MONTH.log"); };
>>>
>>> destination vpn {
>>> file("/logging/network/vpn_$MONTH.log" flush_lines(10));
>>> };
>>>
>>> destination fw-intern-all {
>>> file("/logging/network/fw-intern_$MONTH.log" flush_lines(10));
>>> };
>>>
>>> destination fw-extern-all {
>>> file("/logging/network/fw-extern_$MONTH.log" flush_lines(10));
>>> };
>>>
>>>
>>> destination fw-extern-new {
>>> file("/logging/network/fw-new_$MONTH.log" flush_lines(10));
>>> };
>>>
>>>
>>> destination dump {
>>> file("/logging/network/dump.log" template
>>> ("$R_YEAR-$R_MONTH-$R_DAY
>>> $R_HOUR:$R_MIN:$R_SEC, $HOST, $FIREWALL_SEQ, $MSGHDR, 0, $FIREWALL_IO,
>>> $FIREWALL_PROTO, $FIREWALL_SCR_LAN, $FIREWALL_SRC_IP,
>>> $FIREWALL_SRC_PORT,
>>> $FIREWALL_DST_LAN, $FIREWALL_DST_IP, $FIREWALL_DST_PORT,
>>> $FIREWALL_NAT_SRC_IP, $FIREWALL_NAT_DST_IP, $FIREWALL_RULE,
>>> $FIREWALL_REASON, $FIREWALL_DURATION\n"));
>>> # file("/logging/network/dump.log" template ("$MSGHDR\n")
>>> flush_lines(5));
>>> };
>>>
>>>
>>>
>>> ################################################################################################
>>> ######################### FINAL-LOGS
>>> ##############################
>>> ################################################################################################
>>>
>>> ##### TO FILE
>>>
>>> log { source(s_remote); filter(http-official); destination(http-log); };
>>> log { source(s_remote); filter(mail-proxy-internal);
>>> destination(mail-out); };
>>> log { source(s_remote); filter(mail-relay-internal);
>>> destination(mail-out); };
>>> log { source(s_remote); filter(mail-relay-alpha-external-out);
>>> destination(mail-out); };
>>> log { source(s_remote); filter(mail-relay-beta-external-out);
>>> destination(mail-out); };
>>> log { source(s_remote); filter(mail-proxy-node1-external);
>>> destination(mail-out); };
>>> log { source(s_remote); filter(mail-proxy-node2-external);
>>> destination(mail-out); };
>>> log { source(s_remote); filter(mail-relay-alpha-external-in);
>>> destination(mail-in); };
>>> log { source(s_remote); filter(mail-relay-beta-external-in);
>>> destination(mail-in); };
>>> log { source(s_remote); filter(vpn); destination(vpn); };
>>> log { source(s_remote); filter(fw-intern-all);
>>> destination(fw-intern-all); };
>>> log { source(s_remote); filter(fw-extern-new);
>>> destination(fw-extern-new); };
>>> log { source(s_remote); filter(fw-extern-all);
>>> destination(fw-extern-all);
>>> flags(final); };
>>>
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>>
>>>
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>>
>>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20100315/0ae14f6f/attachment-0001.htm
More information about the syslog-ng
mailing list