[syslog-ng] Memory not freeing up in syslog-ng-3.6.2 during reload

Wed Jan 8 05:41:20 UTC 2020

Hi,

I was going through syslog-ng administrators guide and came across the
following point from the given link:

*"NOTE:Message contexts are persistent and are not lost when syslog-ng OSE
is reloaded (SIGHUP), but are lost when syslog-ng OSE is restarted."*

https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/72#TOPIC-1044162

1) Is this because of the message contexts that are persisted on reload,
the memory is not freeing up at the time of reload of patterndb ?

2) Is  there a way we can free-up these message contexts without a restart ?

Please correct me if I am wrong or missing something here.

Thanks,
Nitish

On Tue, Jan 7, 2020 at 5:37 PM Nitish Saboo <nitish.saboo55 at gmail.com>
wrote:

> Hi Peter,
>
> Thanks for your response :)
> I am seeing the memory usage getting increased ONLY at the time of reload
> of patterndb.
> All other operation are having constant memory consumption.
> Hence, my question is specific to reload operation if it is by default
> that patterndb component is not freeing up the memory on reload operation ?
> If it is by default, why is it so ?
>
>
> *The init and deinit can be repeated multiple times (init, deinit, init,
> deinit, ...) also usually the repeation of single init is fine (init, init,
> ...).*
> >> With this you meant initialising syslog-ng  every-time for every reload
> operation?
>
> Thanks*,*
> Nitish
>
> On Tue, Jan 7, 2020 at 5:05 PM Peter Kokai (pkokai) <
> Peter.Kokai at oneidentity.com> wrote:
>
>> Hello,
>>
>> Yes, we like to keep memory for ourselfs ;)[1]
>>
>> This is in fact not syslog-ng reload, and mostly about patterndb
>> component. You could verify leak with valgrind or sanitizer. Please if you
>> find a leak check it on upstream and report if you have any.
>>
>> Regarding the question that your application does leak or not is hard to
>> tell withouth knowing it.
>> Please share your application and we might be able to help you.
>>
>> Just a hint: things in syslog-ng usually like to be called like this:
>>
>> new
>> init
>> deinit
>> free
>>
>> In case of reference counted object also use unref instead of free, and
>> ref well to reference stuff.
>>
>> The init and deinit can be repeated multiple times (init, deinit, init,
>> deinit, ...) also usually the repeation of single init is fine (init, init,
>> ...).
>> (This is of course not a contract by any means.)
>>
>> Have fun.
>>
>> [1] Not really true.
>>
>> --
>> Kokan
>>
>> On Tue, Jan 07, 2020 at 02:35:43PM +0530, Nitish Saboo wrote:
>> > CAUTION: This email originated from outside of the organization. Do not
>> follow guidance, click links, or open attachments unless you recognize the
>> sender and know the content is safe.
>> >
>> > Hi,
>> >
>> > I am using syslog-ng version 3.6.2:
>> >
>> > syslog-ng --version
>> > syslog-ng 3.6.2
>> > Installer-Version: 3.6.2
>> > Revision:
>> > Compile-Date: May  2 2019 01:20:32
>> > Available-Modules:
>> afmongodb,afstomp,afuser,csvparser,afprog,afamqp,cryptofuncs,afsocket,graphite,dbparser,affile,syslogformat,afsocket-notls,system-source,confgen,pseudofile,linux-kmsg-format,basicfuncs
>> > Enable-Debug: off
>> > Enable-GProf: off
>> > Enable-Memtrace: off
>> > Enable-IPv6: on
>> > Enable-Spoof-Source: off
>> > Enable-TCP-Wrapper: off
>> > Enable-Linux-Caps: off
>> >
>> >
>> > I initialised syslog-ng with the following configuration:
>> >
>> > int initialize_engine(const gchar* filename, const gchar* _module_path)
>> > {
>> >   module_path = _module_path;
>> >   app_startup();
>> >   msg_init(TRUE);
>> >
>> >   configuration = cfg_new(0x0302);
>> >   plugin_load_module("basicfuncs", configuration, NULL);
>> >   plugin_load_module("syslogformat", configuration, NULL);
>> >
>> >   pattern_db_global_init();
>> >   return 0;
>> >   }
>> >
>> >   int reload_pattern_db(const gchar* filename, key_value_cb cb)
>> > {
>> >   if(patterndb!= NULL){
>> >     pattern_db_free(patterndb);
>> >   }
>> >   patterndb = pattern_db_new();
>> >   pattern_db_reload_ruleset(patterndb, configuration, filename);
>> >   pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
>> >   return 0;
>> > }
>> >
>> > When I reload the syslog-ng, the memory keeps on increasing.Looks like
>> syslog-ng is not freeing up the memory.It continues to occupy the memory
>> unless the process is restarted.
>> > Just want to understand if it is a sign of memory leak or default
>> behavior of syslog-ng to not free-up memory during reload ?
>> >
>> > Thanks,
>> > Nitish
>> >
>>
>> >
>> ______________________________________________________________________________
>> > Member info:
>> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7C787bcb55bf024c043c6308d79350d004%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637139847625536021&sdata=siZ9%2BKokl%2FvxLmhWsuoL4PZcmzGFKk7PNW3vhfEQMRg%3D&reserved=0
>> > Documentation:
>> https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7C787bcb55bf024c043c6308d79350d004%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637139847625536021&sdata=%2FdBMM4ZmjzNQI2NYt8T3s3WXx1XTd7QtGmVjEvLDBA0%3D&reserved=0
>> > FAQ:
>> https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7C787bcb55bf024c043c6308d79350d004%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637139847625536021&sdata=JcZw66v9GCGWFly1RKTeCdgVaHzOaL1oXenDCQIpLqs%3D&reserved=0
>> >
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20200108/fdf44319/attachment.html>