[syslog-ng] Memory not freeing up in syslog-ng-3.6.2 during reload
Nitish Saboo
nitish.saboo55 at gmail.com
Thu Jan 9 09:42:03 UTC 2020
Hi Kokan,
>You should destroy, free patterndb via this function. This destroy
context, and unreference LogMessage it holds a reference to.
>From the following link, I am using the same 'pattern_db_free' function.
https://github.com/syslog-ng/syslog-ng/blob/c220d1364ce338725097dabd171819660530018f/modules/dbparser/patterndb.c#L806-L815
As you said 'pattern_db_free' function first destroys and then free
patterndb, it also destroys context... make sense.
>As LogMessages are reference counted ref/unref, it is possible that
somebody still holds a reference to your LogMessages.
void match(const gchar *pattern, size_t len, const gchar *program, size_t
p_len)
{
LogMessage *msg = log_msg_new_empty();
PDBInput in;
log_msg_set_value(msg, LM_V_MESSAGE, pattern, len);
log_msg_set_value(msg, LM_V_PROGRAM, program, p_len);
pattern_db_process(patterndb, PDB_INPUT_WRAP_MESSAGE(&in, msg));
log_msg_unref(msg);
}
I am removing the reference to the the LogMessage via this function after
parsing each one of them.It doesn't look like I am keeping any reference to
the LogMessage atleast from the C code.
I am calling these C methods from Go.
Any other thought that comes to your mind that points to increase in memory
usage ONLY at the time of reload of patterndb ?
>Valgrind is your friend, use it :)
Can we use valgrind for Go and C combination (CGO) ?
Thanks,
Nitish
On Thu, Jan 9, 2020 at 12:17 PM Peter Kokai (pkokai) <
Peter.Kokai at oneidentity.com> wrote:
> Hello,
>
> > Is this the correct way to free up the existing patterndb instance ?
> > pattern_db_free(patterndb);
> You should destroy, free patterndb via this function. This destroy
> context, and unreference LogMessage it holds a reference to.
> As LogMessages are reference counted ref/unref, it is possible that
> somebody still holds a reference to your LogMessages.
>
> > Is this because the message and context are not getting deleted?
> I do not think so.
>
> Valgrind is your friend, use it :)
>
> --
> Kokan
>
> On Wed, Jan 08, 2020 at 05:36:10PM +0530, Nitish Saboo wrote:
> > CAUTION: This email originated from outside of the organization. Do not
> follow guidance, click links, or open attachments unless you recognize the
> sender and know the content is safe.
> >
> > Hi Kokan,
> >
> > >Please first of all correct me if I am wrong. But my assumption is that
> you build an application using one of the library in syslog-ng that
> provides the patterndb functionality.
> >
> > Yes, I have built a service with some of the libraries in syslog-ng that
> provides the patterndb functionality.I am using the following method to
> initialise syslog-ng and patterndb:
> >
> > PatternDB *patterndb;
> >
> > int initialize_engine(const gchar* filename, const gchar* _module_path)
> > {
> > module_path = _module_path;
> > app_startup();
> > msg_init(TRUE);
> >
> > configuration = cfg_new(0x0302);
> > plugin_load_module("basicfuncs", configuration, NULL);
> > plugin_load_module("syslogformat", configuration, NULL);
> >
> > pattern_db_global_init();
> > return 0;
> > }
> >
> > >In case of syslog-ng the LogDBParser own's a PatternDB. In case of
> reload the LogDBParser are destroyed (deinit, free) and the PatternDB is
> stored in a kinda map.
> > When the new LogDBParser is created, it looks up that map if there is a
> PatternDB in it and if there is it uses the PatternDB (which holds the
> context and messages).
> >
> > I am reloading the patterndb using the following method, where I am
> initially freeing up the existing patterndb instance using
> 'pattern_db_free' method.Will this free-up the context and messages ?
> > Is this the correct way to free up the existing patterndb instance ?
> >
> > int reload_pattern_db(const gchar* filename, key_value_cb cb)
> > {
> > if(patterndb!= NULL){
> > pattern_db_free(patterndb);
> > }
> > patterndb = pattern_db_new();
> > pattern_db_reload_ruleset(patterndb, configuration, filename);
> > pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
> > return 0;
> > }
> >
> > Let me clarify my issue to you and hope then you will be able to guide
> me correctly.
> >
> > ISSUE:
> >
> > 1) Suppose the patterndb.xml file size is 10MB. When I start the service
> the memory usage shows 2% for 10MB file size.
> >
> > 2)With every reload of patterndb(I am increasing the file size of
> patterndb.xml for example from 10MB to 11MB), the memory usage keeps on
> increasing like 2% to 3%.The increase in memory usage is seen ONLY during
> reload and not while parsing.This process is repeated and now the file size
> is 20 MB and memory usage has shot up to say 8%.
> >
> > 3)But now when I restart the service, the memory usage for the same
> patterndb.xml file of size 20MB shows only 4%.
> >
> > So my concern is why the memory usage is increasing with each reload of
> patterndb.xml file but when I restart the service the same size of
> patterndb.xml shows less memory usage.
> > Is this because the message and context are not getting deleted?
> >
> > Thanks,
> > Nitish
> >
> >
> >
> >
> > On Wed, Jan 8, 2020 at 4:28 PM Peter Kokai (pkokai) <
> Peter.Kokai at oneidentity.com<mailto:Peter.Kokai at oneidentity.com>> wrote:
> > Hello,
> >
> > Please first of all correct me if I am wrong. But my assumption is that
> you build an application using one of the library in syslog-ng that
> provides the patterndb functionality.
> >
> > In that case reading the syslog-ng administrators guide is not the most
> useful thing to read, well there is no real development documentation
> focusing on patterndb.
> >
> >
> > > 1) Is this because of the message contexts that are persisted on
> reload, the memory is not freeing up at the time of reload of patterndb ?
> >
> > In case of syslog-ng the LogDBParser own's a PatternDB. In case of
> reload the LogDBParser are destroyed (deinit, free) and the PatternDB is
> stored in a kinda map.
> > When the new LogDBParser is created, it looks up that map if there is a
> PatternDB in it and if there is it uses the PatternDB (which holds the
> context and messages).
> >
> > > 2) Is there a way we can free-up these message contexts without a
> restart ?
> >
> > Probably there is. A proper free of PatternDB object should suffiece.
> Try that and if not, check out what is leaked.
> > Please not that we do not update old releases like your 3.6.2 version.
> Even if you found a valid leak, it won't be fixed by us.
> >
> > On the other hand if that same issue applies to upstream, that is going
> to be solved sooner or later on upstream so reporting that might be
> beneficial.
> >
> >
> > --
> > Kokan
> >
> > On Wed, Jan 08, 2020 at 11:11:20AM +0530, Nitish Saboo wrote:
> > > CAUTION: This email originated from outside of the organization. Do
> not follow guidance, click links, or open attachments unless you recognize
> the sender and know the content is safe.
> > >
> > > Hi,
> > >
> > > I was going through syslog-ng administrators guide and came across the
> following point from the given link:
> > >
> > > "NOTE:Message contexts are persistent and are not lost when syslog-ng
> OSE is reloaded (SIGHUP), but are lost when syslog-ng OSE is restarted."
> > >
> > >
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.syslog-ng.com%2Ftechnical-documents%2Fdoc%2Fsyslog-ng-open-source-edition%2F3.18%2Fadministration-guide%2F72%23TOPIC-1044162&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7C051c8d91f1c047f1b4b308d794332fde%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140819902855703&sdata=DshVLk37ucUfoCPvLWfHxCAUqN%2B1VW%2BsLSK1kAIT4D0%3D&reserved=0
> > >
> > > 1) Is this because of the message contexts that are persisted on
> reload, the memory is not freeing up at the time of reload of patterndb ?
> > >
> > > 2) Is there a way we can free-up these message contexts without a
> restart ?
> > >
> > > Please correct me if I am wrong or missing something here.
> > >
> > > Thanks,
> > > Nitish
> > >
> > >
> > >
> > > On Tue, Jan 7, 2020 at 5:37 PM Nitish Saboo <nitish.saboo55 at gmail.com
> <mailto:nitish.saboo55 at gmail.com><mailto:nitish.saboo55 at gmail.com<mailto:
> nitish.saboo55 at gmail.com>>> wrote:
> > > Hi Peter,
> > >
> > > Thanks for your response :)
> > > I am seeing the memory usage getting increased ONLY at the time of
> reload of patterndb.
> > > All other operation are having constant memory consumption.
> > > Hence, my question is specific to reload operation if it is by default
> that patterndb component is not freeing up the memory on reload operation ?
> If it is by default, why is it so ?
> > >
> > > The init and deinit can be repeated multiple times (init, deinit,
> init, deinit, ...) also usually the repeation of single init is fine (init,
> init, ...).
> > > >> With this you meant initialising syslog-ng every-time for every
> reload operation?
> > >
> > > Thanks,
> > > Nitish
> > >
> > > On Tue, Jan 7, 2020 at 5:05 PM Peter Kokai (pkokai) <
> Peter.Kokai at oneidentity.com<mailto:Peter.Kokai at oneidentity.com><mailto:
> Peter.Kokai at oneidentity.com<mailto:Peter.Kokai at oneidentity.com>>> wrote:
> > > Hello,
> > >
> > > Yes, we like to keep memory for ourselfs ;)[1]
> > >
> > > This is in fact not syslog-ng reload, and mostly about patterndb
> component. You could verify leak with valgrind or sanitizer. Please if you
> find a leak check it on upstream and report if you have any.
> > >
> > > Regarding the question that your application does leak or not is hard
> to tell withouth knowing it.
> > > Please share your application and we might be able to help you.
> > >
> > > Just a hint: things in syslog-ng usually like to be called like this:
> > >
> > > new
> > > init
> > > deinit
> > > free
> > >
> > > In case of reference counted object also use unref instead of free,
> and ref well to reference stuff.
> > >
> > > The init and deinit can be repeated multiple times (init, deinit,
> init, deinit, ...) also usually the repeation of single init is fine (init,
> init, ...).
> > > (This is of course not a contract by any means.)
> > >
> > > Have fun.
> > >
> > > [1] Not really true.
> > >
> > > --
> > > Kokan
> > >
> > > On Tue, Jan 07, 2020 at 02:35:43PM +0530, Nitish Saboo wrote:
> > > > CAUTION: This email originated from outside of the organization. Do
> not follow guidance, click links, or open attachments unless you recognize
> the sender and know the content is safe.
> > > >
> > > > Hi,
> > > >
> > > > I am using syslog-ng version 3.6.2:
> > > >
> > > > syslog-ng --version
> > > > syslog-ng 3.6.2
> > > > Installer-Version: 3.6.2
> > > > Revision:
> > > > Compile-Date: May 2 2019 01:20:32
> > > > Available-Modules:
> afmongodb,afstomp,afuser,csvparser,afprog,afamqp,cryptofuncs,afsocket,graphite,dbparser,affile,syslogformat,afsocket-notls,system-source,confgen,pseudofile,linux-kmsg-format,basicfuncs
> > > > Enable-Debug: off
> > > > Enable-GProf: off
> > > > Enable-Memtrace: off
> > > > Enable-IPv6: on
> > > > Enable-Spoof-Source: off
> > > > Enable-TCP-Wrapper: off
> > > > Enable-Linux-Caps: off
> > > >
> > > >
> > > > I initialised syslog-ng with the following configuration:
> > > >
> > > > int initialize_engine(const gchar* filename, const gchar*
> _module_path)
> > > > {
> > > > module_path = _module_path;
> > > > app_startup();
> > > > msg_init(TRUE);
> > > >
> > > > configuration = cfg_new(0x0302);
> > > > plugin_load_module("basicfuncs", configuration, NULL);
> > > > plugin_load_module("syslogformat", configuration, NULL);
> > > >
> > > > pattern_db_global_init();
> > > > return 0;
> > > > }
> > > >
> > > > int reload_pattern_db(const gchar* filename, key_value_cb cb)
> > > > {
> > > > if(patterndb!= NULL){
> > > > pattern_db_free(patterndb);
> > > > }
> > > > patterndb = pattern_db_new();
> > > > pattern_db_reload_ruleset(patterndb, configuration, filename);
> > > > pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate,
> cb);
> > > > return 0;
> > > > }
> > > >
> > > > When I reload the syslog-ng, the memory keeps on increasing.Looks
> like syslog-ng is not freeing up the memory.It continues to occupy the
> memory unless the process is restarted.
> > > > Just want to understand if it is a sign of memory leak or default
> behavior of syslog-ng to not free-up memory during reload ?
> > > >
> > > > Thanks,
> > > > Nitish
> > > >
> > >
> > > >
> ______________________________________________________________________________
> > > > Member info:
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7C051c8d91f1c047f1b4b308d794332fde%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140819902855703&sdata=E468zeYrQ%2FRwxlzzH%2BsDxjUlYRnlXaQyfLli%2FVgo0z8%3D&reserved=0
> > > > Documentation:
> https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7C051c8d91f1c047f1b4b308d794332fde%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140819902855703&sdata=mJTGCZzU0YgzT3S58PxVMQAqtcsEA8DXn2yrBejOb%2F4%3D&reserved=0
> > > > FAQ:
> https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7C051c8d91f1c047f1b4b308d794332fde%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140819902855703&sdata=TSF%2FrgjN3%2BpAirEjrDWiPZWDplyWc1VsWd7giZd1urs%3D&reserved=0
> > > >
> > >
> ______________________________________________________________________________
> > > Member info:
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7C051c8d91f1c047f1b4b308d794332fde%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140819902855703&sdata=E468zeYrQ%2FRwxlzzH%2BsDxjUlYRnlXaQyfLli%2FVgo0z8%3D&reserved=0
> > > Documentation:
> https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7C051c8d91f1c047f1b4b308d794332fde%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140819902855703&sdata=mJTGCZzU0YgzT3S58PxVMQAqtcsEA8DXn2yrBejOb%2F4%3D&reserved=0
> > > FAQ:
> https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7C051c8d91f1c047f1b4b308d794332fde%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140819902865693&sdata=tJSBPD4kTASrCZWR8eh7oXbvh74hvvKwqpZDXhgvcmg%3D&reserved=0
> > >
> >
> > >
> ______________________________________________________________________________
> > > Member info:
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7C051c8d91f1c047f1b4b308d794332fde%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140819902865693&sdata=rAlYc8iMZ0pot5zjAYxvyReqLxgLyXHYCBOxQdY0NPI%3D&reserved=0
> > > Documentation:
> https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7C051c8d91f1c047f1b4b308d794332fde%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140819902865693&sdata=IKiN%2Bq1%2B2X17lsCKTKwf0FjnZCMZcJ7DvJsAwLgF3B4%3D&reserved=0
> > > FAQ:
> https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7C051c8d91f1c047f1b4b308d794332fde%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140819902865693&sdata=tJSBPD4kTASrCZWR8eh7oXbvh74hvvKwqpZDXhgvcmg%3D&reserved=0
> > >
> >
> ______________________________________________________________________________
> > Member info:
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7C051c8d91f1c047f1b4b308d794332fde%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140819902865693&sdata=rAlYc8iMZ0pot5zjAYxvyReqLxgLyXHYCBOxQdY0NPI%3D&reserved=0
> > Documentation:
> https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7C051c8d91f1c047f1b4b308d794332fde%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140819902865693&sdata=IKiN%2Bq1%2B2X17lsCKTKwf0FjnZCMZcJ7DvJsAwLgF3B4%3D&reserved=0
> > FAQ:
> https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7C051c8d91f1c047f1b4b308d794332fde%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140819902865693&sdata=tJSBPD4kTASrCZWR8eh7oXbvh74hvvKwqpZDXhgvcmg%3D&reserved=0
> >
>
> >
> ______________________________________________________________________________
> > Member info:
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7C051c8d91f1c047f1b4b308d794332fde%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140819903045587&sdata=zAJJe9loFIQcHGvgaNcbAaI6gfCMEQlLyMfqTINBn2k%3D&reserved=0
> > Documentation:
> https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7C051c8d91f1c047f1b4b308d794332fde%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140819903045587&sdata=Y3%2Fzr0%2BT5E8DOTAod1Mgvqzkg2EeXoiw2c4DTh9bezQ%3D&reserved=0
> > FAQ:
> https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Cpeter.kokai%40oneidentity.com%7C051c8d91f1c047f1b4b308d794332fde%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140819903045587&sdata=ymD0rmBGQZgKhcB0hpfbfBTasSF%2Fi23bHl6QoHCDg98%3D&reserved=0
> >
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20200109/ee8e5e3b/attachment-0001.html>
More information about the syslog-ng
mailing list