<div dir="ltr">Hi Kokan,<div><br></div><div>>Please first of all correct me if I am wrong. But my assumption is that you build an application using one of the library in syslog-ng that provides the patterndb functionality.</div><div><br></div><div>Yes, I have built a service with some of the libraries in syslog-ng that provides the patterndb functionality.I am using the following method to initialise syslog-ng and patterndb:<br><br><span style="color:rgb(36,41,46);font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:12px;white-space:pre">PatternDB *patterndb;</span></div><div><font color="#24292e" face="SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier, monospace"><span style="font-size:12px;white-space:pre"><br></span></font>int initialize_engine(const gchar* filename, const gchar* _module_path)<br>{<br> module_path = _module_path;<br> app_startup();<br> msg_init(TRUE);<br><br> configuration = cfg_new(0x0302);<br> plugin_load_module("basicfuncs", configuration, NULL);<br> plugin_load_module("syslogformat", configuration, NULL);<br><br> pattern_db_global_init();<br> return 0;<br> }<br></div><div><br></div><div>>In case of syslog-ng the LogDBParser own's a PatternDB. In case of reload the LogDBParser are destroyed (deinit, free) and the PatternDB is stored in a kinda map.<br>When the new LogDBParser is created, it looks up that map if there is a PatternDB in it and if there is it uses the PatternDB (which holds the context and messages).<br></div><div><br></div><div>I am reloading the patterndb using the following method, where I am initially freeing up the existing patterndb instance using 'pattern_db_free' method.Will this free-up the context and messages ?</div><div>Is this the correct way to free up the existing patterndb instance ?<br><br>int reload_pattern_db(const gchar* filename, key_value_cb cb)<br>{<br> if(patterndb!= NULL){<br> pattern_db_free(patterndb);<br> }<br> patterndb = pattern_db_new();<br> pattern_db_reload_ruleset(patterndb, configuration, filename);<br> pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);<br> return 0;<br>}<br><br>Let me clarify my issue to you and hope then you will be able to guide me correctly.<br><br><b>ISSUE:</b></div><div><b><br></b>1) Suppose the patterndb.xml file size is 10MB. When I start the service the memory usage shows 2% for 10MB file size.<br><br>2)With every reload of patterndb(I am increasing the file size of patterndb.xml for example from 10MB to 11MB), the memory usage keeps on increasing like 2% to 3%.The increase in memory usage is seen <b>ONLY</b> during reload and not while parsing<b></b>.This process is repeated and now the file size is 20 MB and memory usage has shot up to say 8%.<br><br>3)But now when I restart the service, the memory usage for the same patterndb.xml file of size 20MB shows only 4%.<br><br>So my concern is why the memory usage is increasing with each reload of patterndb.xml file but when I restart the service the same size of patterndb.xml shows less memory usage.</div><div>Is this because the message and context are not getting deleted?</div><div><br></div><div>Thanks,</div><div>Nitish<br><br><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jan 8, 2020 at 4:28 PM Peter Kokai (pkokai) <<a href="mailto:Peter.Kokai@oneidentity.com">Peter.Kokai@oneidentity.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello,<br>
<br>
Please first of all correct me if I am wrong. But my assumption is that you build an application using one of the library in syslog-ng that provides the patterndb functionality.<br>
<br>
In that case reading the syslog-ng administrators guide is not the most useful thing to read, well there is no real development documentation focusing on patterndb.<br>
<br>
<br>
> 1) Is this because of the message contexts that are persisted on reload, the memory is not freeing up at the time of reload of patterndb ?<br>
<br>
In case of syslog-ng the LogDBParser own's a PatternDB. In case of reload the LogDBParser are destroyed (deinit, free) and the PatternDB is stored in a kinda map.<br>
When the new LogDBParser is created, it looks up that map if there is a PatternDB in it and if there is it uses the PatternDB (which holds the context and messages).<br>
<br>
> 2) Is there a way we can free-up these message contexts without a restart ?<br>
<br>
Probably there is. A proper free of PatternDB object should suffiece. Try that and if not, check out what is leaked.<br>
Please not that we do not update old releases like your 3.6.2 version. Even if you found a valid leak, it won't be fixed by us.<br>
<br>
On the other hand if that same issue applies to upstream, that is going to be solved sooner or later on upstream so reporting that might be beneficial.<br>
<br>
<br>
--<br>
Kokan<br>
<br>
On Wed, Jan 08, 2020 at 11:11:20AM +0530, Nitish Saboo wrote:<br>
> CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.<br>
> <br>
> Hi,<br>
> <br>
> I was going through syslog-ng administrators guide and came across the following point from the given link:<br>
> <br>
> "NOTE:Message contexts are persistent and are not lost when syslog-ng OSE is reloaded (SIGHUP), but are lost when syslog-ng OSE is restarted."<br>
> <br>
> <a href="https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.syslog-ng.com%2Ftechnical-documents%2Fdoc%2Fsyslog-ng-open-source-edition%2F3.18%2Fadministration-guide%2F72%23TOPIC-1044162&data=02%7C01%7CPeter.Kokai%40oneidentity.com%7C98e50b9febb94766906408d793fd6cc9%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140588987560053&sdata=TYRUPe088j8cP%2Br8e034XO9J46yY%2FMtlbcEeRisuyno%3D&reserved=0" rel="noreferrer" target="_blank">https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.syslog-ng.com%2Ftechnical-documents%2Fdoc%2Fsyslog-ng-open-source-edition%2F3.18%2Fadministration-guide%2F72%23TOPIC-1044162&data=02%7C01%7CPeter.Kokai%40oneidentity.com%7C98e50b9febb94766906408d793fd6cc9%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140588987560053&sdata=TYRUPe088j8cP%2Br8e034XO9J46yY%2FMtlbcEeRisuyno%3D&reserved=0</a><br>
> <br>
> 1) Is this because of the message contexts that are persisted on reload, the memory is not freeing up at the time of reload of patterndb ?<br>
> <br>
> 2) Is there a way we can free-up these message contexts without a restart ?<br>
> <br>
> Please correct me if I am wrong or missing something here.<br>
> <br>
> Thanks,<br>
> Nitish<br>
> <br>
> <br>
> <br>
> On Tue, Jan 7, 2020 at 5:37 PM Nitish Saboo <<a href="mailto:nitish.saboo55@gmail.com" target="_blank">nitish.saboo55@gmail.com</a><mailto:<a href="mailto:nitish.saboo55@gmail.com" target="_blank">nitish.saboo55@gmail.com</a>>> wrote:<br>
> Hi Peter,<br>
> <br>
> Thanks for your response :)<br>
> I am seeing the memory usage getting increased ONLY at the time of reload of patterndb.<br>
> All other operation are having constant memory consumption.<br>
> Hence, my question is specific to reload operation if it is by default that patterndb component is not freeing up the memory on reload operation ? If it is by default, why is it so ?<br>
> <br>
> The init and deinit can be repeated multiple times (init, deinit, init, deinit, ...) also usually the repeation of single init is fine (init, init, ...).<br>
> >> With this you meant initialising syslog-ng every-time for every reload operation?<br>
> <br>
> Thanks,<br>
> Nitish<br>
> <br>
> On Tue, Jan 7, 2020 at 5:05 PM Peter Kokai (pkokai) <<a href="mailto:Peter.Kokai@oneidentity.com" target="_blank">Peter.Kokai@oneidentity.com</a><mailto:<a href="mailto:Peter.Kokai@oneidentity.com" target="_blank">Peter.Kokai@oneidentity.com</a>>> wrote:<br>
> Hello,<br>
> <br>
> Yes, we like to keep memory for ourselfs ;)[1]<br>
> <br>
> This is in fact not syslog-ng reload, and mostly about patterndb component. You could verify leak with valgrind or sanitizer. Please if you find a leak check it on upstream and report if you have any.<br>
> <br>
> Regarding the question that your application does leak or not is hard to tell withouth knowing it.<br>
> Please share your application and we might be able to help you.<br>
> <br>
> Just a hint: things in syslog-ng usually like to be called like this:<br>
> <br>
> new<br>
> init<br>
> deinit<br>
> free<br>
> <br>
> In case of reference counted object also use unref instead of free, and ref well to reference stuff.<br>
> <br>
> The init and deinit can be repeated multiple times (init, deinit, init, deinit, ...) also usually the repeation of single init is fine (init, init, ...).<br>
> (This is of course not a contract by any means.)<br>
> <br>
> Have fun.<br>
> <br>
> [1] Not really true.<br>
> <br>
> --<br>
> Kokan<br>
> <br>
> On Tue, Jan 07, 2020 at 02:35:43PM +0530, Nitish Saboo wrote:<br>
> > CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.<br>
> ><br>
> > Hi,<br>
> ><br>
> > I am using syslog-ng version 3.6.2:<br>
> ><br>
> > syslog-ng --version<br>
> > syslog-ng 3.6.2<br>
> > Installer-Version: 3.6.2<br>
> > Revision:<br>
> > Compile-Date: May 2 2019 01:20:32<br>
> > Available-Modules: afmongodb,afstomp,afuser,csvparser,afprog,afamqp,cryptofuncs,afsocket,graphite,dbparser,affile,syslogformat,afsocket-notls,system-source,confgen,pseudofile,linux-kmsg-format,basicfuncs<br>
> > Enable-Debug: off<br>
> > Enable-GProf: off<br>
> > Enable-Memtrace: off<br>
> > Enable-IPv6: on<br>
> > Enable-Spoof-Source: off<br>
> > Enable-TCP-Wrapper: off<br>
> > Enable-Linux-Caps: off<br>
> ><br>
> ><br>
> > I initialised syslog-ng with the following configuration:<br>
> ><br>
> > int initialize_engine(const gchar* filename, const gchar* _module_path)<br>
> > {<br>
> > module_path = _module_path;<br>
> > app_startup();<br>
> > msg_init(TRUE);<br>
> ><br>
> > configuration = cfg_new(0x0302);<br>
> > plugin_load_module("basicfuncs", configuration, NULL);<br>
> > plugin_load_module("syslogformat", configuration, NULL);<br>
> ><br>
> > pattern_db_global_init();<br>
> > return 0;<br>
> > }<br>
> ><br>
> > int reload_pattern_db(const gchar* filename, key_value_cb cb)<br>
> > {<br>
> > if(patterndb!= NULL){<br>
> > pattern_db_free(patterndb);<br>
> > }<br>
> > patterndb = pattern_db_new();<br>
> > pattern_db_reload_ruleset(patterndb, configuration, filename);<br>
> > pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);<br>
> > return 0;<br>
> > }<br>
> ><br>
> > When I reload the syslog-ng, the memory keeps on increasing.Looks like syslog-ng is not freeing up the memory.It continues to occupy the memory unless the process is restarted.<br>
> > Just want to understand if it is a sign of memory leak or default behavior of syslog-ng to not free-up memory during reload ?<br>
> ><br>
> > Thanks,<br>
> > Nitish<br>
> ><br>
> <br>
> > ______________________________________________________________________________<br>
> > Member info: <a href="https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7CPeter.Kokai%40oneidentity.com%7C98e50b9febb94766906408d793fd6cc9%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140588987560053&sdata=80UrMalZmN5IOI8fIcul18oeWhcSmGC5Qm3sOATy%2BGo%3D&reserved=0" rel="noreferrer" target="_blank">https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7CPeter.Kokai%40oneidentity.com%7C98e50b9febb94766906408d793fd6cc9%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140588987560053&sdata=80UrMalZmN5IOI8fIcul18oeWhcSmGC5Qm3sOATy%2BGo%3D&reserved=0</a><br>
> > Documentation: <a href="https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7CPeter.Kokai%40oneidentity.com%7C98e50b9febb94766906408d793fd6cc9%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140588987560053&sdata=h2VSO0%2FE6Jya03KEkZHW6EHtZ7VxdWBaO8677wvc73Q%3D&reserved=0" rel="noreferrer" target="_blank">https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7CPeter.Kokai%40oneidentity.com%7C98e50b9febb94766906408d793fd6cc9%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140588987560053&sdata=h2VSO0%2FE6Jya03KEkZHW6EHtZ7VxdWBaO8677wvc73Q%3D&reserved=0</a><br>
> > FAQ: <a href="https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7CPeter.Kokai%40oneidentity.com%7C98e50b9febb94766906408d793fd6cc9%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140588987560053&sdata=RiKxeKjeug6xqe6rbgjM182HcRY8s%2FfEwXOoRB1hK0I%3D&reserved=0" rel="noreferrer" target="_blank">https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7CPeter.Kokai%40oneidentity.com%7C98e50b9febb94766906408d793fd6cc9%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140588987560053&sdata=RiKxeKjeug6xqe6rbgjM182HcRY8s%2FfEwXOoRB1hK0I%3D&reserved=0</a><br>
> ><br>
> ______________________________________________________________________________<br>
> Member info: <a href="https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7CPeter.Kokai%40oneidentity.com%7C98e50b9febb94766906408d793fd6cc9%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140588987570048&sdata=Z3g3ZFjOX96kzak8gx87cjSW6on2lLquFk2ZhR7ND%2FE%3D&reserved=0" rel="noreferrer" target="_blank">https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7CPeter.Kokai%40oneidentity.com%7C98e50b9febb94766906408d793fd6cc9%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140588987570048&sdata=Z3g3ZFjOX96kzak8gx87cjSW6on2lLquFk2ZhR7ND%2FE%3D&reserved=0</a><br>
> Documentation: <a href="https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7CPeter.Kokai%40oneidentity.com%7C98e50b9febb94766906408d793fd6cc9%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140588987570048&sdata=hzVDeYj%2BYUniHgbmzlmbIhucKeweGlR5%2BJc%2Bm70lFvY%3D&reserved=0" rel="noreferrer" target="_blank">https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7CPeter.Kokai%40oneidentity.com%7C98e50b9febb94766906408d793fd6cc9%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140588987570048&sdata=hzVDeYj%2BYUniHgbmzlmbIhucKeweGlR5%2BJc%2Bm70lFvY%3D&reserved=0</a><br>
> FAQ: <a href="https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7CPeter.Kokai%40oneidentity.com%7C98e50b9febb94766906408d793fd6cc9%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140588987570048&sdata=wakZRruKQpuezhA1fU8LPnL5SOpin6bvHB2O5mbf8Ls%3D&reserved=0" rel="noreferrer" target="_blank">https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7CPeter.Kokai%40oneidentity.com%7C98e50b9febb94766906408d793fd6cc9%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140588987570048&sdata=wakZRruKQpuezhA1fU8LPnL5SOpin6bvHB2O5mbf8Ls%3D&reserved=0</a><br>
> <br>
<br>
> ______________________________________________________________________________<br>
> Member info: <a href="https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7CPeter.Kokai%40oneidentity.com%7C98e50b9febb94766906408d793fd6cc9%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140588987700002&sdata=iY32Y9gzDT72OLmOWoZBlDrAmu5%2F5AqXF6mX3cEzfTU%3D&reserved=0" rel="noreferrer" target="_blank">https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7CPeter.Kokai%40oneidentity.com%7C98e50b9febb94766906408d793fd6cc9%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140588987700002&sdata=iY32Y9gzDT72OLmOWoZBlDrAmu5%2F5AqXF6mX3cEzfTU%3D&reserved=0</a><br>
> Documentation: <a href="https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7CPeter.Kokai%40oneidentity.com%7C98e50b9febb94766906408d793fd6cc9%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140588987700002&sdata=fVrlGkOS7wkLp%2FuJ8%2Fvvr%2BF223svv64NxyHabEpJS7w%3D&reserved=0" rel="noreferrer" target="_blank">https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7CPeter.Kokai%40oneidentity.com%7C98e50b9febb94766906408d793fd6cc9%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140588987700002&sdata=fVrlGkOS7wkLp%2FuJ8%2Fvvr%2BF223svv64NxyHabEpJS7w%3D&reserved=0</a><br>
> FAQ: <a href="https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7CPeter.Kokai%40oneidentity.com%7C98e50b9febb94766906408d793fd6cc9%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140588987700002&sdata=GGqYNSsa%2Bnw6fy5EROhlJbzkFaCOI7EpUiuCRFFX%2FOY%3D&reserved=0" rel="noreferrer" target="_blank">https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7CPeter.Kokai%40oneidentity.com%7C98e50b9febb94766906408d793fd6cc9%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637140588987700002&sdata=GGqYNSsa%2Bnw6fy5EROhlJbzkFaCOI7EpUiuCRFFX%2FOY%3D&reserved=0</a><br>
> <br>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div>