<div dir="ltr">Hi Michal!<br><br>Thanks for sharing logs and details.<div>You said that "If I disable the Bro via AMQP the problem goes away, so that must be something there ".</div><div>Do you mean that Bro is not sending any logs to the files or something else?</div><div><br></div><div>I have experimented with your config and with another one as well in which I removed the json-parser and json-format template function.</div><div>Both cases I have seen an increased memory usage even after the queue was empty, but I have not seen a per-message based increasing memory leak.</div><div>Of course when the queue is used it was an even higher memory footprint, but it dropped back when the queue emptied.</div><div><br></div><div>Valgrind logs only showed traces from startup, not during message sending.</div><div><br></div><div>For testing I have used the <b>loggen</b> tool from our repository and the log examples that you have shared:</div><div>bin/loggen -r 6000 -I 10 --active-connections=5 -iSd -R mozilla/conn.log --file-loop-reading localhost 4444<br></div><div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">I have sent the messages into a network() source with 6000 msg/s msg rate through 5 connections for 10 seconds.</div></div><div><br></div><div>Here are some statistics about the tests:</div><div>- after start syslog-ng used around 8k (RES)</div><div>- sending the first pack of messages (~300000 messages) memory usage is 623M (RES) even after the queue emptied!</div><div><div> dst.amqp;d_amqp#0;amqp,/,localhost,5672,eventtask,direct;a;processed;305735</div><div> dst.amqp;d_amqp#0;amqp,/,localhost,5672,eventtask,direct;a;queued;0 </div></div><div>- 2. iteration: processed;611243, 653M usage (the queue is used during receiving messages from network and memory usage topped at 900M)</div><div>- 3. iteration: 658M</div><div>- 4. iteration: processed;1222515, 659M</div><div>- 5. iteration: doubled the message sending interval (20s)</div><div>the memory usage increased to 1212M RES ! </div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">It seems amqp() driver does not free the used memory buffer after the queue emptied.</span><br class="gmail-Apple-interchange-newline"><br></div><div>You asked previously whether the increased memory usage is due to unprocessed messages.</div><div>Based on the statistics you shared, the 'queued' counter was 0. And after the queue is emptied the memory should be freed.</div><div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">I still have a couple of ideas and will investigate further, just wanted to share some information in the mean time.<br></div></div><div><br></div><div>Gabor</div><div><br></div><div><div class="gmail_extra"><div class="gmail_quote">On Wed, Mar 21, 2018 at 7:16 PM, Michal Purzynski <span dir="ltr"><<a href="mailto:michal@mozilla.com" target="_blank">michal@mozilla.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>The CPU usage is now between 40-60%, usually somewhere around 45% so it went down indeed. It spikes every few seconds to 55% then goes down, etc.</div><div><br></div><div>What worries me is that we're already at 21GB of RES so the memory usage grows and that's all syslog-ng's private data.</div><div><br></div><div>Are those messages that aren't processed? TBH I believe all messages are getting to Rabbit on time, I can see what I'm expecting in ES that's pulling from that Rabbit.</div><div><br></div></div><div class="gmail-HOEnZb"><div class="gmail-h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 20, 2018 at 10:49 PM, Michal Purzynski <span dir="ltr"><<a href="mailto:michal@mozilla.com" target="_blank">michal@mozilla.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">cat /proc/`pidof -s syslog-ng`/maps | egrep -i libjemalloc<br>7f0d97978000-7f0d979a8000 r-xp 00000000 08:02 25771415575 /usr/lib/x86_64-linux-gnu/libj<wbr>emalloc.so.1<br>7f0d979a8000-7f0d97ba8000 ---p 00030000 08:02 25771415575 /usr/lib/x86_64-linux-gnu/libj<wbr>emalloc.so.1<br>7f0d97ba8000-7f0d97baa000 r--p 00030000 08:02 25771415575 /usr/lib/x86_64-linux-gnu/libj<wbr>emalloc.so.1<br>7f0d97baa000-7f0d97bab000 rw-p 00032000 08:02 25771415575 /usr/lib/x86_64-linux-gnu/libj<wbr>emalloc.so.1<br><div><br></div><div><br></div><div>jemalloc loaded. Let's see how that helps. I also have tcmalloc up my sleeve.<br></div></div><div class="gmail-m_-8063925455884992131HOEnZb"><div class="gmail-m_-8063925455884992131h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 20, 2018 at 9:06 PM, Scheidler, Balázs <span dir="ltr"><<a href="mailto:balazs.scheidler@balabit.com" target="_blank">balazs.scheidler@balabit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div dir="ltr">Probably the biggest toll is the json-parser(), which would allocate memory when we are parsing the json. On the output side we have a handcrafted json generator that doesnt allocate memory, but on the inbound side libjson-c is doing it.<br></div><div dir="ltr"><br></div><div dir="ltr">That's probably the reason behind the futex numbers, malloc() uses a number of mutexes, which might get contended.</div><div dir="ltr"><br></div><div dir="ltr">Jemalloc would probably help a lot.</div></div><div class="gmail_extra"><span class="gmail-m_-8063925455884992131m_3395947134879140157HOEnZb"><font color="#888888"><br clear="all"><div><div class="gmail-m_-8063925455884992131m_3395947134879140157m_-3543377098694014331m_-1856218819312331434gmail_signature"><div dir="ltr">-- <br>Bazsi<br></div></div></div></font></span><div><div class="gmail-m_-8063925455884992131m_3395947134879140157h5">
<br><div class="gmail_quote">On Wed, Mar 21, 2018 at 2:51 AM, Michal Purzynski <span dir="ltr"><<a href="mailto:michal@mozilla.com" target="_blank">michal@mozilla.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">[ perf record: Captured and wrote 54.230 MB perf.data (1418800 samples) ]<br><div><br></div><div>Eyeballing looks like syslog-ng spends tons of time in malloc, allocating and deallocating memory. Maybe using gperf / jemalloc could help here?</div><div><br></div><div>Let me know if you want entire file. This is without call-graph, with call-graph... hmmm, looks like I will have to rebuild syslog-ng with symbols. Let's trace that leaks first ;-)<br></div><div><br></div><div> 16.77% syslog-ng [kernel.kallsyms] [k] update_blocked_averages<br> 16.37% syslog-ng <a href="http://libpthread-2.19.so" target="_blank">libpthread-2.19.so</a> [.] pthread_mutex_lock<br> 11.84% syslog-ng [kernel.kallsyms] [k] audit_filter_syscall<br> 8.09% syslog-ng [kernel.kallsyms] [k] copy_page<br> 7.50% syslog-ng [kernel.kallsyms] [k] syscall_return_via_sysret<br> 6.32% syslog-ng <a href="http://libc-2.19.so" target="_blank">libc-2.19.so</a> [.] _int_free<br> 5.13% syslog-ng libglib-2.0.so.0.4002.0 [.] 0x0000000000073487<br> 4.14% syslog-ng <a href="http://libc-2.19.so" target="_blank">libc-2.19.so</a> [.] _IO_vfscanf<br> 3.75% syslog-ng libivykis.so.0.5.4 [.] pthread_mutex_unlock@plt<br> 3.35% syslog-ng libsyslog-ng-3.14.so.0.0.0 [.] 0x0000000000026d02<br> 2.97% syslog-ng <a href="http://libc-2.19.so" target="_blank">libc-2.19.so</a> [.] _int_malloc<br> 2.96% syslog-ng libglib-2.0.so.0.4002.0 [.] 0x000000000008a617<br> 2.56% syslog-ng libglib-2.0.so.0.4002.0 [.] 0x000000000008a61a<br> 2.37% syslog-ng <a href="http://libc-2.19.so" target="_blank">libc-2.19.so</a> [.] __memmove_ssse3_back<br> 1.97% syslog-ng libjson-c.so.2.0.0 [.] lh_char_hash<br> 1.97% syslog-ng libsyslog-ng-3.14.so.0.0.0 [.] 0x0000000000036410<br> 0.32% syslog-ng <a href="http://libc-2.19.so" target="_blank">libc-2.19.so</a> [.] _IO_setb<br> 0.18% syslog-ng <a href="http://libc-2.19.so" target="_blank">libc-2.19.so</a> [.] malloc_consolidate<br> 0.17% syslog-ng <a href="http://libc-2.19.so" target="_blank">libc-2.19.so</a> [.] __strchrnul<br> 0.17% syslog-ng libglib-2.0.so.0.4002.0 [.] g_string_append_printf<br> 0.17% syslog-ng [kernel.kallsyms] [k] reschedule_interrupt<br> 0.17% syslog-ng libglib-2.0.so.0.4002.0 [.] 0x00000000000735a1<br> 0.17% syslog-ng libglib-2.0.so.0.4002.0 [.] 0x000000000008a8a6<br> 0.17% syslog-ng libsyslog-ng-3.14.so.0.0.0 [.] 0x000000000006df70<br> 0.15% syslog-ng <a href="http://libc-2.19.so" target="_blank">libc-2.19.so</a> [.] __memcpy_sse2_unaligned<br> 0.15% syslog-ng libglib-2.0.so.0.4002.0 [.] g_string_assign<br> 0.01% syslog-ng <a href="http://libc-2.19.so" target="_blank">libc-2.19.so</a> [.] malloc<br> 0.00% syslog-ng libjson-c.so.2.0.0 [.] json_tokener_parse_ex<br> 0.00% syslog-ng <a href="http://libc-2.19.so" target="_blank">libc-2.19.so</a> [.] vfprintf<br> 0.00% syslog-ng libglib-2.0.so.0.4002.0 [.] g_utf8_get_char_validated<br> 0.00% syslog-ng libglib-2.0.so.0.4002.0 [.] 0x000000000008a8a7<br> 0.00% syslog-ng <a href="http://libc-2.19.so" target="_blank">libc-2.19.so</a> [.] free<br> 0.00% syslog-ng <a href="http://libc-2.19.so" target="_blank">libc-2.19.so</a> [.] __strcmp_sse2_unaligned<br> 0.00% syslog-ng libglib-2.0.so.0.4002.0 [.] g_string_truncate</div><div><br></div><div><br></div><div><br></div><div><br></div><div>With cal graph, like I said, only the library name tell you something here, I can test later with syslog-ng with symbols</div><div><br></div><div><br></div><div>+ 59.22% 0.00% syslog-ng libivykis.so.0.5.4 [.] 0xffff80b3f3309c65 <wbr> <wbr> ▒<br>+ 56.85% 0.00% syslog-ng <a href="http://libc-2.19.so" target="_blank">libc-2.19.so</a> [.] 0xffff80b3f259f6d3 <wbr> <wbr> ▒<br>+ 38.22% 0.00% syslog-ng [unknown] [.] 0x0000000000000029 <wbr> <wbr> ▒<br>+ 29.22% 29.22% syslog-ng <a href="http://libc-2.19.so" target="_blank">libc-2.19.so</a> [.] 0x00000000000fe6d3 <wbr> <wbr> ▒<br>+ 18.46% 0.00% syslog-ng [kernel.kallsyms] [k] entry_SYSCALL_64_fastpath <wbr> <wbr> ▒<br>+ 18.32% 0.00% syslog-ng [kernel.kallsyms] [k] sys_epoll_wait <wbr> <wbr> ▒<br>+ 13.27% 9.00% syslog-ng [kernel.kallsyms] [k] __fget_light <wbr> <wbr> ▒<br>+ 9.32% 0.00% syslog-ng [kernel.kallsyms] [k] int_ret_from_sys_call <wbr> <wbr> ▒<br>+ 9.32% 0.00% syslog-ng [kernel.kallsyms] [k] syscall_return_slowpath <wbr> <wbr> ▒<br>+ 9.32% 0.00% syslog-ng [kernel.kallsyms] [k] syscall_slow_exit_work <wbr> <wbr> ▒<br>+ 9.32% 0.00% syslog-ng [unknown] [.] 0x000000000000002d <wbr> <wbr> ▒<br>+ 9.32% 9.32% syslog-ng [kernel.kallsyms] [k] unroll_tree_refs <wbr> <wbr> ▒<br>+ 7.71% 3.29% syslog-ng <a href="http://libc-2.19.so" target="_blank">libc-2.19.so</a> [.] _int_malloc <wbr> <wbr> ▒<br>+ 7.27% 7.27% syslog-ng libjson-c.so.2.0.0 [.] lh_char_hash <wbr> <wbr> ▒<br>+ 6.79% 0.00% syslog-ng [kernel.kallsyms] [k] apic_timer_interrupt <wbr> <wbr> ▒<br>+ 6.79% 0.00% syslog-ng [kernel.kallsyms] [k] smp_apic_timer_interrupt <wbr> <wbr> ▒<br>+ 6.63% 0.00% syslog-ng [unknown] [k] 0x000000000000002a <wbr> <wbr> ▒<br>+ 5.05% 0.00% syslog-ng [kernel.kallsyms] [k] ep_poll <wbr> <wbr> ▒<br>+ 5.05% 0.00% syslog-ng [kernel.kallsyms] [k] schedule_hrtimeout_range <wbr> <wbr> ▒<br>+ 5.05% 0.00% syslog-ng [kernel.kallsyms] [k] schedule_hrtimeout_range_clock<wbr> <wbr> ▒<br>+ 5.05% 0.00% syslog-ng [kernel.kallsyms] [k] schedule <wbr> <wbr> ▒<br>+ 5.05% 0.00% syslog-ng [kernel.kallsyms] [k] __schedule <wbr> <wbr> ▒<br>+ 5.05% 0.00% syslog-ng [unknown] [k] 0x000000000000002b <wbr> <wbr> ▒<br>+ 5.05% 0.00% syslog-ng [kernel.kallsyms] [k] deactivate_task <wbr> <wbr> ▒<br>+ 5.05% 0.00% syslog-ng [kernel.kallsyms] [k] dequeue_task_fair <wbr> <wbr> ▒<br>+ 5.05% 5.05% syslog-ng [kernel.kallsyms] [k] account_entity_dequeue <wbr> <wbr> ▒<br>+ 4.58% 4.58% syslog-ng <a href="http://libc-2.19.so" target="_blank">libc-2.19.so</a> [.] __ctype_b_loc <wbr> <wbr> ▒<br>+ 4.58% 0.00% syslog-ng [unknown] [.] 0x00007f4bfc4121a0 <wbr> <wbr> ▒<br>+ 4.42% 0.00% syslog-ng [kernel.kallsyms] [k] irq_exit <wbr> <wbr> ▒<br>+ 4.42% 4.42% syslog-ng [kernel.kallsyms] [k] __do_softirq <wbr> <wbr> ▒<br>+ 4.42% 0.00% syslog-ng [unknown] [.] 0x0000000002579bb0 <wbr> <wbr> ▒<br>+ 4.42% 0.00% syslog-ng libglib-2.0.so.0.4002.0 [.] g_static_mutex_get_mutex_impl <wbr> <wbr> ▒<br>+ 4.42% 4.42% syslog-ng libsyslog-ng-3.14.so.0.0.0 [.] log_msg_set_value <wbr> <wbr> ▒<br>+ 4.42% 4.42% syslog-ng libglib-2.0.so.0.4002.0 [.] g_ptr_array_free <wbr> <wbr> ▒<br><br></div></div><div class="gmail_extra"><br><div class="gmail_quote"><span>On Tue, Mar 20, 2018 at 5:57 PM, Scheidler, Balázs <span dir="ltr"><<a href="mailto:balazs.scheidler@balabit.com" target="_blank">balazs.scheidler@balabit.com</a>></span> wrote:<br></span><div><div class="gmail-m_-8063925455884992131m_3395947134879140157m_-3543377098694014331m_-1856218819312331434h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>126% CPU usage? that would be great to know the details there, although I understand that the memory is more of a concern now. :)<br><br></div>can you run a perf record on that process, perhaps once the memory issue is solved? I have my suspicion where it is spending its time, but it would be great to confirm. (my guess is value-pairs while formatting json messages).<br><div class="gmail_extra"><br></div><div class="gmail_extra">cheers,<span class="gmail-m_-8063925455884992131m_3395947134879140157m_-3543377098694014331m_-1856218819312331434m_-3733023120874606572HOEnZb"><font color="#888888"><br clear="all"></font></span></div><div class="gmail_extra"><span class="gmail-m_-8063925455884992131m_3395947134879140157m_-3543377098694014331m_-1856218819312331434m_-3733023120874606572HOEnZb"><font color="#888888"><div><div class="gmail-m_-8063925455884992131m_3395947134879140157m_-3543377098694014331m_-1856218819312331434m_-3733023120874606572m_2750937714692181180gmail_signature"><div dir="ltr">-- <br>Bazsi<br></div></div></div></font></span><div><div class="gmail-m_-8063925455884992131m_3395947134879140157m_-3543377098694014331m_-1856218819312331434m_-3733023120874606572h5">
<br><div class="gmail_quote">On Tue, Mar 20, 2018 at 8:26 PM, Michal Purzynski <span dir="ltr"><<a href="mailto:michal@mozilla.com" target="_blank">michal@mozilla.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hello Gábor!</div><div><br></div><div>Answers inline.</div><div><br></div><div><div class="gmail_extra"><div class="gmail_quote"><span>On Mon, Mar 19, 2018 at 9:09 AM, Nagy, Gábor <span dir="ltr"><<a href="mailto:gabor.nagy@balabit.com" target="_blank">gabor.nagy@balabit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div>- have you built syslog-ng from source or you downloaded a package?</div><div></div></div></blockquote><div><br></div></span><div>It's a package from</div><div><br></div><div>deb <a href="http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/xUbuntu_14.04" target="_blank">http://download.opensuse.org/r<wbr>epositories/home:/laszlo_budai<wbr>:/syslog-ng/xUbuntu_14.04</a> ./<br></div><span><div> </div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>- have you tried to reproduce the issue in a different environment with a minimal config? We are using your configuration, but if you narrowed down the problem it would be helpful.</div></div></blockquote><div><br></div></span><div>If I disable the Bro via AMQP the problem goes away, so that must be something there :/<br></div><span><div> </div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>- we were experimenting with a very simple JSON message, can you show us an example log to see the complexity of it, please? We are thinking to check Bro out for log message structure.</div></div></blockquote><div><br></div></span><div>Example logs follow sent to Peter via a private channel, making them public would be kind of difficult.</div><div><br></div><div>Appreciate you looking into it! And BTW, I just restarted syslog-ng on the most busy server</div><div><br></div><div><span style="color:rgb(204,204,204);font-family:"Source Sans Pro",sans-serif;font-size:15px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:0.2px;text-align:start;text-indent:0px;text-transform:none;white-space:pre-wrap;word-spacing:0px;background-color:rgb(31,31,31);display:inline;float:none">14910 root 20 0 59.899g 0.057t 3784 S 126.4 92.9 1166:22 syslog-ng</span></div><div><br></div><div>;-)<br></div><div><div class="gmail-m_-8063925455884992131m_3395947134879140157m_-3543377098694014331m_-1856218819312331434m_-3733023120874606572m_2750937714692181180h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>I saw that you have gave a huge log-fetch-limit() in the global config compared to the default. Setting log-fetch-limit() in global config is deprecated, you need to set it up per source.</div><div><br></div><div>We have a couple of ideas and will continue to try reproducing the memleak you reported.</div><div><br></div><div>Regards,</div><div>Gabor</div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="gmail-m_-8063925455884992131m_3395947134879140157m_-3543377098694014331m_-1856218819312331434m_-3733023120874606572m_2750937714692181180m_-580635352056405161gmail-h5">On Sat, Mar 17, 2018 at 12:45 AM, Michal Purzynski <span dir="ltr"><<a href="mailto:michal@mozilla.com" target="_blank">michal@mozilla.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail-m_-8063925455884992131m_3395947134879140157m_-3543377098694014331m_-1856218819312331434m_-3733023120874606572m_2750937714692181180m_-580635352056405161gmail-h5"><div dir="ltr"><div><div>Hello!!</div><div><br></div><div>Could you help us troubleshoot a memory leak or a misconfiguration that makes the syslog-ng process memory usage grow? Like, a lot.<br></div><div><br></div><div>We use syslog-ng to read some JSON files and ship them to a RabbitMQ server via AMQP. As you can see, this is just a client, it does not accept connections from other systems, it works mostly with Bro logs (plus it handles a local syslog because that's convenient).</div><div><br></div><div>We have around 6000 events per second on this server. What's interesting, syslog-ng's memory grows quickly without flow control (and slower, but it still continues to grow with it). I'll switch that to TLS soon, a more secure configuration is ready to be deployed.</div><br></div><div>Things look pretty good on the RabbitMQ side. That server is not under pressure and handles the load just fine, the queue is consumed, there's nothing building up that would make me believe we have RabbitMQ server overloaded.</div><div><br></div><div>How much does syslog-ng grow?</div><div><br></div><div>I'd say - if I disable flow-control it will eat 55GB of RAM in less than 24h, if not faster. With flow-control enabled on the most 'busy' files things are way better, but the memory usage still keep growing - <br></div><div><br></div><div>syslog-ng.conf looks like below - BTW that's Ubuntu 14.04 LTS, 3.14.1-3 of syslog-ng</div><div><br></div><div>Let me know what other data you might need.<br></div><div><br></div><div>@version: 3.14<br>@include "scl.conf"<br><br># Syslog-ng configuration file, compatible with default Debian syslogd<br># installation.<br><br># First, set some global options.<br>options {<br> threaded (yes);<br> flush_lines (50000);<br> flush_timeout (1000);<br> time_reopen (10);<br> log_fetch_limit (50000);<br> log_fifo_size (500000);<br> use_dns (yes);<br> dns_cache (5000);<br> dns_cache_expire(87600);<br> use_fqdn (yes);<br> owner("root");<br> group("adm");<br> perm(0640);<br> keep_hostname (yes);<br> chain_hostnames (off);<br>};<br><br>########################<br># Sources<br>########################<br># This is the default behavior of sysklogd package<br># Logs may come from unix stream, but not from another machine.<br>#<br>source s_src {<br> system();<br> internal();<br>};<br><br># If you wish to get logs from remote machine you should uncomment<br># this and comment the above source line.<br>#<br>#source s_net { tcp(ip(127.0.0.1) port(1000)); };<br><br>########################<br># Destinations<br>########################<br># First some standard logfile<br>#<br>destination d_auth { file("/var/log/auth.log"); };<br>destination d_cron { file("/var/log/cron.log"); };<br>destination d_daemon { file("/var/log/daemon.log"); };<br>destination d_kern { file("/var/log/kern.log"); };<br>destination d_mail { file("/var/log/mail.log"); };<br>destination d_syslog { file("/var/log/syslog"); };<br><br># This files are the log come from the mail subsystem.<br>#<br>#destination d_mailinfo { file("/var/log/<a href="http://mail.info" target="_blank">mail.info</a>"); };<br>#destination d_mailwarn { file("/var/log/mail.warn"); };<br>#destination d_mailerr { file("/var/log/mail.err"); };<br><br># Logging for INN news system<br>#<br>#destination d_newscrit { file("/var/log/news/news.crit"<wbr>); };<br>#destination d_newserr { file("/var/log/news/news.err")<wbr>; };<br>#destination d_newsnotice { file("/var/log/news/news.notic<wbr>e"); };<br><br># Some 'catch-all' logfiles.<br>#<br>destination d_debug { file("/var/log/debug"); };<br>destination d_error { file("/var/log/error"); };<br><br># Syslog1 in SCL3<br>destination d_scl3 {<br> udp("<a href="http://syslog1.private.scl3.mozilla.com" target="_blank">syslog1.private.scl3.mozi<wbr>lla.com</a>" port(514));<br>};<br><br>########################<br># Filters<br>########################<br># Here's come the filter options. With this rules, we can set which<br># message go where.<br><br>filter f_dbg { level(debug); };<br>filter f_info { level(info); };<br>filter f_notice { level(notice); };<br>filter f_warn { level(warn); };<br>filter f_err { level(err); };<br>filter f_crit { level(crit .. emerg); };<br>filter f_debug { level(debug) and not facility(auth, authpriv, news, mail); };<br>filter f_error { level(err .. emerg) ; };<br>filter f_auth { facility(auth, authpriv) and not filter(f_debug); };<br>filter f_cron { facility(cron) and not filter(f_debug); };<br>filter f_daemon { facility(daemon) and not filter(f_debug); };<br>filter f_kern { facility(kern) and not filter(f_debug); };<br>filter f_local { facility(local0, local1, local3, local4, local5,<br> local6, local7) and not filter(f_debug); };<br>filter f_mail { facility(mail) and not filter(f_debug); };<br>filter f_syslog3 { not facility(auth, authpriv, mail) and not filter(f_debug); };<br><br>########################<br># Log paths<br>########################<br>log { source(s_src); filter(f_auth); destination(d_auth); };<br>log { source(s_src); filter(f_cron); destination(d_cron); };<br>log { source(s_src); filter(f_daemon); destination(d_daemon); };<br>log { source(s_src); filter(f_kern); destination(d_kern); };<br>log { source(s_src); filter(f_syslog3); destination(d_syslog); };<br>log { source(s_src); filter(f_mail); destination(d_mail); };<br>log { source(s_src); filter(f_debug); destination(d_debug); };<br>log { source(s_src); filter(f_error); destination(d_error); };<br><br><br># All messages send to a remote site<br>#<br>log { source(s_src); destination(d_scl3); };<br><br>###<br># Include all config files in /etc/syslog-ng/conf.d/<br>###<br>@include "/etc/syslog-ng/conf.d/*.conf"<br></div><div><br></div><div><br></div><div><br></div><div>There's another file, amqp.conf where we actually read and ship those Bro logs.<br></div><div><br></div><div><br></div><div>source bro_conn {<br> file( "/nsm/bro/logs/current/conn.lo<wbr>g" flags(no-parse));<br>};<br><br>source bro_ssl {<br> file( "/nsm/bro/logs/current/ssl.log<wbr>" flags(no-parse));<br>};<br><br>source bro_dns {<br> file( "/nsm/bro/logs/current/dns.log<wbr>" flags(no-parse));<br>};<br><br>source bro_smtp {<br> file( "/nsm/bro/logs/current/smtp.lo<wbr>g" flags(no-parse));<br>};<br><br>source bro_ssh {<br> file( "/nsm/bro/logs/current/ssh.log<wbr>" flags(no-parse));<br>};<br><br>source bro_notice {<br> file( "/nsm/bro/logs/current/notice.<wbr>log" flags(no-parse));<br>};<br><br>source bro_intel {<br> file( "/nsm/bro/logs/current/intel.l<wbr>og" flags(no-parse));<br>};<br><br>source bro_dce_rpc {<br> file( "/nsm/bro/logs/current/dce_rpc<wbr>.log" flags(no-parse));<br>};<br><br>source bro_dhcp {<br> file( "/nsm/bro/logs/current/dhcp.lo<wbr>g" flags(no-parse));<br>};<br><br>source bro_files {<br> file( "/nsm/bro/logs/current/files.l<wbr>og" flags(no-parse));<br>};<br><br>source bro_kerberos {<br> file( "/nsm/bro/logs/current/kerbero<wbr>s.log" flags(no-parse));<br>};<br><br>source bro_http {<br> file( "/nsm/bro/logs/current/http.lo<wbr>g" flags(no-parse));<br>};<br><br>source bro_software {<br> file( "/nsm/bro/logs/current/softwar<wbr>e.log" flags(no-parse));<br>};<br><br>source bro_snmp {<br> file( "/nsm/bro/logs/current/snmp.lo<wbr>g" flags(no-parse));<br>};<br><br>source bro_socks {<br> file( "/nsm/bro/logs/current/socks.l<wbr>og" flags(no-parse));<br>};<br><br>source bro_tunnel {<br> file( "/nsm/bro/logs/current/tunnel.<wbr>log" flags(no-parse));<br>};<br><br>source bro_ntlm {<br> file( "/nsm/bro/logs/current/ntlm.lo<wbr>g" flags(no-parse));<br>};<br><br>source bro_pe {<br> file( "/nsm/bro/logs/current/pe.log" flags(no-parse));<br>};<br><br>source bro_sip {<br> file( "/nsm/bro/logs/current/sip.log<wbr>" flags(no-parse));<br>};<br><br>source bro_smb_files {<br> file( "/nsm/bro/logs/current/smb_fil<wbr>es.log" flags(no-parse));<br>};<br><br>source bro_smb_mapping {<br> file( "/nsm/bro/logs/current/smb_map<wbr>ping.log" flags(no-parse));<br>};<br><br>source bro_x509 {<br> file( "/nsm/bro/logs/current/x509.lo<wbr>g" flags(no-parse));<br>};<br><br>source bro_known_certs {<br> file( "/nsm/bro/logs/current/known_c<wbr>erts.log" flags(no-parse));<br>};<br><br>source bro_known_devices {<br> file( "/nsm/bro/logs/current/known_d<wbr>evices.log" flags(no-parse));<br>};<br><br>source bro_known_hosts {<br> file( "/nsm/bro/logs/current/known_h<wbr>osts.log" flags(no-parse));<br>};<br><br>source bro_known_services {<br> file( "/nsm/bro/logs/current/known_s<wbr>ervices.log" flags(no-parse));<br>};<br><br><br>destination d_amqp {<br> amqp(<br> vhost("nsm")<br> host("<a href="http://syslog-proxy1.dmz.mdc1.mozilla.com" target="_blank"><our happy rabbit></a>")<br> port(5672)<br> exchange("eventtask")<br> exchange-type("direct")<br> routing-key("eventtask")<br> body("$(format-json --scope nv_pairs --pair category=\"bro\" --pair source=$source --pair customendpoint=\" \" --pair tags=\"bro\")")<br> persistent(yes)<br> username("USERNAME")<br> password("PASSWORD")<br> );<br>};<br><br><br>parser p_json { json-parser(); };<br><br><br>log { source(bro_conn); parser(p_json); destination(d_amqp); };<br>log { source(bro_http); parser(p_json); destination(d_amqp); };<br>log { source(bro_ssl); parser(p_json); destination(d_amqp); };<br>log { source(bro_dns); parser(p_json); destination(d_amqp); };<br>log { source(bro_smtp); parser(p_json); destination(d_amqp); };<br>log { source(bro_ssh); parser(p_json); destination(d_amqp); };<br>log { source(bro_intel); parser(p_json); destination(d_amqp); };<br>log { source(bro_notice); parser(p_json); destination(d_amqp); };<br>log { source(bro_dce_rpc); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_dhcp); parser(p_json); destination(d_amqp); };<br>log { source(bro_files); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_kerberos); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_software); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_snmp); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_socks); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_tunnel); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_ntlm); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_pe); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_sip); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_smb_files); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_smb_mapping); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_x509); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_known_certs); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_known_devices); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_known_hosts); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_known_services); parser(p_json); destination(d_amqp); flags(flow-control); };</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><b>Statistics</b></div><div><br></div><div><br></div><div>Mar 15 <b>00:17:30</b> nsmserver syslog-ng[11278]: Log statistics; processed='source(bro_conn)=11<wbr>2360513', processed='source(s_src)=22734<wbr>9', processed='source(bro_known_de<wbr>vices)=3791', processed='global(sdata_update<wbr>s)=0', processed='center(received)=31<wbr>0790955', processed='source(bro_ssh)=622<wbr>441', processed='source(bro_smb_file<wbr>s)=5815964', processed='source(bro_socks)=0<wbr>', processed='destination(d_daemo<wbr>n)=21', dropped='dst.amqp(d_amqp#0,amq<wbr>p,nsm,happyrabbit,5672,eventta<wbr>sk,direct)=2', processed='dst.amqp(d_amqp#0,a<wbr>mqp,nsm,happyrabbit,5672,event<wbr>task,direct)=<a href="tel:(31)%20056%203565" value="+36310563565" target="_blank">310563565</a>', queued='dst.amqp(d_amqp#0,amqp<wbr>,nsm,happyrabbit,5672,eventtas<wbr>k,direct)=0', processed='destination(d_error<wbr>)=189386', processed='destination(d_syslo<wbr>g)=207595', processed='source(bro_ssl)=497<wbr>88364', processed='source(bro_kerberos<wbr>)=133177', processed='source(bro_dhcp)=69<wbr>970', processed='destination(d_mail)<wbr>=0', processed='source(bro_http)=60<wbr>085539', processed='global(msg_clones)=<wbr>1576', processed='destination(d_amqp)<wbr>=<a href="tel:(31)%20056%203565" value="+36310563565" target="_blank">310563565</a>', processed='destination(d_kern)<wbr>=146', processed='source(bro_tunnel)=<wbr>520921', processed='source(bro_software<wbr>)=<a href="tel:(1)%20885%201236" value="+3618851236" target="_blank">18851236</a>', processed='source(bro_known_se<wbr>rvices)=13403', processed='source(bro_known_ce<wbr>rts)=2070', processed='source(bro_dce_rpc)<wbr>=501875', processed='destination(d_scl3)<wbr>=227349', processed='source(bro_known_ho<wbr>sts)=14604', processed='source(bro_smb_mapp<wbr>ing)=116412', processed='source(bro_files)=1<wbr>5152100', processed='center(queued)=3112<wbr>10449', processed='destination(d_debug<wbr>)=10280', processed='src.internal(s_src#<wbr>2)=26785', stamp='src.internal(s_src#2)=1<wbr>521073048', processed='source(bro_ntlm)=16<wbr>823', processed='destination(d_auth)<wbr>=9474', processed='global(internal_que<wbr>ue_length)=0', processed='source(bro_smtp)=10<wbr>67448', dropped='dst.udp(d_scl3#0,udp,<wbr>syslog1.private.scl3.mozilla.c<wbr>om:514)=0', processed='dst.udp(d_scl3#0,ud<wbr>p,syslog1.private.scl3.mozilla<wbr>.com:514)=227349', queued='dst.udp(d_scl3#0,udp,s<a href="http://yslog1.private.scl3.mozilla.co" target="_blank"><wbr>yslog1.private.scl3.mozilla.co</a><wbr>m:514)=221705', written='dst.udp(d_scl3#0,udp,<wbr>syslog1.private.scl3.mozilla.c<wbr>om:514)=5644', processed='global(payload_real<wbr>locs)=<a href="tel:(31)%20046%207149" value="+36310467149" target="_blank">310467149</a>', queued='global(scratch_buffers<wbr>_count)=17875655781170', processed='destination(d_cron)<wbr>=2633', processed='source(bro_snmp)=98<wbr>54568', processed='source(bro_notice)=<wbr>119415', processed='source(bro_dns)=193<wbr>03431', processed='source(bro_sip)=978<wbr>22', processed='source(bro_intel)=2<wbr>6969', processed='source(bro_pe)=5311<wbr>03', processed='source(bro_x509)=15<wbr>493647', queued='global(scratch_buffers<wbr>_bytes)=2304'<br></div><div><br></div><div><br></div><div>Mar 15 <b>00:27:30</b> nsmserver<a href="http://nsmserver1.private.scl3.mozilla.com" target="_blank"></a> syslog-ng[11278]: Log statistics; processed='source(bro_conn)=11<wbr>2988941', processed='source(s_src)=22865<wbr>1', processed='source(bro_known_de<wbr>vices)=3791', processed='global(sdata_update<wbr>s)=0', processed='center(received)=31<wbr>2659144', processed='source(bro_ssh)=627<wbr>013', processed='source(bro_smb_file<wbr>s)=5863697', processed='source(bro_socks)=0<wbr>', processed='destination(d_daemo<wbr>n)=21', dropped='dst.amqp(d_amqp#0,amq<wbr>p,nsm,<a href="http://syslog-proxy1.dmz.mdc1.mozilla.com" target="_blank">happyrabbit</a>,5672,eventta<wbr>sk,direct)=2', processed='dst.amqp(d_amqp#0,a<wbr>mqp,nsm,happyrabbit<a href="http://syslog-proxy1.dmz.mdc1.mozilla.com" target="_blank"></a>,5672,event<wbr>task,direct)=<a href="tel:(31)%20243%200452" value="+36312430452" target="_blank">312430452</a>', queued='dst.amqp(d_amqp#0,amqp<wbr>,nsm,happyrabbit<a href="http://syslog-proxy1.dmz.mdc1.mozilla.com" target="_blank"></a>,5672,eventtas<wbr>k,direct)=0', processed='destination(d_error<wbr>)=190429', processed='destination(d_syslo<wbr>g)=208759', processed='source(bro_ssl)=500<wbr>77572', processed='source(bro_kerberos<wbr>)=134215', processed='source(bro_dhcp)=70<wbr>487', processed='destination(d_mail)<wbr>=0', processed='source(bro_http)=60<wbr>446166', processed='global(msg_clones)=<wbr>1594', processed='destination(d_amqp)<wbr>=<a href="tel:(31)%20243%200452" value="+36312430452" target="_blank">312430452</a>', processed='destination(d_kern)<wbr>=146', processed='source(bro_tunnel)=<wbr>524450', processed='source(bro_software<wbr>)=<a href="tel:(1)%20893%208552" value="+3618938552" target="_blank">18938552</a>', processed='source(bro_known_se<wbr>rvices)=13532', processed='source(bro_known_ce<wbr>rts)=2073', processed='source(bro_dce_rpc)<wbr>=505206', processed='destination(d_scl3)<wbr>=228651', processed='source(bro_known_ho<wbr>sts)=14630', processed='source(bro_smb_mapp<wbr>ing)=117177', processed='source(bro_files)=1<wbr>5252368', processed='center(queued)=3130<wbr>80999', processed='destination(d_debug<wbr>)=10352', processed='src.internal(s_src#<wbr>2)=26966', stamp='src.internal(s_src#2)=1<wbr>521073648', processed='source(bro_ntlm)=16<wbr>848', processed='destination(d_auth)<wbr>=9540', processed='global(internal_que<wbr>ue_length)=0', processed='source(bro_smtp)=10<wbr>74012', dropped='dst.udp(d_scl3#0,udp,<wbr>syslog1.private.scl3.mozilla.c<wbr>om:514)=0', processed='dst.udp(d_scl3#0,ud<wbr>p,syslog1.private.scl3.mozilla<wbr>.com:514)=228651', queued='dst.udp(d_scl3#0,udp,s<a href="http://yslog1.private.scl3.mozilla.co" target="_blank"><wbr>yslog1.private.scl3.mozilla.co</a><wbr>m:514)=223007', written='dst.udp(d_scl3#0,udp,<wbr>syslog1.private.scl3.mozilla.c<wbr>om:514)=5644', processed='global(payload_real<wbr>locs)=<a href="tel:(31)%20233%203723" value="+36312333723" target="_blank">312333723</a>', queued='global(scratch_buffers<wbr>_count)=17970145061685', processed='destination(d_cron)<wbr>=2649', processed='source(bro_snmp)=99<wbr>17302', processed='source(bro_notice)=<wbr>120140', processed='source(bro_dns)=194<wbr>62256', processed='source(bro_sip)=985<wbr>65', processed='source(bro_intel)=2<wbr>7061', processed='source(bro_pe)=5357<wbr>53', processed='source(bro_x509)=15<wbr>598686', queued='global(scratch_buffers<wbr>_bytes)=2304'</div><br></div>
<br></div></div>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=syslog<wbr>-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>
<br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=syslog<wbr>-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
<br></blockquote></div></div></div><br></div></div></div>
<br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=syslog<wbr>-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div></div></div></div>
<br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=syslog<wbr>-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
<br></blockquote></div></div></div><br></div>
<br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=syslog<wbr>-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div></div></div>
<br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=syslog<wbr>-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div><br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/<wbr>mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/<wbr>support/documentation/?<wbr>product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/<wbr>syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div></div></div>