[syslog-ng] Syslog-ng shipping logs through AMQP with huge memory leaks
Nagy, Gábor
gabor.nagy at balabit.com
Fri Mar 23 22:47:08 UTC 2018
Hi Michal!
Thanks for sharing logs and details.
You said that "If I disable the Bro via AMQP the problem goes away, so that
must be something there ".
Do you mean that Bro is not sending any logs to the files or something else?
I have experimented with your config and with another one as well in which
I removed the json-parser and json-format template function.
Both cases I have seen an increased memory usage even after the queue was
empty, but I have not seen a per-message based increasing memory leak.
Of course when the queue is used it was an even higher memory footprint,
but it dropped back when the queue emptied.
Valgrind logs only showed traces from startup, not during message sending.
For testing I have used the *loggen* tool from our repository and the log
examples that you have shared:
bin/loggen -r 6000 -I 10 --active-connections=5 -iSd -R mozilla/conn.log
--file-loop-reading localhost 4444
I have sent the messages into a network() source with 6000 msg/s msg rate
through 5 connections for 10 seconds.
Here are some statistics about the tests:
- after start syslog-ng used around 8k (RES)
- sending the first pack of messages (~300000 messages) memory usage is
623M (RES) even after the queue emptied!
dst.amqp;d_amqp#0;amqp,/,localhost,5672,eventtask,direct;a;processed;305735
dst.amqp;d_amqp#0;amqp,/,localhost,5672,eventtask,direct;a;queued;0
- 2. iteration: processed;611243, 653M usage (the queue is used during
receiving messages from network and memory usage topped at 900M)
- 3. iteration: 658M
- 4. iteration: processed;1222515, 659M
- 5. iteration: doubled the message sending interval (20s)
the memory usage increased to 1212M RES !
It seems amqp() driver does not free the used memory buffer after the queue
emptied.
You asked previously whether the increased memory usage is due to
unprocessed messages.
Based on the statistics you shared, the 'queued' counter was 0. And after
the queue is emptied the memory should be freed.
I still have a couple of ideas and will investigate further, just wanted to
share some information in the mean time.
Gabor
On Wed, Mar 21, 2018 at 7:16 PM, Michal Purzynski <michal at mozilla.com>
wrote:
> The CPU usage is now between 40-60%, usually somewhere around 45% so it
> went down indeed. It spikes every few seconds to 55% then goes down, etc.
>
> What worries me is that we're already at 21GB of RES so the memory usage
> grows and that's all syslog-ng's private data.
>
> Are those messages that aren't processed? TBH I believe all messages are
> getting to Rabbit on time, I can see what I'm expecting in ES that's
> pulling from that Rabbit.
>
>
> On Tue, Mar 20, 2018 at 10:49 PM, Michal Purzynski <michal at mozilla.com>
> wrote:
>
>> cat /proc/`pidof -s syslog-ng`/maps | egrep -i libjemalloc
>> 7f0d97978000-7f0d979a8000 r-xp 00000000 08:02 25771415575
>> /usr/lib/x86_64-linux-gnu/libjemalloc.so.1
>> 7f0d979a8000-7f0d97ba8000 ---p 00030000 08:02 25771415575
>> /usr/lib/x86_64-linux-gnu/libjemalloc.so.1
>> 7f0d97ba8000-7f0d97baa000 r--p 00030000 08:02 25771415575
>> /usr/lib/x86_64-linux-gnu/libjemalloc.so.1
>> 7f0d97baa000-7f0d97bab000 rw-p 00032000 08:02 25771415575
>> /usr/lib/x86_64-linux-gnu/libjemalloc.so.1
>>
>>
>> jemalloc loaded. Let's see how that helps. I also have tcmalloc up my
>> sleeve.
>>
>> On Tue, Mar 20, 2018 at 9:06 PM, Scheidler, Balázs <
>> balazs.scheidler at balabit.com> wrote:
>>
>>> Probably the biggest toll is the json-parser(), which would allocate
>>> memory when we are parsing the json. On the output side we have a
>>> handcrafted json generator that doesnt allocate memory, but on the inbound
>>> side libjson-c is doing it.
>>>
>>> That's probably the reason behind the futex numbers, malloc() uses a
>>> number of mutexes, which might get contended.
>>>
>>> Jemalloc would probably help a lot.
>>>
>>> --
>>> Bazsi
>>>
>>> On Wed, Mar 21, 2018 at 2:51 AM, Michal Purzynski <michal at mozilla.com>
>>> wrote:
>>>
>>>> [ perf record: Captured and wrote 54.230 MB perf.data (1418800 samples)
>>>> ]
>>>>
>>>> Eyeballing looks like syslog-ng spends tons of time in malloc,
>>>> allocating and deallocating memory. Maybe using gperf / jemalloc could help
>>>> here?
>>>>
>>>> Let me know if you want entire file. This is without call-graph, with
>>>> call-graph... hmmm, looks like I will have to rebuild syslog-ng with
>>>> symbols. Let's trace that leaks first ;-)
>>>>
>>>> 16.77% syslog-ng [kernel.kallsyms] [k]
>>>> update_blocked_averages
>>>> 16.37% syslog-ng libpthread-2.19.so [.] pthread_mutex_lock
>>>> 11.84% syslog-ng [kernel.kallsyms] [k]
>>>> audit_filter_syscall
>>>> 8.09% syslog-ng [kernel.kallsyms] [k] copy_page
>>>> 7.50% syslog-ng [kernel.kallsyms] [k]
>>>> syscall_return_via_sysret
>>>> 6.32% syslog-ng libc-2.19.so [.] _int_free
>>>> 5.13% syslog-ng libglib-2.0.so.0.4002.0 [.] 0x0000000000073487
>>>> 4.14% syslog-ng libc-2.19.so [.] _IO_vfscanf
>>>> 3.75% syslog-ng libivykis.so.0.5.4 [.]
>>>> pthread_mutex_unlock at plt
>>>> 3.35% syslog-ng libsyslog-ng-3.14.so.0.0.0 [.] 0x0000000000026d02
>>>> 2.97% syslog-ng libc-2.19.so [.] _int_malloc
>>>> 2.96% syslog-ng libglib-2.0.so.0.4002.0 [.] 0x000000000008a617
>>>> 2.56% syslog-ng libglib-2.0.so.0.4002.0 [.] 0x000000000008a61a
>>>> 2.37% syslog-ng libc-2.19.so [.]
>>>> __memmove_ssse3_back
>>>> 1.97% syslog-ng libjson-c.so.2.0.0 [.] lh_char_hash
>>>> 1.97% syslog-ng libsyslog-ng-3.14.so.0.0.0 [.] 0x0000000000036410
>>>> 0.32% syslog-ng libc-2.19.so [.] _IO_setb
>>>> 0.18% syslog-ng libc-2.19.so [.] malloc_consolidate
>>>> 0.17% syslog-ng libc-2.19.so [.] __strchrnul
>>>> 0.17% syslog-ng libglib-2.0.so.0.4002.0 [.]
>>>> g_string_append_printf
>>>> 0.17% syslog-ng [kernel.kallsyms] [k]
>>>> reschedule_interrupt
>>>> 0.17% syslog-ng libglib-2.0.so.0.4002.0 [.] 0x00000000000735a1
>>>> 0.17% syslog-ng libglib-2.0.so.0.4002.0 [.] 0x000000000008a8a6
>>>> 0.17% syslog-ng libsyslog-ng-3.14.so.0.0.0 [.] 0x000000000006df70
>>>> 0.15% syslog-ng libc-2.19.so [.]
>>>> __memcpy_sse2_unaligned
>>>> 0.15% syslog-ng libglib-2.0.so.0.4002.0 [.] g_string_assign
>>>> 0.01% syslog-ng libc-2.19.so [.] malloc
>>>> 0.00% syslog-ng libjson-c.so.2.0.0 [.]
>>>> json_tokener_parse_ex
>>>> 0.00% syslog-ng libc-2.19.so [.] vfprintf
>>>> 0.00% syslog-ng libglib-2.0.so.0.4002.0 [.]
>>>> g_utf8_get_char_validated
>>>> 0.00% syslog-ng libglib-2.0.so.0.4002.0 [.] 0x000000000008a8a7
>>>> 0.00% syslog-ng libc-2.19.so [.] free
>>>> 0.00% syslog-ng libc-2.19.so [.]
>>>> __strcmp_sse2_unaligned
>>>> 0.00% syslog-ng libglib-2.0.so.0.4002.0 [.] g_string_truncate
>>>>
>>>>
>>>>
>>>>
>>>> With cal graph, like I said, only the library name tell you something
>>>> here, I can test later with syslog-ng with symbols
>>>>
>>>>
>>>> + 59.22% 0.00% syslog-ng libivykis.so.0.5.4 [.]
>>>> 0xffff80b3f3309c65 ▒
>>>> + 56.85% 0.00% syslog-ng libc-2.19.so [.]
>>>> 0xffff80b3f259f6d3 ▒
>>>> + 38.22% 0.00% syslog-ng [unknown] [.]
>>>> 0x0000000000000029 ▒
>>>> + 29.22% 29.22% syslog-ng libc-2.19.so [.]
>>>> 0x00000000000fe6d3 ▒
>>>> + 18.46% 0.00% syslog-ng [kernel.kallsyms] [k]
>>>> entry_SYSCALL_64_fastpath ▒
>>>> + 18.32% 0.00% syslog-ng [kernel.kallsyms] [k]
>>>> sys_epoll_wait ▒
>>>> + 13.27% 9.00% syslog-ng [kernel.kallsyms] [k]
>>>> __fget_light ▒
>>>> + 9.32% 0.00% syslog-ng [kernel.kallsyms] [k]
>>>> int_ret_from_sys_call ▒
>>>> + 9.32% 0.00% syslog-ng [kernel.kallsyms] [k]
>>>> syscall_return_slowpath ▒
>>>> + 9.32% 0.00% syslog-ng [kernel.kallsyms] [k]
>>>> syscall_slow_exit_work ▒
>>>> + 9.32% 0.00% syslog-ng [unknown] [.]
>>>> 0x000000000000002d ▒
>>>> + 9.32% 9.32% syslog-ng [kernel.kallsyms] [k]
>>>> unroll_tree_refs ▒
>>>> + 7.71% 3.29% syslog-ng libc-2.19.so [.]
>>>> _int_malloc ▒
>>>> + 7.27% 7.27% syslog-ng libjson-c.so.2.0.0 [.]
>>>> lh_char_hash ▒
>>>> + 6.79% 0.00% syslog-ng [kernel.kallsyms] [k]
>>>> apic_timer_interrupt ▒
>>>> + 6.79% 0.00% syslog-ng [kernel.kallsyms] [k]
>>>> smp_apic_timer_interrupt ▒
>>>> + 6.63% 0.00% syslog-ng [unknown] [k]
>>>> 0x000000000000002a ▒
>>>> + 5.05% 0.00% syslog-ng [kernel.kallsyms] [k]
>>>> ep_poll ▒
>>>> + 5.05% 0.00% syslog-ng [kernel.kallsyms] [k]
>>>> schedule_hrtimeout_range ▒
>>>> + 5.05% 0.00% syslog-ng [kernel.kallsyms] [k]
>>>> schedule_hrtimeout_range_clock ▒
>>>> + 5.05% 0.00% syslog-ng [kernel.kallsyms] [k]
>>>> schedule ▒
>>>> + 5.05% 0.00% syslog-ng [kernel.kallsyms] [k]
>>>> __schedule ▒
>>>> + 5.05% 0.00% syslog-ng [unknown] [k]
>>>> 0x000000000000002b ▒
>>>> + 5.05% 0.00% syslog-ng [kernel.kallsyms] [k]
>>>> deactivate_task ▒
>>>> + 5.05% 0.00% syslog-ng [kernel.kallsyms] [k]
>>>> dequeue_task_fair ▒
>>>> + 5.05% 5.05% syslog-ng [kernel.kallsyms] [k]
>>>> account_entity_dequeue ▒
>>>> + 4.58% 4.58% syslog-ng libc-2.19.so [.]
>>>> __ctype_b_loc ▒
>>>> + 4.58% 0.00% syslog-ng [unknown] [.]
>>>> 0x00007f4bfc4121a0 ▒
>>>> + 4.42% 0.00% syslog-ng [kernel.kallsyms] [k]
>>>> irq_exit ▒
>>>> + 4.42% 4.42% syslog-ng [kernel.kallsyms] [k]
>>>> __do_softirq ▒
>>>> + 4.42% 0.00% syslog-ng [unknown] [.]
>>>> 0x0000000002579bb0 ▒
>>>> + 4.42% 0.00% syslog-ng libglib-2.0.so.0.4002.0 [.]
>>>> g_static_mutex_get_mutex_impl ▒
>>>> + 4.42% 4.42% syslog-ng libsyslog-ng-3.14.so.0.0.0 [.]
>>>> log_msg_set_value ▒
>>>> + 4.42% 4.42% syslog-ng libglib-2.0.so.0.4002.0 [.]
>>>> g_ptr_array_free ▒
>>>>
>>>>
>>>> On Tue, Mar 20, 2018 at 5:57 PM, Scheidler, Balázs <
>>>> balazs.scheidler at balabit.com> wrote:
>>>>
>>>>> 126% CPU usage? that would be great to know the details there,
>>>>> although I understand that the memory is more of a concern now. :)
>>>>>
>>>>> can you run a perf record on that process, perhaps once the memory
>>>>> issue is solved? I have my suspicion where it is spending its time, but it
>>>>> would be great to confirm. (my guess is value-pairs while formatting json
>>>>> messages).
>>>>>
>>>>> cheers,
>>>>> --
>>>>> Bazsi
>>>>>
>>>>> On Tue, Mar 20, 2018 at 8:26 PM, Michal Purzynski <michal at mozilla.com>
>>>>> wrote:
>>>>>
>>>>>> Hello Gábor!
>>>>>>
>>>>>> Answers inline.
>>>>>>
>>>>>> On Mon, Mar 19, 2018 at 9:09 AM, Nagy, Gábor <gabor.nagy at balabit.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> - have you built syslog-ng from source or you downloaded a package?
>>>>>>>
>>>>>>
>>>>>> It's a package from
>>>>>>
>>>>>> deb http://download.opensuse.org/repositories/home:/laszlo_budai
>>>>>> :/syslog-ng/xUbuntu_14.04 ./
>>>>>>
>>>>>>
>>>>>> - have you tried to reproduce the issue in a different environment
>>>>>>> with a minimal config? We are using your configuration, but if you narrowed
>>>>>>> down the problem it would be helpful.
>>>>>>>
>>>>>>
>>>>>> If I disable the Bro via AMQP the problem goes away, so that must be
>>>>>> something there :/
>>>>>>
>>>>>>
>>>>>> - we were experimenting with a very simple JSON message, can you show
>>>>>>> us an example log to see the complexity of it, please? We are thinking to
>>>>>>> check Bro out for log message structure.
>>>>>>>
>>>>>>
>>>>>> Example logs follow sent to Peter via a private channel, making them
>>>>>> public would be kind of difficult.
>>>>>>
>>>>>> Appreciate you looking into it! And BTW, I just restarted syslog-ng
>>>>>> on the most busy server
>>>>>>
>>>>>> 14910 root 20 0 59.899g 0.057t 3784 S 126.4 92.9 1166:22 syslog-ng
>>>>>>
>>>>>> ;-)
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> I saw that you have gave a huge log-fetch-limit() in the global
>>>>>>> config compared to the default. Setting log-fetch-limit() in global config
>>>>>>> is deprecated, you need to set it up per source.
>>>>>>>
>>>>>>> We have a couple of ideas and will continue to try reproducing the
>>>>>>> memleak you reported.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Gabor
>>>>>>>
>>>>>>> On Sat, Mar 17, 2018 at 12:45 AM, Michal Purzynski <
>>>>>>> michal at mozilla.com> wrote:
>>>>>>>
>>>>>>>> Hello!!
>>>>>>>>
>>>>>>>> Could you help us troubleshoot a memory leak or a misconfiguration
>>>>>>>> that makes the syslog-ng process memory usage grow? Like, a lot.
>>>>>>>>
>>>>>>>> We use syslog-ng to read some JSON files and ship them to a
>>>>>>>> RabbitMQ server via AMQP. As you can see, this is just a client, it does
>>>>>>>> not accept connections from other systems, it works mostly with Bro logs
>>>>>>>> (plus it handles a local syslog because that's convenient).
>>>>>>>>
>>>>>>>> We have around 6000 events per second on this server. What's
>>>>>>>> interesting, syslog-ng's memory grows quickly without flow control (and
>>>>>>>> slower, but it still continues to grow with it). I'll switch that to TLS
>>>>>>>> soon, a more secure configuration is ready to be deployed.
>>>>>>>>
>>>>>>>> Things look pretty good on the RabbitMQ side. That server is not
>>>>>>>> under pressure and handles the load just fine, the queue is consumed,
>>>>>>>> there's nothing building up that would make me believe we have RabbitMQ
>>>>>>>> server overloaded.
>>>>>>>>
>>>>>>>> How much does syslog-ng grow?
>>>>>>>>
>>>>>>>> I'd say - if I disable flow-control it will eat 55GB of RAM in less
>>>>>>>> than 24h, if not faster. With flow-control enabled on the most 'busy' files
>>>>>>>> things are way better, but the memory usage still keep growing -
>>>>>>>>
>>>>>>>> syslog-ng.conf looks like below - BTW that's Ubuntu 14.04 LTS,
>>>>>>>> 3.14.1-3 of syslog-ng
>>>>>>>>
>>>>>>>> Let me know what other data you might need.
>>>>>>>>
>>>>>>>> @version: 3.14
>>>>>>>> @include "scl.conf"
>>>>>>>>
>>>>>>>> # Syslog-ng configuration file, compatible with default Debian
>>>>>>>> syslogd
>>>>>>>> # installation.
>>>>>>>>
>>>>>>>> # First, set some global options.
>>>>>>>> options {
>>>>>>>> threaded (yes);
>>>>>>>> flush_lines (50000);
>>>>>>>> flush_timeout (1000);
>>>>>>>> time_reopen (10);
>>>>>>>> log_fetch_limit (50000);
>>>>>>>> log_fifo_size (500000);
>>>>>>>> use_dns (yes);
>>>>>>>> dns_cache (5000);
>>>>>>>> dns_cache_expire(87600);
>>>>>>>> use_fqdn (yes);
>>>>>>>> owner("root");
>>>>>>>> group("adm");
>>>>>>>> perm(0640);
>>>>>>>> keep_hostname (yes);
>>>>>>>> chain_hostnames (off);
>>>>>>>> };
>>>>>>>>
>>>>>>>> ########################
>>>>>>>> # Sources
>>>>>>>> ########################
>>>>>>>> # This is the default behavior of sysklogd package
>>>>>>>> # Logs may come from unix stream, but not from another machine.
>>>>>>>> #
>>>>>>>> source s_src {
>>>>>>>> system();
>>>>>>>> internal();
>>>>>>>> };
>>>>>>>>
>>>>>>>> # If you wish to get logs from remote machine you should uncomment
>>>>>>>> # this and comment the above source line.
>>>>>>>> #
>>>>>>>> #source s_net { tcp(ip(127.0.0.1) port(1000)); };
>>>>>>>>
>>>>>>>> ########################
>>>>>>>> # Destinations
>>>>>>>> ########################
>>>>>>>> # First some standard logfile
>>>>>>>> #
>>>>>>>> destination d_auth { file("/var/log/auth.log"); };
>>>>>>>> destination d_cron { file("/var/log/cron.log"); };
>>>>>>>> destination d_daemon { file("/var/log/daemon.log"); };
>>>>>>>> destination d_kern { file("/var/log/kern.log"); };
>>>>>>>> destination d_mail { file("/var/log/mail.log"); };
>>>>>>>> destination d_syslog { file("/var/log/syslog"); };
>>>>>>>>
>>>>>>>> # This files are the log come from the mail subsystem.
>>>>>>>> #
>>>>>>>> #destination d_mailinfo { file("/var/log/mail.info"); };
>>>>>>>> #destination d_mailwarn { file("/var/log/mail.warn"); };
>>>>>>>> #destination d_mailerr { file("/var/log/mail.err"); };
>>>>>>>>
>>>>>>>> # Logging for INN news system
>>>>>>>> #
>>>>>>>> #destination d_newscrit { file("/var/log/news/news.crit"); };
>>>>>>>> #destination d_newserr { file("/var/log/news/news.err"); };
>>>>>>>> #destination d_newsnotice { file("/var/log/news/news.notice"); };
>>>>>>>>
>>>>>>>> # Some 'catch-all' logfiles.
>>>>>>>> #
>>>>>>>> destination d_debug { file("/var/log/debug"); };
>>>>>>>> destination d_error { file("/var/log/error"); };
>>>>>>>>
>>>>>>>> # Syslog1 in SCL3
>>>>>>>> destination d_scl3 {
>>>>>>>> udp("syslog1.private.scl3.mozilla.com" port(514));
>>>>>>>> };
>>>>>>>>
>>>>>>>> ########################
>>>>>>>> # Filters
>>>>>>>> ########################
>>>>>>>> # Here's come the filter options. With this rules, we can set which
>>>>>>>> # message go where.
>>>>>>>>
>>>>>>>> filter f_dbg { level(debug); };
>>>>>>>> filter f_info { level(info); };
>>>>>>>> filter f_notice { level(notice); };
>>>>>>>> filter f_warn { level(warn); };
>>>>>>>> filter f_err { level(err); };
>>>>>>>> filter f_crit { level(crit .. emerg); };
>>>>>>>> filter f_debug { level(debug) and not facility(auth, authpriv,
>>>>>>>> news, mail); };
>>>>>>>> filter f_error { level(err .. emerg) ; };
>>>>>>>> filter f_auth { facility(auth, authpriv) and not filter(f_debug); };
>>>>>>>> filter f_cron { facility(cron) and not filter(f_debug); };
>>>>>>>> filter f_daemon { facility(daemon) and not filter(f_debug); };
>>>>>>>> filter f_kern { facility(kern) and not filter(f_debug); };
>>>>>>>> filter f_local { facility(local0, local1, local3, local4, local5,
>>>>>>>> local6, local7) and not filter(f_debug); };
>>>>>>>> filter f_mail { facility(mail) and not filter(f_debug); };
>>>>>>>> filter f_syslog3 { not facility(auth, authpriv, mail) and not
>>>>>>>> filter(f_debug); };
>>>>>>>>
>>>>>>>> ########################
>>>>>>>> # Log paths
>>>>>>>> ########################
>>>>>>>> log { source(s_src); filter(f_auth); destination(d_auth); };
>>>>>>>> log { source(s_src); filter(f_cron); destination(d_cron); };
>>>>>>>> log { source(s_src); filter(f_daemon); destination(d_daemon); };
>>>>>>>> log { source(s_src); filter(f_kern); destination(d_kern); };
>>>>>>>> log { source(s_src); filter(f_syslog3); destination(d_syslog); };
>>>>>>>> log { source(s_src); filter(f_mail); destination(d_mail); };
>>>>>>>> log { source(s_src); filter(f_debug); destination(d_debug); };
>>>>>>>> log { source(s_src); filter(f_error); destination(d_error); };
>>>>>>>>
>>>>>>>>
>>>>>>>> # All messages send to a remote site
>>>>>>>> #
>>>>>>>> log { source(s_src); destination(d_scl3); };
>>>>>>>>
>>>>>>>> ###
>>>>>>>> # Include all config files in /etc/syslog-ng/conf.d/
>>>>>>>> ###
>>>>>>>> @include "/etc/syslog-ng/conf.d/*.conf"
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> There's another file, amqp.conf where we actually read and ship
>>>>>>>> those Bro logs.
>>>>>>>>
>>>>>>>>
>>>>>>>> source bro_conn {
>>>>>>>> file( "/nsm/bro/logs/current/conn.log" flags(no-parse));
>>>>>>>> };
>>>>>>>>
>>>>>>>> source bro_ssl {
>>>>>>>> file( "/nsm/bro/logs/current/ssl.log" flags(no-parse));
>>>>>>>> };
>>>>>>>>
>>>>>>>> source bro_dns {
>>>>>>>> file( "/nsm/bro/logs/current/dns.log" flags(no-parse));
>>>>>>>> };
>>>>>>>>
>>>>>>>> source bro_smtp {
>>>>>>>> file( "/nsm/bro/logs/current/smtp.log" flags(no-parse));
>>>>>>>> };
>>>>>>>>
>>>>>>>> source bro_ssh {
>>>>>>>> file( "/nsm/bro/logs/current/ssh.log" flags(no-parse));
>>>>>>>> };
>>>>>>>>
>>>>>>>> source bro_notice {
>>>>>>>> file( "/nsm/bro/logs/current/notice.log" flags(no-parse));
>>>>>>>> };
>>>>>>>>
>>>>>>>> source bro_intel {
>>>>>>>> file( "/nsm/bro/logs/current/intel.log" flags(no-parse));
>>>>>>>> };
>>>>>>>>
>>>>>>>> source bro_dce_rpc {
>>>>>>>> file( "/nsm/bro/logs/current/dce_rpc.log" flags(no-parse));
>>>>>>>> };
>>>>>>>>
>>>>>>>> source bro_dhcp {
>>>>>>>> file( "/nsm/bro/logs/current/dhcp.log" flags(no-parse));
>>>>>>>> };
>>>>>>>>
>>>>>>>> source bro_files {
>>>>>>>> file( "/nsm/bro/logs/current/files.log" flags(no-parse));
>>>>>>>> };
>>>>>>>>
>>>>>>>> source bro_kerberos {
>>>>>>>> file( "/nsm/bro/logs/current/kerberos.log" flags(no-parse));
>>>>>>>> };
>>>>>>>>
>>>>>>>> source bro_http {
>>>>>>>> file( "/nsm/bro/logs/current/http.log" flags(no-parse));
>>>>>>>> };
>>>>>>>>
>>>>>>>> source bro_software {
>>>>>>>> file( "/nsm/bro/logs/current/software.log" flags(no-parse));
>>>>>>>> };
>>>>>>>>
>>>>>>>> source bro_snmp {
>>>>>>>> file( "/nsm/bro/logs/current/snmp.log" flags(no-parse));
>>>>>>>> };
>>>>>>>>
>>>>>>>> source bro_socks {
>>>>>>>> file( "/nsm/bro/logs/current/socks.log" flags(no-parse));
>>>>>>>> };
>>>>>>>>
>>>>>>>> source bro_tunnel {
>>>>>>>> file( "/nsm/bro/logs/current/tunnel.log" flags(no-parse));
>>>>>>>> };
>>>>>>>>
>>>>>>>> source bro_ntlm {
>>>>>>>> file( "/nsm/bro/logs/current/ntlm.log" flags(no-parse));
>>>>>>>> };
>>>>>>>>
>>>>>>>> source bro_pe {
>>>>>>>> file( "/nsm/bro/logs/current/pe.log" flags(no-parse));
>>>>>>>> };
>>>>>>>>
>>>>>>>> source bro_sip {
>>>>>>>> file( "/nsm/bro/logs/current/sip.log" flags(no-parse));
>>>>>>>> };
>>>>>>>>
>>>>>>>> source bro_smb_files {
>>>>>>>> file( "/nsm/bro/logs/current/smb_files.log" flags(no-parse));
>>>>>>>> };
>>>>>>>>
>>>>>>>> source bro_smb_mapping {
>>>>>>>> file( "/nsm/bro/logs/current/smb_mapping.log" flags(no-parse));
>>>>>>>> };
>>>>>>>>
>>>>>>>> source bro_x509 {
>>>>>>>> file( "/nsm/bro/logs/current/x509.log" flags(no-parse));
>>>>>>>> };
>>>>>>>>
>>>>>>>> source bro_known_certs {
>>>>>>>> file( "/nsm/bro/logs/current/known_certs.log" flags(no-parse));
>>>>>>>> };
>>>>>>>>
>>>>>>>> source bro_known_devices {
>>>>>>>> file( "/nsm/bro/logs/current/known_devices.log"
>>>>>>>> flags(no-parse));
>>>>>>>> };
>>>>>>>>
>>>>>>>> source bro_known_hosts {
>>>>>>>> file( "/nsm/bro/logs/current/known_hosts.log" flags(no-parse));
>>>>>>>> };
>>>>>>>>
>>>>>>>> source bro_known_services {
>>>>>>>> file( "/nsm/bro/logs/current/known_services.log"
>>>>>>>> flags(no-parse));
>>>>>>>> };
>>>>>>>>
>>>>>>>>
>>>>>>>> destination d_amqp {
>>>>>>>> amqp(
>>>>>>>> vhost("nsm")
>>>>>>>> host("<our happy rabbit>
>>>>>>>> <http://syslog-proxy1.dmz.mdc1.mozilla.com>")
>>>>>>>> port(5672)
>>>>>>>> exchange("eventtask")
>>>>>>>> exchange-type("direct")
>>>>>>>> routing-key("eventtask")
>>>>>>>> body("$(format-json --scope nv_pairs --pair
>>>>>>>> category=\"bro\" --pair source=$source --pair customendpoint=\" \" --pair
>>>>>>>> tags=\"bro\")")
>>>>>>>> persistent(yes)
>>>>>>>> username("USERNAME")
>>>>>>>> password("PASSWORD")
>>>>>>>> );
>>>>>>>> };
>>>>>>>>
>>>>>>>>
>>>>>>>> parser p_json { json-parser(); };
>>>>>>>>
>>>>>>>>
>>>>>>>> log { source(bro_conn); parser(p_json); destination(d_amqp); };
>>>>>>>> log { source(bro_http); parser(p_json); destination(d_amqp); };
>>>>>>>> log { source(bro_ssl); parser(p_json); destination(d_amqp); };
>>>>>>>> log { source(bro_dns); parser(p_json); destination(d_amqp); };
>>>>>>>> log { source(bro_smtp); parser(p_json); destination(d_amqp); };
>>>>>>>> log { source(bro_ssh); parser(p_json); destination(d_amqp); };
>>>>>>>> log { source(bro_intel); parser(p_json); destination(d_amqp); };
>>>>>>>> log { source(bro_notice); parser(p_json); destination(d_amqp); };
>>>>>>>> log { source(bro_dce_rpc); parser(p_json); destination(d_amqp);
>>>>>>>> flags(flow-control); };
>>>>>>>> log { source(bro_dhcp); parser(p_json); destination(d_amqp); };
>>>>>>>> log { source(bro_files); parser(p_json); destination(d_amqp);
>>>>>>>> flags(flow-control); };
>>>>>>>> log { source(bro_kerberos); parser(p_json); destination(d_amqp);
>>>>>>>> flags(flow-control); };
>>>>>>>> log { source(bro_software); parser(p_json); destination(d_amqp);
>>>>>>>> flags(flow-control); };
>>>>>>>> log { source(bro_snmp); parser(p_json); destination(d_amqp);
>>>>>>>> flags(flow-control); };
>>>>>>>> log { source(bro_socks); parser(p_json); destination(d_amqp);
>>>>>>>> flags(flow-control); };
>>>>>>>> log { source(bro_tunnel); parser(p_json); destination(d_amqp);
>>>>>>>> flags(flow-control); };
>>>>>>>> log { source(bro_ntlm); parser(p_json); destination(d_amqp);
>>>>>>>> flags(flow-control); };
>>>>>>>> log { source(bro_pe); parser(p_json); destination(d_amqp);
>>>>>>>> flags(flow-control); };
>>>>>>>> log { source(bro_sip); parser(p_json); destination(d_amqp);
>>>>>>>> flags(flow-control); };
>>>>>>>> log { source(bro_smb_files); parser(p_json); destination(d_amqp);
>>>>>>>> flags(flow-control); };
>>>>>>>> log { source(bro_smb_mapping); parser(p_json); destination(d_amqp);
>>>>>>>> flags(flow-control); };
>>>>>>>> log { source(bro_x509); parser(p_json); destination(d_amqp);
>>>>>>>> flags(flow-control); };
>>>>>>>> log { source(bro_known_certs); parser(p_json); destination(d_amqp);
>>>>>>>> flags(flow-control); };
>>>>>>>> log { source(bro_known_devices); parser(p_json);
>>>>>>>> destination(d_amqp); flags(flow-control); };
>>>>>>>> log { source(bro_known_hosts); parser(p_json); destination(d_amqp);
>>>>>>>> flags(flow-control); };
>>>>>>>> log { source(bro_known_services); parser(p_json);
>>>>>>>> destination(d_amqp); flags(flow-control); };
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *Statistics*
>>>>>>>>
>>>>>>>>
>>>>>>>> Mar 15 *00:17:30* nsmserver syslog-ng[11278]: Log statistics;
>>>>>>>> processed='source(bro_conn)=112360513',
>>>>>>>> processed='source(s_src)=227349', processed='source(bro_known_devices)=3791',
>>>>>>>> processed='global(sdata_updates)=0', processed='center(received)=310790955',
>>>>>>>> processed='source(bro_ssh)=622441', processed='source(bro_smb_files)=5815964',
>>>>>>>> processed='source(bro_socks)=0', processed='destination(d_daemon)=21',
>>>>>>>> dropped='dst.amqp(d_amqp#0,amqp,nsm,happyrabbit,5672,eventtask,direct)=2',
>>>>>>>> processed='dst.amqp(d_amqp#0,amqp,nsm,happyrabbit,5672,event
>>>>>>>> task,direct)=310563565 <(31)%20056%203565>',
>>>>>>>> queued='dst.amqp(d_amqp#0,amqp,nsm,happyrabbit,5672,eventtask,direct)=0',
>>>>>>>> processed='destination(d_error)=189386',
>>>>>>>> processed='destination(d_syslog)=207595',
>>>>>>>> processed='source(bro_ssl)=49788364',
>>>>>>>> processed='source(bro_kerberos)=133177',
>>>>>>>> processed='source(bro_dhcp)=69970', processed='destination(d_mail)=0',
>>>>>>>> processed='source(bro_http)=60085539',
>>>>>>>> processed='global(msg_clones)=1576', processed='destination(d_amqp)
>>>>>>>> =310563565 <(31)%20056%203565>', processed='destination(d_kern)=146',
>>>>>>>> processed='source(bro_tunnel)=520921',
>>>>>>>> processed='source(bro_software)=18851236 <(1)%20885%201236>',
>>>>>>>> processed='source(bro_known_services)=13403',
>>>>>>>> processed='source(bro_known_certs)=2070',
>>>>>>>> processed='source(bro_dce_rpc)=501875',
>>>>>>>> processed='destination(d_scl3)=227349',
>>>>>>>> processed='source(bro_known_hosts)=14604',
>>>>>>>> processed='source(bro_smb_mapping)=116412',
>>>>>>>> processed='source(bro_files)=15152100',
>>>>>>>> processed='center(queued)=311210449',
>>>>>>>> processed='destination(d_debug)=10280',
>>>>>>>> processed='src.internal(s_src#2)=26785',
>>>>>>>> stamp='src.internal(s_src#2)=1521073048',
>>>>>>>> processed='source(bro_ntlm)=16823', processed='destination(d_auth)=9474',
>>>>>>>> processed='global(internal_queue_length)=0',
>>>>>>>> processed='source(bro_smtp)=1067448',
>>>>>>>> dropped='dst.udp(d_scl3#0,udp,syslog1.private.scl3.mozilla.com:514)=0',
>>>>>>>> processed='dst.udp(d_scl3#0,udp,syslog1.private.scl3.mozilla.com:514)=227349',
>>>>>>>> queued='dst.udp(d_scl3#0,udp,syslog1.private.scl3.mozilla.com:514)=221705',
>>>>>>>> written='dst.udp(d_scl3#0,udp,syslog1.private.scl3.mozilla.com:514)=5644',
>>>>>>>> processed='global(payload_reallocs)=310467149 <(31)%20046%207149>',
>>>>>>>> queued='global(scratch_buffers_count)=17875655781170',
>>>>>>>> processed='destination(d_cron)=2633',
>>>>>>>> processed='source(bro_snmp)=9854568',
>>>>>>>> processed='source(bro_notice)=119415',
>>>>>>>> processed='source(bro_dns)=19303431',
>>>>>>>> processed='source(bro_sip)=97822', processed='source(bro_intel)=26969',
>>>>>>>> processed='source(bro_pe)=531103', processed='source(bro_x509)=15493647',
>>>>>>>> queued='global(scratch_buffers_bytes)=2304'
>>>>>>>>
>>>>>>>>
>>>>>>>> Mar 15 *00:27:30* nsmserver
>>>>>>>> <http://nsmserver1.private.scl3.mozilla.com> syslog-ng[11278]: Log
>>>>>>>> statistics; processed='source(bro_conn)=112988941',
>>>>>>>> processed='source(s_src)=228651', processed='source(bro_known_devices)=3791',
>>>>>>>> processed='global(sdata_updates)=0', processed='center(received)=312659144',
>>>>>>>> processed='source(bro_ssh)=627013', processed='source(bro_smb_files)=5863697',
>>>>>>>> processed='source(bro_socks)=0', processed='destination(d_daemon)=21',
>>>>>>>> dropped='dst.amqp(d_amqp#0,amqp,nsm,happyrabbit
>>>>>>>> <http://syslog-proxy1.dmz.mdc1.mozilla.com>,5672,eventtask,direct)=2',
>>>>>>>> processed='dst.amqp(d_amqp#0,amqp,nsm,happyrabbit
>>>>>>>> <http://syslog-proxy1.dmz.mdc1.mozilla.com>,5672,eventtask,direct)=
>>>>>>>> 312430452 <(31)%20243%200452>', queued='dst.amqp(d_amqp#0,amqp
>>>>>>>> ,nsm,happyrabbit <http://syslog-proxy1.dmz.mdc1.mozilla.com>
>>>>>>>> ,5672,eventtask,direct)=0', processed='destination(d_error)=190429',
>>>>>>>> processed='destination(d_syslog)=208759',
>>>>>>>> processed='source(bro_ssl)=50077572',
>>>>>>>> processed='source(bro_kerberos)=134215',
>>>>>>>> processed='source(bro_dhcp)=70487', processed='destination(d_mail)=0',
>>>>>>>> processed='source(bro_http)=60446166',
>>>>>>>> processed='global(msg_clones)=1594', processed='destination(d_amqp)
>>>>>>>> =312430452 <(31)%20243%200452>', processed='destination(d_kern)=146',
>>>>>>>> processed='source(bro_tunnel)=524450',
>>>>>>>> processed='source(bro_software)=18938552 <(1)%20893%208552>',
>>>>>>>> processed='source(bro_known_services)=13532',
>>>>>>>> processed='source(bro_known_certs)=2073',
>>>>>>>> processed='source(bro_dce_rpc)=505206',
>>>>>>>> processed='destination(d_scl3)=228651',
>>>>>>>> processed='source(bro_known_hosts)=14630',
>>>>>>>> processed='source(bro_smb_mapping)=117177',
>>>>>>>> processed='source(bro_files)=15252368',
>>>>>>>> processed='center(queued)=313080999',
>>>>>>>> processed='destination(d_debug)=10352',
>>>>>>>> processed='src.internal(s_src#2)=26966',
>>>>>>>> stamp='src.internal(s_src#2)=1521073648',
>>>>>>>> processed='source(bro_ntlm)=16848', processed='destination(d_auth)=9540',
>>>>>>>> processed='global(internal_queue_length)=0',
>>>>>>>> processed='source(bro_smtp)=1074012',
>>>>>>>> dropped='dst.udp(d_scl3#0,udp,syslog1.private.scl3.mozilla.com:514)=0',
>>>>>>>> processed='dst.udp(d_scl3#0,udp,syslog1.private.scl3.mozilla.com:514)=228651',
>>>>>>>> queued='dst.udp(d_scl3#0,udp,syslog1.private.scl3.mozilla.com:514)=223007',
>>>>>>>> written='dst.udp(d_scl3#0,udp,syslog1.private.scl3.mozilla.com:514)=5644',
>>>>>>>> processed='global(payload_reallocs)=312333723 <(31)%20233%203723>',
>>>>>>>> queued='global(scratch_buffers_count)=17970145061685',
>>>>>>>> processed='destination(d_cron)=2649',
>>>>>>>> processed='source(bro_snmp)=9917302',
>>>>>>>> processed='source(bro_notice)=120140',
>>>>>>>> processed='source(bro_dns)=19462256',
>>>>>>>> processed='source(bro_sip)=98565', processed='source(bro_intel)=27061',
>>>>>>>> processed='source(bro_pe)=535753', processed='source(bro_x509)=15598686',
>>>>>>>> queued='global(scratch_buffers_bytes)=2304'
>>>>>>>>
>>>>>>>>
>>>>>>>> ____________________________________________________________
>>>>>>>> __________________
>>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>>> Documentation: http://www.balabit.com/support
>>>>>>>> /documentation/?product=syslog-ng
>>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> ____________________________________________________________
>>>>>>> __________________
>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>> Documentation: http://www.balabit.com/support
>>>>>>> /documentation/?product=syslog-ng
>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ____________________________________________________________
>>>>>> __________________
>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>> Documentation: http://www.balabit.com/support
>>>>>> /documentation/?product=syslog-ng
>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> ____________________________________________________________
>>>>> __________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation: http://www.balabit.com/support
>>>>> /documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>>>>
>>>>
>>>> ____________________________________________________________
>>>> __________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation: http://www.balabit.com/support
>>>> /documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>>
>>>
>>> ____________________________________________________________
>>> __________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support
>>> /documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180323/7f77bce3/attachment-0001.html>
More information about the syslog-ng
mailing list