[syslog-ng] Syslog-ng shipping logs through AMQP with huge memory leaks
Michal Purzynski
michal at mozilla.com
Wed Mar 21 18:16:54 UTC 2018
The CPU usage is now between 40-60%, usually somewhere around 45% so it
went down indeed. It spikes every few seconds to 55% then goes down, etc.
What worries me is that we're already at 21GB of RES so the memory usage
grows and that's all syslog-ng's private data.
Are those messages that aren't processed? TBH I believe all messages are
getting to Rabbit on time, I can see what I'm expecting in ES that's
pulling from that Rabbit.
On Tue, Mar 20, 2018 at 10:49 PM, Michal Purzynski <michal at mozilla.com>
wrote:
> cat /proc/`pidof -s syslog-ng`/maps | egrep -i libjemalloc
> 7f0d97978000-7f0d979a8000 r-xp 00000000 08:02 25771415575
> /usr/lib/x86_64-linux-gnu/libjemalloc.so.1
> 7f0d979a8000-7f0d97ba8000 ---p 00030000 08:02 25771415575
> /usr/lib/x86_64-linux-gnu/libjemalloc.so.1
> 7f0d97ba8000-7f0d97baa000 r--p 00030000 08:02 25771415575
> /usr/lib/x86_64-linux-gnu/libjemalloc.so.1
> 7f0d97baa000-7f0d97bab000 rw-p 00032000 08:02 25771415575
> /usr/lib/x86_64-linux-gnu/libjemalloc.so.1
>
>
> jemalloc loaded. Let's see how that helps. I also have tcmalloc up my
> sleeve.
>
> On Tue, Mar 20, 2018 at 9:06 PM, Scheidler, Balázs <
> balazs.scheidler at balabit.com> wrote:
>
>> Probably the biggest toll is the json-parser(), which would allocate
>> memory when we are parsing the json. On the output side we have a
>> handcrafted json generator that doesnt allocate memory, but on the inbound
>> side libjson-c is doing it.
>>
>> That's probably the reason behind the futex numbers, malloc() uses a
>> number of mutexes, which might get contended.
>>
>> Jemalloc would probably help a lot.
>>
>> --
>> Bazsi
>>
>> On Wed, Mar 21, 2018 at 2:51 AM, Michal Purzynski <michal at mozilla.com>
>> wrote:
>>
>>> [ perf record: Captured and wrote 54.230 MB perf.data (1418800 samples) ]
>>>
>>> Eyeballing looks like syslog-ng spends tons of time in malloc,
>>> allocating and deallocating memory. Maybe using gperf / jemalloc could help
>>> here?
>>>
>>> Let me know if you want entire file. This is without call-graph, with
>>> call-graph... hmmm, looks like I will have to rebuild syslog-ng with
>>> symbols. Let's trace that leaks first ;-)
>>>
>>> 16.77% syslog-ng [kernel.kallsyms] [k]
>>> update_blocked_averages
>>> 16.37% syslog-ng libpthread-2.19.so [.] pthread_mutex_lock
>>> 11.84% syslog-ng [kernel.kallsyms] [k] audit_filter_syscall
>>> 8.09% syslog-ng [kernel.kallsyms] [k] copy_page
>>> 7.50% syslog-ng [kernel.kallsyms] [k]
>>> syscall_return_via_sysret
>>> 6.32% syslog-ng libc-2.19.so [.] _int_free
>>> 5.13% syslog-ng libglib-2.0.so.0.4002.0 [.] 0x0000000000073487
>>> 4.14% syslog-ng libc-2.19.so [.] _IO_vfscanf
>>> 3.75% syslog-ng libivykis.so.0.5.4 [.]
>>> pthread_mutex_unlock at plt
>>> 3.35% syslog-ng libsyslog-ng-3.14.so.0.0.0 [.] 0x0000000000026d02
>>> 2.97% syslog-ng libc-2.19.so [.] _int_malloc
>>> 2.96% syslog-ng libglib-2.0.so.0.4002.0 [.] 0x000000000008a617
>>> 2.56% syslog-ng libglib-2.0.so.0.4002.0 [.] 0x000000000008a61a
>>> 2.37% syslog-ng libc-2.19.so [.]
>>> __memmove_ssse3_back
>>> 1.97% syslog-ng libjson-c.so.2.0.0 [.] lh_char_hash
>>> 1.97% syslog-ng libsyslog-ng-3.14.so.0.0.0 [.] 0x0000000000036410
>>> 0.32% syslog-ng libc-2.19.so [.] _IO_setb
>>> 0.18% syslog-ng libc-2.19.so [.] malloc_consolidate
>>> 0.17% syslog-ng libc-2.19.so [.] __strchrnul
>>> 0.17% syslog-ng libglib-2.0.so.0.4002.0 [.]
>>> g_string_append_printf
>>> 0.17% syslog-ng [kernel.kallsyms] [k] reschedule_interrupt
>>> 0.17% syslog-ng libglib-2.0.so.0.4002.0 [.] 0x00000000000735a1
>>> 0.17% syslog-ng libglib-2.0.so.0.4002.0 [.] 0x000000000008a8a6
>>> 0.17% syslog-ng libsyslog-ng-3.14.so.0.0.0 [.] 0x000000000006df70
>>> 0.15% syslog-ng libc-2.19.so [.]
>>> __memcpy_sse2_unaligned
>>> 0.15% syslog-ng libglib-2.0.so.0.4002.0 [.] g_string_assign
>>> 0.01% syslog-ng libc-2.19.so [.] malloc
>>> 0.00% syslog-ng libjson-c.so.2.0.0 [.]
>>> json_tokener_parse_ex
>>> 0.00% syslog-ng libc-2.19.so [.] vfprintf
>>> 0.00% syslog-ng libglib-2.0.so.0.4002.0 [.]
>>> g_utf8_get_char_validated
>>> 0.00% syslog-ng libglib-2.0.so.0.4002.0 [.] 0x000000000008a8a7
>>> 0.00% syslog-ng libc-2.19.so [.] free
>>> 0.00% syslog-ng libc-2.19.so [.]
>>> __strcmp_sse2_unaligned
>>> 0.00% syslog-ng libglib-2.0.so.0.4002.0 [.] g_string_truncate
>>>
>>>
>>>
>>>
>>> With cal graph, like I said, only the library name tell you something
>>> here, I can test later with syslog-ng with symbols
>>>
>>>
>>> + 59.22% 0.00% syslog-ng libivykis.so.0.5.4 [.]
>>> 0xffff80b3f3309c65 ▒
>>> + 56.85% 0.00% syslog-ng libc-2.19.so [.]
>>> 0xffff80b3f259f6d3 ▒
>>> + 38.22% 0.00% syslog-ng [unknown] [.]
>>> 0x0000000000000029 ▒
>>> + 29.22% 29.22% syslog-ng libc-2.19.so [.]
>>> 0x00000000000fe6d3 ▒
>>> + 18.46% 0.00% syslog-ng [kernel.kallsyms] [k]
>>> entry_SYSCALL_64_fastpath ▒
>>> + 18.32% 0.00% syslog-ng [kernel.kallsyms] [k]
>>> sys_epoll_wait ▒
>>> + 13.27% 9.00% syslog-ng [kernel.kallsyms] [k]
>>> __fget_light ▒
>>> + 9.32% 0.00% syslog-ng [kernel.kallsyms] [k]
>>> int_ret_from_sys_call ▒
>>> + 9.32% 0.00% syslog-ng [kernel.kallsyms] [k]
>>> syscall_return_slowpath ▒
>>> + 9.32% 0.00% syslog-ng [kernel.kallsyms] [k]
>>> syscall_slow_exit_work ▒
>>> + 9.32% 0.00% syslog-ng [unknown] [.]
>>> 0x000000000000002d ▒
>>> + 9.32% 9.32% syslog-ng [kernel.kallsyms] [k]
>>> unroll_tree_refs ▒
>>> + 7.71% 3.29% syslog-ng libc-2.19.so [.]
>>> _int_malloc ▒
>>> + 7.27% 7.27% syslog-ng libjson-c.so.2.0.0 [.]
>>> lh_char_hash ▒
>>> + 6.79% 0.00% syslog-ng [kernel.kallsyms] [k]
>>> apic_timer_interrupt ▒
>>> + 6.79% 0.00% syslog-ng [kernel.kallsyms] [k]
>>> smp_apic_timer_interrupt ▒
>>> + 6.63% 0.00% syslog-ng [unknown] [k]
>>> 0x000000000000002a ▒
>>> + 5.05% 0.00% syslog-ng [kernel.kallsyms] [k]
>>> ep_poll ▒
>>> + 5.05% 0.00% syslog-ng [kernel.kallsyms] [k]
>>> schedule_hrtimeout_range ▒
>>> + 5.05% 0.00% syslog-ng [kernel.kallsyms] [k]
>>> schedule_hrtimeout_range_clock ▒
>>> + 5.05% 0.00% syslog-ng [kernel.kallsyms] [k]
>>> schedule ▒
>>> + 5.05% 0.00% syslog-ng [kernel.kallsyms] [k]
>>> __schedule ▒
>>> + 5.05% 0.00% syslog-ng [unknown] [k]
>>> 0x000000000000002b ▒
>>> + 5.05% 0.00% syslog-ng [kernel.kallsyms] [k]
>>> deactivate_task ▒
>>> + 5.05% 0.00% syslog-ng [kernel.kallsyms] [k]
>>> dequeue_task_fair ▒
>>> + 5.05% 5.05% syslog-ng [kernel.kallsyms] [k]
>>> account_entity_dequeue ▒
>>> + 4.58% 4.58% syslog-ng libc-2.19.so [.]
>>> __ctype_b_loc ▒
>>> + 4.58% 0.00% syslog-ng [unknown] [.]
>>> 0x00007f4bfc4121a0 ▒
>>> + 4.42% 0.00% syslog-ng [kernel.kallsyms] [k]
>>> irq_exit ▒
>>> + 4.42% 4.42% syslog-ng [kernel.kallsyms] [k]
>>> __do_softirq ▒
>>> + 4.42% 0.00% syslog-ng [unknown] [.]
>>> 0x0000000002579bb0 ▒
>>> + 4.42% 0.00% syslog-ng libglib-2.0.so.0.4002.0 [.]
>>> g_static_mutex_get_mutex_impl ▒
>>> + 4.42% 4.42% syslog-ng libsyslog-ng-3.14.so.0.0.0 [.]
>>> log_msg_set_value ▒
>>> + 4.42% 4.42% syslog-ng libglib-2.0.so.0.4002.0 [.]
>>> g_ptr_array_free ▒
>>>
>>>
>>> On Tue, Mar 20, 2018 at 5:57 PM, Scheidler, Balázs <
>>> balazs.scheidler at balabit.com> wrote:
>>>
>>>> 126% CPU usage? that would be great to know the details there, although
>>>> I understand that the memory is more of a concern now. :)
>>>>
>>>> can you run a perf record on that process, perhaps once the memory
>>>> issue is solved? I have my suspicion where it is spending its time, but it
>>>> would be great to confirm. (my guess is value-pairs while formatting json
>>>> messages).
>>>>
>>>> cheers,
>>>> --
>>>> Bazsi
>>>>
>>>> On Tue, Mar 20, 2018 at 8:26 PM, Michal Purzynski <michal at mozilla.com>
>>>> wrote:
>>>>
>>>>> Hello Gábor!
>>>>>
>>>>> Answers inline.
>>>>>
>>>>> On Mon, Mar 19, 2018 at 9:09 AM, Nagy, Gábor <gabor.nagy at balabit.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> - have you built syslog-ng from source or you downloaded a package?
>>>>>>
>>>>>
>>>>> It's a package from
>>>>>
>>>>> deb http://download.opensuse.org/repositories/home:/laszlo_budai
>>>>> :/syslog-ng/xUbuntu_14.04 ./
>>>>>
>>>>>
>>>>> - have you tried to reproduce the issue in a different environment
>>>>>> with a minimal config? We are using your configuration, but if you narrowed
>>>>>> down the problem it would be helpful.
>>>>>>
>>>>>
>>>>> If I disable the Bro via AMQP the problem goes away, so that must be
>>>>> something there :/
>>>>>
>>>>>
>>>>> - we were experimenting with a very simple JSON message, can you show
>>>>>> us an example log to see the complexity of it, please? We are thinking to
>>>>>> check Bro out for log message structure.
>>>>>>
>>>>>
>>>>> Example logs follow sent to Peter via a private channel, making them
>>>>> public would be kind of difficult.
>>>>>
>>>>> Appreciate you looking into it! And BTW, I just restarted syslog-ng on
>>>>> the most busy server
>>>>>
>>>>> 14910 root 20 0 59.899g 0.057t 3784 S 126.4 92.9 1166:22 syslog-ng
>>>>>
>>>>> ;-)
>>>>>
>>>>>
>>>>>>
>>>>>> I saw that you have gave a huge log-fetch-limit() in the global
>>>>>> config compared to the default. Setting log-fetch-limit() in global config
>>>>>> is deprecated, you need to set it up per source.
>>>>>>
>>>>>> We have a couple of ideas and will continue to try reproducing the
>>>>>> memleak you reported.
>>>>>>
>>>>>> Regards,
>>>>>> Gabor
>>>>>>
>>>>>> On Sat, Mar 17, 2018 at 12:45 AM, Michal Purzynski <
>>>>>> michal at mozilla.com> wrote:
>>>>>>
>>>>>>> Hello!!
>>>>>>>
>>>>>>> Could you help us troubleshoot a memory leak or a misconfiguration
>>>>>>> that makes the syslog-ng process memory usage grow? Like, a lot.
>>>>>>>
>>>>>>> We use syslog-ng to read some JSON files and ship them to a RabbitMQ
>>>>>>> server via AMQP. As you can see, this is just a client, it does not accept
>>>>>>> connections from other systems, it works mostly with Bro logs (plus it
>>>>>>> handles a local syslog because that's convenient).
>>>>>>>
>>>>>>> We have around 6000 events per second on this server. What's
>>>>>>> interesting, syslog-ng's memory grows quickly without flow control (and
>>>>>>> slower, but it still continues to grow with it). I'll switch that to TLS
>>>>>>> soon, a more secure configuration is ready to be deployed.
>>>>>>>
>>>>>>> Things look pretty good on the RabbitMQ side. That server is not
>>>>>>> under pressure and handles the load just fine, the queue is consumed,
>>>>>>> there's nothing building up that would make me believe we have RabbitMQ
>>>>>>> server overloaded.
>>>>>>>
>>>>>>> How much does syslog-ng grow?
>>>>>>>
>>>>>>> I'd say - if I disable flow-control it will eat 55GB of RAM in less
>>>>>>> than 24h, if not faster. With flow-control enabled on the most 'busy' files
>>>>>>> things are way better, but the memory usage still keep growing -
>>>>>>>
>>>>>>> syslog-ng.conf looks like below - BTW that's Ubuntu 14.04 LTS,
>>>>>>> 3.14.1-3 of syslog-ng
>>>>>>>
>>>>>>> Let me know what other data you might need.
>>>>>>>
>>>>>>> @version: 3.14
>>>>>>> @include "scl.conf"
>>>>>>>
>>>>>>> # Syslog-ng configuration file, compatible with default Debian
>>>>>>> syslogd
>>>>>>> # installation.
>>>>>>>
>>>>>>> # First, set some global options.
>>>>>>> options {
>>>>>>> threaded (yes);
>>>>>>> flush_lines (50000);
>>>>>>> flush_timeout (1000);
>>>>>>> time_reopen (10);
>>>>>>> log_fetch_limit (50000);
>>>>>>> log_fifo_size (500000);
>>>>>>> use_dns (yes);
>>>>>>> dns_cache (5000);
>>>>>>> dns_cache_expire(87600);
>>>>>>> use_fqdn (yes);
>>>>>>> owner("root");
>>>>>>> group("adm");
>>>>>>> perm(0640);
>>>>>>> keep_hostname (yes);
>>>>>>> chain_hostnames (off);
>>>>>>> };
>>>>>>>
>>>>>>> ########################
>>>>>>> # Sources
>>>>>>> ########################
>>>>>>> # This is the default behavior of sysklogd package
>>>>>>> # Logs may come from unix stream, but not from another machine.
>>>>>>> #
>>>>>>> source s_src {
>>>>>>> system();
>>>>>>> internal();
>>>>>>> };
>>>>>>>
>>>>>>> # If you wish to get logs from remote machine you should uncomment
>>>>>>> # this and comment the above source line.
>>>>>>> #
>>>>>>> #source s_net { tcp(ip(127.0.0.1) port(1000)); };
>>>>>>>
>>>>>>> ########################
>>>>>>> # Destinations
>>>>>>> ########################
>>>>>>> # First some standard logfile
>>>>>>> #
>>>>>>> destination d_auth { file("/var/log/auth.log"); };
>>>>>>> destination d_cron { file("/var/log/cron.log"); };
>>>>>>> destination d_daemon { file("/var/log/daemon.log"); };
>>>>>>> destination d_kern { file("/var/log/kern.log"); };
>>>>>>> destination d_mail { file("/var/log/mail.log"); };
>>>>>>> destination d_syslog { file("/var/log/syslog"); };
>>>>>>>
>>>>>>> # This files are the log come from the mail subsystem.
>>>>>>> #
>>>>>>> #destination d_mailinfo { file("/var/log/mail.info"); };
>>>>>>> #destination d_mailwarn { file("/var/log/mail.warn"); };
>>>>>>> #destination d_mailerr { file("/var/log/mail.err"); };
>>>>>>>
>>>>>>> # Logging for INN news system
>>>>>>> #
>>>>>>> #destination d_newscrit { file("/var/log/news/news.crit"); };
>>>>>>> #destination d_newserr { file("/var/log/news/news.err"); };
>>>>>>> #destination d_newsnotice { file("/var/log/news/news.notice"); };
>>>>>>>
>>>>>>> # Some 'catch-all' logfiles.
>>>>>>> #
>>>>>>> destination d_debug { file("/var/log/debug"); };
>>>>>>> destination d_error { file("/var/log/error"); };
>>>>>>>
>>>>>>> # Syslog1 in SCL3
>>>>>>> destination d_scl3 {
>>>>>>> udp("syslog1.private.scl3.mozilla.com" port(514));
>>>>>>> };
>>>>>>>
>>>>>>> ########################
>>>>>>> # Filters
>>>>>>> ########################
>>>>>>> # Here's come the filter options. With this rules, we can set which
>>>>>>> # message go where.
>>>>>>>
>>>>>>> filter f_dbg { level(debug); };
>>>>>>> filter f_info { level(info); };
>>>>>>> filter f_notice { level(notice); };
>>>>>>> filter f_warn { level(warn); };
>>>>>>> filter f_err { level(err); };
>>>>>>> filter f_crit { level(crit .. emerg); };
>>>>>>> filter f_debug { level(debug) and not facility(auth, authpriv, news,
>>>>>>> mail); };
>>>>>>> filter f_error { level(err .. emerg) ; };
>>>>>>> filter f_auth { facility(auth, authpriv) and not filter(f_debug); };
>>>>>>> filter f_cron { facility(cron) and not filter(f_debug); };
>>>>>>> filter f_daemon { facility(daemon) and not filter(f_debug); };
>>>>>>> filter f_kern { facility(kern) and not filter(f_debug); };
>>>>>>> filter f_local { facility(local0, local1, local3, local4, local5,
>>>>>>> local6, local7) and not filter(f_debug); };
>>>>>>> filter f_mail { facility(mail) and not filter(f_debug); };
>>>>>>> filter f_syslog3 { not facility(auth, authpriv, mail) and not
>>>>>>> filter(f_debug); };
>>>>>>>
>>>>>>> ########################
>>>>>>> # Log paths
>>>>>>> ########################
>>>>>>> log { source(s_src); filter(f_auth); destination(d_auth); };
>>>>>>> log { source(s_src); filter(f_cron); destination(d_cron); };
>>>>>>> log { source(s_src); filter(f_daemon); destination(d_daemon); };
>>>>>>> log { source(s_src); filter(f_kern); destination(d_kern); };
>>>>>>> log { source(s_src); filter(f_syslog3); destination(d_syslog); };
>>>>>>> log { source(s_src); filter(f_mail); destination(d_mail); };
>>>>>>> log { source(s_src); filter(f_debug); destination(d_debug); };
>>>>>>> log { source(s_src); filter(f_error); destination(d_error); };
>>>>>>>
>>>>>>>
>>>>>>> # All messages send to a remote site
>>>>>>> #
>>>>>>> log { source(s_src); destination(d_scl3); };
>>>>>>>
>>>>>>> ###
>>>>>>> # Include all config files in /etc/syslog-ng/conf.d/
>>>>>>> ###
>>>>>>> @include "/etc/syslog-ng/conf.d/*.conf"
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> There's another file, amqp.conf where we actually read and ship
>>>>>>> those Bro logs.
>>>>>>>
>>>>>>>
>>>>>>> source bro_conn {
>>>>>>> file( "/nsm/bro/logs/current/conn.log" flags(no-parse));
>>>>>>> };
>>>>>>>
>>>>>>> source bro_ssl {
>>>>>>> file( "/nsm/bro/logs/current/ssl.log" flags(no-parse));
>>>>>>> };
>>>>>>>
>>>>>>> source bro_dns {
>>>>>>> file( "/nsm/bro/logs/current/dns.log" flags(no-parse));
>>>>>>> };
>>>>>>>
>>>>>>> source bro_smtp {
>>>>>>> file( "/nsm/bro/logs/current/smtp.log" flags(no-parse));
>>>>>>> };
>>>>>>>
>>>>>>> source bro_ssh {
>>>>>>> file( "/nsm/bro/logs/current/ssh.log" flags(no-parse));
>>>>>>> };
>>>>>>>
>>>>>>> source bro_notice {
>>>>>>> file( "/nsm/bro/logs/current/notice.log" flags(no-parse));
>>>>>>> };
>>>>>>>
>>>>>>> source bro_intel {
>>>>>>> file( "/nsm/bro/logs/current/intel.log" flags(no-parse));
>>>>>>> };
>>>>>>>
>>>>>>> source bro_dce_rpc {
>>>>>>> file( "/nsm/bro/logs/current/dce_rpc.log" flags(no-parse));
>>>>>>> };
>>>>>>>
>>>>>>> source bro_dhcp {
>>>>>>> file( "/nsm/bro/logs/current/dhcp.log" flags(no-parse));
>>>>>>> };
>>>>>>>
>>>>>>> source bro_files {
>>>>>>> file( "/nsm/bro/logs/current/files.log" flags(no-parse));
>>>>>>> };
>>>>>>>
>>>>>>> source bro_kerberos {
>>>>>>> file( "/nsm/bro/logs/current/kerberos.log" flags(no-parse));
>>>>>>> };
>>>>>>>
>>>>>>> source bro_http {
>>>>>>> file( "/nsm/bro/logs/current/http.log" flags(no-parse));
>>>>>>> };
>>>>>>>
>>>>>>> source bro_software {
>>>>>>> file( "/nsm/bro/logs/current/software.log" flags(no-parse));
>>>>>>> };
>>>>>>>
>>>>>>> source bro_snmp {
>>>>>>> file( "/nsm/bro/logs/current/snmp.log" flags(no-parse));
>>>>>>> };
>>>>>>>
>>>>>>> source bro_socks {
>>>>>>> file( "/nsm/bro/logs/current/socks.log" flags(no-parse));
>>>>>>> };
>>>>>>>
>>>>>>> source bro_tunnel {
>>>>>>> file( "/nsm/bro/logs/current/tunnel.log" flags(no-parse));
>>>>>>> };
>>>>>>>
>>>>>>> source bro_ntlm {
>>>>>>> file( "/nsm/bro/logs/current/ntlm.log" flags(no-parse));
>>>>>>> };
>>>>>>>
>>>>>>> source bro_pe {
>>>>>>> file( "/nsm/bro/logs/current/pe.log" flags(no-parse));
>>>>>>> };
>>>>>>>
>>>>>>> source bro_sip {
>>>>>>> file( "/nsm/bro/logs/current/sip.log" flags(no-parse));
>>>>>>> };
>>>>>>>
>>>>>>> source bro_smb_files {
>>>>>>> file( "/nsm/bro/logs/current/smb_files.log" flags(no-parse));
>>>>>>> };
>>>>>>>
>>>>>>> source bro_smb_mapping {
>>>>>>> file( "/nsm/bro/logs/current/smb_mapping.log" flags(no-parse));
>>>>>>> };
>>>>>>>
>>>>>>> source bro_x509 {
>>>>>>> file( "/nsm/bro/logs/current/x509.log" flags(no-parse));
>>>>>>> };
>>>>>>>
>>>>>>> source bro_known_certs {
>>>>>>> file( "/nsm/bro/logs/current/known_certs.log" flags(no-parse));
>>>>>>> };
>>>>>>>
>>>>>>> source bro_known_devices {
>>>>>>> file( "/nsm/bro/logs/current/known_devices.log"
>>>>>>> flags(no-parse));
>>>>>>> };
>>>>>>>
>>>>>>> source bro_known_hosts {
>>>>>>> file( "/nsm/bro/logs/current/known_hosts.log" flags(no-parse));
>>>>>>> };
>>>>>>>
>>>>>>> source bro_known_services {
>>>>>>> file( "/nsm/bro/logs/current/known_services.log"
>>>>>>> flags(no-parse));
>>>>>>> };
>>>>>>>
>>>>>>>
>>>>>>> destination d_amqp {
>>>>>>> amqp(
>>>>>>> vhost("nsm")
>>>>>>> host("<our happy rabbit>
>>>>>>> <http://syslog-proxy1.dmz.mdc1.mozilla.com>")
>>>>>>> port(5672)
>>>>>>> exchange("eventtask")
>>>>>>> exchange-type("direct")
>>>>>>> routing-key("eventtask")
>>>>>>> body("$(format-json --scope nv_pairs --pair category=\"bro\"
>>>>>>> --pair source=$source --pair customendpoint=\" \" --pair tags=\"bro\")")
>>>>>>> persistent(yes)
>>>>>>> username("USERNAME")
>>>>>>> password("PASSWORD")
>>>>>>> );
>>>>>>> };
>>>>>>>
>>>>>>>
>>>>>>> parser p_json { json-parser(); };
>>>>>>>
>>>>>>>
>>>>>>> log { source(bro_conn); parser(p_json); destination(d_amqp); };
>>>>>>> log { source(bro_http); parser(p_json); destination(d_amqp); };
>>>>>>> log { source(bro_ssl); parser(p_json); destination(d_amqp); };
>>>>>>> log { source(bro_dns); parser(p_json); destination(d_amqp); };
>>>>>>> log { source(bro_smtp); parser(p_json); destination(d_amqp); };
>>>>>>> log { source(bro_ssh); parser(p_json); destination(d_amqp); };
>>>>>>> log { source(bro_intel); parser(p_json); destination(d_amqp); };
>>>>>>> log { source(bro_notice); parser(p_json); destination(d_amqp); };
>>>>>>> log { source(bro_dce_rpc); parser(p_json); destination(d_amqp);
>>>>>>> flags(flow-control); };
>>>>>>> log { source(bro_dhcp); parser(p_json); destination(d_amqp); };
>>>>>>> log { source(bro_files); parser(p_json); destination(d_amqp);
>>>>>>> flags(flow-control); };
>>>>>>> log { source(bro_kerberos); parser(p_json); destination(d_amqp);
>>>>>>> flags(flow-control); };
>>>>>>> log { source(bro_software); parser(p_json); destination(d_amqp);
>>>>>>> flags(flow-control); };
>>>>>>> log { source(bro_snmp); parser(p_json); destination(d_amqp);
>>>>>>> flags(flow-control); };
>>>>>>> log { source(bro_socks); parser(p_json); destination(d_amqp);
>>>>>>> flags(flow-control); };
>>>>>>> log { source(bro_tunnel); parser(p_json); destination(d_amqp);
>>>>>>> flags(flow-control); };
>>>>>>> log { source(bro_ntlm); parser(p_json); destination(d_amqp);
>>>>>>> flags(flow-control); };
>>>>>>> log { source(bro_pe); parser(p_json); destination(d_amqp);
>>>>>>> flags(flow-control); };
>>>>>>> log { source(bro_sip); parser(p_json); destination(d_amqp);
>>>>>>> flags(flow-control); };
>>>>>>> log { source(bro_smb_files); parser(p_json); destination(d_amqp);
>>>>>>> flags(flow-control); };
>>>>>>> log { source(bro_smb_mapping); parser(p_json); destination(d_amqp);
>>>>>>> flags(flow-control); };
>>>>>>> log { source(bro_x509); parser(p_json); destination(d_amqp);
>>>>>>> flags(flow-control); };
>>>>>>> log { source(bro_known_certs); parser(p_json); destination(d_amqp);
>>>>>>> flags(flow-control); };
>>>>>>> log { source(bro_known_devices); parser(p_json);
>>>>>>> destination(d_amqp); flags(flow-control); };
>>>>>>> log { source(bro_known_hosts); parser(p_json); destination(d_amqp);
>>>>>>> flags(flow-control); };
>>>>>>> log { source(bro_known_services); parser(p_json);
>>>>>>> destination(d_amqp); flags(flow-control); };
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *Statistics*
>>>>>>>
>>>>>>>
>>>>>>> Mar 15 *00:17:30* nsmserver syslog-ng[11278]: Log statistics;
>>>>>>> processed='source(bro_conn)=112360513',
>>>>>>> processed='source(s_src)=227349', processed='source(bro_known_devices)=3791',
>>>>>>> processed='global(sdata_updates)=0', processed='center(received)=310790955',
>>>>>>> processed='source(bro_ssh)=622441', processed='source(bro_smb_files)=5815964',
>>>>>>> processed='source(bro_socks)=0', processed='destination(d_daemon)=21',
>>>>>>> dropped='dst.amqp(d_amqp#0,amqp,nsm,happyrabbit,5672,eventtask,direct)=2',
>>>>>>> processed='dst.amqp(d_amqp#0,amqp,nsm,happyrabbit,5672,event
>>>>>>> task,direct)=310563565 <(31)%20056%203565>',
>>>>>>> queued='dst.amqp(d_amqp#0,amqp,nsm,happyrabbit,5672,eventtask,direct)=0',
>>>>>>> processed='destination(d_error)=189386',
>>>>>>> processed='destination(d_syslog)=207595',
>>>>>>> processed='source(bro_ssl)=49788364', processed='source(bro_kerberos)=133177',
>>>>>>> processed='source(bro_dhcp)=69970', processed='destination(d_mail)=0',
>>>>>>> processed='source(bro_http)=60085539',
>>>>>>> processed='global(msg_clones)=1576', processed='destination(d_amqp)=
>>>>>>> 310563565 <(31)%20056%203565>', processed='destination(d_kern)=146',
>>>>>>> processed='source(bro_tunnel)=520921',
>>>>>>> processed='source(bro_software)=18851236 <(1)%20885%201236>',
>>>>>>> processed='source(bro_known_services)=13403',
>>>>>>> processed='source(bro_known_certs)=2070',
>>>>>>> processed='source(bro_dce_rpc)=501875',
>>>>>>> processed='destination(d_scl3)=227349',
>>>>>>> processed='source(bro_known_hosts)=14604',
>>>>>>> processed='source(bro_smb_mapping)=116412',
>>>>>>> processed='source(bro_files)=15152100',
>>>>>>> processed='center(queued)=311210449', processed='destination(d_debug)=10280',
>>>>>>> processed='src.internal(s_src#2)=26785',
>>>>>>> stamp='src.internal(s_src#2)=1521073048',
>>>>>>> processed='source(bro_ntlm)=16823', processed='destination(d_auth)=9474',
>>>>>>> processed='global(internal_queue_length)=0',
>>>>>>> processed='source(bro_smtp)=1067448', dropped='dst.udp(d_scl3#0,udp,
>>>>>>> syslog1.private.scl3.mozilla.com:514)=0',
>>>>>>> processed='dst.udp(d_scl3#0,udp,syslog1.private.scl3.mozilla.com:514)=227349',
>>>>>>> queued='dst.udp(d_scl3#0,udp,syslog1.private.scl3.mozilla.com:514)=221705',
>>>>>>> written='dst.udp(d_scl3#0,udp,syslog1.private.scl3.mozilla.com:514)=5644',
>>>>>>> processed='global(payload_reallocs)=310467149 <(31)%20046%207149>',
>>>>>>> queued='global(scratch_buffers_count)=17875655781170',
>>>>>>> processed='destination(d_cron)=2633', processed='source(bro_snmp)=9854568',
>>>>>>> processed='source(bro_notice)=119415',
>>>>>>> processed='source(bro_dns)=19303431', processed='source(bro_sip)=97822',
>>>>>>> processed='source(bro_intel)=26969', processed='source(bro_pe)=531103',
>>>>>>> processed='source(bro_x509)=15493647',
>>>>>>> queued='global(scratch_buffers_bytes)=2304'
>>>>>>>
>>>>>>>
>>>>>>> Mar 15 *00:27:30* nsmserver
>>>>>>> <http://nsmserver1.private.scl3.mozilla.com> syslog-ng[11278]: Log
>>>>>>> statistics; processed='source(bro_conn)=112988941',
>>>>>>> processed='source(s_src)=228651', processed='source(bro_known_devices)=3791',
>>>>>>> processed='global(sdata_updates)=0', processed='center(received)=312659144',
>>>>>>> processed='source(bro_ssh)=627013', processed='source(bro_smb_files)=5863697',
>>>>>>> processed='source(bro_socks)=0', processed='destination(d_daemon)=21',
>>>>>>> dropped='dst.amqp(d_amqp#0,amqp,nsm,happyrabbit
>>>>>>> <http://syslog-proxy1.dmz.mdc1.mozilla.com>,5672,eventtask,direct)=2',
>>>>>>> processed='dst.amqp(d_amqp#0,amqp,nsm,happyrabbit
>>>>>>> <http://syslog-proxy1.dmz.mdc1.mozilla.com>,5672,eventtask,direct)=
>>>>>>> 312430452 <(31)%20243%200452>', queued='dst.amqp(d_amqp#0,amqp
>>>>>>> ,nsm,happyrabbit <http://syslog-proxy1.dmz.mdc1.mozilla.com>
>>>>>>> ,5672,eventtask,direct)=0', processed='destination(d_error)=190429',
>>>>>>> processed='destination(d_syslog)=208759',
>>>>>>> processed='source(bro_ssl)=50077572', processed='source(bro_kerberos)=134215',
>>>>>>> processed='source(bro_dhcp)=70487', processed='destination(d_mail)=0',
>>>>>>> processed='source(bro_http)=60446166',
>>>>>>> processed='global(msg_clones)=1594', processed='destination(d_amqp)=
>>>>>>> 312430452 <(31)%20243%200452>', processed='destination(d_kern)=146',
>>>>>>> processed='source(bro_tunnel)=524450',
>>>>>>> processed='source(bro_software)=18938552 <(1)%20893%208552>',
>>>>>>> processed='source(bro_known_services)=13532',
>>>>>>> processed='source(bro_known_certs)=2073',
>>>>>>> processed='source(bro_dce_rpc)=505206',
>>>>>>> processed='destination(d_scl3)=228651',
>>>>>>> processed='source(bro_known_hosts)=14630',
>>>>>>> processed='source(bro_smb_mapping)=117177',
>>>>>>> processed='source(bro_files)=15252368',
>>>>>>> processed='center(queued)=313080999', processed='destination(d_debug)=10352',
>>>>>>> processed='src.internal(s_src#2)=26966',
>>>>>>> stamp='src.internal(s_src#2)=1521073648',
>>>>>>> processed='source(bro_ntlm)=16848', processed='destination(d_auth)=9540',
>>>>>>> processed='global(internal_queue_length)=0',
>>>>>>> processed='source(bro_smtp)=1074012', dropped='dst.udp(d_scl3#0,udp,
>>>>>>> syslog1.private.scl3.mozilla.com:514)=0',
>>>>>>> processed='dst.udp(d_scl3#0,udp,syslog1.private.scl3.mozilla.com:514)=228651',
>>>>>>> queued='dst.udp(d_scl3#0,udp,syslog1.private.scl3.mozilla.com:514)=223007',
>>>>>>> written='dst.udp(d_scl3#0,udp,syslog1.private.scl3.mozilla.com:514)=5644',
>>>>>>> processed='global(payload_reallocs)=312333723 <(31)%20233%203723>',
>>>>>>> queued='global(scratch_buffers_count)=17970145061685',
>>>>>>> processed='destination(d_cron)=2649', processed='source(bro_snmp)=9917302',
>>>>>>> processed='source(bro_notice)=120140',
>>>>>>> processed='source(bro_dns)=19462256', processed='source(bro_sip)=98565',
>>>>>>> processed='source(bro_intel)=27061', processed='source(bro_pe)=535753',
>>>>>>> processed='source(bro_x509)=15598686',
>>>>>>> queued='global(scratch_buffers_bytes)=2304'
>>>>>>>
>>>>>>>
>>>>>>> ____________________________________________________________
>>>>>>> __________________
>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>> Documentation: http://www.balabit.com/support
>>>>>>> /documentation/?product=syslog-ng
>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ____________________________________________________________
>>>>>> __________________
>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>> Documentation: http://www.balabit.com/support
>>>>>> /documentation/?product=syslog-ng
>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> ____________________________________________________________
>>>>> __________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation: http://www.balabit.com/support
>>>>> /documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>>>>
>>>>
>>>> ____________________________________________________________
>>>> __________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation: http://www.balabit.com/support
>>>> /documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>>
>>>
>>> ____________________________________________________________
>>> __________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support
>>> /documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=
>> syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180321/b01922ae/attachment-0001.html>
More information about the syslog-ng
mailing list