<div dir="ltr"><div>Yeah, focusing on the memory usage would be great for now and I can promise to help troubleshoot the CPU usage later, to make syslog-ng better for everyone :)</div><div><br></div><div><br></div><div>Quick statistics, to satisfy your curiosity. Both -s for statistics + the more verbose option will be next.<br></div><div><br></div><div><br></div><div> Summary of events:<br><br> syslog-ng (21324), 1624 events, 0.2%, 0.000 msec<br><br> syscall calls total min avg max stddev<br> (msec) (msec) (msec) (msec) (%)<br> --------------- -------- --------- --------- --------- --------- ------<br> read 24 0.298 0.004 0.012 0.037 15.55%<br> close 1 0.003 0.003 0.003 0.003 0.00%<br> madvise 1 0.014 0.014 0.014 0.014 0.00%<br> futex 172 0.926 0.002 0.005 0.046 7.69%<br> epoll_wait 303 15467.279 0.000 51.047 10010.236 68.51%<br> epoll_ctl 311 2.207 0.003 0.007 0.039 2.53%<br><br><br> syslog-ng (21337), 40816 events, 3.9%, 0.000 msec<br><br> syscall calls total min avg max stddev<br> (msec) (msec) (msec) (msec) (%)<br> --------------- -------- --------- --------- --------- --------- ------<br> read 1191 11.601 0.002 0.010 0.322 4.42%<br> open 342 2.325 0.004 0.007 0.022 2.84%<br> close 343 0.848 0.002 0.002 0.014 2.94%<br> stat 5 0.035 0.005 0.007 0.009 12.34%<br> writev 222 2.299 0.005 0.010 0.040 2.93%<br> madvise 1 0.012 0.012 0.012 0.012 0.00%<br> recvmsg 226 1.444 0.003 0.006 0.024 3.31%<br> readlink 114 0.662 0.004 0.006 0.015 2.78%<br> futex 6586 29.215 0.002 0.004 0.600 2.36%<br> epoll_wait 5625 34934.479 0.000 6.211 10008.432 30.25%<br> epoll_ctl 5753 42.784 0.002 0.007 0.051 0.45%<br><br><br> syslog-ng (21403), 63917 events, 6.1%, 0.000 msec<br><br> syscall calls total min avg max stddev<br> (msec) (msec) (msec) (msec) (%)<br> --------------- -------- --------- --------- --------- --------- ------<br> read 1661 17.125 0.002 0.010 0.052 2.36%<br> open 426 2.455 0.004 0.006 0.021 2.12%<br> close 426 0.902 0.002 0.002 0.011 1.51%<br> stat 4 0.030 0.006 0.008 0.009 8.88%<br> writev 291 2.671 0.005 0.009 0.033 2.69%<br> sendto 2 0.009 0.004 0.005 0.005 18.82%<br> recvmsg 284 1.549 0.004 0.005 0.020 2.47%<br> readlink 142 0.745 0.004 0.005 0.016 2.52%<br> futex 11142 47.025 0.002 0.004 0.076 0.59%<br> epoll_wait 8695 39758.929 0.000 4.573 8462.289 23.40%<br> epoll_ctl 8886 68.424 0.002 0.008 0.057 0.31%<br><br><br> syslog-ng (45743), 119187 events, 11.4%, 0.000 msec<br><br> syscall calls total min avg max stddev<br> (msec) (msec) (msec) (msec) (%)<br> --------------- -------- --------- --------- --------- --------- ------<br> read 537 2.146 0.002 0.004 0.030 2.86%<br> open 12 0.150 0.008 0.013 0.020 9.11%<br> close 21 0.109 0.002 0.005 0.033 26.53%<br> stat 489 6.383 0.004 0.013 0.042 1.87%<br> fstat 1076 3.730 0.002 0.003 0.018 1.85%<br> poll 8 2.100 0.003 0.263 0.597 37.74%<br> lseek 1064 4.928 0.002 0.005 0.023 2.27%<br> mmap 8 0.089 0.006 0.011 0.020 16.85%<br> munmap 8 0.104 0.008 0.013 0.019 10.89%<br> ioctl 4 0.018 0.004 0.005 0.005 5.77%<br> socket 8 0.074 0.006 0.009 0.018 15.23%<br> connect 8 0.067 0.004 0.008 0.013 16.40%<br> sendto 4 0.086 0.020 0.021 0.023 2.77%<br> recvfrom 4 0.020 0.004 0.005 0.006 5.57%<br> bind 4 0.017 0.004 0.004 0.005 7.98%<br> setsockopt 575 1.774 0.002 0.003 0.023 2.25%<br> clone 2 0.114 0.031 0.057 0.084 46.29%<br> fcntl 1158 3.505 0.002 0.003 0.026 2.19%<br> fchmod 4 0.031 0.004 0.008 0.015 30.83%<br> fchown 8 0.083 0.004 0.010 0.017 17.27%<br> capget 56 0.130 0.002 0.002 0.005 3.56%<br> capset 28 0.083 0.002 0.003 0.005 4.60%<br> futex 785 3.376 0.002 0.004 0.067 4.53%<br> epoll_wait 30958 42810.853 0.000 1.383 207.240 3.57%<br> epoll_ctl 22239 104.310 0.002 0.005 0.300 0.39%<br> timerfd_settime 526 1.847 0.002 0.004 0.024 2.01%<br><br><br> syslog-ng (45746), 821179 events, 78.5%, 0.000 msec<br><br> syscall calls total min avg max stddev<br> (msec) (msec) (msec) (msec) (%)<br> --------------- -------- --------- --------- --------- --------- ------<br> sendto 380040 2073.061 0.002 0.005 0.365 0.12%<br> futex 22112 83.157 0.002 0.004 3.919 4.83%<br> epoll_wait 571 25533.479 0.000 44.717 386.682 6.52%<br> epoll_ctl 7867 43.704 0.002 0.006 0.030 0.32%<br></div><div><br></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 20, 2018 at 5:57 PM, Scheidler, Balázs <span dir="ltr"><<a href="mailto:balazs.scheidler@balabit.com" target="_blank">balazs.scheidler@balabit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>126% CPU usage? that would be great to know the details there, although I understand that the memory is more of a concern now. :)<br><br></div>can you run a perf record on that process, perhaps once the memory issue is solved? I have my suspicion where it is spending its time, but it would be great to confirm. (my guess is value-pairs while formatting json messages).<br><div class="gmail_extra"><br></div><div class="gmail_extra">cheers,<span class="HOEnZb"><font color="#888888"><br clear="all"></font></span></div><div class="gmail_extra"><span class="HOEnZb"><font color="#888888"><div><div class="m_-810667320680045734gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">-- <br>Bazsi<br></div></div></div></font></span><div><div class="h5">
<br><div class="gmail_quote">On Tue, Mar 20, 2018 at 8:26 PM, Michal Purzynski <span dir="ltr"><<a href="mailto:michal@mozilla.com" target="_blank">michal@mozilla.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hello Gábor!</div><div><br></div><div>Answers inline.</div><div><br></div><div><div class="gmail_extra"><div class="gmail_quote"><span>On Mon, Mar 19, 2018 at 9:09 AM, Nagy, Gábor <span dir="ltr"><<a href="mailto:gabor.nagy@balabit.com" target="_blank">gabor.nagy@balabit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div>- have you built syslog-ng from source or you downloaded a package?</div><div></div></div></blockquote><div><br></div></span><div>It's a package from</div><div><br></div><div>deb <a href="http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/xUbuntu_14.04" target="_blank">http://download.opensuse.org/r<wbr>epositories/home:/laszlo_budai<wbr>:/syslog-ng/xUbuntu_14.04</a> ./<br></div><span><div> </div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>- have you tried to reproduce the issue in a different environment with a minimal config? We are using your configuration, but if you narrowed down the problem it would be helpful.</div></div></blockquote><div><br></div></span><div>If I disable the Bro via AMQP the problem goes away, so that must be something there :/<br></div><span><div> </div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>- we were experimenting with a very simple JSON message, can you show us an example log to see the complexity of it, please? We are thinking to check Bro out for log message structure.</div></div></blockquote><div><br></div></span><div>Example logs follow sent to Peter via a private channel, making them public would be kind of difficult.</div><div><br></div><div>Appreciate you looking into it! And BTW, I just restarted syslog-ng on the most busy server</div><div><br></div><div><span style="color:rgb(204,204,204);font-family:"Source Sans Pro",sans-serif;font-size:15px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:0.2px;text-align:start;text-indent:0px;text-transform:none;white-space:pre-wrap;word-spacing:0px;background-color:rgb(31,31,31);display:inline;float:none">14910 root 20 0 59.899g 0.057t 3784 S 126.4 92.9 1166:22 syslog-ng</span></div><div><br></div><div>;-)<br></div><div><div class="m_-810667320680045734h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>I saw that you have gave a huge log-fetch-limit() in the global config compared to the default. Setting log-fetch-limit() in global config is deprecated, you need to set it up per source.</div><div><br></div><div>We have a couple of ideas and will continue to try reproducing the memleak you reported.</div><div><br></div><div>Regards,</div><div>Gabor</div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_-810667320680045734m_-580635352056405161gmail-h5">On Sat, Mar 17, 2018 at 12:45 AM, Michal Purzynski <span dir="ltr"><<a href="mailto:michal@mozilla.com" target="_blank">michal@mozilla.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="m_-810667320680045734m_-580635352056405161gmail-h5"><div dir="ltr"><div><div>Hello!!</div><div><br></div><div>Could you help us troubleshoot a memory leak or a misconfiguration that makes the syslog-ng process memory usage grow? Like, a lot.<br></div><div><br></div><div>We use syslog-ng to read some JSON files and ship them to a RabbitMQ server via AMQP. As you can see, this is just a client, it does not accept connections from other systems, it works mostly with Bro logs (plus it handles a local syslog because that's convenient).</div><div><br></div><div>We have around 6000 events per second on this server. What's interesting, syslog-ng's memory grows quickly without flow control (and slower, but it still continues to grow with it). I'll switch that to TLS soon, a more secure configuration is ready to be deployed.</div><br></div><div>Things look pretty good on the RabbitMQ side. That server is not under pressure and handles the load just fine, the queue is consumed, there's nothing building up that would make me believe we have RabbitMQ server overloaded.</div><div><br></div><div>How much does syslog-ng grow?</div><div><br></div><div>I'd say - if I disable flow-control it will eat 55GB of RAM in less than 24h, if not faster. With flow-control enabled on the most 'busy' files things are way better, but the memory usage still keep growing - <br></div><div><br></div><div>syslog-ng.conf looks like below - BTW that's Ubuntu 14.04 LTS, 3.14.1-3 of syslog-ng</div><div><br></div><div>Let me know what other data you might need.<br></div><div><br></div><div>@version: 3.14<br>@include "scl.conf"<br><br># Syslog-ng configuration file, compatible with default Debian syslogd<br># installation.<br><br># First, set some global options.<br>options {<br> threaded (yes);<br> flush_lines (50000);<br> flush_timeout (1000);<br> time_reopen (10);<br> log_fetch_limit (50000);<br> log_fifo_size (500000);<br> use_dns (yes);<br> dns_cache (5000);<br> dns_cache_expire(87600);<br> use_fqdn (yes);<br> owner("root");<br> group("adm");<br> perm(0640);<br> keep_hostname (yes);<br> chain_hostnames (off);<br>};<br><br>########################<br># Sources<br>########################<br># This is the default behavior of sysklogd package<br># Logs may come from unix stream, but not from another machine.<br>#<br>source s_src {<br> system();<br> internal();<br>};<br><br># If you wish to get logs from remote machine you should uncomment<br># this and comment the above source line.<br>#<br>#source s_net { tcp(ip(127.0.0.1) port(1000)); };<br><br>########################<br># Destinations<br>########################<br># First some standard logfile<br>#<br>destination d_auth { file("/var/log/auth.log"); };<br>destination d_cron { file("/var/log/cron.log"); };<br>destination d_daemon { file("/var/log/daemon.log"); };<br>destination d_kern { file("/var/log/kern.log"); };<br>destination d_mail { file("/var/log/mail.log"); };<br>destination d_syslog { file("/var/log/syslog"); };<br><br># This files are the log come from the mail subsystem.<br>#<br>#destination d_mailinfo { file("/var/log/<a href="http://mail.info" target="_blank">mail.info</a>"); };<br>#destination d_mailwarn { file("/var/log/mail.warn"); };<br>#destination d_mailerr { file("/var/log/mail.err"); };<br><br># Logging for INN news system<br>#<br>#destination d_newscrit { file("/var/log/news/news.crit"<wbr>); };<br>#destination d_newserr { file("/var/log/news/news.err")<wbr>; };<br>#destination d_newsnotice { file("/var/log/news/news.notic<wbr>e"); };<br><br># Some 'catch-all' logfiles.<br>#<br>destination d_debug { file("/var/log/debug"); };<br>destination d_error { file("/var/log/error"); };<br><br># Syslog1 in SCL3<br>destination d_scl3 {<br> udp("<a href="http://syslog1.private.scl3.mozilla.com" target="_blank">syslog1.private.scl3.mozi<wbr>lla.com</a>" port(514));<br>};<br><br>########################<br># Filters<br>########################<br># Here's come the filter options. With this rules, we can set which<br># message go where.<br><br>filter f_dbg { level(debug); };<br>filter f_info { level(info); };<br>filter f_notice { level(notice); };<br>filter f_warn { level(warn); };<br>filter f_err { level(err); };<br>filter f_crit { level(crit .. emerg); };<br>filter f_debug { level(debug) and not facility(auth, authpriv, news, mail); };<br>filter f_error { level(err .. emerg) ; };<br>filter f_auth { facility(auth, authpriv) and not filter(f_debug); };<br>filter f_cron { facility(cron) and not filter(f_debug); };<br>filter f_daemon { facility(daemon) and not filter(f_debug); };<br>filter f_kern { facility(kern) and not filter(f_debug); };<br>filter f_local { facility(local0, local1, local3, local4, local5,<br> local6, local7) and not filter(f_debug); };<br>filter f_mail { facility(mail) and not filter(f_debug); };<br>filter f_syslog3 { not facility(auth, authpriv, mail) and not filter(f_debug); };<br><br>########################<br># Log paths<br>########################<br>log { source(s_src); filter(f_auth); destination(d_auth); };<br>log { source(s_src); filter(f_cron); destination(d_cron); };<br>log { source(s_src); filter(f_daemon); destination(d_daemon); };<br>log { source(s_src); filter(f_kern); destination(d_kern); };<br>log { source(s_src); filter(f_syslog3); destination(d_syslog); };<br>log { source(s_src); filter(f_mail); destination(d_mail); };<br>log { source(s_src); filter(f_debug); destination(d_debug); };<br>log { source(s_src); filter(f_error); destination(d_error); };<br><br><br># All messages send to a remote site<br>#<br>log { source(s_src); destination(d_scl3); };<br><br>###<br># Include all config files in /etc/syslog-ng/conf.d/<br>###<br>@include "/etc/syslog-ng/conf.d/*.conf"<br></div><div><br></div><div><br></div><div><br></div><div>There's another file, amqp.conf where we actually read and ship those Bro logs.<br></div><div><br></div><div><br></div><div>source bro_conn {<br> file( "/nsm/bro/logs/current/conn.lo<wbr>g" flags(no-parse));<br>};<br><br>source bro_ssl {<br> file( "/nsm/bro/logs/current/ssl.log<wbr>" flags(no-parse));<br>};<br><br>source bro_dns {<br> file( "/nsm/bro/logs/current/dns.log<wbr>" flags(no-parse));<br>};<br><br>source bro_smtp {<br> file( "/nsm/bro/logs/current/smtp.lo<wbr>g" flags(no-parse));<br>};<br><br>source bro_ssh {<br> file( "/nsm/bro/logs/current/ssh.log<wbr>" flags(no-parse));<br>};<br><br>source bro_notice {<br> file( "/nsm/bro/logs/current/notice.<wbr>log" flags(no-parse));<br>};<br><br>source bro_intel {<br> file( "/nsm/bro/logs/current/intel.l<wbr>og" flags(no-parse));<br>};<br><br>source bro_dce_rpc {<br> file( "/nsm/bro/logs/current/dce_rpc<wbr>.log" flags(no-parse));<br>};<br><br>source bro_dhcp {<br> file( "/nsm/bro/logs/current/dhcp.lo<wbr>g" flags(no-parse));<br>};<br><br>source bro_files {<br> file( "/nsm/bro/logs/current/files.l<wbr>og" flags(no-parse));<br>};<br><br>source bro_kerberos {<br> file( "/nsm/bro/logs/current/kerbero<wbr>s.log" flags(no-parse));<br>};<br><br>source bro_http {<br> file( "/nsm/bro/logs/current/http.lo<wbr>g" flags(no-parse));<br>};<br><br>source bro_software {<br> file( "/nsm/bro/logs/current/softwar<wbr>e.log" flags(no-parse));<br>};<br><br>source bro_snmp {<br> file( "/nsm/bro/logs/current/snmp.lo<wbr>g" flags(no-parse));<br>};<br><br>source bro_socks {<br> file( "/nsm/bro/logs/current/socks.l<wbr>og" flags(no-parse));<br>};<br><br>source bro_tunnel {<br> file( "/nsm/bro/logs/current/tunnel.<wbr>log" flags(no-parse));<br>};<br><br>source bro_ntlm {<br> file( "/nsm/bro/logs/current/ntlm.lo<wbr>g" flags(no-parse));<br>};<br><br>source bro_pe {<br> file( "/nsm/bro/logs/current/pe.log" flags(no-parse));<br>};<br><br>source bro_sip {<br> file( "/nsm/bro/logs/current/sip.log<wbr>" flags(no-parse));<br>};<br><br>source bro_smb_files {<br> file( "/nsm/bro/logs/current/smb_fil<wbr>es.log" flags(no-parse));<br>};<br><br>source bro_smb_mapping {<br> file( "/nsm/bro/logs/current/smb_map<wbr>ping.log" flags(no-parse));<br>};<br><br>source bro_x509 {<br> file( "/nsm/bro/logs/current/x509.lo<wbr>g" flags(no-parse));<br>};<br><br>source bro_known_certs {<br> file( "/nsm/bro/logs/current/known_c<wbr>erts.log" flags(no-parse));<br>};<br><br>source bro_known_devices {<br> file( "/nsm/bro/logs/current/known_d<wbr>evices.log" flags(no-parse));<br>};<br><br>source bro_known_hosts {<br> file( "/nsm/bro/logs/current/known_h<wbr>osts.log" flags(no-parse));<br>};<br><br>source bro_known_services {<br> file( "/nsm/bro/logs/current/known_s<wbr>ervices.log" flags(no-parse));<br>};<br><br><br>destination d_amqp {<br> amqp(<br> vhost("nsm")<br> host("<a href="http://syslog-proxy1.dmz.mdc1.mozilla.com" target="_blank"><our happy rabbit></a>")<br> port(5672)<br> exchange("eventtask")<br> exchange-type("direct")<br> routing-key("eventtask")<br> body("$(format-json --scope nv_pairs --pair category=\"bro\" --pair source=$source --pair customendpoint=\" \" --pair tags=\"bro\")")<br> persistent(yes)<br> username("USERNAME")<br> password("PASSWORD")<br> );<br>};<br><br><br>parser p_json { json-parser(); };<br><br><br>log { source(bro_conn); parser(p_json); destination(d_amqp); };<br>log { source(bro_http); parser(p_json); destination(d_amqp); };<br>log { source(bro_ssl); parser(p_json); destination(d_amqp); };<br>log { source(bro_dns); parser(p_json); destination(d_amqp); };<br>log { source(bro_smtp); parser(p_json); destination(d_amqp); };<br>log { source(bro_ssh); parser(p_json); destination(d_amqp); };<br>log { source(bro_intel); parser(p_json); destination(d_amqp); };<br>log { source(bro_notice); parser(p_json); destination(d_amqp); };<br>log { source(bro_dce_rpc); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_dhcp); parser(p_json); destination(d_amqp); };<br>log { source(bro_files); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_kerberos); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_software); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_snmp); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_socks); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_tunnel); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_ntlm); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_pe); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_sip); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_smb_files); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_smb_mapping); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_x509); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_known_certs); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_known_devices); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_known_hosts); parser(p_json); destination(d_amqp); flags(flow-control); };<br>log { source(bro_known_services); parser(p_json); destination(d_amqp); flags(flow-control); };</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><b>Statistics</b></div><div><br></div><div><br></div><div>Mar 15 <b>00:17:30</b> nsmserver syslog-ng[11278]: Log statistics; processed='source(bro_conn)=11<wbr>2360513', processed='source(s_src)=22734<wbr>9', processed='source(bro_known_de<wbr>vices)=3791', processed='global(sdata_update<wbr>s)=0', processed='center(received)=31<wbr>0790955', processed='source(bro_ssh)=622<wbr>441', processed='source(bro_smb_file<wbr>s)=5815964', processed='source(bro_socks)=0<wbr>', processed='destination(d_daemo<wbr>n)=21', dropped='dst.amqp(d_amqp#0,amq<wbr>p,nsm,happyrabbit,5672,eventta<wbr>sk,direct)=2', processed='dst.amqp(d_amqp#0,a<wbr>mqp,nsm,happyrabbit,5672,event<wbr>task,direct)=<a href="tel:(31)%20056%203565" value="+36310563565" target="_blank">310563565</a>', queued='dst.amqp(d_amqp#0,amqp<wbr>,nsm,happyrabbit,5672,eventtas<wbr>k,direct)=0', processed='destination(d_error<wbr>)=189386', processed='destination(d_syslo<wbr>g)=207595', processed='source(bro_ssl)=497<wbr>88364', processed='source(bro_kerberos<wbr>)=133177', processed='source(bro_dhcp)=69<wbr>970', processed='destination(d_mail)<wbr>=0', processed='source(bro_http)=60<wbr>085539', processed='global(msg_clones)=<wbr>1576', processed='destination(d_amqp)<wbr>=<a href="tel:(31)%20056%203565" value="+36310563565" target="_blank">310563565</a>', processed='destination(d_kern)<wbr>=146', processed='source(bro_tunnel)=<wbr>520921', processed='source(bro_software<wbr>)=<a href="tel:(1)%20885%201236" value="+3618851236" target="_blank">18851236</a>', processed='source(bro_known_se<wbr>rvices)=13403', processed='source(bro_known_ce<wbr>rts)=2070', processed='source(bro_dce_rpc)<wbr>=501875', processed='destination(d_scl3)<wbr>=227349', processed='source(bro_known_ho<wbr>sts)=14604', processed='source(bro_smb_mapp<wbr>ing)=116412', processed='source(bro_files)=1<wbr>5152100', processed='center(queued)=3112<wbr>10449', processed='destination(d_debug<wbr>)=10280', processed='src.internal(s_src#<wbr>2)=26785', stamp='src.internal(s_src#2)=1<wbr>521073048', processed='source(bro_ntlm)=16<wbr>823', processed='destination(d_auth)<wbr>=9474', processed='global(internal_que<wbr>ue_length)=0', processed='source(bro_smtp)=10<wbr>67448', dropped='dst.udp(d_scl3#0,udp,<wbr>syslog1.private.scl3.mozilla.c<wbr>om:514)=0', processed='dst.udp(d_scl3#0,ud<wbr>p,syslog1.private.scl3.mozilla<wbr>.com:514)=227349', queued='dst.udp(d_scl3#0,udp,s<a href="http://yslog1.private.scl3.mozilla.co" target="_blank"><wbr>yslog1.private.scl3.mozilla.co</a><wbr>m:514)=221705', written='dst.udp(d_scl3#0,udp,<wbr>syslog1.private.scl3.mozilla.c<wbr>om:514)=5644', processed='global(payload_real<wbr>locs)=<a href="tel:(31)%20046%207149" value="+36310467149" target="_blank">310467149</a>', queued='global(scratch_buffers<wbr>_count)=17875655781170', processed='destination(d_cron)<wbr>=2633', processed='source(bro_snmp)=98<wbr>54568', processed='source(bro_notice)=<wbr>119415', processed='source(bro_dns)=193<wbr>03431', processed='source(bro_sip)=978<wbr>22', processed='source(bro_intel)=2<wbr>6969', processed='source(bro_pe)=5311<wbr>03', processed='source(bro_x509)=15<wbr>493647', queued='global(scratch_buffers<wbr>_bytes)=2304'<br></div><div><br></div><div><br></div><div>Mar 15 <b>00:27:30</b> nsmserver<a href="http://nsmserver1.private.scl3.mozilla.com" target="_blank"></a> syslog-ng[11278]: Log statistics; processed='source(bro_conn)=11<wbr>2988941', processed='source(s_src)=22865<wbr>1', processed='source(bro_known_de<wbr>vices)=3791', processed='global(sdata_update<wbr>s)=0', processed='center(received)=31<wbr>2659144', processed='source(bro_ssh)=627<wbr>013', processed='source(bro_smb_file<wbr>s)=5863697', processed='source(bro_socks)=0<wbr>', processed='destination(d_daemo<wbr>n)=21', dropped='dst.amqp(d_amqp#0,amq<wbr>p,nsm,<a href="http://syslog-proxy1.dmz.mdc1.mozilla.com" target="_blank">happyrabbit</a>,5672,eventta<wbr>sk,direct)=2', processed='dst.amqp(d_amqp#0,a<wbr>mqp,nsm,happyrabbit<a href="http://syslog-proxy1.dmz.mdc1.mozilla.com" target="_blank"></a>,5672,event<wbr>task,direct)=<a href="tel:(31)%20243%200452" value="+36312430452" target="_blank">312430452</a>', queued='dst.amqp(d_amqp#0,amqp<wbr>,nsm,happyrabbit<a href="http://syslog-proxy1.dmz.mdc1.mozilla.com" target="_blank"></a>,5672,eventtas<wbr>k,direct)=0', processed='destination(d_error<wbr>)=190429', processed='destination(d_syslo<wbr>g)=208759', processed='source(bro_ssl)=500<wbr>77572', processed='source(bro_kerberos<wbr>)=134215', processed='source(bro_dhcp)=70<wbr>487', processed='destination(d_mail)<wbr>=0', processed='source(bro_http)=60<wbr>446166', processed='global(msg_clones)=<wbr>1594', processed='destination(d_amqp)<wbr>=<a href="tel:(31)%20243%200452" value="+36312430452" target="_blank">312430452</a>', processed='destination(d_kern)<wbr>=146', processed='source(bro_tunnel)=<wbr>524450', processed='source(bro_software<wbr>)=<a href="tel:(1)%20893%208552" value="+3618938552" target="_blank">18938552</a>', processed='source(bro_known_se<wbr>rvices)=13532', processed='source(bro_known_ce<wbr>rts)=2073', processed='source(bro_dce_rpc)<wbr>=505206', processed='destination(d_scl3)<wbr>=228651', processed='source(bro_known_ho<wbr>sts)=14630', processed='source(bro_smb_mapp<wbr>ing)=117177', processed='source(bro_files)=1<wbr>5252368', processed='center(queued)=3130<wbr>80999', processed='destination(d_debug<wbr>)=10352', processed='src.internal(s_src#<wbr>2)=26966', stamp='src.internal(s_src#2)=1<wbr>521073648', processed='source(bro_ntlm)=16<wbr>848', processed='destination(d_auth)<wbr>=9540', processed='global(internal_que<wbr>ue_length)=0', processed='source(bro_smtp)=10<wbr>74012', dropped='dst.udp(d_scl3#0,udp,<wbr>syslog1.private.scl3.mozilla.c<wbr>om:514)=0', processed='dst.udp(d_scl3#0,ud<wbr>p,syslog1.private.scl3.mozilla<wbr>.com:514)=228651', queued='dst.udp(d_scl3#0,udp,s<a href="http://yslog1.private.scl3.mozilla.co" target="_blank"><wbr>yslog1.private.scl3.mozilla.co</a><wbr>m:514)=223007', written='dst.udp(d_scl3#0,udp,<wbr>syslog1.private.scl3.mozilla.c<wbr>om:514)=5644', processed='global(payload_real<wbr>locs)=<a href="tel:(31)%20233%203723" value="+36312333723" target="_blank">312333723</a>', queued='global(scratch_buffers<wbr>_count)=17970145061685', processed='destination(d_cron)<wbr>=2649', processed='source(bro_snmp)=99<wbr>17302', processed='source(bro_notice)=<wbr>120140', processed='source(bro_dns)=194<wbr>62256', processed='source(bro_sip)=985<wbr>65', processed='source(bro_intel)=2<wbr>7061', processed='source(bro_pe)=5357<wbr>53', processed='source(bro_x509)=15<wbr>598686', queued='global(scratch_buffers<wbr>_bytes)=2304'</div><br></div>
<br></div></div>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=syslog<wbr>-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>
<br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=syslog<wbr>-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
<br></blockquote></div></div></div><br></div></div></div>
<br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=<wbr>syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div></div></div></div>
<br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/<wbr>mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/<wbr>support/documentation/?<wbr>product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/<wbr>syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>