<div dir="ltr">Hi Soumyadip,<div><br></div><div>This symptoms are very similar to the problem solved which is solved by PR: <a href="https://github.com/balabit/syslog-ng/pull/1183">https://github.com/balabit/syslog-ng/pull/1183</a></div><div><br></div><div>It is a known bug the next 3.8 release will contain the fix.</div><div><br></div><div>BR,</div><div>Viktor Juhász</div><div><br></div><div><br><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 25, 2016 at 1:01 PM, Soumyadip Das Mahapatra <span dir="ltr"><<a href="mailto:soumyadip.bt@gmail.com" target="_blank">soumyadip.bt@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr">Hi,<div><br></div><div>We have installed syslog-ng 3.7.3 on RHEL 6.6 from COPR repo: </div><div><br></div><div>From dmesg: </div><div><br></div><div><div>[Thu Sep 1 02:24:08 2016] syslog-ng[10374]: segfault at 1a116c10 ip 000000001a116c10 sp 00007f26eebfa928 error 15</div><div>[Sun Sep 4 20:10:47 2016] syslog-ng[27797]: segfault at 0 ip (null) sp 00007f998b5fb928 error 14 in syslog-ng[400000+3000]</div><div>[Mon Sep 5 19:04:15 2016] syslog-ng[12727]: segfault at 36aaa42080 ip 00000039b8618df0 sp 00007f996fffc878 error 7 in libglib-2.0.so.0.2800.8[39b860<wbr>0000+115000]</div><div>[Tue Sep 6 03:01:21 2016] syslog-ng[3827]: segfault at 15cb09d0 ip 0000000015cb09d0 sp 00007f99aebfa928 error 15</div><div>[Fri Sep 9 05:03:41 2016] syslog-ng[28275] general protection ip:39b8618df0 sp:7f29c57f8878 error:0 in libglib-2.0.so.0.2800.8[39b860<wbr>0000+115000]</div><div>[Sun Sep 11 00:48:01 2016] syslog-ng[14479]: segfault at 0 ip (null) sp 00007f2a03caf928 error 14 in syslog-ng[400000+3000]</div><div>[Sun Sep 11 10:15:37 2016] syslog-ng[2012]: segfault at 0 ip 00000036aaa3cc1c sp 00007feed2bfa880 error 6 in libsyslog-ng-3.7.so.0.0.0[36aa<wbr>a00000+a2000]<br><br>Core 1:</div></div><div><br></div><div><div>(gdb) bt full</div><div>#0 0x00000039b8618df0 in g_atomic_int_add () from /lib64/libglib-2.0.so.0</div><div>No symbol table info available.</div><div>#1 0x00000036aaa3cbe4 in stats_counter_add (self=0xabe68ea0, thread_id=5)</div><div> at lib/stats/stats-counter.h:39</div><div>No locals.</div><div>#2 log_queue_fifo_move_input_unl<wbr>ocked (self=0xabe68ea0, thread_id=5)</div><div> at lib/logqueue-fifo.c:193</div><div> queue_len = 1755882337</div><div>#3 0x00000036aaa3cd6e in log_queue_fifo_move_input (user_data=0xabe68ea0)</div><div> at lib/logqueue-fifo.c:215</div><div> self = 0xabe68ea0</div><div> thread_id = 5</div><div> __PRETTY_FUNCTION__ = "log_queue_fifo_move_input"</div><div>#4 0x00000036aaa456ca in main_loop_worker_invoke_batch_<wbr>callbacks ()</div><div> at lib/mainloop-worker.c:270</div><div> cb = 0xabe690e0</div><div> lh = 0xabe690e0</div><div> lh2 = 0xabe690e0</div><div>#5 0x00000036aaa6fa2a in iv_work_thread_do_work (_thr=0x9d6bd530) at iv_work.c:118</div><div> work = 0x6fb160</div><div> thr = 0x9d6bd530</div><div> pool = 0x6e48c0</div><div> last_seq = 1477269410</div><div>#6 0x00000036aaa6ed4a in iv_run_tasks (st=0x7f29b44d3c00) at iv_task.c:48</div><div> t = <value optimized out></div><div> tasks = {next = 0x7f29c57f89a0, prev = 0x7f29c57f89a0}</div><div>#7 0x00000036aaa7100c in iv_main () at iv_main_posix.c:106</div><div> to = {tv_sec = 10, tv_nsec = 0}</div><div> st = 0x7f29b44d3c00</div><div>#8 0x00000036aaa6f841 in iv_work_thread (_thr=0x9d6bd530) at iv_work.c:200</div><div> thr = 0x9d6bd530</div><div> pool = 0x6e48c0</div><div>#9 0x00000036aaa71a1f in iv_thread_handler (_thr=0x75242f90)</div><div> at iv_thread_posix.c:142</div><div> __clframe = {__cancel_routine = 0x36aaa71a80 <iv_thread_cleanup_handler>,</div><div> __cancel_arg = 0x75242f90, __do_it = 1, __cancel_type = 0}</div><div> thr = 0x75242f90</div><div>#10 0x00000039b6e079d1 in start_thread () from /lib64/libpthread.so.0</div><div>No symbol table info available.</div><div>#11 0x00000039b6ae88fd in clone () from /lib64/libc.so.6</div><div>No symbol table info available.<br><br>Core 2:</div></div><div><br></div><div><div>(gdb) bt full</div><div>#0 0x0000000000000000 in ?? ()</div><div>No symbol table info available.</div><div>#1 0x00000036aaa456ca in main_loop_worker_invoke_batch_<wbr>callbacks ()</div><div> at lib/mainloop-worker.c:270</div><div> cb = 0x87e96b10</div><div> lh = 0x87e96b10</div><div> lh2 = 0x0</div><div>#2 0x00000036aaa6fa2a in iv_work_thread_do_work (_thr=0x7034ffd0) at iv_work.c:118</div><div> work = 0x6fb390</div><div> thr = 0x7034ffd0</div><div> pool = 0x6e48b0</div><div> last_seq = 1323426114</div><div>#3 0x00000036aaa6ed4a in iv_run_tasks (st=0x7f29d4360200) at iv_task.c:48</div><div> t = <value optimized out></div><div> tasks = {next = 0x7f2a03caf9a0, prev = 0x7f2a03caf9a0}</div><div>#4 0x00000036aaa7100c in iv_main () at iv_main_posix.c:106</div><div> to = {tv_sec = 10, tv_nsec = 0}</div><div> st = 0x7f29d4360200</div><div>#5 0x00000036aaa6f841 in iv_work_thread (_thr=0x7034ffd0) at iv_work.c:200</div><div> thr = 0x7034ffd0</div><div> pool = 0x6e48b0</div><div>#6 0x00000036aaa71a1f in iv_thread_handler (_thr=0x426b490)</div><div> at iv_thread_posix.c:142</div><div> __clframe = {__cancel_routine = 0x36aaa71a80 <iv_thread_cleanup_handler>,</div><div> __cancel_arg = 0x426b490, __do_it = 1, __cancel_type = 0}</div><div> thr = 0x426b490</div><div>#7 0x00000039b6e079d1 in start_thread () from /lib64/libpthread.so.0</div><div>No symbol table info available.</div><div>#8 0x00000039b6ae88fd in clone () from /lib64/libc.so.6</div><div>No symbol table info available.</div></div><div><br></div><div>So looks like segfault is happening at random part of the code.</div><div><br></div><div># cat /etc/sysconfig/syslog-ng</div><div><div>#---</div><div># Syslog-ng command line options</div><div># See syslog-ng(8) for more details</div><div>#---</div><div>SYSLOGNG_PID="/var/run/syslog-<wbr>ng.pid"</div><div>SYSLOGNG_OPTIONS="-p $SYSLOGNG_PID --fd-limit 30000"</div><div>SYSLOGNG_COMPAT_PID="/var/run/<wbr>syslogd.pid"<br><br><br># cat /etc/syslog-ng/syslog-ng.conf<br><br><div>@version:3.6</div><div>options {</div><div> flush_lines (0);</div><div> time_reopen (10);</div><div> log_fifo_size (20000);</div><div> use_dns (yes);</div><div> use_fqdn (yes);</div><div> create_dirs (yes);</div><div> dir_group("wheel");</div><div> dir_owner("nobody");</div><div> dir_perm(0755);</div><div> owner("nobody");</div><div> group("nobody");</div><div> perm(0644);</div><div> threaded(yes);</div><div> keep_hostname (yes);</div><div> chain_hostnames(yes);</div><div> bad_hostname("[^[:print:]]");</div><div> dns_cache(yes);</div><div> dns_cache_expire(300);</div><div> dns_cache_expire_failed(30);</div><div> dns_cache_size(1000);</div><div> stats_freq(3600);</div><div> flush_lines(0);</div><div>};</div><div>source s_sys {</div><div> file ("/proc/kmsg" program_override("kernel: "));</div><div> unix-stream ("/dev/log");</div><div> # udp(ip(0.0.0.0) port(514));</div><div>};</div><div>source s_tcp {</div><div> network(transport("tcp") port(514) so_rcvbuf(8388608) max-connections(200) log-iw-size(20000));</div><div>};</div><div>source s_udp {</div><div> network(transport("udp") port(514) so_rcvbuf(8388608) log-iw-size(20000));</div><div>};</div><div>source s_net {</div><div> network(transport("udp") port(515) so_rcvbuf(8388608) log-iw-size(20000) tags("source_net"));</div><div> network(transport("tcp") port(515) so_rcvbuf(8388608) max-connections(200) log-iw-size(20000) tags("source_net"));</div><div>};</div><div>source s_windows {</div><div> network(transport("tcp") port(516) so_rcvbuf(8388608) max-connections(200) log-iw-size(20000) tags("windows"));</div><div>};</div><div>destination d_cons { file("/dev/console"); };</div><div>destination d_mesg { file("/var/log/messages"); };</div><div>destination d_auth { file("/var/log/secure"); };</div><div>destination d_mail { file("/var/log/maillog" flush_lines(10)); };</div><div>destination d_spol { file("/var/log/spooler"); };</div><div>destination d_boot { file("/var/log/boot.log"); };</div><div>destination d_cron { file("/var/log/cron"); };</div><div>destination d_kern { file("/var/log/kern"); };</div><div>destination d_mlal { usertty("*"); };</div><div>destination d_userx { file("/u1/syslog/$HOST/$YEAR$M<wbr>ONTH$DAY/user.log"); };</div><div>destination d_kernx { file("/u1/syslog/$HOST/$YEAR$M<wbr>ONTH$DAY/kern.log"); };</div><div>destination d_mailx { file("/u1/syslog/$HOST/$YEAR$M<wbr>ONTH$DAY/mail.log"); };</div><div>destination d_mp { file("/u1/syslog/$HOST/$YEAR$M<wbr>ONTH$DAY/mp.log"); };</div><div>destination d_daemonx { file("/u1/syslog/$HOST/$YEAR$M<wbr>ONTH$DAY/daemon.log"); };</div><div>destination d_authx { file("/u1/syslog/$HOST/$YEAR$M<wbr>ONTH$DAY/auth.log"); };</div><div>destination d_lprx { file("/u1/syslog/$HOST/$YEAR$M<wbr>ONTH$DAY/lpr.log"); };</div><div>destination d_cronx { file("/u1/syslog/$HOST/$YEAR$M<wbr>ONTH$DAY/cron.log"); };</div><div>destination d_messagesx { file("/u1/syslog/$HOST/$YEAR$M<wbr>ONTH$DAY/messages.log"); };</div><div>destination d_networkx { file("/u1/syslog/$HOST/$YEAR$M<wbr>ONTH$DAY/network.log"); };</div><div>destination d_firewallx { file("/u1/syslog/$HOST/$YEAR$M<wbr>ONTH$DAY/firewall.log"); };</div><div>destination d_local0 { file("/u1/syslog/$HOST/$YEAR$M<wbr>ONTH$DAY/sp.log"); };</div><div>destination d_local1 { file("/u1/syslog/$HOST/$YEAR$M<wbr>ONTH$DAY/app.log"); };</div><div>destination d_localx { file("/u1/syslog/$HOST/$YEAR$M<wbr>ONTH$DAY/local.log"); };</div><div>destination d_local6 { file("/u1/syslog/$HOST/$YEAR$M<wbr>ONTH$DAY/httpd.log"); };</div><div>destination d_dnslogs { file("/u1/syslog/$HOST/$YEAR$M<wbr>ONTH$DAY/dnslogs.log"); };</div><div>destination d_logstash {</div><div> tcp("127.0.0.1" port(5140)</div><div> template("$R_ISODATE <$FACILITY_NUM.$LEVEL_NUM> $HOST $MSGHDR $MESSAGE\n")</div><div> frac_digits(3) template-escape(no));</div><div> pipe("/var/log/logstash-pipe"</div><div> template("$R_ISODATE <$FACILITY_NUM.$LEVEL_NUM> $HOST $MSGHDR $MESSAGE\n")</div><div> frac_digits(3) template-escape(no));</div><div>};</div><div>destination d_userauth {</div><div> tcp("<a href="http://abc.xyz.com" target="_blank">abc.xyz.com</a>" port(13456));</div><div>};</div><div>destination d_otto {</div><div> tcp("<a href="http://abc.xyz.com" target="_blank">abc.xyz.com</a>" port(13456));</div><div> tcp("<a href="http://abc.xyz.com" target="_blank">abc.xyz.com</a>" port(13456));</div><div>};</div><div>destination d_notifier {</div><div> tcp("<a href="http://abc.xyz.com" target="_blank">abc.xyz.com</a>" port(6000));</div><div> tcp("10.255.xx.yy" port(1248));</div><div>};</div><div>destination d_graylog {</div><div> syslog("<a href="http://abc.xyz.com" target="_blank">abc.xyz.com</a>" transport("tcp") port(9514));</div><div> pipe("/var/log/graylog-pipe" ts-format(iso));</div><div>};</div><div>destination d_gtsadfeed {</div><div> syslog("<a href="http://abc.xyz.com" target="_blank">abc.xyz.com</a>" transport("udp") port(514));</div><div>};</div><div>filter f_kernel { facility(kern); };</div><div>filter f_default { level(info..emerg)</div><div> and not (facility(mail)</div><div> or facility(authpriv)</div><div> or facility(cron))</div><div> and not message("Syslog connection (accepted|closed)"); };</div><div>filter f_auth { facility(auth,authpriv); };</div><div>filter f_mail { facility(mail); };</div><div>filter f_emergency { level(emerg); };</div><div>filter f_news { facility(uucp) or</div><div> (facility(news)</div><div> and level(crit..emerg)); };</div><div>filter f_boot { facility(local7); };</div><div>filter f_cron { facility(cron,solaris-cron); };</div><div>filter f_user { facility(user); };</div><div>filter f_daemon { facility(daemon); };</div><div>filter f_lpr { facility(lpr); };</div><div>filter f_dnslogs { program("mydns_logger"); };</div><div>filter f_local0 { facility(local0); };</div><div>filter f_local1 { facility(local1) and not program("mydns_logger"); };</div><div>filter f_local2 { facility(local2); };</div><div>filter f_local3 { facility(local3); };</div><div>filter f_local4 { facility(local4); };</div><div>filter f_local5 { facility(local5) or tags("source_net"); }; # local5 is network, so is port 515</div><div>filter f_local6 { facility(local6); };</div><div>filter f_local7 { facility(local7); };</div><div>filter f_alert { level(alert); };</div><div>filter f_crit { level(crit); };</div><div>filter f_err { level(err); };</div><div>filter f_warn { level(warn); };</div><div>filter f_notice { level(notice); };</div><div>filter f_info { level(info); };</div><div>filter f_debug { level(debug); };</div><div>filter f_nonet { not facility(local4) and not facility(local5) and not tags("source_net"); };</div><div>filter f_nocf3 { not program("cf3"); };</div><div>filter f_nodhcpd { not program("dhcpd"); };</div><div>filter f_windows { tags("windows"); };</div><div>filter f_snmptrapd { program("snmptrapd"); };</div><div>filter f_userauth { not facility(local3); };</div><div>filter f_notifier_filter { not match("ASA-4-302015|ASA-4-3020<wbr>13|TRAFFIC|permitted|Deny|Deni<wbr>ed|denied", value("MESSAGE")); };</div><div>filter f_gtsadfeed_filter { host("*-adc*.<a href="http://abc.xyz.com" target="_blank">abc.xyz.com</a>" type(glob)); };</div><div>rewrite f_rewrite_name { set("$FULLHOST_FROM", value("HOST")</div><div> condition(not tags("windows")</div><div> and not match("REMOTELOG", value("MESSAGE")))); };</div><div>rewrite r_rewrite_name_windows { set("$FULLHOST_FROM", value("HOST")</div><div> condition(not match("REMOTELOG", value("MESSAGE")))); };</div><div>rewrite r_snmptrapd { subst("^([^ ]+) (.*)", "${2}", value("MESSAGE"));</div><div> set("${1}", value("HOST")); };</div><div>log { source(s_sys); filter(f_kernel); destination(d_kern); };</div><div>log { source(s_sys); filter(f_default); destination(d_mesg); };</div><div>log { source(s_sys); filter(f_auth); destination(d_auth); };</div><div>log { source(s_sys); filter(f_mail); destination(d_mail); };</div><div>log { source(s_sys); filter(f_emergency); destination(d_mlal); };</div><div>log { source(s_sys); filter(f_news); destination(d_spol); };</div><div>log { source(s_sys); filter(f_boot); destination(d_boot); };</div><div>log { source(s_sys); filter(f_cron); destination(d_cron); };</div><div>log {</div><div> source(s_sys);</div><div> filter(f_snmptrapd);</div><div> rewrite(r_snmptrapd);</div><div> destination(d_networkx);</div><div> destination(d_logstash);</div><div> log {</div><div> filter(f_notifier_filter);</div><div> destination(d_notifier);</div><div> };</div><div>};</div><div>log {</div><div> source(s_net);</div><div> source(s_udp);</div><div> source(s_tcp);</div><div> source(s_sys);</div><div> source(s_windows);</div><div> log {</div><div> filter(f_userauth);</div><div> destination(d_userauth);</div><div> destination(d_otto);</div><div> };</div><div> log {</div><div> rewrite(f_rewrite_name);</div><div> log {</div><div> destination(d_logstash);</div><div> };</div><div> log {</div><div> filter(f_nocf3);</div><div> filter(f_nodhcpd);</div><div> destination(d_graylog);</div><div> };</div><div> log {</div><div> filter(f_user);</div><div> destination(d_userx);</div><div> };</div><div> log {</div><div> filter(f_kernel);</div><div> destination(d_kernx);</div><div> };</div><div> log {</div><div> filter(f_mail);</div><div> destination(d_mailx);</div><div> };</div><div> log {</div><div> filter(f_daemon);</div><div> destination(d_daemonx);</div><div> };</div><div> log {</div><div> filter(f_auth);</div><div> destination(d_authx);</div><div> };</div><div> log {</div><div> filter(f_lpr);</div><div> destination(d_lprx);</div><div> };</div><div> log {</div><div> filter(f_cron);</div><div> destination(d_cronx);</div><div> };</div><div> log {</div><div> filter(f_local0);</div><div> destination(d_local0);</div><div> };</div><div> log {</div><div> filter(f_local1);</div><div> destination(d_local1);</div><div> };</div><div> log {</div><div> filter(f_local2);</div><div> destination(d_authx);</div><div> };</div><div> log {</div><div> filter(f_dnslogs);</div><div> destination(d_dnslogs);</div><div> };</div><div> log {</div><div> filter(f_local4);</div><div> destination(d_firewallx);</div><div> log {</div><div> filter(f_notifier_filter);</div><div> destination(d_notifier);</div><div> };</div><div> };</div><div> log {</div><div> filter(f_local5);</div><div> destination(d_networkx);</div><div> log {</div><div> filter(f_notifier_filter);</div><div> destination(d_notifier);</div><div> };</div><div> };</div><div> log {</div><div> filter(f_local6);</div><div> destination(d_local6);</div><div> };</div><div> log {</div><div> filter(f_local7);</div><div> destination(d_mp);</div><div> };</div><div> log {</div><div> filter(f_windows);</div><div> filter(f_gtsadfeed_filter);</div><div> rewrite(r_rewrite_name_windows<wbr>);</div><div> destination(d_gtsadfeed);</div><div> };</div><div> };</div><div>};</div></div></div><div><br></div><div><br></div><div>Could you please advise?</div><div><br></div><div>Regards,</div><div>Soumyadip</div></div>
</div><br></div>
<br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/<wbr>mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/<wbr>support/documentation/?<wbr>product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/<wbr>syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>