<div dir="ltr"><div><div>Hi, Janos<br><br></div>Thanks for your reminder. The directory is correct, but I realize the jar files inside the lib is read-only. After I change it to executable, the error messages are different, no longer hang at opening HDFS.<br><br></div>Thank you.<br></div><br><div class="gmail_quote"><div dir="ltr">On Wed, Aug 22, 2018 at 11:08 PM SZIGETVÁRI János <<a href="mailto:jszigetvari@gmail.com">jszigetvari@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hello,</div><div><br></div><div>I remember seeing this very error message, when syslog-ng was not able to find the hdfs libraries at the directories I specified as client-lib-dir().</div><div>Then I realized, my directory was called "libs", and syslog-ng was looking for them under "lib".<br></div><div><br></div><div>Regards,</div><div>János<br></div><div><br></div><div><div dir="ltr" class="m_-6539793064078013541gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">--</div><div dir="ltr">Janos SZIGETVARI<br><span>RHCE, License no. <a href="https://www.redhat.com/rhtapps/verify/?certId=150-053-692" target="_blank">150-053-692</a></span><br></div><div dir="ltr"><span><br></span></div><div dir="ltr"><span>LinkedIn: <a href="http://linkedin.com/in/janosszigetvari" target="_blank">linkedin.com/in/janosszigetvari</a></span><br><br>__@__˚V˚<br>Make the switch to open (source) applications, protocols, formats now:<br>- windows -> Linux, iexplore -> Firefox, msoffice -> LibreOffice<br>- msn -> jabber protocol (Pidgin, Google Talk)<br>- mp3 -> ogg, wmv -> ogg, jpg -> png, doc/xls/ppt -> odt/ods/odp</div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr">Nagy, Gábor <<a href="mailto:gabor.nagy@oneidentity.com" target="_blank">gabor.nagy@oneidentity.com</a>> ezt írta (időpont: 2018. aug. 21., K, 12:48):<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>Can you tell ne what is the version of the hadoop lib you use with syslog-ng, please?</div><div><br></div><div>Can you share your syslog-ng configuration, mainly the hdfs part, please?</div><div><br></div><div>Regards,</div><div>Gabor</div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Aug 21, 2018 at 4:15 AM Lee Keng Ket <<a href="mailto:kengket@gmail.com" target="_blank">kengket@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>Hi, Gabor<br><br></div>I have run it, seems like it stops at the HDFS side.<br><br>[2018-08-21T10:07:51.212015] Worker thread started; driver='d_hdfs#0'<br>[2018-08-21T10:07:51.212499] Running application hooks; hook='1'<br>[2018-08-21T10:07:51.212516] Running application hooks; hook='3'<br>[2018-08-21T10:07:51.212595] syslog-ng starting up; version='3.14.1'<br>[2018-08-21T10:07:51.214113] Opening hdfs;<br>[2018-08-21T10:08:01.215622] Opening hdfs;<br>[2018-08-21T10:08:11.216050] Opening hdfs;<br>[2018-08-21T10:08:21.226340] Opening hdfs;<br>[2018-08-21T10:08:31.236589] Opening hdfs;<br>[2018-08-21T10:08:41.240623] Opening hdfs;<br>[2018-08-21T10:08:51.250879] Opening hdfs;<br>[2018-08-21T10:09:01.261172] Opening hdfs;<br>[2018-08-21T10:09:11.271410] Opening hdfs;<br>[2018-08-21T10:09:21.281685] Opening hdfs;<br>[2018-08-21T10:09:31.290765] Opening hdfs;<br>[2018-08-21T10:09:41.301098] Opening hdfs;<br>[2018-08-21T10:09:51.311362] Opening hdfs;<br>[2018-08-21T10:10:01.321152] Opening hdfs;<br>[2018-08-21T10:10:11.321818] Opening hdfs;<br>[2018-08-21T10:10:21.330114] Opening hdfs;<br>[2018-08-21T10:10:31.340413] Opening hdfs;<br>[2018-08-21T10:10:41.350654] Opening hdfs;<br>[2018-08-21T10:10:51.354016] Opening hdfs;<br>[2018-08-21T10:11:01.364267] Opening hdfs;<br>[2018-08-21T10:11:11.374516] Opening hdfs;<br>[2018-08-21T10:11:21.384761] Opening hdfs;<br>[2018-08-21T10:11:31.395017] Opening hdfs;<br>[2018-08-21T10:11:41.402256] Opening hdfs;<br>[2018-08-21T10:11:51.404097] Opening hdfs;<br>^C[2018-08-21T10:11:59.672252] syslog-ng shutting down; version='3.14.1'<br>Exception in thread "" java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration<br>        at org.syslog_ng.hdfs.HdfsDestination.open(HdfsDestination.java:92)<br>        at org.syslog_ng.LogDestination.openProxy(LogDestination.java:65)<br>[2018-08-21T10:11:59.774895] Worker thread finished; driver='d_hdfs#0'<br>[2018-08-21T10:11:59.775384] Closing log transport fd; fd='13'<br>[2018-08-21T10:11:59.775508] Deinitialize hdfs destination;<br>[2018-08-21T10:11:59.776534] Java machine free;<br>[2018-08-21T10:11:59.778421] Running application hooks; hook='4'<br><br></div>Any idea what to be checked further?<br><br></div>Thank you.<br></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Aug 17, 2018 at 4:45 PM Nagy, Gábor <<a href="mailto:gabor.nagy@oneidentity.com" target="_blank">gabor.nagy@oneidentity.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hello!<div><br></div><div>In the statistics it can be seen that the log message is not sent to the HDFS server:<br></div><div><div>dropped='dst.java(d_hdfs#0 java_dst hdfs hdfs://x.x.x.x:25000 /user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=0'</div><div>processed='dst.java(d_hdfs#0 java_dst hdfs hdfs://x.x.x.x:25000 /user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=1'</div><div>queued='dst.java(d_hdfs#0 java_dst hdfs hdfs://x.x.x.x:25000 /user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=1'</div></div><div><br></div><div>Well, generally on write error there should be an exception that results in an error message.</div><div><br></div><div>You should try debugging it either in running syslog-ng in foreground (-F option), forwarding internal logs to stderr (-e) and with debug mode (-dv) on.<br>Or in service mode use the internal() source in your config and connect it to a destination (e.g. file()) which you prefer.</div><div><br></div><div>You could turn on debug messages on java side too using jvm_options() in syslog-ng config and configuring the log4j logging service, e.g.:<br><div>options {</div><div>        jvm_options("-Dlog4j.configuration=file:/etc/hadoop/log4j.properties -Dlog4j.debug=true");</div><div>};</div></div><div><br></div><div>Regards,</div><div>Gabor</div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Aug 17, 2018 at 10:34 AM Czanik, Péter <<a href="mailto:peter.czanik@balabit.com" target="_blank">peter.czanik@balabit.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi,</div><div><br></div><div>As <a href="https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng314/" target="_blank">https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng314/</a> also writes: "Java is enabled, but JAR dependencies are not provided in package, except for Elasticsearch http mode." The syslog-ng-java-deps.noarch contains build time dependencies. Probably I should rename the package to syslog-ng-java-build-deps...</div><div><br></div><div>Check the documentation at <a href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/33#TOPIC-956506" target="_blank">https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/33#TOPIC-956506</a> on how to download and configure HDFS related JAR dependencies.</div><div><br></div><div>Bye,<br></div></div><div class="gmail_extra"><br clear="all"><div><div class="m_-6539793064078013541m_-8697699300578092212m_-8289036816759268877m_-4468223703778826464m_9221686417707055935gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Peter Czanik (CzP) <<a href="mailto:peter.czanik@balabit.com" target="_blank">peter.czanik@balabit.com</a>><br>Balabit / syslog-ng upstream<br><a href="https://syslog-ng.com/community/" target="_blank">https://syslog-ng.com/community/</a><br><a href="https://twitter.com/PCzanik" target="_blank">https://twitter.com/PCzanik</a></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Fri, Aug 17, 2018 at 10:22 AM, Lee Keng Ket <span dir="ltr"><<a href="mailto:kengket@gmail.com" target="_blank">kengket@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>Hi,<br><br></div>I'm trying to connect syslog-ng 3.14.1 to HDFS to store the syslog messages. The syslog-ng can start without error, and it's able to write into local file. However, the log is not written to the HDFS. As there is no single error, I'm not sure how I should troubleshoot on this.<br><br></div><div>I have installed the syslog-ng from this repo, <a href="https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng314/repo/epel-7/czanik-syslog-ng314-epel-7.repo" target="_blank">https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng314/repo/epel-7/czanik-syslog-ng314-epel-7.repo</a><br><br>Installed Packages<br>syslog-ng.x86_64                                                                        3.14.1-4.el7.centos                                                              @czanik-syslog-ng314<br>syslog-ng-java.x86_64                                                                   3.14.1-4.el7.centos                                                              @czanik-syslog-ng314<br>syslog-ng-java-deps.noarch                                                              1.0-2                                                                            @czanik-syslog-ng314<br></div><div><br></div>This is the message from /var/log/message:<br>Log statistics; processed='src.internal(s_sys#0)=1', stamp='src.internal(s_sys#0)=1534491834', processed='destination(d_spol)=0', processed='destination(d_mlal)=0', processed='center(received)=2', processed='destination(d_mesg)=1', processed='destination(d_mail)=0', processed='destination(d_auth)=0', processed='destination(d_cron)=0', processed='destination(d_hdfs)=1', processed='center(queued)=3', queued='global(scratch_buffers_count)=0', processed='source(remote_log)=1', dropped='dst.java(d_hdfs#0,java_dst,hdfs,hdfs://x.x.x.x:25000,/user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=0', processed='dst.java(d_hdfs#0,java_dst,hdfs,hdfs://x.x.x.x:25000,/user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=1', queued='dst.java(d_hdfs#0,java_dst,hdfs,hdfs://x.x.x.x:25000,/user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=1', processed='global(payload_reallocs)=0', processed='src.journald(journal)=0', stamp='src.journald(journal)=0', processed='global(sdata_updates)=0', queued='global(scratch_buffers_bytes)=0', processed='destination(d_boot)=0', processed='destination(d_kern)=0', processed='source(s_sys)=1', processed='destination(remote)=1', processed='global(internal_queue_length)=0', processed='global(msg_clones)=0'<br><br></div>Anyone has any idea how should I proceed the troubleshooting?<br></div>
<br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div>