[syslog-ng] Connect syslog-ng to HDFS
SZIGETVÁRI János
jszigetvari at gmail.com
Wed Aug 22 15:08:03 UTC 2018
Hello,
I remember seeing this very error message, when syslog-ng was not able to
find the hdfs libraries at the directories I specified as client-lib-dir().
Then I realized, my directory was called "libs", and syslog-ng was looking
for them under "lib".
Regards,
János
--
Janos SZIGETVARI
RHCE, License no. 150-053-692
<https://www.redhat.com/rhtapps/verify/?certId=150-053-692>
LinkedIn: linkedin.com/in/janosszigetvari
__ at __˚V˚
Make the switch to open (source) applications, protocols, formats now:
- windows -> Linux, iexplore -> Firefox, msoffice -> LibreOffice
- msn -> jabber protocol (Pidgin, Google Talk)
- mp3 -> ogg, wmv -> ogg, jpg -> png, doc/xls/ppt -> odt/ods/odp
Nagy, Gábor <gabor.nagy at oneidentity.com> ezt írta (időpont: 2018. aug. 21.,
K, 12:48):
> Hi,
>
> Can you tell ne what is the version of the hadoop lib you use with
> syslog-ng, please?
>
> Can you share your syslog-ng configuration, mainly the hdfs part, please?
>
> Regards,
> Gabor
>
> On Tue, Aug 21, 2018 at 4:15 AM Lee Keng Ket <kengket at gmail.com> wrote:
>
>> Hi, Gabor
>>
>> I have run it, seems like it stops at the HDFS side.
>>
>> [2018-08-21T10:07:51.212015] Worker thread started; driver='d_hdfs#0'
>> [2018-08-21T10:07:51.212499] Running application hooks; hook='1'
>> [2018-08-21T10:07:51.212516] Running application hooks; hook='3'
>> [2018-08-21T10:07:51.212595] syslog-ng starting up; version='3.14.1'
>> [2018-08-21T10:07:51.214113] Opening hdfs;
>> [2018-08-21T10:08:01.215622] Opening hdfs;
>> [2018-08-21T10:08:11.216050] Opening hdfs;
>> [2018-08-21T10:08:21.226340] Opening hdfs;
>> [2018-08-21T10:08:31.236589] Opening hdfs;
>> [2018-08-21T10:08:41.240623] Opening hdfs;
>> [2018-08-21T10:08:51.250879] Opening hdfs;
>> [2018-08-21T10:09:01.261172] Opening hdfs;
>> [2018-08-21T10:09:11.271410] Opening hdfs;
>> [2018-08-21T10:09:21.281685] Opening hdfs;
>> [2018-08-21T10:09:31.290765] Opening hdfs;
>> [2018-08-21T10:09:41.301098] Opening hdfs;
>> [2018-08-21T10:09:51.311362] Opening hdfs;
>> [2018-08-21T10:10:01.321152] Opening hdfs;
>> [2018-08-21T10:10:11.321818] Opening hdfs;
>> [2018-08-21T10:10:21.330114] Opening hdfs;
>> [2018-08-21T10:10:31.340413] Opening hdfs;
>> [2018-08-21T10:10:41.350654] Opening hdfs;
>> [2018-08-21T10:10:51.354016] Opening hdfs;
>> [2018-08-21T10:11:01.364267] Opening hdfs;
>> [2018-08-21T10:11:11.374516] Opening hdfs;
>> [2018-08-21T10:11:21.384761] Opening hdfs;
>> [2018-08-21T10:11:31.395017] Opening hdfs;
>> [2018-08-21T10:11:41.402256] Opening hdfs;
>> [2018-08-21T10:11:51.404097] Opening hdfs;
>> ^C[2018-08-21T10:11:59.672252] syslog-ng shutting down; version='3.14.1'
>> Exception in thread "" java.lang.NoClassDefFoundError:
>> org/apache/hadoop/conf/Configuration
>> at
>> org.syslog_ng.hdfs.HdfsDestination.open(HdfsDestination.java:92)
>> at org.syslog_ng.LogDestination.openProxy(LogDestination.java:65)
>> [2018-08-21T10:11:59.774895] Worker thread finished; driver='d_hdfs#0'
>> [2018-08-21T10:11:59.775384] Closing log transport fd; fd='13'
>> [2018-08-21T10:11:59.775508] Deinitialize hdfs destination;
>> [2018-08-21T10:11:59.776534] Java machine free;
>> [2018-08-21T10:11:59.778421] Running application hooks; hook='4'
>>
>> Any idea what to be checked further?
>>
>> Thank you.
>>
>> On Fri, Aug 17, 2018 at 4:45 PM Nagy, Gábor <gabor.nagy at oneidentity.com>
>> wrote:
>>
>>> Hello!
>>>
>>> In the statistics it can be seen that the log message is not sent to the
>>> HDFS server:
>>> dropped='dst.java(d_hdfs#0 java_dst hdfs hdfs://x.x.x.x:25000
>>> /user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=0'
>>> processed='dst.java(d_hdfs#0 java_dst hdfs hdfs://x.x.x.x:25000
>>> /user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=1'
>>> queued='dst.java(d_hdfs#0 java_dst hdfs hdfs://x.x.x.x:25000
>>> /user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=1'
>>>
>>> Well, generally on write error there should be an exception that results
>>> in an error message.
>>>
>>> You should try debugging it either in running syslog-ng in foreground
>>> (-F option), forwarding internal logs to stderr (-e) and with debug mode
>>> (-dv) on.
>>> Or in service mode use the internal() source in your config and connect
>>> it to a destination (e.g. file()) which you prefer.
>>>
>>> You could turn on debug messages on java side too using jvm_options() in
>>> syslog-ng config and configuring the log4j logging service, e.g.:
>>> options {
>>>
>>> jvm_options("-Dlog4j.configuration=file:/etc/hadoop/log4j.properties
>>> -Dlog4j.debug=true");
>>> };
>>>
>>> Regards,
>>> Gabor
>>>
>>> On Fri, Aug 17, 2018 at 10:34 AM Czanik, Péter <peter.czanik at balabit.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> As https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng314/ also
>>>> writes: "Java is enabled, but JAR dependencies are not provided in package,
>>>> except for Elasticsearch http mode." The syslog-ng-java-deps.noarch
>>>> contains build time dependencies. Probably I should rename the package to
>>>> syslog-ng-java-build-deps...
>>>>
>>>> Check the documentation at
>>>> https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/33#TOPIC-956506
>>>> on how to download and configure HDFS related JAR dependencies.
>>>>
>>>> Bye,
>>>>
>>>> Peter Czanik (CzP) <peter.czanik at balabit.com>
>>>> Balabit / syslog-ng upstream
>>>> https://syslog-ng.com/community/
>>>> https://twitter.com/PCzanik
>>>>
>>>> On Fri, Aug 17, 2018 at 10:22 AM, Lee Keng Ket <kengket at gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I'm trying to connect syslog-ng 3.14.1 to HDFS to store the syslog
>>>>> messages. The syslog-ng can start without error, and it's able to write
>>>>> into local file. However, the log is not written to the HDFS. As there is
>>>>> no single error, I'm not sure how I should troubleshoot on this.
>>>>>
>>>>> I have installed the syslog-ng from this repo,
>>>>> https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng314/repo/epel-7/czanik-syslog-ng314-epel-7.repo
>>>>>
>>>>> Installed Packages
>>>>> syslog-ng.x86_64
>>>>> 3.14.1-4.el7.centos
>>>>> @czanik-syslog-ng314
>>>>> syslog-ng-java.x86_64
>>>>> 3.14.1-4.el7.centos
>>>>> @czanik-syslog-ng314
>>>>> syslog-ng-java-deps.noarch
>>>>> 1.0-2
>>>>> @czanik-syslog-ng314
>>>>>
>>>>> This is the message from /var/log/message:
>>>>> Log statistics; processed='src.internal(s_sys#0)=1',
>>>>> stamp='src.internal(s_sys#0)=1534491834',
>>>>> processed='destination(d_spol)=0', processed='destination(d_mlal)=0',
>>>>> processed='center(received)=2', processed='destination(d_mesg)=1',
>>>>> processed='destination(d_mail)=0', processed='destination(d_auth)=0',
>>>>> processed='destination(d_cron)=0', processed='destination(d_hdfs)=1',
>>>>> processed='center(queued)=3', queued='global(scratch_buffers_count)=0',
>>>>> processed='source(remote_log)=1',
>>>>> dropped='dst.java(d_hdfs#0,java_dst,hdfs,hdfs://x.x.x.x:25000,/user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=0',
>>>>> processed='dst.java(d_hdfs#0,java_dst,hdfs,hdfs://x.x.x.x:25000,/user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=1',
>>>>> queued='dst.java(d_hdfs#0,java_dst,hdfs,hdfs://x.x.x.x:25000,/user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=1',
>>>>> processed='global(payload_reallocs)=0',
>>>>> processed='src.journald(journal)=0', stamp='src.journald(journal)=0',
>>>>> processed='global(sdata_updates)=0',
>>>>> queued='global(scratch_buffers_bytes)=0',
>>>>> processed='destination(d_boot)=0', processed='destination(d_kern)=0',
>>>>> processed='source(s_sys)=1', processed='destination(remote)=1',
>>>>> processed='global(internal_queue_length)=0',
>>>>> processed='global(msg_clones)=0'
>>>>>
>>>>> Anyone has any idea how should I proceed the troubleshooting?
>>>>>
>>>>>
>>>>> ______________________________________________________________________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation:
>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>>>>
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180822/2f30f2b2/attachment-0001.html>
More information about the syslog-ng
mailing list