[syslog-ng] Connect syslog-ng to HDFS

[Czanik, Péter]

Hi,
Yes, that's right. Just seeing syslog-ng-java-deps installed I assumed that
you skipped that step.
Bye,

Peter Czanik (CzP) <peter.czanik at balabit.com>
Balabit / syslog-ng upstream
https://syslog-ng.com/community/
https://twitter.com/PCzanik

On Fri, Aug 17, 2018 at 11:27 AM, Lee Keng Ket <kengket at gmail.com> wrote:

> Hi, Peter
>
> I have put in all the jar files from the Hadoop client to the
> /opt/hadoop/lib and declare this in the conf file.
> This should solve the JAR dependencies, right? Or please correct me if I'm
> wrong.
>
> Thank you.
>
> Regards,
> KK
>
> On Fri, Aug 17, 2018 at 4:34 PM Czanik, Péter <peter.czanik at balabit.com>
> wrote:
>
>> Hi,
>>
>> As https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng314/ also
>> writes: "Java is enabled, but JAR dependencies are not provided in package,
>> except for Elasticsearch http mode." The syslog-ng-java-deps.noarch
>> contains build time dependencies. Probably I should rename the package to
>> syslog-ng-java-build-deps...
>>
>> Check the documentation at https://www.syslog-ng.com/
>> technical-documents/doc/syslog-ng-open-source-edition/
>> 3.16/administration-guide/33#TOPIC-956506 on how to download and
>> configure HDFS related JAR dependencies.
>>
>> Bye,
>>
>> Peter Czanik (CzP) <peter.czanik at balabit.com>
>> Balabit / syslog-ng upstream
>> https://syslog-ng.com/community/
>> https://twitter.com/PCzanik
>>
>> On Fri, Aug 17, 2018 at 10:22 AM, Lee Keng Ket <kengket at gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I'm trying to connect syslog-ng 3.14.1 to HDFS to store the syslog
>>> messages. The syslog-ng can start without error, and it's able to write
>>> into local file. However, the log is not written to the HDFS. As there is
>>> no single error, I'm not sure how I should troubleshoot on this.
>>>
>>> I have installed the syslog-ng from this repo,
>>> https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng314/
>>> repo/epel-7/czanik-syslog-ng314-epel-7.repo
>>>
>>> Installed Packages
>>> syslog-ng.x86_64
>>> 3.14.1-4.el7.centos
>>> @czanik-syslog-ng314
>>> syslog-ng-java.x86_64
>>> 3.14.1-4.el7.centos
>>> @czanik-syslog-ng314
>>> syslog-ng-java-deps.noarch
>>> 1.0-2
>>> @czanik-syslog-ng314
>>>
>>> This is the message from /var/log/message:
>>> Log statistics; processed='src.internal(s_sys#0)=1',
>>> stamp='src.internal(s_sys#0)=1534491834', processed='destination(d_spol)=0',
>>> processed='destination(d_mlal)=0', processed='center(received)=2',
>>> processed='destination(d_mesg)=1', processed='destination(d_mail)=0',
>>> processed='destination(d_auth)=0', processed='destination(d_cron)=0',
>>> processed='destination(d_hdfs)=1', processed='center(queued)=3',
>>> queued='global(scratch_buffers_count)=0', processed='source(remote_log)=1',
>>> dropped='dst.java(d_hdfs#0,java_dst,hdfs,hdfs://x.x.x.x:
>>> 25000,/user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=0',
>>> processed='dst.java(d_hdfs#0,java_dst,hdfs,hdfs://x.x.x.x:
>>> 25000,/user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=1',
>>> queued='dst.java(d_hdfs#0,java_dst,hdfs,hdfs://x.x.x.x:
>>> 25000,/user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=1',
>>> processed='global(payload_reallocs)=0', processed='src.journald(journal)=0',
>>> stamp='src.journald(journal)=0', processed='global(sdata_updates)=0',
>>> queued='global(scratch_buffers_bytes)=0', processed='destination(d_boot)=0',
>>> processed='destination(d_kern)=0', processed='source(s_sys)=1',
>>> processed='destination(remote)=1', processed='global(internal_queue_length)=0',
>>> processed='global(msg_clones)=0'
>>>
>>> Anyone has any idea how should I proceed the troubleshooting?
>>>
>>> ____________________________________________________________
>>> __________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support/documentation/?
>>> product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?
>> product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180817/191a54f0/attachment-0001.html>