[syslog-ng] Connect syslog-ng to HDFS

[Lee Keng Ket]

Hi,

I'm trying to connect syslog-ng 3.14.1 to HDFS to store the syslog
messages. The syslog-ng can start without error, and it's able to write
into local file. However, the log is not written to the HDFS. As there is
no single error, I'm not sure how I should troubleshoot on this.

I have installed the syslog-ng from this repo,
https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng314/repo/epel-7/czanik-syslog-ng314-epel-7.repo

Installed Packages
syslog-ng.x86_64
3.14.1-4.el7.centos
@czanik-syslog-ng314
syslog-ng-java.x86_64
3.14.1-4.el7.centos
@czanik-syslog-ng314
syslog-ng-java-deps.noarch
1.0-2
@czanik-syslog-ng314

This is the message from /var/log/message:
Log statistics; processed='src.internal(s_sys#0)=1',
stamp='src.internal(s_sys#0)=1534491834',
processed='destination(d_spol)=0', processed='destination(d_mlal)=0',
processed='center(received)=2', processed='destination(d_mesg)=1',
processed='destination(d_mail)=0', processed='destination(d_auth)=0',
processed='destination(d_cron)=0', processed='destination(d_hdfs)=1',
processed='center(queued)=3', queued='global(scratch_buffers_count)=0',
processed='source(remote_log)=1',
dropped='dst.java(d_hdfs#0,java_dst,hdfs,hdfs://x.x.x.x:25000,/user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=0',
processed='dst.java(d_hdfs#0,java_dst,hdfs,hdfs://x.x.x.x:25000,/user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=1',
queued='dst.java(d_hdfs#0,java_dst,hdfs,hdfs://x.x.x.x:25000,/user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=1',
processed='global(payload_reallocs)=0',
processed='src.journald(journal)=0', stamp='src.journald(journal)=0',
processed='global(sdata_updates)=0',
queued='global(scratch_buffers_bytes)=0',
processed='destination(d_boot)=0', processed='destination(d_kern)=0',
processed='source(s_sys)=1', processed='destination(remote)=1',
processed='global(internal_queue_length)=0',
processed='global(msg_clones)=0'

Anyone has any idea how should I proceed the troubleshooting?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180817/f0acbe75/attachment.html>