Hello!

In the statistics it can be seen that the log message is not sent to the HDFS server:
dropped='dst.java(d_hdfs#0 java_dst hdfs hdfs://x.x.x.x:25000 /user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=0'
processed='dst.java(d_hdfs#0 java_dst hdfs hdfs://x.x.x.x:25000 /user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=1'
queued='dst.java(d_hdfs#0 java_dst hdfs hdfs://x.x.x.x:25000 /user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=1'

Well, generally on write error there should be an exception that results in an error message.

You should try debugging it either in running syslog-ng in foreground (-F option), forwarding internal logs to stderr (-e) and with debug mode (-dv) on.
Or in service mode use the internal() source in your config and connect it to a destination (e.g. file()) which you prefer.

You could turn on debug messages on java side too using jvm_options() in syslog-ng config and configuring the log4j logging service, e.g.:
options {
        jvm_options("-Dlog4j.configuration=file:/etc/hadoop/log4j.properties -Dlog4j.debug=true");
};

Regards,
Gabor

On Fri, Aug 17, 2018 at 10:34 AM Czanik, Péter <peter.czanik@balabit.com> wrote:
Hi,

As https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng314/ also writes: "Java is enabled, but JAR dependencies are not provided in package, except for Elasticsearch http mode." The syslog-ng-java-deps.noarch contains build time dependencies. Probably I should rename the package to syslog-ng-java-build-deps...

Check the documentation at https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/33#TOPIC-956506 on how to download and configure HDFS related JAR dependencies.

Bye,


On Fri, Aug 17, 2018 at 10:22 AM, Lee Keng Ket <kengket@gmail.com> wrote:
Hi,

I'm trying to connect syslog-ng 3.14.1 to HDFS to store the syslog messages. The syslog-ng can start without error, and it's able to write into local file. However, the log is not written to the HDFS. As there is no single error, I'm not sure how I should troubleshoot on this.

I have installed the syslog-ng from this repo, https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng314/repo/epel-7/czanik-syslog-ng314-epel-7.repo

Installed Packages
syslog-ng.x86_64                                                                        3.14.1-4.el7.centos                                                              @czanik-syslog-ng314
syslog-ng-java.x86_64                                                                   3.14.1-4.el7.centos                                                              @czanik-syslog-ng314
syslog-ng-java-deps.noarch                                                              1.0-2                                                                            @czanik-syslog-ng314

This is the message from /var/log/message:
Log statistics; processed='src.internal(s_sys#0)=1', stamp='src.internal(s_sys#0)=1534491834', processed='destination(d_spol)=0', processed='destination(d_mlal)=0', processed='center(received)=2', processed='destination(d_mesg)=1', processed='destination(d_mail)=0', processed='destination(d_auth)=0', processed='destination(d_cron)=0', processed='destination(d_hdfs)=1', processed='center(queued)=3', queued='global(scratch_buffers_count)=0', processed='source(remote_log)=1', dropped='dst.java(d_hdfs#0,java_dst,hdfs,hdfs://x.x.x.x:25000,/user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=0', processed='dst.java(d_hdfs#0,java_dst,hdfs,hdfs://x.x.x.x:25000,/user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=1', queued='dst.java(d_hdfs#0,java_dst,hdfs,hdfs://x.x.x.x:25000,/user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=1', processed='global(payload_reallocs)=0', processed='src.journald(journal)=0', stamp='src.journald(journal)=0', processed='global(sdata_updates)=0', queued='global(scratch_buffers_bytes)=0', processed='destination(d_boot)=0', processed='destination(d_kern)=0', processed='source(s_sys)=1', processed='destination(remote)=1', processed='global(internal_queue_length)=0', processed='global(msg_clones)=0'

Anyone has any idea how should I proceed the troubleshooting?

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq



______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq