[syslog-ng] Connect syslog-ng to HDFS

[Nagy, Gábor]

Hello!

In the statistics it can be seen that the log message is not sent to the
HDFS server:
dropped='dst.java(d_hdfs#0 java_dst hdfs hdfs://x.x.x.x:25000
/user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=0'
processed='dst.java(d_hdfs#0 java_dst hdfs hdfs://x.x.x.x:25000
/user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=1'
queued='dst.java(d_hdfs#0 java_dst hdfs hdfs://x.x.x.x:25000
/user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=1'

Well, generally on write error there should be an exception that results in
an error message.

You should try debugging it either in running syslog-ng in foreground (-F
option), forwarding internal logs to stderr (-e) and with debug mode (-dv)
on.
Or in service mode use the internal() source in your config and connect it
to a destination (e.g. file()) which you prefer.

You could turn on debug messages on java side too using jvm_options() in
syslog-ng config and configuring the log4j logging service, e.g.:
options {

jvm_options("-Dlog4j.configuration=file:/etc/hadoop/log4j.properties
-Dlog4j.debug=true");
};

Regards,
Gabor

On Fri, Aug 17, 2018 at 10:34 AM Czanik, Péter <peter.czanik at balabit.com>
wrote:

> Hi,
>
> As https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng314/ also
> writes: "Java is enabled, but JAR dependencies are not provided in package,
> except for Elasticsearch http mode." The syslog-ng-java-deps.noarch
> contains build time dependencies. Probably I should rename the package to
> syslog-ng-java-build-deps...
>
> Check the documentation at
> https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/33#TOPIC-956506
> on how to download and configure HDFS related JAR dependencies.
>
> Bye,
>
> Peter Czanik (CzP) <peter.czanik at balabit.com>
> Balabit / syslog-ng upstream
> https://syslog-ng.com/community/
> https://twitter.com/PCzanik
>
> On Fri, Aug 17, 2018 at 10:22 AM, Lee Keng Ket <kengket at gmail.com> wrote:
>
>> Hi,
>>
>> I'm trying to connect syslog-ng 3.14.1 to HDFS to store the syslog
>> messages. The syslog-ng can start without error, and it's able to write
>> into local file. However, the log is not written to the HDFS. As there is
>> no single error, I'm not sure how I should troubleshoot on this.
>>
>> I have installed the syslog-ng from this repo,
>> https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng314/repo/epel-7/czanik-syslog-ng314-epel-7.repo
>>
>> Installed Packages
>> syslog-ng.x86_64
>> 3.14.1-4.el7.centos
>> @czanik-syslog-ng314
>> syslog-ng-java.x86_64
>> 3.14.1-4.el7.centos
>> @czanik-syslog-ng314
>> syslog-ng-java-deps.noarch
>> 1.0-2
>> @czanik-syslog-ng314
>>
>> This is the message from /var/log/message:
>> Log statistics; processed='src.internal(s_sys#0)=1',
>> stamp='src.internal(s_sys#0)=1534491834',
>> processed='destination(d_spol)=0', processed='destination(d_mlal)=0',
>> processed='center(received)=2', processed='destination(d_mesg)=1',
>> processed='destination(d_mail)=0', processed='destination(d_auth)=0',
>> processed='destination(d_cron)=0', processed='destination(d_hdfs)=1',
>> processed='center(queued)=3', queued='global(scratch_buffers_count)=0',
>> processed='source(remote_log)=1',
>> dropped='dst.java(d_hdfs#0,java_dst,hdfs,hdfs://x.x.x.x:25000,/user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=0',
>> processed='dst.java(d_hdfs#0,java_dst,hdfs,hdfs://x.x.x.x:25000,/user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=1',
>> queued='dst.java(d_hdfs#0,java_dst,hdfs,hdfs://x.x.x.x:25000,/user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=1',
>> processed='global(payload_reallocs)=0',
>> processed='src.journald(journal)=0', stamp='src.journald(journal)=0',
>> processed='global(sdata_updates)=0',
>> queued='global(scratch_buffers_bytes)=0',
>> processed='destination(d_boot)=0', processed='destination(d_kern)=0',
>> processed='source(s_sys)=1', processed='destination(remote)=1',
>> processed='global(internal_queue_length)=0',
>> processed='global(msg_clones)=0'
>>
>> Anyone has any idea how should I proceed the troubleshooting?
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180817/3cfc89f7/attachment-0001.html>