[syslog-ng] Connect syslog-ng to HDFS

Nagy, Gábor gabor.nagy at oneidentity.com
Mon Aug 27 08:08:45 UTC 2018 

I've asked about append to know whether you will write to the same file
again.

On Mon, Aug 27, 2018 at 10:02 AM Lee Keng Ket <kengket at gmail.com> wrote:

> Hi, Gabor
>
> Yes, I see this note as well, so I'm monitoring how long it takes to write
> into the disk.
>
> I'm using append in hdfs, but I'm not really sure what's the difference
> between no append and append.
>
> Thank you.
>
> Regards,
> KK
>
>
>
> On Mon, 27 Aug 2018, 15:41 Nagy, Gábor, <gabor.nagy at oneidentity.com>
> wrote:
>
>> Hi,
>>
>> As I know we have no control over flushing the file in HDFS.
>> I've checked our admin guide, let me quote it:
>> > You cannot set when log messages are flushed. Hadoop performs this
>> action
>> > automatically, depending on its configured block size, and the amount
>> of data
>> > received. There is no way for the syslog-ng OSE application to
>> influence when the
>> > messages are actually written to disk. This means that syslog-ng OSE
>> cannot
>> > guarantee that a message sent to HDFS is actually written to disk.
>>
>> You can reload syslog-ng periodically. During reload the used files are
>> closed and HDFS should flushes them to the datanodes.
>> Do you use append in your HDFS environment?
>>
>> Regards,
>> Gabor
>>
>> On Mon, Aug 27, 2018 at 9:13 AM Lee Keng Ket <kengket at gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I'm able to send the syslog to HDFS, but it seems not real time. The
>>> file in HDFS does not increase size, until I stop the syslog-ng process,
>>> then all the messages go into HDFS and we can see the file size increase.
>>> Any idea how frequent the syslog-ng send the syslog to HDFS? If the
>>> syslog-ng incoming syslog is a lot, is there any fine-tuning required? The
>>> syslog-ng also write log file in local server, I can see the file size
>>> increase by 40MB in 20min. However, the file in HDFS remains same size
>>> until I stop the syslog-ng process. Once I stop the process, the log is
>>> written into the HDFS.
>>>
>>> This is log stats:
>>> Log statistics; processed='src.internal(s_sys#1)=4',
>>> stamp='src.internal(s_sys#1)=1535352937',
>>> processed='destination(d_mlal)=0', processed='center(received)=30694',
>>> processed='destination(d_spol)=0', processed='destination(d_mesg)=4',
>>> processed='destination(d_mail)=0', processed='destination(d_auth)=0',
>>> processed='destination(d_cron)=0', processed='destination(d_hdfs)=30690',
>>> processed='center(queued)=61384', queued='global(scratch_buffers_count)=0',
>>> processed='source(remote_log)=30690',
>>> dropped='dst.java(d_hdfs#0,java_dst,hdfs,hdfs://CN03:25000,/user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=0',
>>> processed='dst.java(d_hdfs#0,java_dst,hdfs,hdfs://CN03:25000,/user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=30690',
>>> queued='dst.java(d_hdfs#0,java_dst,hdfs,hdfs://CN03:25000,/user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=0',
>>> processed='global(payload_reallocs)=0',
>>> processed='global(sdata_updates)=0',
>>> queued='global(scratch_buffers_bytes)=0',
>>> processed='src.journald(s_sys#0,journal)=0',
>>> stamp='src.journald(s_sys#0,journal)=0', processed='destination(d_boot)=0',
>>> processed='destination(d_kern)=0', processed='source(s_sys)=4',
>>> processed='destination(remote)=30690',
>>> processed='global(internal_queue_length)=0',
>>> processed='global(msg_clones)=0'
>>>
>>> Thank you.
>>>
>>> Regards,
>>> KK
>>>
>>>
>>>
>>> On Mon, Aug 27, 2018 at 10:41 AM Lee Keng Ket <kengket at gmail.com> wrote:
>>>
>>>> Hi, Gabor
>>>>
>>>> I am able to send the syslog to HDFS after adding in the hdfs-resources
>>>> option.
>>>> Thanks a lot for your help.
>>>>
>>>> Thank you.
>>>>
>>>> Regards,
>>>> KK
>>>>
>>>> On Fri, Aug 24, 2018 at 10:29 PM Nagy, Gábor <
>>>> gabor.nagy at oneidentity.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> As I was searching for answer what i saw was that "SaslException: No
>>>>> common protection layer between client and server;" can occur
>>>>> if the configuration differs on client-server side.
>>>>>
>>>>> You should configure the "hadoop.rpc.protection" value in
>>>>> core-site.xml on the client side to the same one as on the server side.
>>>>> To use that xml with hdfs, you should put the filename in syslog-ng
>>>>> config in the hdfs destination block:
>>>>> destination d_hdfs {
>>>>>   ...
>>>>>   hdfs-resources("/path/to/xmlconfig/core-site.xml");
>>>>> };
>>>>>
>>>>> Regards,
>>>>> Gabor
>>>>>
>>>>> On Thu, Aug 23, 2018 at 3:53 AM Lee Keng Ket <kengket at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi, Gabor
>>>>>>
>>>>>> After I change the lib to executable as reminded by Janos, now there
>>>>>> is different error. Any idea if this is Kerberos issue or still the Hadoop
>>>>>> lib issue?
>>>>>>
>>>>>> The Hadoop that I'm using is Huawei FusionInsight. Hadoop HDFS
>>>>>> version should be 2.7.2.
>>>>>>
>>>>>> This is the syslog-ng.conf for HDFS part:
>>>>>> destination d_hdfs {
>>>>>>         hdfs(client_lib_dir("/opt/hadoop/lib")
>>>>>>         hdfs_uri("hdfs://x.x.x.x:25000")
>>>>>>         kerberos-keytab-file("/etc/syslog.keytab")
>>>>>>         kerberos-principal("syslog at HADOOP.COM")
>>>>>>         hdfs_file("/user/syslog/$HOST-$DAY-$MONTH-$YEAR.log")
>>>>>>         template(t_cgnat)
>>>>>>         hdfs-append-enabled(true)
>>>>>>         );
>>>>>> };
>>>>>>
>>>>>> There are the messages I get when starting in debug:
>>>>>>
>>>>>> [2018-08-23T09:40:09.210168] Running application hooks; hook='1'
>>>>>> [2018-08-23T09:40:09.210192] Running application hooks; hook='3'
>>>>>> [2018-08-23T09:40:09.210501] syslog-ng starting up; version='3.14.1'
>>>>>> [2018-08-23T09:40:09.213049] Worker thread started; driver='d_hdfs#0'
>>>>>> [2018-08-23T09:40:09.214922] Opening hdfs;
>>>>>> [2018-08-23T09:40:09.548286] field
>>>>>> org.apache.hadoop.metrics2.lib.MutableRate
>>>>>> org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess
>>>>>> with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=,
>>>>>> sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of
>>>>>> successful kerberos logins and latency (milliseconds)]);
>>>>>> [2018-08-23T09:40:09.561345] field
>>>>>> org.apache.hadoop.metrics2.lib.MutableRate
>>>>>> org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure
>>>>>> with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=,
>>>>>> sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of
>>>>>> failed kerberos logins and latency (milliseconds)]);
>>>>>> [2018-08-23T09:40:09.561608] field
>>>>>> org.apache.hadoop.metrics2.lib.MutableRate
>>>>>> org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with
>>>>>> annotation @org.apache.hadoop.metrics2.annotation.Metric(about=,
>>>>>> sampleName=Ops, always=false, type=DEFAULT, valueName=Time,
>>>>>> value=[GetGroups]);
>>>>>> [2018-08-23T09:40:09.562485] UgiMetrics, User and group related
>>>>>> metrics;
>>>>>> [2018-08-23T09:40:09.604037] Failed to detect a valid hadoop home
>>>>>> directory;
>>>>>> [2018-08-23T09:40:09.687386] setsid exited with exit code 0;
>>>>>> [2018-08-23T09:40:09.715804]  Creating new Groups object;
>>>>>> [2018-08-23T09:40:09.717743] Trying to load the custom-built
>>>>>> native-hadoop library...;
>>>>>> [2018-08-23T09:40:09.718065] Failed to load native-hadoop with error:
>>>>>> java.lang.UnsatisfiedLinkError: no hadoop in java.library.path;
>>>>>> [2018-08-23T09:40:09.718095] java.library.path=//usr/lib64/syslog-ng;
>>>>>> [2018-08-23T09:40:09.718117] Unable to load native-hadoop library for
>>>>>> your platform... using builtin-java classes where applicable;
>>>>>> [2018-08-23T09:40:09.718418] Falling back to shell based;
>>>>>> [2018-08-23T09:40:09.718997] Group mapping
>>>>>> impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
>>>>>> [2018-08-23T09:40:09.753615] Group mapping
>>>>>> impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback\;
>>>>>> cacheTimeout=300000\; warningDeltaMs=5000;
>>>>>> [2018-08-23T09:40:09.905060] hadoop login;
>>>>>> [2018-08-23T09:40:09.906493] hadoop login commit;
>>>>>> [2018-08-23T09:40:09.907300] using kerberos user:syslog at HADOOP.COM;
>>>>>> [2018-08-23T09:40:09.907333] Using user: "syslog at HADOOP.COM" with
>>>>>> name syslog at HADOOP.COM;
>>>>>> [2018-08-23T09:40:09.907592] User entry: "syslog at HADOOP.COM";
>>>>>> [2018-08-23T09:40:09.908005] Login successful for user
>>>>>> syslog at HADOOP.COM using keytab file syslog.keytab;
>>>>>> [2018-08-23T09:40:10.104386] dfs.client.use.legacy.blockreader.local
>>>>>> = false;
>>>>>> [2018-08-23T09:40:10.104436] dfs.client.read.shortcircuit = false;
>>>>>> [2018-08-23T09:40:10.104450] dfs.client.domain.socket.data.traffic =
>>>>>> false;
>>>>>> [2018-08-23T09:40:10.104461] dfs.domain.socket.path = ;
>>>>>> [2018-08-23T09:40:10.121280] Sets
>>>>>> dfs.client.block.write.replace-datanode-on-failure.replication to 0;
>>>>>> [2018-08-23T09:40:10.144892] multipleLinearRandomRetry = null;
>>>>>> [2018-08-23T09:40:10.168901] rpcKind=RPC_PROTOCOL_BUFFER,
>>>>>> rpcRequestWrapperClass=class
>>>>>> org.apache.hadoop.ipc.ProtobufRpcEngine$RpcProtobufRequest,
>>>>>> rpcInvoker=org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker at 4379f1a4
>>>>>> ;
>>>>>> [2018-08-23T09:40:10.173738] getting client out of cache:
>>>>>> org.apache.hadoop.ipc.Client at 5b52b909;
>>>>>> [2018-08-23T09:40:10.238466] finalize() called.;
>>>>>> [2018-08-23T09:40:10.238705] finalize() called.;
>>>>>> [2018-08-23T09:40:10.526021] Both short-circuit local reads and UNIX
>>>>>> domain socket are disabled.;
>>>>>> [2018-08-23T09:40:10.532037] DataTransferProtocol not using
>>>>>> SaslPropertiesResolver, no QOP found in configuration for
>>>>>> dfs.data.transfer.protection;
>>>>>> [2018-08-23T09:40:10.555581] The ping interval is 60000 ms.;
>>>>>> [2018-08-23T09:40:10.556336] Connecting to /x.x.x.x:25000;
>>>>>> [2018-08-23T09:40:10.572385] PrivilegedAction as:syslog at HADOOP.COM
>>>>>> (auth:KERBEROS)
>>>>>> from:org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:771);
>>>>>> [2018-08-23T09:40:10.613823] Get kerberos info proto:interface
>>>>>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB
>>>>>> info:@org.apache.hadoop.security.KerberosInfo(clientPrincipal=,
>>>>>> serverPrincipal=dfs.namenode.kerberos.principal);
>>>>>> [2018-08-23T09:40:10.723447] RPC Server's Kerberos principal name for
>>>>>> protocol=org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB is hdfs/
>>>>>> hadoop.hadoop.com at HADOOP.COM;
>>>>>> [2018-08-23T09:40:10.723514] Creating SASL GSSAPI(KERBEROS)  client
>>>>>> to authenticate to service at hadoop.hadoop.com;
>>>>>> [2018-08-23T09:40:10.730466] Use KERBEROS authentication for protocol
>>>>>> ClientNamenodeProtocolPB;
>>>>>> [2018-08-23T09:40:10.758296] PrivilegedActionException
>>>>>> as:syslog at HADOOP.COM (auth:KERBEROS)
>>>>>> cause:javax.security.sasl.SaslException: No common protection layer between
>>>>>> client and server;
>>>>>> [2018-08-23T09:40:10.759031] PrivilegedAction as:syslog at HADOOP.COM
>>>>>> (auth:KERBEROS)
>>>>>> from:org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:694);
>>>>>> [2018-08-23T09:40:10.759531] Exception encountered while connecting
>>>>>> to the server : javax.security.sasl.SaslException: No common protection
>>>>>> layer between client and server;
>>>>>> [2018-08-23T09:40:14.446925] PrivilegedAction as:syslog at HADOOP.COM
>>>>>> (auth:KERBEROS)
>>>>>> from:org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:771);
>>>>>> [2018-08-23T09:40:14.447824] Get kerberos info proto:interface
>>>>>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB
>>>>>> info:@org.apache.hadoop.security.KerberosInfo(clientPrincipal=,
>>>>>> serverPrincipal=dfs.namenode.kerberos.principal);
>>>>>> [2018-08-23T09:40:14.447982] RPC Server's Kerberos principal name for
>>>>>> protocol=org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB is hdfs/
>>>>>> hadoop.hadoop.com at HADOOP.COM;
>>>>>> [2018-08-23T09:40:14.448001] Creating SASL GSSAPI(KERBEROS)  client
>>>>>> to authenticate to service at hadoop.hadoop.com;
>>>>>> [2018-08-23T09:40:14.449087] Use KERBEROS authentication for protocol
>>>>>> ClientNamenodeProtocolPB;
>>>>>> [2018-08-23T09:40:14.455070] PrivilegedActionException
>>>>>> as:syslog at HADOOP.COM (auth:KERBEROS)
>>>>>> cause:javax.security.sasl.SaslException: No common protection layer between
>>>>>> client and server;
>>>>>> [2018-08-23T09:40:14.455190] PrivilegedAction as:syslog at HADOOP.COM
>>>>>> (auth:KERBEROS)
>>>>>> from:org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:694);
>>>>>> [2018-08-23T09:40:14.455476] Exception encountered while connecting
>>>>>> to the server : javax.security.sasl.SaslException: No common protection
>>>>>> layer between client and server;
>>>>>> [2018-08-23T09:40:17.206928] PrivilegedAction as:syslog at HADOOP.COM
>>>>>> (auth:KERBEROS)
>>>>>> from:org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:771);
>>>>>> [2018-08-23T09:40:17.207978] Get kerberos info proto:interface
>>>>>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB
>>>>>> info:@org.apache.hadoop.security.KerberosInfo(clientPrincipal=,
>>>>>> serverPrincipal=dfs.namenode.kerberos.principal);
>>>>>> [2018-08-23T09:40:17.208115] RPC Server's Kerberos principal name for
>>>>>> protocol=org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB is hdfs/
>>>>>> hadoop.hadoop.com at HADOOP.COM;
>>>>>> [2018-08-23T09:40:17.208133] Creating SASL GSSAPI(KERBEROS)  client
>>>>>> to authenticate to service at hadoop.hadoop.com;
>>>>>> [2018-08-23T09:40:17.208877] Use KERBEROS authentication for protocol
>>>>>> ClientNamenodeProtocolPB;
>>>>>> [2018-08-23T09:40:17.214382] PrivilegedActionException
>>>>>> as:syslog at HADOOP.COM (auth:KERBEROS)
>>>>>> cause:javax.security.sasl.SaslException: No common protection layer between
>>>>>> client and server;
>>>>>> [2018-08-23T09:40:17.214536] PrivilegedAction as:syslog at HADOOP.COM
>>>>>> (auth:KERBEROS)
>>>>>> from:org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:694);
>>>>>> [2018-08-23T09:40:17.214845] Exception encountered while connecting
>>>>>> to the server : javax.security.sasl.SaslException: No common protection
>>>>>> layer between client and server;
>>>>>> [2018-08-23T09:40:17.535313] PrivilegedAction as:syslog at HADOOP.COM
>>>>>> (auth:KERBEROS)
>>>>>> from:org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:771);
>>>>>> [2018-08-23T09:40:17.536419] Get kerberos info proto:interface
>>>>>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB
>>>>>> info:@org.apache.hadoop.security.KerberosInfo(clientPrincipal=,
>>>>>> serverPrincipal=dfs.namenode.kerberos.principal);
>>>>>> [2018-08-23T09:40:17.536615] RPC Server's Kerberos principal name for
>>>>>> protocol=org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB is hdfs/
>>>>>> hadoop.hadoop.com at HADOOP.COM;
>>>>>> [2018-08-23T09:40:17.536634] Creating SASL GSSAPI(KERBEROS)  client
>>>>>> to authenticate to service at hadoop.hadoop.com;
>>>>>> [2018-08-23T09:40:17.537679] Use KERBEROS authentication for protocol
>>>>>> ClientNamenodeProtocolPB;
>>>>>> [2018-08-23T09:40:17.546375] PrivilegedActionException
>>>>>> as:syslog at HADOOP.COM (auth:KERBEROS)
>>>>>> cause:javax.security.sasl.SaslException: No common protection layer between
>>>>>> client and server;
>>>>>> [2018-08-23T09:40:17.546587] PrivilegedAction as:syslog at HADOOP.COM
>>>>>> (auth:KERBEROS)
>>>>>> from:org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:694);
>>>>>> [2018-08-23T09:40:17.546963] Exception encountered while connecting
>>>>>> to the server : javax.security.sasl.SaslException: No common protection
>>>>>> layer between client and server;
>>>>>> [2018-08-23T09:40:21.891382] PrivilegedAction as:syslog at HADOOP.COM
>>>>>> (auth:KERBEROS)
>>>>>> from:org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:771);
>>>>>> [2018-08-23T09:40:21.892476] Get kerberos info proto:interface
>>>>>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB
>>>>>> info:@org.apache.hadoop.security.KerberosInfo(clientPrincipal=,
>>>>>> serverPrincipal=dfs.namenode.kerberos.principal);
>>>>>> [2018-08-23T09:40:21.909267] RPC Server's Kerberos principal name for
>>>>>> protocol=org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB is hdfs/
>>>>>> hadoop.hadoop.com at HADOOP.COM;
>>>>>> [2018-08-23T09:40:21.909306] Creating SASL GSSAPI(KERBEROS)  client
>>>>>> to authenticate to service at hadoop.hadoop.com;
>>>>>> [2018-08-23T09:40:21.910125] Use KERBEROS authentication for protocol
>>>>>> ClientNamenodeProtocolPB;
>>>>>> [2018-08-23T09:40:21.915324] PrivilegedActionException
>>>>>> as:syslog at HADOOP.COM (auth:KERBEROS)
>>>>>> cause:javax.security.sasl.SaslException: No common protection layer between
>>>>>> client and server;
>>>>>> [2018-08-23T09:40:21.915540] PrivilegedAction as:syslog at HADOOP.COM
>>>>>> (auth:KERBEROS)
>>>>>> from:org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:694);
>>>>>> [2018-08-23T09:40:21.915889] Exception encountered while connecting
>>>>>> to the server : javax.security.sasl.SaslException: No common protection
>>>>>> layer between client and server;
>>>>>> [2018-08-23T09:40:26.095299] PrivilegedAction as:syslog at HADOOP.COM
>>>>>> (auth:KERBEROS)
>>>>>> from:org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:771);
>>>>>> [2018-08-23T09:40:26.096234] Get kerberos info proto:interface
>>>>>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB
>>>>>> info:@org.apache.hadoop.security.KerberosInfo(clientPrincipal=,
>>>>>> serverPrincipal=dfs.namenode.kerberos.principal);
>>>>>> [2018-08-23T09:40:26.096391] RPC Server's Kerberos principal name for
>>>>>> protocol=org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB is hdfs/
>>>>>> hadoop.hadoop.com at HADOOP.COM;
>>>>>> [2018-08-23T09:40:26.096411] Creating SASL GSSAPI(KERBEROS)  client
>>>>>> to authenticate to service at hadoop.hadoop.com;
>>>>>> [2018-08-23T09:40:26.097116] Use KERBEROS authentication for protocol
>>>>>> ClientNamenodeProtocolPB;
>>>>>> [2018-08-23T09:40:26.101823] PrivilegedActionException
>>>>>> as:syslog at HADOOP.COM (auth:KERBEROS)
>>>>>> cause:javax.security.sasl.SaslException: No common protection layer between
>>>>>> client and server;
>>>>>> [2018-08-23T09:40:26.101960] PrivilegedAction as:syslog at HADOOP.COM
>>>>>> (auth:KERBEROS)
>>>>>> from:org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:694);
>>>>>> [2018-08-23T09:40:26.102283] Couldn't setup connection for
>>>>>> syslog at HADOOP.COM to /x.x.x.x:25000;
>>>>>> [2018-08-23T09:40:26.102365] PrivilegedActionException
>>>>>> as:syslog at HADOOP.COM (auth:KERBEROS) cause:java.io.IOException:
>>>>>> Couldn't setup connection for syslog at HADOOP.COM to /x.x.x.x:25000;
>>>>>> [2018-08-23T09:40:26.103844] closing ipc connection to
>>>>>> /x.x.x.x:25000: Couldn't setup connection for syslog at HADOOP.COM to
>>>>>> /x.x.x.x:25000;
>>>>>> [2018-08-23T09:40:26.103994] IPC Client (1766482975) connection to
>>>>>> /x.x.x.x:25000 from syslog at HADOOP.COM: closed;
>>>>>>
>>>>>> Thank you.
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 21, 2018 at 6:48 PM Nagy, Gábor <
>>>>>> gabor.nagy at oneidentity.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Can you tell ne what is the version of the hadoop lib you use with
>>>>>>> syslog-ng, please?
>>>>>>>
>>>>>>> Can you share your syslog-ng configuration, mainly the hdfs part,
>>>>>>> please?
>>>>>>>
>>>>>>> Regards,
>>>>>>> Gabor
>>>>>>>
>>>>>>> On Tue, Aug 21, 2018 at 4:15 AM Lee Keng Ket <kengket at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi, Gabor
>>>>>>>>
>>>>>>>> I have run it, seems like it stops at the HDFS side.
>>>>>>>>
>>>>>>>> [2018-08-21T10:07:51.212015] Worker thread started;
>>>>>>>> driver='d_hdfs#0'
>>>>>>>> [2018-08-21T10:07:51.212499] Running application hooks; hook='1'
>>>>>>>> [2018-08-21T10:07:51.212516] Running application hooks; hook='3'
>>>>>>>> [2018-08-21T10:07:51.212595] syslog-ng starting up; version='3.14.1'
>>>>>>>> [2018-08-21T10:07:51.214113] Opening hdfs;
>>>>>>>> [2018-08-21T10:08:01.215622] Opening hdfs;
>>>>>>>> [2018-08-21T10:08:11.216050] Opening hdfs;
>>>>>>>> [2018-08-21T10:08:21.226340] Opening hdfs;
>>>>>>>> [2018-08-21T10:08:31.236589] Opening hdfs;
>>>>>>>> [2018-08-21T10:08:41.240623] Opening hdfs;
>>>>>>>> [2018-08-21T10:08:51.250879] Opening hdfs;
>>>>>>>> [2018-08-21T10:09:01.261172] Opening hdfs;
>>>>>>>> [2018-08-21T10:09:11.271410] Opening hdfs;
>>>>>>>> [2018-08-21T10:09:21.281685] Opening hdfs;
>>>>>>>> [2018-08-21T10:09:31.290765] Opening hdfs;
>>>>>>>> [2018-08-21T10:09:41.301098] Opening hdfs;
>>>>>>>> [2018-08-21T10:09:51.311362] Opening hdfs;
>>>>>>>> [2018-08-21T10:10:01.321152] Opening hdfs;
>>>>>>>> [2018-08-21T10:10:11.321818] Opening hdfs;
>>>>>>>> [2018-08-21T10:10:21.330114] Opening hdfs;
>>>>>>>> [2018-08-21T10:10:31.340413] Opening hdfs;
>>>>>>>> [2018-08-21T10:10:41.350654] Opening hdfs;
>>>>>>>> [2018-08-21T10:10:51.354016] Opening hdfs;
>>>>>>>> [2018-08-21T10:11:01.364267] Opening hdfs;
>>>>>>>> [2018-08-21T10:11:11.374516] Opening hdfs;
>>>>>>>> [2018-08-21T10:11:21.384761] Opening hdfs;
>>>>>>>> [2018-08-21T10:11:31.395017] Opening hdfs;
>>>>>>>> [2018-08-21T10:11:41.402256] Opening hdfs;
>>>>>>>> [2018-08-21T10:11:51.404097] Opening hdfs;
>>>>>>>> ^C[2018-08-21T10:11:59.672252] syslog-ng shutting down;
>>>>>>>> version='3.14.1'
>>>>>>>> Exception in thread "" java.lang.NoClassDefFoundError:
>>>>>>>> org/apache/hadoop/conf/Configuration
>>>>>>>>         at
>>>>>>>> org.syslog_ng.hdfs.HdfsDestination.open(HdfsDestination.java:92)
>>>>>>>>         at
>>>>>>>> org.syslog_ng.LogDestination.openProxy(LogDestination.java:65)
>>>>>>>> [2018-08-21T10:11:59.774895] Worker thread finished;
>>>>>>>> driver='d_hdfs#0'
>>>>>>>> [2018-08-21T10:11:59.775384] Closing log transport fd; fd='13'
>>>>>>>> [2018-08-21T10:11:59.775508] Deinitialize hdfs destination;
>>>>>>>> [2018-08-21T10:11:59.776534] Java machine free;
>>>>>>>> [2018-08-21T10:11:59.778421] Running application hooks; hook='4'
>>>>>>>>
>>>>>>>> Any idea what to be checked further?
>>>>>>>>
>>>>>>>> Thank you.
>>>>>>>>
>>>>>>>> On Fri, Aug 17, 2018 at 4:45 PM Nagy, Gábor <
>>>>>>>> gabor.nagy at oneidentity.com> wrote:
>>>>>>>>
>>>>>>>>> Hello!
>>>>>>>>>
>>>>>>>>> In the statistics it can be seen that the log message is not sent
>>>>>>>>> to the HDFS server:
>>>>>>>>> dropped='dst.java(d_hdfs#0 java_dst hdfs hdfs://x.x.x.x:25000
>>>>>>>>> /user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=0'
>>>>>>>>> processed='dst.java(d_hdfs#0 java_dst hdfs hdfs://x.x.x.x:25000
>>>>>>>>> /user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=1'
>>>>>>>>> queued='dst.java(d_hdfs#0 java_dst hdfs hdfs://x.x.x.x:25000
>>>>>>>>> /user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=1'
>>>>>>>>>
>>>>>>>>> Well, generally on write error there should be an exception that
>>>>>>>>> results in an error message.
>>>>>>>>>
>>>>>>>>> You should try debugging it either in running syslog-ng in
>>>>>>>>> foreground (-F option), forwarding internal logs to stderr (-e) and with
>>>>>>>>> debug mode (-dv) on.
>>>>>>>>> Or in service mode use the internal() source in your config and
>>>>>>>>> connect it to a destination (e.g. file()) which you prefer.
>>>>>>>>>
>>>>>>>>> You could turn on debug messages on java side too using
>>>>>>>>> jvm_options() in syslog-ng config and configuring the log4j logging
>>>>>>>>> service, e.g.:
>>>>>>>>> options {
>>>>>>>>>
>>>>>>>>> jvm_options("-Dlog4j.configuration=file:/etc/hadoop/log4j.properties
>>>>>>>>> -Dlog4j.debug=true");
>>>>>>>>> };
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Gabor
>>>>>>>>>
>>>>>>>>> On Fri, Aug 17, 2018 at 10:34 AM Czanik, Péter <
>>>>>>>>> peter.czanik at balabit.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> As https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng314/
>>>>>>>>>> also writes: "Java is enabled, but JAR dependencies are not provided in
>>>>>>>>>> package, except for Elasticsearch http mode." The
>>>>>>>>>> syslog-ng-java-deps.noarch contains build time dependencies. Probably I
>>>>>>>>>> should rename the package to syslog-ng-java-build-deps...
>>>>>>>>>>
>>>>>>>>>> Check the documentation at
>>>>>>>>>> https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/33#TOPIC-956506
>>>>>>>>>> on how to download and configure HDFS related JAR dependencies.
>>>>>>>>>>
>>>>>>>>>> Bye,
>>>>>>>>>>
>>>>>>>>>> Peter Czanik (CzP) <peter.czanik at balabit.com>
>>>>>>>>>> Balabit / syslog-ng upstream
>>>>>>>>>> https://syslog-ng.com/community/
>>>>>>>>>> https://twitter.com/PCzanik
>>>>>>>>>>
>>>>>>>>>> On Fri, Aug 17, 2018 at 10:22 AM, Lee Keng Ket <kengket at gmail.com
>>>>>>>>>> > wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I'm trying to connect syslog-ng 3.14.1 to HDFS to store the
>>>>>>>>>>> syslog messages. The syslog-ng can start without error, and it's able to
>>>>>>>>>>> write into local file. However, the log is not written to the HDFS. As
>>>>>>>>>>> there is no single error, I'm not sure how I should troubleshoot on this.
>>>>>>>>>>>
>>>>>>>>>>> I have installed the syslog-ng from this repo,
>>>>>>>>>>> https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng314/repo/epel-7/czanik-syslog-ng314-epel-7.repo
>>>>>>>>>>>
>>>>>>>>>>> Installed Packages
>>>>>>>>>>> syslog-ng.x86_64
>>>>>>>>>>> 3.14.1-4.el7.centos
>>>>>>>>>>> @czanik-syslog-ng314
>>>>>>>>>>> syslog-ng-java.x86_64
>>>>>>>>>>> 3.14.1-4.el7.centos
>>>>>>>>>>> @czanik-syslog-ng314
>>>>>>>>>>> syslog-ng-java-deps.noarch
>>>>>>>>>>> 1.0-2
>>>>>>>>>>> @czanik-syslog-ng314
>>>>>>>>>>>
>>>>>>>>>>> This is the message from /var/log/message:
>>>>>>>>>>> Log statistics; processed='src.internal(s_sys#0)=1',
>>>>>>>>>>> stamp='src.internal(s_sys#0)=1534491834',
>>>>>>>>>>> processed='destination(d_spol)=0', processed='destination(d_mlal)=0',
>>>>>>>>>>> processed='center(received)=2', processed='destination(d_mesg)=1',
>>>>>>>>>>> processed='destination(d_mail)=0', processed='destination(d_auth)=0',
>>>>>>>>>>> processed='destination(d_cron)=0', processed='destination(d_hdfs)=1',
>>>>>>>>>>> processed='center(queued)=3', queued='global(scratch_buffers_count)=0',
>>>>>>>>>>> processed='source(remote_log)=1',
>>>>>>>>>>> dropped='dst.java(d_hdfs#0,java_dst,hdfs,hdfs://x.x.x.x:25000,/user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=0',
>>>>>>>>>>> processed='dst.java(d_hdfs#0,java_dst,hdfs,hdfs://x.x.x.x:25000,/user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=1',
>>>>>>>>>>> queued='dst.java(d_hdfs#0,java_dst,hdfs,hdfs://x.x.x.x:25000,/user/syslog/$HOST-$DAY-$MONTH-$YEAR.log)=1',
>>>>>>>>>>> processed='global(payload_reallocs)=0',
>>>>>>>>>>> processed='src.journald(journal)=0', stamp='src.journald(journal)=0',
>>>>>>>>>>> processed='global(sdata_updates)=0',
>>>>>>>>>>> queued='global(scratch_buffers_bytes)=0',
>>>>>>>>>>> processed='destination(d_boot)=0', processed='destination(d_kern)=0',
>>>>>>>>>>> processed='source(s_sys)=1', processed='destination(remote)=1',
>>>>>>>>>>> processed='global(internal_queue_length)=0',
>>>>>>>>>>> processed='global(msg_clones)=0'
>>>>>>>>>>>
>>>>>>>>>>> Anyone has any idea how should I proceed the troubleshooting?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ______________________________________________________________________________
>>>>>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>>>>>> Documentation:
>>>>>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ______________________________________________________________________________
>>>>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>>>>> Documentation:
>>>>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> ______________________________________________________________________________
>>>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>>>> Documentation:
>>>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>>>
>>>>>>>>>
>>>>>>>> ______________________________________________________________________________
>>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>>> Documentation:
>>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>>
>>>>>>>>
>>>>>>> ______________________________________________________________________________
>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>> Documentation:
>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>
>>>>>>>
>>>>>> ______________________________________________________________________________
>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>> Documentation:
>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>
>>>>>>
>>>>> ______________________________________________________________________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation:
>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180827/5ef73b3c/attachment-0001.html>

More information about the syslog-ng mailing list