<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi all,</div><div><br></div><div>I'm trying to push logs from files to Kafka using syslog-ng container provided there: <a href="https://hub.docker.com/r/balabit/syslog-ng/">https://hub.docker.com/r/balabit/syslog-ng/</a></div><div><br></div><div>I used also :</div><div><a href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/37#TOPIC-1044021">https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/37#TOPIC-1044021</a></div><div><br></div><div>The ERROR:<br></div><div>When running the container including Kafka configuration I get:</div><div>[2018-11-30T12:20:51.324773] WARNING: With use-dns(no), dns-cache() will be forced to 'no' too!;<br>[2018-11-30T12:20:51.478631] Can't find class; class_name='org.syslog_ng.kafka.KafkaDestination'<br>[2018-11-30T12:20:51.478667] If you have reloaded syslog-ng, the JVM is not used anymore, but it is still running. If you want to stop JVM, stop syslog-ng and then start syslog-ng again;<br>[2018-11-30T12:20:51.478680] Error initializing message pipeline; plugin_name='java', location='#buffer:4:3'<br></div><div><br></div><div>Here are my configuration files:</div><div>/etc/syslog-ng/syslog-ng.conf :</div><div>@version: 3.18<br>@include "scl.conf"<br><br># Syslog-ng configuration file, compatible with default Debian syslogd<br># installation.<br><br># First, set some global options.<br>options {<br>  chain_hostnames(off);<br>  flush_lines(0);<br>  #use_dns(yes);<br>  #use_fqdn(yes);<br>  use_dns(no);<br>  use_fqdn(no);<br>  #owner("root");<br>  #group("adm");<br>  perm(0640);<br>  # The default action of syslog-ng is to log a STATS line<br>  # to the file every 10 minutes.  That's pretty ugly after a while.<br>  # Change it to every 12 hours so you get a nice daily update of<br>  # how many messages syslog-ng missed (0).<br>  stats_freq(0);<br>  stats_level(1);<br>  # The default action of syslog-ng is to log a MARK line<br>  # to the file every 20 minutes.  That's seems high for most<br>  # people so turn it down to once an hour.  Set it to zero<br>  # if you don't want the functionality at all.<br>  mark_freq(0);<br>  bad_hostname("^gconfd$");<br>  threaded(yes);<br><br>  log_msg_size (16384);<br>  log_fifo_size (8000000);<br><br>  #<br>  # Time management:<br>  #<br>  # The goal is to have in logs the original date and time set by<br>  # the syslog which effectively receives the message<br>  # and that this timestamp is not changed during it's journey<br>  # to reach the database.<br>  #<br>  # To achieve that all clients are configured with keep_timestamp(no)<br>  # for syslog-ng marks when it receives the message.<br>  # This is done globally because system() and internal() sources are not<br>  # both supporting this keep_timestamp option when the syslog() source<br>  # used for networked syslog support that option. This options is forced to "yes"<br>  # in remote source().<br>  keep_timestamp(no);<br>  #<br>  # Time precision is increased to microseconds<br>  frac_digits(6);<br>  # And format is set to "iso": YYYY-MM-DDThh:mm:ss.0123456+02.00<br>  # where "T" is the separator between date and time<br>  # and +02.00 GMT decay (as an example)<br>  ts_format("iso");<br>};<br><br>########################<br># Sources<br>########################<br># This is the default behavior of sysklogd package<br># Logs may come from unix stream, but not from another machine.<br>#<br>source s_src {<br>  system();<br>  internal();<br>};<br># Log from any hosts around<br>#<br>source s_net {<br>  network(<br>    transport("tcp")<br>    port(601)<br>    flags("syslog-protocol", "threaded")<br>    keep_timestamp(yes)<br>    max-connections(250)<br>    log_iw_size(25000)<br>    log_fetch_limit(100)<br>    flags("threaded")<br>  );<br>};<br><br>source s_app_logs {<br>  file(/opt/test.log,flags(no-parse),follow_freq(1),keep-timestamp(yes));<br>};<br><br>#################################################################################################<br># Destinations<br>#################################################################################################<br>destination d_messages { file("/var/log/messages"); };<br><br>destination d_clients { file("/var/log/syslog_clients/$HOST"); };<br>destination d_clients_all { file("/var/log/syslog_clients/all_logs"); };<br><br>@include "kafka_destination.conf"<br><br>#################################################################################################<br># Destinations<br>#################################################################################################<br>log {<br>  source(s_net);<br>  #<br>  destination(d_clients);<br>};<br>log {<br>  source(s_src);<br>  #<br>  destination(d_messages);<br>};<br><br>#################################################################################################<br># Filters<br>#################################################################################################<br>filter f_no_debug { not level(debug); };<br></div><div><br></div><div>---------------------------------------------------------------------------------------------</div><div>kafka_destination.conf:</div><div><br></div><div>@module mod-java<br>@define java-module-dir "/usr/lib/syslog-ng/3.18/java-modules:/usr/lib/syslog-ng/3.18/java-modules/:/usr/lib/syslog-ng/3.18/java-modules/*.jar"<br><br>destination d_kafka {<br>  kafka(<br>    client-lib-dir("/usr/lib/syslog-ng/3.18/java-modules:/usr/lib/syslog-ng/3.18/java-modules/:/usr/lib/syslog-ng/3.18/java-modules/*.jar:/etc/syslog-ng/kafka_libs/libs/:/etc/syslog-ng/kafka_libs/libs/*.jar:/etc/syslog-ng/kafka_libs/libs")<br>    #client-lib-dir("/usr/lib/syslog-ng/3.18/java-modules/:/usr/lib/x86_64-linux-gnu/")<br>    kafka-bootstrap-servers("<a href="http://10.11.12.129:9093">10.11.12.129:9093</a>,<a href="http://10.11.12.130:9093">10.11.12.130:9093</a>,<a href="http://10.11.12.131:9093">10.11.12.131:9093</a>,<a href="http://10.11.12.132:9093">10.11.12.132:9093</a>,<a href="http://10.11.12.133:9093">10.11.12.133:9093</a>,<a href="http://10.11.12.134:9093">10.11.12.134:9093</a>")<br>    topic("roma")<br>    properties-file("/etc/syslog-ng/kafka_producer.properties")<br>  );<br>};<br><br>log {<br>  source(s_app_logs);<br>  destination(d_kafka);<br>};<br><br></div><div>---------------------------------------------------------------------------------------------</div><div><br></div><div><br></div><div>Directory /etc/syslog-ng/kafka_libs/libs is in fact "libs" directory from kafka_2.11-2.1.0.tgz which comes from <a href="http://kafka.apache.org/downloads">http://kafka.apache.org/downloads</a>.</div><div><br></div><div>Here is the content of this libs directory :</div><div><br>ls kafka_libs/libs/ -1<br>activation-1.1.1.jar<br>aopalliance-repackaged-2.5.0-b42.jar<br>argparse4j-0.7.0.jar<br>audience-annotations-0.5.0.jar<br>commons-lang3-3.5.jar<br>compileScala.mapping<br>compileScala.mapping.asc<br>connect-api-2.1.0.jar<br>connect-basic-auth-extension-2.1.0.jar<br>connect-file-2.1.0.jar<br>connect-json-2.1.0.jar<br>connect-runtime-2.1.0.jar<br>connect-transforms-2.1.0.jar<br>guava-20.0.jar<br>hk2-api-2.5.0-b42.jar<br>hk2-locator-2.5.0-b42.jar<br>hk2-utils-2.5.0-b42.jar<br>jackson-annotations-2.9.7.jar<br>jackson-core-2.9.7.jar<br>jackson-databind-2.9.7.jar<br>jackson-jaxrs-base-2.9.7.jar<br>jackson-jaxrs-json-provider-2.9.7.jar<br>jackson-module-jaxb-annotations-2.9.7.jar<br>javassist-3.22.0-CR2.jar<br>javax.annotation-api-1.2.jar<br>javax.inject-1.jar<br>javax.inject-2.5.0-b42.jar<br>javax.servlet-api-3.1.0.jar<br>javax.ws.rs-api-2.1.1.jar<br>javax.ws.rs-api-2.1.jar<br>jaxb-api-2.3.0.jar<br>jersey-client-2.27.jar<br>jersey-common-2.27.jar<br>jersey-container-servlet-2.27.jar<br>jersey-container-servlet-core-2.27.jar<br>jersey-hk2-2.27.jar<br>jersey-media-jaxb-2.27.jar<br>jersey-server-2.27.jar<br>jetty-client-9.4.12.v20180830.jar<br>jetty-continuation-9.4.12.v20180830.jar<br>jetty-http-9.4.12.v20180830.jar<br>jetty-io-9.4.12.v20180830.jar<br>jetty-security-9.4.12.v20180830.jar<br>jetty-server-9.4.12.v20180830.jar<br>jetty-servlet-9.4.12.v20180830.jar<br>jetty-servlets-9.4.12.v20180830.jar<br>jetty-util-9.4.12.v20180830.jar<br>jopt-simple-5.0.4.jar<br>kafka_2.11-2.1.0.jar<br>kafka_2.11-2.1.0.jar.asc<br>kafka_2.11-2.1.0-javadoc.jar<br>kafka_2.11-2.1.0-javadoc.jar.asc<br>kafka_2.11-2.1.0-scaladoc.jar<br>kafka_2.11-2.1.0-scaladoc.jar.asc<br>kafka_2.11-2.1.0-sources.jar<br>kafka_2.11-2.1.0-sources.jar.asc<br>kafka_2.11-2.1.0-test.jar<br>kafka_2.11-2.1.0-test.jar.asc<br>kafka_2.11-2.1.0-test-sources.jar<br>kafka_2.11-2.1.0-test-sources.jar.asc<br>kafka-clients-2.1.0.jar<br>kafka-log4j-appender-2.1.0.jar<br>kafka-streams-2.1.0.jar<br>kafka-streams-examples-2.1.0.jar<br>kafka-streams-scala_2.11-2.1.0.jar<br>kafka-streams-test-utils-2.1.0.jar<br>kafka-tools-2.1.0.jar<br>log4j-1.2.17.jar<br>lz4-java-1.5.0.jar<br>maven-artifact-3.5.4.jar<br>metrics-core-2.2.0.jar<br>osgi-resource-locator-1.0.1.jar<br>plexus-utils-3.1.0.jar<br>reflections-0.9.11.jar<br>rocksdbjni-5.14.2.jar<br>scala-library-2.11.12.jar<br>scala-logging_2.11-3.9.0.jar<br>scala-reflect-2.11.12.jar<br>slf4j-api-1.7.25.jar<br>slf4j-log4j12-1.7.25.jar<br>snappy-java-1.1.7.2.jar<br>validation-api-1.1.0.Final.jar<br>zkclient-0.10.jar<br>zookeeper-3.4.13.jar<br>zstd-jni-1.3.5-4.jar<br></div><div><br></div><div>----------------------------------------------------------------------------------------------</div><div><br></div><div>Here is "syslog-ng --version" inside the conainer:</div><div>root@e1ca5fdc292a:/# syslog-ng --version<br>syslog-ng 3 (3.18.1)<br>Config version: 3.18<br>Installer-Version: 3.18.1<br>Revision: 3.18.1-1<br>Compile-Date: Oct 24 2018 10:04:41<br>Module-Directory: /usr/lib/syslog-ng/3.18<br>Module-Path: /usr/lib/syslog-ng/3.18<br>Available-Modules: add-contextual-data,afamqp,affile,afmongodb,afprog,afsmtp,afsocket,afsql,afstomp,afuser,appmodel,basicfuncs,cef,confgen,cryptofuncs,csvparser,date,dbparser,disk-buffer,examples,geoip-plugin,geoip2-plugin,graphite,hook-commands,http,json-plugin,kvformat,linux-kmsg-format,map-value-pairs,mod-java,mod-python,pacctformat,pseudofile,redis,riemann,sdjournal,snmptrapd-parser,stardate,syslogformat,system-source,tags-parser,tfgetent,xml<br>Enable-Debug: off<br>Enable-GProf: off<br>Enable-Memtrace: off<br>Enable-IPv6: on<br>Enable-Spoof-Source: on<br>Enable-TCP-Wrapper: on<br>Enable-Linux-Caps: on<br>Enable-Systemd: on<br>root@e1ca5fdc292a:/#<br></div><div><br></div><div>mod-java is there but that's not enough and I can"t understand what I am missing.</div><div><br></div><div>-------------------------------------------</div><div>Packages:</div><div>root@e1ca5fdc292a:/# dpkg -l | grep kafka<br>ii  librdkafka-dev:amd64              0.8.5-2                          amd64        library implementing the Apache Kafka protocol (development headers)<br>ii  librdkafka1:amd64                 0.8.5-2                          amd64        library implementing the Apache Kafka protocol<br>ii  syslog-ng-mod-kafka               3.18.1-1                         amd64        Enhanced system logging daemon (Kafka destination)<br>root@e1ca5fdc292a:/#<br></div><div><br></div><div>If information are missing, please tell me. If you have some clue, that would be really appreciated.</div><div><br></div><div>Best regards,</div><div><br></div><div>mathias<br></div><div><br></div><div><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>