November 2018 - syslog-ng - lists.balabit.hu

balabit/syslog-ng:latest container and kafka output
by Mathias Dufresne 30 Nov '18

30 Nov '18

Hi all, I'm trying to push logs from files to Kafka using syslog-ng container provided there: https://hub.docker.com/r/balabit/syslog-ng/ I used also : https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edi… The ERROR: When running the container including Kafka configuration I get: [2018-11-30T12:20:51.324773] WARNING: With use-dns(no), dns-cache() will be forced to 'no' too!; [2018-11-30T12:20:51.478631] Can't find class; class_name='org.syslog_ng.kafka.KafkaDestination' [2018-11-30T12:20:51.478667] If you have reloaded syslog-ng, the JVM is not used anymore, but it is still running. If you want to stop JVM, stop syslog-ng and then start syslog-ng again; [2018-11-30T12:20:51.478680] Error initializing message pipeline; plugin_name='java', location='#buffer:4:3' Here are my configuration files: /etc/syslog-ng/syslog-ng.conf : @version: 3.18 @include "scl.conf" # Syslog-ng configuration file, compatible with default Debian syslogd # installation. # First, set some global options. options { chain_hostnames(off); flush_lines(0); #use_dns(yes); #use_fqdn(yes); use_dns(no); use_fqdn(no); #owner("root"); #group("adm"); perm(0640); # The default action of syslog-ng is to log a STATS line # to the file every 10 minutes. That's pretty ugly after a while. # Change it to every 12 hours so you get a nice daily update of # how many messages syslog-ng missed (0). stats_freq(0); stats_level(1); # The default action of syslog-ng is to log a MARK line # to the file every 20 minutes. That's seems high for most # people so turn it down to once an hour. Set it to zero # if you don't want the functionality at all. mark_freq(0); bad_hostname("^gconfd$"); threaded(yes); log_msg_size (16384); log_fifo_size (8000000); # # Time management: # # The goal is to have in logs the original date and time set by # the syslog which effectively receives the message # and that this timestamp is not changed during it's journey # to reach the database. # # To achieve that all clients are configured with keep_timestamp(no) # for syslog-ng marks when it receives the message. # This is done globally because system() and internal() sources are not # both supporting this keep_timestamp option when the syslog() source # used for networked syslog support that option. This options is forced to "yes" # in remote source(). keep_timestamp(no); # # Time precision is increased to microseconds frac_digits(6); # And format is set to "iso": YYYY-MM-DDThh:mm:ss.0123456+02.00 # where "T" is the separator between date and time # and +02.00 GMT decay (as an example) ts_format("iso"); }; ######################## # Sources ######################## # This is the default behavior of sysklogd package # Logs may come from unix stream, but not from another machine. # source s_src { system(); internal(); }; # Log from any hosts around # source s_net { network( transport("tcp") port(601) flags("syslog-protocol", "threaded") keep_timestamp(yes) max-connections(250) log_iw_size(25000) log_fetch_limit(100) flags("threaded") ); }; source s_app_logs { file(/opt/test.log,flags(no-parse),follow_freq(1),keep-timestamp(yes)); }; ################################################################################################# # Destinations ################################################################################################# destination d_messages { file("/var/log/messages"); }; destination d_clients { file("/var/log/syslog_clients/$HOST"); }; destination d_clients_all { file("/var/log/syslog_clients/all_logs"); }; @include "kafka_destination.conf" ################################################################################################# # Destinations ################################################################################################# log { source(s_net); # destination(d_clients); }; log { source(s_src); # destination(d_messages); }; ################################################################################################# # Filters ################################################################################################# filter f_no_debug { not level(debug); }; --------------------------------------------------------------------------------------------- kafka_destination.conf: @module mod-java @define java-module-dir "/usr/lib/syslog-ng/3.18/java-modules:/usr/lib/syslog-ng/3.18/java-modules/:/usr/lib/syslog-ng/3.18/java-modules/*.jar" destination d_kafka { kafka( client-lib-dir("/usr/lib/syslog-ng/3.18/java-modules:/usr/lib/syslog-ng/3.18/java-modules/:/usr/lib/syslog-ng/3.18/java-modules/*.jar:/etc/syslog-ng/kafka_libs/libs/:/etc/syslog-ng/kafka_libs/libs/*.jar:/etc/syslog-ng/kafka_libs/libs") #client-lib-dir("/usr/lib/syslog-ng/3.18/java-modules/:/usr/lib/x86_64-linux-gnu/") kafka-bootstrap-servers("10.11.12.129:9093,10.11.12.130:9093, 10.11.12.131:9093,10.11.12.132:9093,10.11.12.133:9093,10.11.12.134:9093") topic("roma") properties-file("/etc/syslog-ng/kafka_producer.properties") ); }; log { source(s_app_logs); destination(d_kafka); }; --------------------------------------------------------------------------------------------- Directory /etc/syslog-ng/kafka_libs/libs is in fact "libs" directory from kafka_2.11-2.1.0.tgz which comes from http://kafka.apache.org/downloads. Here is the content of this libs directory : ls kafka_libs/libs/ -1 activation-1.1.1.jar aopalliance-repackaged-2.5.0-b42.jar argparse4j-0.7.0.jar audience-annotations-0.5.0.jar commons-lang3-3.5.jar compileScala.mapping compileScala.mapping.asc connect-api-2.1.0.jar connect-basic-auth-extension-2.1.0.jar connect-file-2.1.0.jar connect-json-2.1.0.jar connect-runtime-2.1.0.jar connect-transforms-2.1.0.jar guava-20.0.jar hk2-api-2.5.0-b42.jar hk2-locator-2.5.0-b42.jar hk2-utils-2.5.0-b42.jar jackson-annotations-2.9.7.jar jackson-core-2.9.7.jar jackson-databind-2.9.7.jar jackson-jaxrs-base-2.9.7.jar jackson-jaxrs-json-provider-2.9.7.jar jackson-module-jaxb-annotations-2.9.7.jar javassist-3.22.0-CR2.jar javax.annotation-api-1.2.jar javax.inject-1.jar javax.inject-2.5.0-b42.jar javax.servlet-api-3.1.0.jar javax.ws.rs-api-2.1.1.jar javax.ws.rs-api-2.1.jar jaxb-api-2.3.0.jar jersey-client-2.27.jar jersey-common-2.27.jar jersey-container-servlet-2.27.jar jersey-container-servlet-core-2.27.jar jersey-hk2-2.27.jar jersey-media-jaxb-2.27.jar jersey-server-2.27.jar jetty-client-9.4.12.v20180830.jar jetty-continuation-9.4.12.v20180830.jar jetty-http-9.4.12.v20180830.jar jetty-io-9.4.12.v20180830.jar jetty-security-9.4.12.v20180830.jar jetty-server-9.4.12.v20180830.jar jetty-servlet-9.4.12.v20180830.jar jetty-servlets-9.4.12.v20180830.jar jetty-util-9.4.12.v20180830.jar jopt-simple-5.0.4.jar kafka_2.11-2.1.0.jar kafka_2.11-2.1.0.jar.asc kafka_2.11-2.1.0-javadoc.jar kafka_2.11-2.1.0-javadoc.jar.asc kafka_2.11-2.1.0-scaladoc.jar kafka_2.11-2.1.0-scaladoc.jar.asc kafka_2.11-2.1.0-sources.jar kafka_2.11-2.1.0-sources.jar.asc kafka_2.11-2.1.0-test.jar kafka_2.11-2.1.0-test.jar.asc kafka_2.11-2.1.0-test-sources.jar kafka_2.11-2.1.0-test-sources.jar.asc kafka-clients-2.1.0.jar kafka-log4j-appender-2.1.0.jar kafka-streams-2.1.0.jar kafka-streams-examples-2.1.0.jar kafka-streams-scala_2.11-2.1.0.jar kafka-streams-test-utils-2.1.0.jar kafka-tools-2.1.0.jar log4j-1.2.17.jar lz4-java-1.5.0.jar maven-artifact-3.5.4.jar metrics-core-2.2.0.jar osgi-resource-locator-1.0.1.jar plexus-utils-3.1.0.jar reflections-0.9.11.jar rocksdbjni-5.14.2.jar scala-library-2.11.12.jar scala-logging_2.11-3.9.0.jar scala-reflect-2.11.12.jar slf4j-api-1.7.25.jar slf4j-log4j12-1.7.25.jar snappy-java-1.1.7.2.jar validation-api-1.1.0.Final.jar zkclient-0.10.jar zookeeper-3.4.13.jar zstd-jni-1.3.5-4.jar ---------------------------------------------------------------------------------------------- Here is "syslog-ng --version" inside the conainer: root@e1ca5fdc292a:/# syslog-ng --version syslog-ng 3 (3.18.1) Config version: 3.18 Installer-Version: 3.18.1 Revision: 3.18.1-1 Compile-Date: Oct 24 2018 10:04:41 Module-Directory: /usr/lib/syslog-ng/3.18 Module-Path: /usr/lib/syslog-ng/3.18 Available-Modules: add-contextual-data,afamqp,affile,afmongodb,afprog,afsmtp,afsocket,afsql,afstomp,afuser,appmodel,basicfuncs,cef,confgen,cryptofuncs,csvparser,date,dbparser,disk-buffer,examples,geoip-plugin,geoip2-plugin,graphite,hook-commands,http,json-plugin,kvformat,linux-kmsg-format,map-value-pairs,mod-java,mod-python,pacctformat,pseudofile,redis,riemann,sdjournal,snmptrapd-parser,stardate,syslogformat,system-source,tags-parser,tfgetent,xml Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Systemd: on root@e1ca5fdc292a:/# mod-java is there but that's not enough and I can"t understand what I am missing. ------------------------------------------- Packages: root@e1ca5fdc292a:/# dpkg -l | grep kafka ii librdkafka-dev:amd64 0.8.5-2 amd64 library implementing the Apache Kafka protocol (development headers) ii librdkafka1:amd64 0.8.5-2 amd64 library implementing the Apache Kafka protocol ii syslog-ng-mod-kafka 3.18.1-1 amd64 Enhanced system logging daemon (Kafka destination) root@e1ca5fdc292a:/# If information are missing, please tell me. If you have some clue, that would be really appreciated. Best regards, mathias

1 0

Verifying local logs and remote logs are being sent to destination graylog
by Rodney Bizzell 23 Nov '18

23 Nov '18

I want to make sure that my local logs on my syslog box are being sent to my graylog host what config changes do I need to make to ensure that is configured correctly. Can I have an example for illustration. Thanks. @version:3.5 @include "scl.conf" # syslog-ng configuration file. # # This should behave pretty much like the original syslog on RedHat. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # # Note: it also sources additional configuration files (*.conf) # located in /etc/syslog-ng/conf.d/ options { flush_lines (0); time_reopen (10); log_fifo_size (250000); chain_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source s_sys { system(); internal(); # udp(ip(0.0.0.0) port(514)); }; source s_net { udp(ip(0.0.0.0) port(514)); #tcp(ip(0.0.0.0) port(514) max-connections(256)); }; destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" flush_lines(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_kern { file("/var/log/kern"); }; destination d_mlal { usertty("*"); }; destination d_graylog { tcp("graylog.host" port (12201) ); }; filter f_kernel { facility(kern); }; filter f_default { level(info..emerg) and not (facility(mail) or facility(authpriv) or facility(cron)); }; filter f_auth { facility(authpriv); }; filter f_mail { facility(mail); }; filter f_emergency { level(emerg); }; filter f_news { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_boot { facility(local7); }; filter f_cron { facility(cron); }; log { source(s_sys); filter(f_kernel); destination(d_cons); }; log { source(s_sys); filter(f_kernel); destination(d_kern); }; log { source(s_sys); filter(f_default); destination(d_mesg); }; log { source(s_sys); filter(f_auth); destination(d_auth); }; log { source(s_sys); filter(f_mail); destination(d_mail); }; log { source(s_sys); filter(f_emergency); destination(d_mlal); }; log { source(s_sys); filter(f_news); destination(d_spol); }; log { source(s_sys); filter(f_boot); destination(d_boot); }; log { source(s_sys); filter(f_cron); destination(d_cron); }; log { source(s_net); destination(d_graylog); }; log { source(s_sys); filter(f_default); destination(d_graylog);}; # Source additional configuration files (.conf extension only) @include "/etc/syslog-ng/conf.d/*.conf"

4 20

IPV6 and IPV4 syslog-ng issue
by venkateswarlu vinjamuri 22 Nov '18

22 Nov '18

Hi, *Issue:* *when configured both IPV4 and IPV6 syslog server IP's, I could see syslog-ng messages going to only one syslog server, either IPV4 or IPV6 syslog-ng server and that too to the latest server IP configured only.* Is there any known issue ? Please help me on this. Thanks & Regards, Venkat.

2 3

Syslog-ng open source usage.
by vijay amruth 14 Nov '18

14 Nov '18

Hello team, does open source versions use is any different after syslog-ng acquisition? Vijay Amrut. -- Thanks, Vijay Amrut.

3 2

Global flush-lines option VS elasticsearch2 destination flush-limit param?
by Jose Angel Santiago 11 Nov '18

11 Nov '18

Hi, It seems both parameters are made for the same purpose, but I guess they don't. ¿Could you explain to me what is the difference between these two parameters? Thanks in advance. -- | Jose Angel Santiago [image: Logo_signature2.png] <http://www.stratio.com/> Vía de las dos Castillas, 33, Ática 4, 3ª Planta 28224 Pozuelo de Alarcón, Madrid, Spain +34 918 286 473 | www.stratio.com <https://twitter.com/stratiobd> <https://www.linkedin.com/company/stratiobd> <https://www.youtube.com/c/StratioBD>

2 2

sending log messages to graylog
by Rodney Bizzell 06 Nov '18

06 Nov '18

I got syslog to send and echo test message from syslog server to my graylog box. How do I ensure that my syslog box will send other servers logs to my graylog box through my syslog server. I am going to setup ipvsadm as load-balancer to point my legacy application to my syslog server and then they should get shipped through to graylog. Any information is greatly appreciated

4 16

log drop-and-delete directory
by Nik Ambrosch 06 Nov '18

06 Nov '18

Looking to create a drop-off directory that syslog-ng handles instead of needing to execute in a separate script.. flow would go something like this: 1) mv file.log /syslog-tmp/ 2) syslog-ng reads /syslog-tmp/file.log 3) syslog-ng deletes /syslog-tmp/file.log when done consuming Sounds simple but I can't seem to figure out a good way to do this. The other option is to read file with a script, send out with logger (or whatever), and hope that syslog-ng is running & healthy. Thanks.

6 10

Incremental update of syslog-ng config
by Monitto Pitchaimani sebastin (mopitcha) 05 Nov '18

05 Nov '18

Hi All, Is there a way to update the syslog-ng config without restarting the syslog-ng? Regards, Monitto

4 5

Unable to make syslog-ng listen to port 514
by Fosiul Alam 05 Nov '18

05 Nov '18

Hello i have installed syslog-ng in our ubuntu server with version : 3.13 its running fine, I am using the default syslog-ng file and i have change this line as per documentation source s_net { syslog(ip(192.168.1.19) transport("tcp")); }; but its not listing on port 514 What Do i have to do make it listen ? Thanks for the help ● syslog-ng.service - System Logger Daemon Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2018-11-02 18:22:46 UTC; 4min 5s ago Docs: man:syslog-ng(8) Main PID: 16470 (syslog-ng) Tasks: 1 (limit: 2361) CGroup: /system.slice/syslog-ng.service └─16470 /usr/sbin/syslog-ng -F Nov 02 18:22:46 syslog systemd[1]: Starting System Logger Daemon... Nov 02 18:22:46 syslog systemd[1]: Started System Logger Daemon. -- Regards Fosiul Alam

7 12