Apache Kafka output module
Hi list, I submitted a PR for an Apache Kafka [1] output module. This is my first syslog-ng module, so it might need a bit of guidance. This provides the ability to create a partitioned stream of logs based on a message field or static key. A typical destination configuration will look like: destination d_kafka { kafka( properties(metadata.broker.list("localhost:9092")) topic("syslog") payload("$(format-json --scope all-nv-pairs --scope core)") partition("static-key") # partition(random) # partition(field("$PROGRAM")) ); }; For reference, this allows us to work on logs locally with syslog, extract fields, then publish on a queue whose consumers will index logs in our data stores. [1]. http://kafka.apache.org
Hi! Pierre-Yves Ritschard <pyr@spootnik.org> writes:
I submitted a PR for an Apache Kafka [1] output module. This is my first syslog-ng module, so it might need a bit of guidance.
I did a first skim of the code[2], and made a few suggestions on the pull request. Very nice work, thank you! Looking forward to merging it! [2]: https://github.com/balabit/syslog-ng/pull/162 -- |8]
participants (2)
-
Gergely Nagy
-
Pierre-Yves Ritschard