[syslog-ng] Apache Kafka output module

Pierre-Yves Ritschard pyr at spootnik.org
Fri Jul 11 10:11:42 CEST 2014


Hi list,

I submitted a PR for an Apache Kafka [1] output module. This is my first
syslog-ng module, so it might need a bit of guidance.

This provides the ability to create a partitioned stream of logs based
on a message field or static key.

A typical destination configuration will look like:

destination d_kafka {
  kafka( properties(metadata.broker.list("localhost:9092"))
       topic("syslog")
       payload("$(format-json --scope all-nv-pairs --scope core)")

       partition("static-key")
#      partition(random)
#      partition(field("$PROGRAM"))
); 

};

For reference, this allows us to work on logs locally with syslog,
extract fields, then publish on a queue whose consumers will index logs
in our data stores.

[1]. http://kafka.apache.org





More information about the syslog-ng mailing list