[syslog-ng] Apache Kafka output module
Pierre-Yves Ritschard
pyr at spootnik.org
Fri Jul 11 10:11:42 CEST 2014
Hi list,
I submitted a PR for an Apache Kafka [1] output module. This is my first
syslog-ng module, so it might need a bit of guidance.
This provides the ability to create a partitioned stream of logs based
on a message field or static key.
A typical destination configuration will look like:
destination d_kafka {
kafka( properties(metadata.broker.list("localhost:9092"))
topic("syslog")
payload("$(format-json --scope all-nv-pairs --scope core)")
partition("static-key")
# partition(random)
# partition(field("$PROGRAM"))
);
};
For reference, this allows us to work on logs locally with syslog,
extract fields, then publish on a queue whose consumers will index logs
in our data stores.
[1]. http://kafka.apache.org
More information about the syslog-ng
mailing list