[syslog-ng] Finding feature owner for Kafka source project

[Laszlo Budai]

Hi Vincent,

the currently available Java features are
* Kafka
* Elasticsearch
* HDFS

(we have HTTP  but we also have curl destination in C).

As you mentioned we have Kafka destination driver written in C in syslog-ng incubator, so users can use it. It should not be a problem if a feature is available only in the syslog-ng incubator.

As the Kafka C client is stable enough, I don't see any problems in integrating it to the syslog-ng master (just someone should create a pull request).

In case of Elasticsearch we are happy that we are able to support even the latest Elasticsearch 5 but we don't have the resource to re-implement it in C after every new Elasticsearch release.

If someone wants to implement a REST Elasticsearch client in C and based on that also implements an ES dest. driver in C then, we will happily integrate it into syslog-ng.

HDFS: users who uses HDFS will have Java on their machines, so it should not be a problem.

I think a feature implemented in Java is much-much better than an unimplemented feature...


regards,
Laszlo Budai

Sent from my HTC

----- Reply message -----
From: "Vincent Bernat" <bernat at luffy.cx>
To: "Vithulan MV" <vithulanmv.12 at cse.mrt.ac.lk>
Cc: "Syslog-ng users' and developers' mailing list" <syslog-ng at lists.balabit.hu>
Subject: [syslog-ng] Finding feature owner for Kafka source project
Date: Sat, May 21, 2016 19:38


 ❦ 21 mai 2016 20:33 +0530, Vithulan MV <vithulanmv.12 at cse.mrt.ac.lk> :

> A simple abstract idea of this project is,
> The syslog-ng application can read messages from the sources. It
> processes them with filters, rewrite rules, parsers and finally sends
> messages to their destinations. The syslog-ng application already has
> a Kafka destination that is implemented in Java. The Kafka source will
> allow syslog-ng to read messages from Kafka, for an example this can
> be used for example as a queue between several syslog-ng instances.

It's a bit annoying that so many interesting features have to go through
Java. One of the major point of syslog-ng is its light
footprint. librdkafka has support for both source and destination and is
officially supported by Confluent. Its development matches Kafka one and
it supports every feature in Kafka 0.9, including SSL and the new
consumer.

It's a bit late to complain, notably if you chose this project because
you were comfortable with Java (and your mentor is obviously interested
in doing that in Java too). However, this has already been pointed
out in March without much response from Balabit. It's not just a matter
of proposing code since such code is likely to get stuck in the
incubator forever.
--
Make sure special cases are truly special.
            - The Elements of Programming Style (Kernighan & Plauger)
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq