Finding feature owner for Kafka source project
Hi all, As you may have all known already, We are going to develop a new feature, Kafka source for syslog-ng product during this GSoC 2016 under the guidance of Viktor Juhasz. A simple abstract idea of this project is, The syslog-ng application can read messages from the sources. It processes them with filters, rewrite rules, parsers and finally sends messages to their destinations. The syslog-ng application already has a Kafka destination that is implemented in Java. The Kafka source will allow syslog-ng to read messages from Kafka, for an example this can be used for example as a queue between several syslog-ng instances. Kafka source project will allow you to, -Use Kafka consumer which uses the high level Kafka group consuming API. -Continue reading from the last message from which syslog-ng reloaded/restarted. -Avoid message loosing or message duplication as much as possible whenever there is multiple syslog-ng reading the same kafka input. If you are interested to become a feature owner for Kafka source project and give us guidance on user's perspective, it will be very easy for us to build a effective component. Appreciate if you can contact us by sending mail to us or to this thread. Thanks, Best regards, Vithulan. -- Vithulan MV. Undergraduate, Department of Computer Science & Engineering, University of Moratuwa, Sri lanka.
❦ 21 mai 2016 20:33 +0530, Vithulan MV <vithulanmv.12@cse.mrt.ac.lk> :
A simple abstract idea of this project is, The syslog-ng application can read messages from the sources. It processes them with filters, rewrite rules, parsers and finally sends messages to their destinations. The syslog-ng application already has a Kafka destination that is implemented in Java. The Kafka source will allow syslog-ng to read messages from Kafka, for an example this can be used for example as a queue between several syslog-ng instances.
It's a bit annoying that so many interesting features have to go through Java. One of the major point of syslog-ng is its light footprint. librdkafka has support for both source and destination and is officially supported by Confluent. Its development matches Kafka one and it supports every feature in Kafka 0.9, including SSL and the new consumer. It's a bit late to complain, notably if you chose this project because you were comfortable with Java (and your mentor is obviously interested in doing that in Java too). However, this has already been pointed out in March without much response from Balabit. It's not just a matter of proposing code since such code is likely to get stuck in the incubator forever. -- Make sure special cases are truly special. - The Elements of Programming Style (Kernighan & Plauger)
Hi, I got exactly the same feedback last week at the DORS/CLUC conference. My feeling based on many F2F discussions (but not backed by proper research), that Java is OK with large enterprise syslog-ng users, but the rest prefers to avoid Java as much as possible. I must admit, that with my SMB (small and medium-sized businesses) sysadmin background I'm also biased towards C-based solutions, that's why I'd like to do a proper user research on the topic instead relying on my feelings... Bye, Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik On Sat, May 21, 2016 at 7:38 PM, Vincent Bernat <bernat@luffy.cx> wrote:
❦ 21 mai 2016 20:33 +0530, Vithulan MV <vithulanmv.12@cse.mrt.ac.lk> :
A simple abstract idea of this project is, The syslog-ng application can read messages from the sources. It processes them with filters, rewrite rules, parsers and finally sends messages to their destinations. The syslog-ng application already has a Kafka destination that is implemented in Java. The Kafka source will allow syslog-ng to read messages from Kafka, for an example this can be used for example as a queue between several syslog-ng instances.
It's a bit annoying that so many interesting features have to go through Java. One of the major point of syslog-ng is its light footprint. librdkafka has support for both source and destination and is officially supported by Confluent. Its development matches Kafka one and it supports every feature in Kafka 0.9, including SSL and the new consumer.
It's a bit late to complain, notably if you chose this project because you were comfortable with Java (and your mentor is obviously interested in doing that in Java too). However, this has already been pointed out in March without much response from Balabit. It's not just a matter of proposing code since such code is likely to get stuck in the incubator forever. -- Make sure special cases are truly special. - The Elements of Programming Style (Kernighan & Plauger)
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (3)
-
Czanik, Péter
-
Vincent Bernat
-
Vithulan MV