[syslog-ng] [GSoC] Project: Kafka source

Wed Apr 27 15:15:29 CEST 2016

Hi all,

First of all, I'd like to thank you Syslog-ng community and my mentor
Viktor Juhasz for accepting my proposal to implement Kafka source in Java!.
Some of you may already know me but for others, I'll introduce myself
again. I'm Vithulan, 3rd year undergraduate at University of Moratuwa, Sri
lanka. My areas of interests are, Data analytics, Machine Learning,
Networking and Security.

As a first step into this GSoC program I already started following the
documentation [1], and going through the code base of syslog-ng (I'm
curious what IDE you all are using?). What are the other areas that you
want me to know to polish my knowledge before we get into the coding period?

I guess, this is going to be very interesting 4 months with good new
experiences :)
Thank you!

[1]
https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/preface.html

Best regards,
Vithulan.

On 23 April 2016 at 08:42, Vithulan MV <vithulanmv.12 at cse.mrt.ac.lk> wrote:

> Hi Viktor,
>
> I am very excited and happy after seeing my morning mail from Google
> summer of code! Thank you very much for accepting my proposal! :)
> How shall we proceed into this project? and any tips before going deep
> into the project? :)
>
> Thank you.
>
> Best regards,
> Vithulan.
>
> On 26 March 2016 at 10:52, Vithulan MV <vithulanmv.12 at cse.mrt.ac.lk>
> wrote:
>
>> Hi Viktor,
>>
>> I have submitted my proposal in GSoC page and Syslog-ng wiki page aswell.
>>
>> Thanks.
>> Best regards,
>> Vithulan
>> On Mar 10, 2016 8:42 PM, "Vithulan MV" <vithulanmv.12 at cse.mrt.ac.lk>
>> wrote:
>>
>>> Hi Viktor,
>>>
>>> Thanks for the link :) I'll check that.
>>>
>>> Regards,
>>> Vithulan.
>>>
>>> On 10 March 2016 at 20:22, Juhász, Viktor <viktor.juhasz at balabit.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> Sorry I missed the link :)
>>>> https://syslog-ng.org/gsoc-2016/
>>>>
>>>> BR,
>>>> Viktor
>>>>
>>>> On Thu, Mar 10, 2016 at 3:46 PM, Vithulan MV <
>>>> vithulanmv.12 at cse.mrt.ac.lk> wrote:
>>>>
>>>>> Hi Viktor,
>>>>>
>>>>> Do you mean creating proposals for GSOC 2016 page [1] ? If not please
>>>>> kindly share the page link with me?
>>>>>
>>>>> [1]
>>>>> https://github.com/balabit/syslog-ng/wiki/Creating-a-proposal-for-GSoC-2016
>>>>>
>>>>> Thanks.
>>>>> Regards,
>>>>> Vithulan.
>>>>>
>>>>> On 10 March 2016 at 19:40, Juhász, Viktor <viktor.juhasz at balabit.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Vithulan,
>>>>>>
>>>>>> We've finished our GSoC process description. Please read it
>>>>>> carefully, and fill our Student Application Form.
>>>>>>
>>>>>> BR,
>>>>>> Viktor
>>>>>>
>>>>>> On Thu, Mar 10, 2016 at 3:06 PM, Vithulan MV <
>>>>>> vithulanmv.12 at cse.mrt.ac.lk> wrote:
>>>>>>
>>>>>>> HI Viktor,
>>>>>>>
>>>>>>> Thanks for the overview and giving the overall idea. I'm working on
>>>>>>> this and report you back on the progress.
>>>>>>>
>>>>>>> Thanks.
>>>>>>> Regards,
>>>>>>> Vithulan.
>>>>>>>
>>>>>>> On 8 March 2016 at 18:05, Juhász, Viktor <viktor.juhasz at balabit.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Vithulan,
>>>>>>>>
>>>>>>>> Yes, this should be plugin (you can find the plugins in the modules
>>>>>>>> directory in syslog-ng source).
>>>>>>>>
>>>>>>>> Just a very briefly overview:
>>>>>>>>
>>>>>>>> syslog-ng read messages from the sources, process them with
>>>>>>>> filters, rewrite rules, parsers, etc, and finally send messages to the
>>>>>>>> destinations.
>>>>>>>> We have a Kafka destination, it is implemented in java. The kafka
>>>>>>>> source also should be implemented in java, but we our java source base is
>>>>>>>> under construction, so till I can't give you an exact API, it is comming in
>>>>>>>> the next 2-3 weeks.
>>>>>>>>
>>>>>>>> The ovarall idea is the following.
>>>>>>>>  - We should implement a Kafka consumer which uses the high level
>>>>>>>> kafka group consuming API (
>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
>>>>>>>> )
>>>>>>>>  - We have to process data read from the Kafka (create LogMessage
>>>>>>>> from this. LogMessage is an internal struct in syslog-ng representing the
>>>>>>>> in the software)
>>>>>>>>  - After reload or restart syslog-ng have to be able to continue
>>>>>>>> reading messages from the last read message.
>>>>>>>>  - If there are more syslog-ng's reading the same Kafka input
>>>>>>>> (these syslog-ng's have the same group name) avoid message loosing or
>>>>>>>> message duplication as much as possible
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Viktor
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Mar 7, 2016 at 6:43 PM, Vithulan MV <
>>>>>>>> vithulanmv.12 at cse.mrt.ac.lk> wrote:
>>>>>>>>
>>>>>>>>> Hi Viktor,
>>>>>>>>>
>>>>>>>>> Oh I see! I've sent a mail to the mail list. Can you kindly give
>>>>>>>>> me an overall idea of project Kafka source please? (Architecture of
>>>>>>>>> syslog-ng/ current APIs) Is this is going to be an extension of syslog-ng?
>>>>>>>>>
>>>>>>>>> Thank you.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Vithulan.
>>>>>>>>> Hi Vithulan,
>>>>>>>>>
>>>>>>>>> I've never faced with this problem, during compiling syslog-ng.
>>>>>>>>> I suggest, that ask about your problem in the mail list of
>>>>>>>>> syslog-ng ;)
>>>>>>>>>
>>>>>>>>> Best Regards,
>>>>>>>>> Viktor
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Mar 7, 2016 at 12:46 PM, Vithulan MV <
>>>>>>>>> vithulanmv.12 at cse.mrt.ac.lk> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I've been going through the git book that you referred. It was
>>>>>>>>>> helpful. Can you please explain a bit about the project structure and
>>>>>>>>>> deliverables of Kafka-source project?
>>>>>>>>>>
>>>>>>>>>> I added required dependencies and tried to build syslog-ng from
>>>>>>>>>> the source, there was an error saying,
>>>>>>>>>> *${syslog-ng_HOME}/modules/afmongodb/mongo-c-driver/src/mongoc/mongoc-scram.c:
>>>>>>>>>> In function '_mongoc_scram_sha1':* *storage size of 'digest_ctx'
>>>>>>>>>> isn't known** (EVP_MD_CTX digest_ctx)*
>>>>>>>>>> I confirmed OpenSSL [1] headers are included in the file.Can you
>>>>>>>>>> please point out whats wrong?
>>>>>>>>>>
>>>>>>>>>> In the meantime I'm going through the syslog-ng documentation
>>>>>>>>>> [2]. Can you give me directives on what I can be doing while preparing my
>>>>>>>>>> proposals?
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>> https://github.com/mongodb/mongo-c-driver/blob/7415d3468de0c500fd6e630f7b52be181cbb66b3/src/mongoc/mongoc-scram.c#L311
>>>>>>>>>> [2]
>>>>>>>>>> https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/so-contents.html
>>>>>>>>>>
>>>>>>>>>> Thank you.
>>>>>>>>>> Regards,
>>>>>>>>>> Vithulan.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 4 March 2016 at 18:56, Vithulan MV <
>>>>>>>>>> vithulanmv.12 at cse.mrt.ac.lk> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> Thanks for the reference, I will go through it and report you
>>>>>>>>>>> back on the progress.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Regards,
>>>>>>>>>>> Vithulan.
>>>>>>>>>>>
>>>>>>>>>>> On 4 March 2016 at 18:45, Juhász, Viktor <
>>>>>>>>>>> viktor.juhasz at balabit.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>
>>>>>>>>>>>> Sorry for the long response time. We are working on a wiki
>>>>>>>>>>>> page, this page will describe all the required information. Till please
>>>>>>>>>>>> read our documentation (getting started)
>>>>>>>>>>>> https://www.gitbook.com/book/syslog-ng/getting-started/details
>>>>>>>>>>>>
>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>> Viktor Juhasz
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Mar 4, 2016 at 2:09 PM, Vithulan MV <
>>>>>>>>>>>> vithulanmv.12 at cse.mrt.ac.lk> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I sent the following mail to the syslog-ng mailing list, but
>>>>>>>>>>>>> didn't get any responses. So I thought it didn't reach to you. I'm highly
>>>>>>>>>>>>> motivated on contributing for syslog-ng in GSoC 2016. I also have prior
>>>>>>>>>>>>> experience in syslog-ng domain so I feel like this project is going to be
>>>>>>>>>>>>> very interesting. Please kindly guide me on how I should start?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Vithulan.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------- Forwarded message ----------
>>>>>>>>>>>>> From: Vithulan MV <vithulanmv.12 at cse.mrt.ac.lk>
>>>>>>>>>>>>> Date: 3 March 2016 at 15:33
>>>>>>>>>>>>> Subject: [GSoC] Project: Kafka source
>>>>>>>>>>>>> To: syslog-ng at lists.balabit.hu
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm currently a 3rd year undergraduate at University of
>>>>>>>>>>>>> Moratuwa, Sri lanka. I have attached my Linkedin profile with this mail
>>>>>>>>>>>>> [1]. I'd like to contribute to a GSoC 2016 project with the open source
>>>>>>>>>>>>> community of syslog-ng.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have contributed in log analyzing project LogAnalyzer [2]
>>>>>>>>>>>>> and also worked with logstash, kafka. Therefore, I think I can contribute
>>>>>>>>>>>>> syslog-ng project kafka source as I'm much interested in that.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please kindly make any suggestions on how I should proceed on
>>>>>>>>>>>>> this?
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1] https://lk.linkedin.com/in/vithulamv
>>>>>>>>>>>>> [2] https://github.com/VIthulan/product-la
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Vithulan
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Vithulan MV.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Undergraduate,
>>>>>>>>>>>>> Department of Computer Science & Engineering,
>>>>>>>>>>>>> University of Moratuwa,
>>>>>>>>>>>>> Sri lanka.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Vithulan MV.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Undergraduate,
>>>>>>>>>>>>> Department of Computer Science & Engineering,
>>>>>>>>>>>>> University of Moratuwa,
>>>>>>>>>>>>> Sri lanka.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Vithulan MV.
>>>>>>>>>>>
>>>>>>>>>>> Undergraduate,
>>>>>>>>>>> Department of Computer Science & Engineering,
>>>>>>>>>>> University of Moratuwa,
>>>>>>>>>>> Sri lanka.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Vithulan MV.
>>>>>>>>>>
>>>>>>>>>> Undergraduate,
>>>>>>>>>> Department of Computer Science & Engineering,
>>>>>>>>>> University of Moratuwa,
>>>>>>>>>> Sri lanka.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Vithulan MV.
>>>>>>>
>>>>>>> Undergraduate,
>>>>>>> Department of Computer Science & Engineering,
>>>>>>> University of Moratuwa,
>>>>>>> Sri lanka.
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Vithulan MV.
>>>>>
>>>>> Undergraduate,
>>>>> Department of Computer Science & Engineering,
>>>>> University of Moratuwa,
>>>>> Sri lanka.
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Vithulan MV.
>>>
>>> Undergraduate,
>>> Department of Computer Science & Engineering,
>>> University of Moratuwa,
>>> Sri lanka.
>>>
>>
>
>
> --
> Vithulan MV.
>
> Undergraduate,
> Department of Computer Science & Engineering,
> University of Moratuwa,
> Sri lanka.
>

-- 
Vithulan MV.

Undergraduate,
Department of Computer Science & Engineering,
University of Moratuwa,
Sri lanka.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160427/caef2e25/attachment-0001.htm 