[syslog-ng] [GSoC] Project: Kafka source

Thu Apr 28 00:25:58 CEST 2016

Great to have you here :)

PS: I am not using an IDE but a plain text editor and a few tabs of my
terminal app.
On Apr 27, 2016 14:15, "Vithulan MV" <vithulanmv.12 at cse.mrt.ac.lk> wrote:

> Hi all,
>
> First of all, I'd like to thank you Syslog-ng community and my mentor
> Viktor Juhasz for accepting my proposal to implement Kafka source in Java!.
> Some of you may already know me but for others, I'll introduce myself
> again. I'm Vithulan, 3rd year undergraduate at University of Moratuwa, Sri
> lanka. My areas of interests are, Data analytics, Machine Learning,
> Networking and Security.
>
> As a first step into this GSoC program I already started following the
> documentation [1], and going through the code base of syslog-ng (I'm
> curious what IDE you all are using?). What are the other areas that you
> want me to know to polish my knowledge before we get into the coding period?
>
> I guess, this is going to be very interesting 4 months with good new
> experiences :)
> Thank you!
>
> [1]
> https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/preface.html
>
> Best regards,
> Vithulan.
>
> On 23 April 2016 at 08:42, Vithulan MV <vithulanmv.12 at cse.mrt.ac.lk>
> wrote:
>
>> Hi Viktor,
>>
>> I am very excited and happy after seeing my morning mail from Google
>> summer of code! Thank you very much for accepting my proposal! :)
>> How shall we proceed into this project? and any tips before going deep
>> into the project? :)
>>
>> Thank you.
>>
>> Best regards,
>> Vithulan.
>>
>> On 26 March 2016 at 10:52, Vithulan MV <vithulanmv.12 at cse.mrt.ac.lk>
>> wrote:
>>
>>> Hi Viktor,
>>>
>>> I have submitted my proposal in GSoC page and Syslog-ng wiki page
>>> aswell.
>>>
>>> Thanks.
>>> Best regards,
>>> Vithulan
>>> On Mar 10, 2016 8:42 PM, "Vithulan MV" <vithulanmv.12 at cse.mrt.ac.lk>
>>> wrote:
>>>
>>>> Hi Viktor,
>>>>
>>>> Thanks for the link :) I'll check that.
>>>>
>>>> Regards,
>>>> Vithulan.
>>>>
>>>> On 10 March 2016 at 20:22, Juhász, Viktor <viktor.juhasz at balabit.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Sorry I missed the link :)
>>>>> https://syslog-ng.org/gsoc-2016/
>>>>>
>>>>> BR,
>>>>> Viktor
>>>>>
>>>>> On Thu, Mar 10, 2016 at 3:46 PM, Vithulan MV <
>>>>> vithulanmv.12 at cse.mrt.ac.lk> wrote:
>>>>>
>>>>>> Hi Viktor,
>>>>>>
>>>>>> Do you mean creating proposals for GSOC 2016 page [1] ? If not please
>>>>>> kindly share the page link with me?
>>>>>>
>>>>>> [1]
>>>>>> https://github.com/balabit/syslog-ng/wiki/Creating-a-proposal-for-GSoC-2016
>>>>>>
>>>>>> Thanks.
>>>>>> Regards,
>>>>>> Vithulan.
>>>>>>
>>>>>> On 10 March 2016 at 19:40, Juhász, Viktor <viktor.juhasz at balabit.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Vithulan,
>>>>>>>
>>>>>>> We've finished our GSoC process description. Please read it
>>>>>>> carefully, and fill our Student Application Form.
>>>>>>>
>>>>>>> BR,
>>>>>>> Viktor
>>>>>>>
>>>>>>> On Thu, Mar 10, 2016 at 3:06 PM, Vithulan MV <
>>>>>>> vithulanmv.12 at cse.mrt.ac.lk> wrote:
>>>>>>>
>>>>>>>> HI Viktor,
>>>>>>>>
>>>>>>>> Thanks for the overview and giving the overall idea. I'm working on
>>>>>>>> this and report you back on the progress.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>> Regards,
>>>>>>>> Vithulan.
>>>>>>>>
>>>>>>>> On 8 March 2016 at 18:05, Juhász, Viktor <viktor.juhasz at balabit.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> Hi Vithulan,
>>>>>>>>>
>>>>>>>>> Yes, this should be plugin (you can find the plugins in the
>>>>>>>>> modules directory in syslog-ng source).
>>>>>>>>>
>>>>>>>>> Just a very briefly overview:
>>>>>>>>>
>>>>>>>>> syslog-ng read messages from the sources, process them with
>>>>>>>>> filters, rewrite rules, parsers, etc, and finally send messages to the
>>>>>>>>> destinations.
>>>>>>>>> We have a Kafka destination, it is implemented in java. The kafka
>>>>>>>>> source also should be implemented in java, but we our java source base is
>>>>>>>>> under construction, so till I can't give you an exact API, it is comming in
>>>>>>>>> the next 2-3 weeks.
>>>>>>>>>
>>>>>>>>> The ovarall idea is the following.
>>>>>>>>>  - We should implement a Kafka consumer which uses the high level
>>>>>>>>> kafka group consuming API (
>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
>>>>>>>>> )
>>>>>>>>>  - We have to process data read from the Kafka (create LogMessage
>>>>>>>>> from this. LogMessage is an internal struct in syslog-ng representing the
>>>>>>>>> in the software)
>>>>>>>>>  - After reload or restart syslog-ng have to be able to continue
>>>>>>>>> reading messages from the last read message.
>>>>>>>>>  - If there are more syslog-ng's reading the same Kafka input
>>>>>>>>> (these syslog-ng's have the same group name) avoid message loosing or
>>>>>>>>> message duplication as much as possible
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Viktor
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Mar 7, 2016 at 6:43 PM, Vithulan MV <
>>>>>>>>> vithulanmv.12 at cse.mrt.ac.lk> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Viktor,
>>>>>>>>>>
>>>>>>>>>> Oh I see! I've sent a mail to the mail list. Can you kindly give
>>>>>>>>>> me an overall idea of project Kafka source please? (Architecture of
>>>>>>>>>> syslog-ng/ current APIs) Is this is going to be an extension of syslog-ng?
>>>>>>>>>>
>>>>>>>>>> Thank you.
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Vithulan.
>>>>>>>>>> Hi Vithulan,
>>>>>>>>>>
>>>>>>>>>> I've never faced with this problem, during compiling syslog-ng.
>>>>>>>>>> I suggest, that ask about your problem in the mail list of
>>>>>>>>>> syslog-ng ;)
>>>>>>>>>>
>>>>>>>>>> Best Regards,
>>>>>>>>>> Viktor
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Mar 7, 2016 at 12:46 PM, Vithulan MV <
>>>>>>>>>> vithulanmv.12 at cse.mrt.ac.lk> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I've been going through the git book that you referred. It was
>>>>>>>>>>> helpful. Can you please explain a bit about the project structure and
>>>>>>>>>>> deliverables of Kafka-source project?
>>>>>>>>>>>
>>>>>>>>>>> I added required dependencies and tried to build syslog-ng from
>>>>>>>>>>> the source, there was an error saying,
>>>>>>>>>>> *${syslog-ng_HOME}/modules/afmongodb/mongo-c-driver/src/mongoc/mongoc-scram.c:
>>>>>>>>>>> In function '_mongoc_scram_sha1':* *storage size of
>>>>>>>>>>> 'digest_ctx' isn't known** (EVP_MD_CTX digest_ctx)*
>>>>>>>>>>> I confirmed OpenSSL [1] headers are included in the file.Can you
>>>>>>>>>>> please point out whats wrong?
>>>>>>>>>>>
>>>>>>>>>>> In the meantime I'm going through the syslog-ng documentation
>>>>>>>>>>> [2]. Can you give me directives on what I can be doing while preparing my
>>>>>>>>>>> proposals?
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>> https://github.com/mongodb/mongo-c-driver/blob/7415d3468de0c500fd6e630f7b52be181cbb66b3/src/mongoc/mongoc-scram.c#L311
>>>>>>>>>>> [2]
>>>>>>>>>>> https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/so-contents.html
>>>>>>>>>>>
>>>>>>>>>>> Thank you.
>>>>>>>>>>> Regards,
>>>>>>>>>>> Vithulan.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 4 March 2016 at 18:56, Vithulan MV <
>>>>>>>>>>> vithulanmv.12 at cse.mrt.ac.lk> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for the reference, I will go through it and report you
>>>>>>>>>>>> back on the progress.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Vithulan.
>>>>>>>>>>>>
>>>>>>>>>>>> On 4 March 2016 at 18:45, Juhász, Viktor <
>>>>>>>>>>>> viktor.juhasz at balabit.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sorry for the long response time. We are working on a wiki
>>>>>>>>>>>>> page, this page will describe all the required information. Till please
>>>>>>>>>>>>> read our documentation (getting started)
>>>>>>>>>>>>> https://www.gitbook.com/book/syslog-ng/getting-started/details
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>> Viktor Juhasz
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Mar 4, 2016 at 2:09 PM, Vithulan MV <
>>>>>>>>>>>>> vithulanmv.12 at cse.mrt.ac.lk> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I sent the following mail to the syslog-ng mailing list, but
>>>>>>>>>>>>>> didn't get any responses. So I thought it didn't reach to you. I'm highly
>>>>>>>>>>>>>> motivated on contributing for syslog-ng in GSoC 2016. I also have prior
>>>>>>>>>>>>>> experience in syslog-ng domain so I feel like this project is going to be
>>>>>>>>>>>>>> very interesting. Please kindly guide me on how I should start?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Vithulan.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---------- Forwarded message ----------
>>>>>>>>>>>>>> From: Vithulan MV <vithulanmv.12 at cse.mrt.ac.lk>
>>>>>>>>>>>>>> Date: 3 March 2016 at 15:33
>>>>>>>>>>>>>> Subject: [GSoC] Project: Kafka source
>>>>>>>>>>>>>> To: syslog-ng at lists.balabit.hu
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm currently a 3rd year undergraduate at University of
>>>>>>>>>>>>>> Moratuwa, Sri lanka. I have attached my Linkedin profile with this mail
>>>>>>>>>>>>>> [1]. I'd like to contribute to a GSoC 2016 project with the open source
>>>>>>>>>>>>>> community of syslog-ng.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have contributed in log analyzing project LogAnalyzer [2]
>>>>>>>>>>>>>> and also worked with logstash, kafka. Therefore, I think I can contribute
>>>>>>>>>>>>>> syslog-ng project kafka source as I'm much interested in that.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Please kindly make any suggestions on how I should proceed on
>>>>>>>>>>>>>> this?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1] https://lk.linkedin.com/in/vithulamv
>>>>>>>>>>>>>> [2] https://github.com/VIthulan/product-la
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Vithulan
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Vithulan MV.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Undergraduate,
>>>>>>>>>>>>>> Department of Computer Science & Engineering,
>>>>>>>>>>>>>> University of Moratuwa,
>>>>>>>>>>>>>> Sri lanka.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Vithulan MV.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Undergraduate,
>>>>>>>>>>>>>> Department of Computer Science & Engineering,
>>>>>>>>>>>>>> University of Moratuwa,
>>>>>>>>>>>>>> Sri lanka.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Vithulan MV.
>>>>>>>>>>>>
>>>>>>>>>>>> Undergraduate,
>>>>>>>>>>>> Department of Computer Science & Engineering,
>>>>>>>>>>>> University of Moratuwa,
>>>>>>>>>>>> Sri lanka.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Vithulan MV.
>>>>>>>>>>>
>>>>>>>>>>> Undergraduate,
>>>>>>>>>>> Department of Computer Science & Engineering,
>>>>>>>>>>> University of Moratuwa,
>>>>>>>>>>> Sri lanka.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Vithulan MV.
>>>>>>>>
>>>>>>>> Undergraduate,
>>>>>>>> Department of Computer Science & Engineering,
>>>>>>>> University of Moratuwa,
>>>>>>>> Sri lanka.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Vithulan MV.
>>>>>>
>>>>>> Undergraduate,
>>>>>> Department of Computer Science & Engineering,
>>>>>> University of Moratuwa,
>>>>>> Sri lanka.
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Vithulan MV.
>>>>
>>>> Undergraduate,
>>>> Department of Computer Science & Engineering,
>>>> University of Moratuwa,
>>>> Sri lanka.
>>>>
>>>
>>
>>
>> --
>> Vithulan MV.
>>
>> Undergraduate,
>> Department of Computer Science & Engineering,
>> University of Moratuwa,
>> Sri lanka.
>>
>
>
>
> --
> Vithulan MV.
>
> Undergraduate,
> Department of Computer Science & Engineering,
> University of Moratuwa,
> Sri lanka.
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160428/e61667dd/attachment-0001.htm 