- syslog-ng - lists.balabit.hu

Syslog-NG with MongoDB
by Ivan Adji - Krstev 24 May '16

24 May '16

Hi all, What is the best practice for storing all those logs in one central environment. I have one Linux Box running Syslog-NG with LogAnalyzer and MongoDB ( for now ), and is the best way to configure and use it with MongoDB or with MariaDB ( MySQL ) ? I have once install MySQL but it was getting very slow as the logs getting bigger and bigger ( for one week ). Now i have done with MongoDB ( still testing ) but i have problem as LogAnalyzer does not show me the real pictures, i have no Date info, no Facility, no serverity, Hosts, syslogtag, i just have ProcessID. Any hints on this ? I have the following configuration on the syslog-ng.cfg: destination d_mongodb { mongodb( servers("localhost:27017") database("logs") # uri('mongodb://localhost/syslog-ng') collection("syslog") value-pairs( scope("selected-macros" "nv-pairs" "sdata") ) ); }; Kind regards Ivan

5 10

[GSoC]Finding Feature owner for "Web-based syslog-ng configuration editor"
by Raymund Ács 23 May '16

23 May '16

Hi, I am PoOwAa (http://github.com/poowaa) I could not write here before because I had some issues with subscribing. I already subscribed and unsubscribed few months ago, and I was not able to subscribe again. Finally I got my confirmation mail, so I can join to the community! :) In the summer I will work on the Web-based syslog-ng configuration editor project in GSoC 2016. Here is my proposal link: https://github.com/balabit/syslog-ng/wiki/GSoC-2016-Proposal:-Web-based-sys… This project will provide a Web Application which can create and save syslog-ng configuration files by drag&drop technique. I would like to follow the trends of 2016 in design, and give a detailed documentation. If you are interested in being the feature owner of this project, please reply in this thread. Advices from you will help us greatly and make the final project more user friendly. Best Regards, Raymund Acs University of Debrecen

1 0

[Release Automation Project] Finding feature owners
by Ankush Sharma 23 May '16

23 May '16

Hi, I am taking this as an opportunity to introduce myself to everyone in the community. I got a bit late as I was busy in relocating back and finishing college paper work. Sorry for it ;-) I am* Ankush Sharma (black-perl <https://github.com/black-perl>)* from India. I contributed to Mailman <http://list.org> last year. So, I will be working on the Automation Release <https://github.com/balabit/syslog-ng/wiki/%5BGSoC'16%5D-Automated-release-generation-for-Syslog-ng> project for syslog-ng this time. The project will mostly involve Python usage and Flask/Django later on. The potential areas where any feature owner can look into are: 1) *Web interface: *An experimental web interface to track things during the release automation cycle would be handy. I am considering to use Flask/Django (web frameworks of Python) for it. You can suggest integrations to it or information you would like to see on the interface. 2) *Overall architecture: *We can discuss the flow from user perspective as well as the architectural view of the tool and you can suggest changes to it ;-) Here is the link to the intro post : http://black-perl.in/black-perl-gsoc-with-syslog-ng/ <http://black-perl.in/black-perl-gsoc-with-syslog-ng/>. You can find a brief introduction to the project here and links to the detailed proposal. I would love to talk to other guys who will be working throughout the summers as part of the GSoC program: @litterbear, @kvch, @PoOwAa, @Vithulan. It would be great if we can have chat sometime and get to know about each others' work ! Thanks ! Ankush Sharma ECE IV IIT-BHU Varanasi-221005 http://black-perl.in Linkedin <https://www.linkedin.com/in/ankushsharma003>

2 1

[GSOC]Finding Feature owner for WebSocket for syslog ng
by Yilin Li 23 May '16

23 May '16

Hi I'm Yilin(https://github.com/litterbear). I'm working on the "WebSocket for syslog ng" project in GSOC 2016. Here is the link to my proposal: https://github.com/balabit/syslog-ng/wiki/GSoC-2016-Proposal:-WebSocket-for… This project will provide - a websocket plugin for syslog-ng source and destination. - user-friendly and detailed documents and examples. The draft design is shared here : https://docs.google.com/document/d/1FVc4EySiTu9RHdlsPd-NGQ9JzvV0WInQrM2VBA4… Any suggestions will be appreciated :) If you are interested in being the feature owner of the project, please reply in this thread. Advices on user's perspective from you will help us greatly and make the final deliverables more user friendly. Thanks in advance. -- Best wishes, Yilin Li Institute of Software Chinese Academy of Sciences

2 2

Re: [syslog-ng] Finding feature owner for Kafka source project
by Laszlo Budai 22 May '16

22 May '16

Sent from my HTC ----- Reply message ----- From: "Vincent Bernat" <bernat(a)luffy.cx> To: "Laszlo Budai" <laszlo.budai(a)outlook.com> Cc: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu>, "Vithulan MV" <vithulanmv.12(a)cse.mrt.ac.lk> Subject: [syslog-ng] Finding feature owner for Kafka source project Date: Sat, May 21, 2016 23:58 ❦ 21 mai 2016 11:58 -0700, Laszlo Budai <laszlo.budai(a)outlook.com> : > As you mentioned we have Kafka destination driver written in C in > syslog-ng incubator, so users can use it. It should not be a problem > if a feature is available only in the syslog-ng incubator. Well, it is because most users will never use it. I don't know why... maybe we should make it clear what is incubator.For me it contains two kind of features:1) experimental features that are not really tested, and maybe the code would also neee some polishment before merge into syslog-ng master2) well implemented features that are using some unstable dependencies> As the Kafka C client is stable enough, I don't see any problems in > integrating it to the syslog-ng master (just someone should create a > pull request). Good to know. That's not what you told me in August. I will fix the performance limitations and do a PR for master.I said that my plan is to keep the C kafka dst in incubator. Last year ;)But Kafka C lib seems to be a stable lib during the last 8-9 months which makes it possible to move forward from incubator. L.-- Make it right before you make it faster. - The Elements of Programming Style (Kernighan & Plauger)

1 0

Re: [syslog-ng] Finding feature owner for Kafka source project
by Laszlo Budai 21 May '16

21 May '16

Hi Vincent, the currently available Java features are * Kafka * Elasticsearch * HDFS (we have HTTP but we also have curl destination in C). As you mentioned we have Kafka destination driver written in C in syslog-ng incubator, so users can use it. It should not be a problem if a feature is available only in the syslog-ng incubator. As the Kafka C client is stable enough, I don't see any problems in integrating it to the syslog-ng master (just someone should create a pull request). In case of Elasticsearch we are happy that we are able to support even the latest Elasticsearch 5 but we don't have the resource to re-implement it in C after every new Elasticsearch release. If someone wants to implement a REST Elasticsearch client in C and based on that also implements an ES dest. driver in C then, we will happily integrate it into syslog-ng. HDFS: users who uses HDFS will have Java on their machines, so it should not be a problem. I think a feature implemented in Java is much-much better than an unimplemented feature... regards, Laszlo Budai Sent from my HTC ----- Reply message ----- From: "Vincent Bernat" <bernat(a)luffy.cx> To: "Vithulan MV" <vithulanmv.12(a)cse.mrt.ac.lk> Cc: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] Finding feature owner for Kafka source project Date: Sat, May 21, 2016 19:38 ❦ 21 mai 2016 20:33 +0530, Vithulan MV <vithulanmv.12(a)cse.mrt.ac.lk> : > A simple abstract idea of this project is, > The syslog-ng application can read messages from the sources. It > processes them with filters, rewrite rules, parsers and finally sends > messages to their destinations. The syslog-ng application already has > a Kafka destination that is implemented in Java. The Kafka source will > allow syslog-ng to read messages from Kafka, for an example this can > be used for example as a queue between several syslog-ng instances. It's a bit annoying that so many interesting features have to go through Java. One of the major point of syslog-ng is its light footprint. librdkafka has support for both source and destination and is officially supported by Confluent. Its development matches Kafka one and it supports every feature in Kafka 0.9, including SSL and the new consumer. It's a bit late to complain, notably if you chose this project because you were comfortable with Java (and your mentor is obviously interested in doing that in Java too). However, this has already been pointed out in March without much response from Balabit. It's not just a matter of proposing code since such code is likely to get stuck in the incubator forever. -- Make sure special cases are truly special. - The Elements of Programming Style (Kernighan & Plauger) ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

2 1

Finding feature owner for Kafka source project
by Vithulan MV 21 May '16

21 May '16

Hi all, As you may have all known already, We are going to develop a new feature, Kafka source for syslog-ng product during this GSoC 2016 under the guidance of Viktor Juhasz. A simple abstract idea of this project is, The syslog-ng application can read messages from the sources. It processes them with filters, rewrite rules, parsers and finally sends messages to their destinations. The syslog-ng application already has a Kafka destination that is implemented in Java. The Kafka source will allow syslog-ng to read messages from Kafka, for an example this can be used for example as a queue between several syslog-ng instances. Kafka source project will allow you to, -Use Kafka consumer which uses the high level Kafka group consuming API. -Continue reading from the last message from which syslog-ng reloaded/restarted. -Avoid message loosing or message duplication as much as possible whenever there is multiple syslog-ng reading the same kafka input. If you are interested to become a feature owner for Kafka source project and give us guidance on user's perspective, it will be very easy for us to build a effective component. Appreciate if you can contact us by sending mail to us or to this thread. Thanks, Best regards, Vithulan. -- Vithulan MV. Undergraduate, Department of Computer Science & Engineering, University of Moratuwa, Sri lanka.

3 2

Re: [syslog-ng] message loss using multi-line-mode
by Laszlo Budai 19 May '16

19 May '16

Hi, opening a new issue is not necessary. I hope that a GSoC student will solve this issue. BTW GSoC: we are still looking for feature owners to our GSoC[1] projects[2]. As an experienced syslog-ng user you could join to us. What do you think? [1] https://syslog-ng.org/gsoc-2016/ [2] https://lists.balabit.hu/pipermail/syslog-ng/2016-April/022862.html regards, Laszlo Budai Sent from my HTC ----- Reply message ----- From: "Patrick Hemmer" <syslogng(a)stormcloud9.net> To: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] message loss using multi-line-mode Date: Tue, May 17, 2016 00:17 Anybody? Should I open a new issue? -Patrick On 2016/3/28 12:01, Patrick Hemmer wrote: > I raised issue on an existing bug on the github tracker, but it seems > to have gone unnoticed, so I'm repeating it here to try and get some > attention on the issue. > > Using multi-line-mode without the multi-line-suffix option WILL result > in message loss. > When syslog-ng is running in multi-line-mode, it buffers multi-line > messages until it sees the start of a new message. When it sees the > start of a new message, it flushes the buffered message, and puts the > first line of the new message in the buffer. However if syslog-ng > shuts down, or receives a SIGHUP (reload), any lines currently > buffered are discarded. Given that syslog-ng can't stay running > forever, and it will get shut down or SIGHUPd eventually, using this > feature will result in messages getting lost. > > The message on the github issue where I brought this up is: > https://github.com/balabit/syslog-ng/issues/140#issuecomment-197673887 > > -Patrick > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

2 1

Re: [syslog-ng] message loss using multi-line-mode
by Laszlo Budai 19 May '16

19 May '16

Hi, it is an ideal topic for GSoC students during community bonding period (or those who want to participate in GSoC in the future). Any volunteer? L. Sent from my HTC ----- Reply message ----- From: syslogng(a)stormcloud9.net To: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] message loss using multi-line-mode Date: Wed, May 18, 2016 15:58 That was my original intent (to write a patch), but after looking at the code, I didn't think it would be a good idea (I used to code against syslog-ng back in the around the 3.1 days, but it appears the code has gotten significantly more complex). Given that there is message loss involved here, I just thought it might raise the priority of the issue. -Patrick On 2016/5/17 08:47, Scheidler, Balázs wrote: > hi, > > I don't think that would help, sorry. Everyone is working on other > stuff as it seems. Would you try to cook up a patch? I know it might > intimidating at first, but you know it's all just code and computers > :) not more complicated than a shell script. * > > [*] ok, a bit more complicated, but everyone has to start somewhere :) > > > -- > Bazsi > > On Tue, May 17, 2016 at 12:16 AM, Patrick Hemmer > <syslogng(a)stormcloud9.net <mailto:syslogng@stormcloud9.net>> wrote: > > Anybody? Should I open a new issue? > > -Patrick > > > On 2016/3/28 12:01, Patrick Hemmer wrote: >> I raised issue on an existing bug on the github tracker, but it >> seems to have gone unnoticed, so I'm repeating it here to try and >> get some attention on the issue. >> >> Using multi-line-mode without the multi-line-suffix option WILL >> result in message loss. >> When syslog-ng is running in multi-line-mode, it buffers >> multi-line messages until it sees the start of a new message. >> When it sees the start of a new message, it flushes the buffered >> message, and puts the first line of the new message in the >> buffer. However if syslog-ng shuts down, or receives a SIGHUP >> (reload), any lines currently buffered are discarded. Given that >> syslog-ng can't stay running forever, and it will get shut down >> or SIGHUPd eventually, using this feature will result in messages >> getting lost. >> >> The message on the github issue where I brought this up is: >> https://github.com/balabit/syslog-ng/issues/140#issuecomment-197673887 >> >> -Patrick >> >> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

message loss using multi-line-mode
by Patrick Hemmer 18 May '16

18 May '16

I raised issue on an existing bug on the github tracker, but it seems to have gone unnoticed, so I'm repeating it here to try and get some attention on the issue. Using multi-line-mode without the multi-line-suffix option WILL result in message loss. When syslog-ng is running in multi-line-mode, it buffers multi-line messages until it sees the start of a new message. When it sees the start of a new message, it flushes the buffered message, and puts the first line of the new message in the buffer. However if syslog-ng shuts down, or receives a SIGHUP (reload), any lines currently buffered are discarded. Given that syslog-ng can't stay running forever, and it will get shut down or SIGHUPd eventually, using this feature will result in messages getting lost. The message on the github issue where I brought this up is: https://github.com/balabit/syslog-ng/issues/140#issuecomment-197673887 -Patrick

4 4