- syslog-ng - lists.balabit.hu

Syslog-NG with Elasticsearch
by Ivan Adji - Krstev 30 May '16

30 May '16

Does syslog-ng 3.8 supports the latest Elasticsearch version 2 ? Kind regards Ivan

2 3

Web Visualize logs
by Ivan Adji - Krstev 26 May '16

26 May '16

Hi all, Me again :) ! Is there any other tool, free open source that can show me the log in some kind of graphs and can make some triggers etc. Like LogAnalyzer ? I want to change the LogAnalyzer and try other different web interfaces etc. Any suggestions ? I see Elasticsearch or Kibana but, as i see it now, they use some different protocols and styles (i still can't understand how is working) and im not sure if i can integrate with syslog-ng, but also i need a free tool :( . As im using Syslog-NG with MongoDB all i need is something that can present all this log and have options for sending e-mails if we have disaster or something and can query fast all this logs. LogAnalyzers is ok but i want to see different styles. Thanks in advanced ! Ivan

3 13

Transform file path
by Alexey Vlasov 25 May '16

25 May '16

Hi, I intend to manage the distribution of the Apache log-files for each virtualhost using syslog-ng. I write the following in Apache vhost configue: <VirtualHost *> SetEnv V3WUSER w_test-l26-apache-_b8649b LogFormat "%{V3WUSER}e %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" xcombined CustomLog /var/log/apache_aux2_worker2/access_pipe.log xcombined </IfModule> as a result I get this kind of format of the log file: w_test-l26-apache-_b8649b 10.0.2.24 - - [24/May/2016:12:41:33 +0300] "GET / HTTP/1.1" 200 - "http://example.com/" "ELinks (0.11.7; Linux 3.14.46-1gb-csm x86_64; 158x45)" where the first field is the unique identifier of the virtualhost. Next step I write the following in syslog-ng: source src_apache_piped_logs { pipe("/var/log/apache_aux2_worker2/access_pipe.log" keep_timestamp(no)); }; destination dst_apache_piped_logs { file("/var/log/virtwww/${V3WUSER}/access.log" template("$MSGONLY\n") template-escape(no)); }; log { source(src_apache_piped_logs); destination(dst_apache_piped_logs); }; Сonsequently I would like to have each log file of the virtualhost placed in its own catalog and the path should contain the first field of the message (${V3WUSER} = w_test-l26-apache-_b8649b) Is it actually possible? Thanks in advance for the help.

4 3

WANTED: feature owner of syslog-ng command line tool :)
by Noémi Ványi 24 May '16

24 May '16

Hello! A new feature is going to be developed, as a part of GSOC, so syslog-ng can be started as a command line tool. What could the CLI do for you? - pipelines of complex filtering, rewriting could be tested more easily, thus configuration could become simpler than before - existing logs could be transformed The format of the new command line options is not decided yet. So your help and insight is needed to design the interface and the usage of the tool that you and other people around the world would use. :) If you are intereted in contributing to the CLI, thus contributing to the syslog-ng community, please send an email to this thread. :) I am available at gitter for further info. :) Regards kvch

4 5

Syslog-NG with MongoDB
by Ivan Adji - Krstev 24 May '16

24 May '16

Hi all, What is the best practice for storing all those logs in one central environment. I have one Linux Box running Syslog-NG with LogAnalyzer and MongoDB ( for now ), and is the best way to configure and use it with MongoDB or with MariaDB ( MySQL ) ? I have once install MySQL but it was getting very slow as the logs getting bigger and bigger ( for one week ). Now i have done with MongoDB ( still testing ) but i have problem as LogAnalyzer does not show me the real pictures, i have no Date info, no Facility, no serverity, Hosts, syslogtag, i just have ProcessID. Any hints on this ? I have the following configuration on the syslog-ng.cfg: destination d_mongodb { mongodb( servers("localhost:27017") database("logs") # uri('mongodb://localhost/syslog-ng') collection("syslog") value-pairs( scope("selected-macros" "nv-pairs" "sdata") ) ); }; Kind regards Ivan

5 10

[GSoC]Finding Feature owner for "Web-based syslog-ng configuration editor"
by Raymund Ács 23 May '16

23 May '16

Hi, I am PoOwAa (http://github.com/poowaa) I could not write here before because I had some issues with subscribing. I already subscribed and unsubscribed few months ago, and I was not able to subscribe again. Finally I got my confirmation mail, so I can join to the community! :) In the summer I will work on the Web-based syslog-ng configuration editor project in GSoC 2016. Here is my proposal link: https://github.com/balabit/syslog-ng/wiki/GSoC-2016-Proposal:-Web-based-sys… This project will provide a Web Application which can create and save syslog-ng configuration files by drag&drop technique. I would like to follow the trends of 2016 in design, and give a detailed documentation. If you are interested in being the feature owner of this project, please reply in this thread. Advices from you will help us greatly and make the final project more user friendly. Best Regards, Raymund Acs University of Debrecen

1 0

[Release Automation Project] Finding feature owners
by Ankush Sharma 23 May '16

23 May '16

Hi, I am taking this as an opportunity to introduce myself to everyone in the community. I got a bit late as I was busy in relocating back and finishing college paper work. Sorry for it ;-) I am* Ankush Sharma (black-perl <https://github.com/black-perl>)* from India. I contributed to Mailman <http://list.org> last year. So, I will be working on the Automation Release <https://github.com/balabit/syslog-ng/wiki/%5BGSoC'16%5D-Automated-release-generation-for-Syslog-ng> project for syslog-ng this time. The project will mostly involve Python usage and Flask/Django later on. The potential areas where any feature owner can look into are: 1) *Web interface: *An experimental web interface to track things during the release automation cycle would be handy. I am considering to use Flask/Django (web frameworks of Python) for it. You can suggest integrations to it or information you would like to see on the interface. 2) *Overall architecture: *We can discuss the flow from user perspective as well as the architectural view of the tool and you can suggest changes to it ;-) Here is the link to the intro post : http://black-perl.in/black-perl-gsoc-with-syslog-ng/ <http://black-perl.in/black-perl-gsoc-with-syslog-ng/>. You can find a brief introduction to the project here and links to the detailed proposal. I would love to talk to other guys who will be working throughout the summers as part of the GSoC program: @litterbear, @kvch, @PoOwAa, @Vithulan. It would be great if we can have chat sometime and get to know about each others' work ! Thanks ! Ankush Sharma ECE IV IIT-BHU Varanasi-221005 http://black-perl.in Linkedin <https://www.linkedin.com/in/ankushsharma003>

2 1

[GSOC]Finding Feature owner for WebSocket for syslog ng
by Yilin Li 23 May '16

23 May '16

Hi I'm Yilin(https://github.com/litterbear). I'm working on the "WebSocket for syslog ng" project in GSOC 2016. Here is the link to my proposal: https://github.com/balabit/syslog-ng/wiki/GSoC-2016-Proposal:-WebSocket-for… This project will provide - a websocket plugin for syslog-ng source and destination. - user-friendly and detailed documents and examples. The draft design is shared here : https://docs.google.com/document/d/1FVc4EySiTu9RHdlsPd-NGQ9JzvV0WInQrM2VBA4… Any suggestions will be appreciated :) If you are interested in being the feature owner of the project, please reply in this thread. Advices on user's perspective from you will help us greatly and make the final deliverables more user friendly. Thanks in advance. -- Best wishes, Yilin Li Institute of Software Chinese Academy of Sciences

2 2

Re: [syslog-ng] Finding feature owner for Kafka source project
by Laszlo Budai 22 May '16

22 May '16

Sent from my HTC ----- Reply message ----- From: "Vincent Bernat" <bernat(a)luffy.cx> To: "Laszlo Budai" <laszlo.budai(a)outlook.com> Cc: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu>, "Vithulan MV" <vithulanmv.12(a)cse.mrt.ac.lk> Subject: [syslog-ng] Finding feature owner for Kafka source project Date: Sat, May 21, 2016 23:58 ❦ 21 mai 2016 11:58 -0700, Laszlo Budai <laszlo.budai(a)outlook.com> : > As you mentioned we have Kafka destination driver written in C in > syslog-ng incubator, so users can use it. It should not be a problem > if a feature is available only in the syslog-ng incubator. Well, it is because most users will never use it. I don't know why... maybe we should make it clear what is incubator.For me it contains two kind of features:1) experimental features that are not really tested, and maybe the code would also neee some polishment before merge into syslog-ng master2) well implemented features that are using some unstable dependencies> As the Kafka C client is stable enough, I don't see any problems in > integrating it to the syslog-ng master (just someone should create a > pull request). Good to know. That's not what you told me in August. I will fix the performance limitations and do a PR for master.I said that my plan is to keep the C kafka dst in incubator. Last year ;)But Kafka C lib seems to be a stable lib during the last 8-9 months which makes it possible to move forward from incubator. L.-- Make it right before you make it faster. - The Elements of Programming Style (Kernighan & Plauger)

1 0

Re: [syslog-ng] Finding feature owner for Kafka source project
by Laszlo Budai 21 May '16

21 May '16

Hi Vincent, the currently available Java features are * Kafka * Elasticsearch * HDFS (we have HTTP but we also have curl destination in C). As you mentioned we have Kafka destination driver written in C in syslog-ng incubator, so users can use it. It should not be a problem if a feature is available only in the syslog-ng incubator. As the Kafka C client is stable enough, I don't see any problems in integrating it to the syslog-ng master (just someone should create a pull request). In case of Elasticsearch we are happy that we are able to support even the latest Elasticsearch 5 but we don't have the resource to re-implement it in C after every new Elasticsearch release. If someone wants to implement a REST Elasticsearch client in C and based on that also implements an ES dest. driver in C then, we will happily integrate it into syslog-ng. HDFS: users who uses HDFS will have Java on their machines, so it should not be a problem. I think a feature implemented in Java is much-much better than an unimplemented feature... regards, Laszlo Budai Sent from my HTC ----- Reply message ----- From: "Vincent Bernat" <bernat(a)luffy.cx> To: "Vithulan MV" <vithulanmv.12(a)cse.mrt.ac.lk> Cc: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] Finding feature owner for Kafka source project Date: Sat, May 21, 2016 19:38 ❦ 21 mai 2016 20:33 +0530, Vithulan MV <vithulanmv.12(a)cse.mrt.ac.lk> : > A simple abstract idea of this project is, > The syslog-ng application can read messages from the sources. It > processes them with filters, rewrite rules, parsers and finally sends > messages to their destinations. The syslog-ng application already has > a Kafka destination that is implemented in Java. The Kafka source will > allow syslog-ng to read messages from Kafka, for an example this can > be used for example as a queue between several syslog-ng instances. It's a bit annoying that so many interesting features have to go through Java. One of the major point of syslog-ng is its light footprint. librdkafka has support for both source and destination and is officially supported by Confluent. Its development matches Kafka one and it supports every feature in Kafka 0.9, including SSL and the new consumer. It's a bit late to complain, notably if you chose this project because you were comfortable with Java (and your mentor is obviously interested in doing that in Java too). However, this has already been pointed out in March without much response from Balabit. It's not just a matter of proposing code since such code is likely to get stuck in the incubator forever. -- Make sure special cases are truly special. - The Elements of Programming Style (Kernighan & Plauger) ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

2 1