- syslog-ng - lists.balabit.hu

Syslog-NG with Elasticsearch
by Ivan Adji - Krstev 30 May '16

30 May '16

Does syslog-ng 3.8 supports the latest Elasticsearch version 2 ? Kind regards Ivan

2 3

Web Visualize logs
by Ivan Adji - Krstev 26 May '16

26 May '16

Hi all, Me again :) ! Is there any other tool, free open source that can show me the log in some kind of graphs and can make some triggers etc. Like LogAnalyzer ? I want to change the LogAnalyzer and try other different web interfaces etc. Any suggestions ? I see Elasticsearch or Kibana but, as i see it now, they use some different protocols and styles (i still can't understand how is working) and im not sure if i can integrate with syslog-ng, but also i need a free tool :( . As im using Syslog-NG with MongoDB all i need is something that can present all this log and have options for sending e-mails if we have disaster or something and can query fast all this logs. LogAnalyzers is ok but i want to see different styles. Thanks in advanced ! Ivan

3 13

Transform file path
by Alexey Vlasov 25 May '16

25 May '16

Hi, I intend to manage the distribution of the Apache log-files for each virtualhost using syslog-ng. I write the following in Apache vhost configue: <VirtualHost *> SetEnv V3WUSER w_test-l26-apache-_b8649b LogFormat "%{V3WUSER}e %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" xcombined CustomLog /var/log/apache_aux2_worker2/access_pipe.log xcombined </IfModule> as a result I get this kind of format of the log file: w_test-l26-apache-_b8649b 10.0.2.24 - - [24/May/2016:12:41:33 +0300] "GET / HTTP/1.1" 200 - "http://example.com/" "ELinks (0.11.7; Linux 3.14.46-1gb-csm x86_64; 158x45)" where the first field is the unique identifier of the virtualhost. Next step I write the following in syslog-ng: source src_apache_piped_logs { pipe("/var/log/apache_aux2_worker2/access_pipe.log" keep_timestamp(no)); }; destination dst_apache_piped_logs { file("/var/log/virtwww/${V3WUSER}/access.log" template("$MSGONLY\n") template-escape(no)); }; log { source(src_apache_piped_logs); destination(dst_apache_piped_logs); }; Сonsequently I would like to have each log file of the virtualhost placed in its own catalog and the path should contain the first field of the message (${V3WUSER} = w_test-l26-apache-_b8649b) Is it actually possible? Thanks in advance for the help.

4 3

WANTED: feature owner of syslog-ng command line tool :)
by Noémi Ványi 24 May '16

24 May '16

Hello! A new feature is going to be developed, as a part of GSOC, so syslog-ng can be started as a command line tool. What could the CLI do for you? - pipelines of complex filtering, rewriting could be tested more easily, thus configuration could become simpler than before - existing logs could be transformed The format of the new command line options is not decided yet. So your help and insight is needed to design the interface and the usage of the tool that you and other people around the world would use. :) If you are intereted in contributing to the CLI, thus contributing to the syslog-ng community, please send an email to this thread. :) I am available at gitter for further info. :) Regards kvch

4 5

Syslog-NG with MongoDB
by Ivan Adji - Krstev 24 May '16

24 May '16

Hi all, What is the best practice for storing all those logs in one central environment. I have one Linux Box running Syslog-NG with LogAnalyzer and MongoDB ( for now ), and is the best way to configure and use it with MongoDB or with MariaDB ( MySQL ) ? I have once install MySQL but it was getting very slow as the logs getting bigger and bigger ( for one week ). Now i have done with MongoDB ( still testing ) but i have problem as LogAnalyzer does not show me the real pictures, i have no Date info, no Facility, no serverity, Hosts, syslogtag, i just have ProcessID. Any hints on this ? I have the following configuration on the syslog-ng.cfg: destination d_mongodb { mongodb( servers("localhost:27017") database("logs") # uri('mongodb://localhost/syslog-ng') collection("syslog") value-pairs( scope("selected-macros" "nv-pairs" "sdata") ) ); }; Kind regards Ivan

5 10

[GSoC]Finding Feature owner for "Web-based syslog-ng configuration editor"
by Raymund Ács 23 May '16

23 May '16

Hi, I am PoOwAa (http://github.com/poowaa) I could not write here before because I had some issues with subscribing. I already subscribed and unsubscribed few months ago, and I was not able to subscribe again. Finally I got my confirmation mail, so I can join to the community! :) In the summer I will work on the Web-based syslog-ng configuration editor project in GSoC 2016. Here is my proposal link: https://github.com/balabit/syslog-ng/wiki/GSoC-2016-Proposal:-Web-based-sys… This project will provide a Web Application which can create and save syslog-ng configuration files by drag&drop technique. I would like to follow the trends of 2016 in design, and give a detailed documentation. If you are interested in being the feature owner of this project, please reply in this thread. Advices from you will help us greatly and make the final project more user friendly. Best Regards, Raymund Acs University of Debrecen

1 0

[Release Automation Project] Finding feature owners
by Ankush Sharma 23 May '16

23 May '16

Hi, I am taking this as an opportunity to introduce myself to everyone in the community. I got a bit late as I was busy in relocating back and finishing college paper work. Sorry for it ;-) I am* Ankush Sharma (black-perl <https://github.com/black-perl>)* from India. I contributed to Mailman <http://list.org> last year. So, I will be working on the Automation Release <https://github.com/balabit/syslog-ng/wiki/%5BGSoC'16%5D-Automated-release-generation-for-Syslog-ng> project for syslog-ng this time. The project will mostly involve Python usage and Flask/Django later on. The potential areas where any feature owner can look into are: 1) *Web interface: *An experimental web interface to track things during the release automation cycle would be handy. I am considering to use Flask/Django (web frameworks of Python) for it. You can suggest integrations to it or information you would like to see on the interface. 2) *Overall architecture: *We can discuss the flow from user perspective as well as the architectural view of the tool and you can suggest changes to it ;-) Here is the link to the intro post : http://black-perl.in/black-perl-gsoc-with-syslog-ng/ <http://black-perl.in/black-perl-gsoc-with-syslog-ng/>. You can find a brief introduction to the project here and links to the detailed proposal. I would love to talk to other guys who will be working throughout the summers as part of the GSoC program: @litterbear, @kvch, @PoOwAa, @Vithulan. It would be great if we can have chat sometime and get to know about each others' work ! Thanks ! Ankush Sharma ECE IV IIT-BHU Varanasi-221005 http://black-perl.in Linkedin <https://www.linkedin.com/in/ankushsharma003>

2 1

[GSOC]Finding Feature owner for WebSocket for syslog ng
by Yilin Li 23 May '16

23 May '16

Hi I'm Yilin(https://github.com/litterbear). I'm working on the "WebSocket for syslog ng" project in GSOC 2016. Here is the link to my proposal: https://github.com/balabit/syslog-ng/wiki/GSoC-2016-Proposal:-WebSocket-for… This project will provide - a websocket plugin for syslog-ng source and destination. - user-friendly and detailed documents and examples. The draft design is shared here : https://docs.google.com/document/d/1FVc4EySiTu9RHdlsPd-NGQ9JzvV0WInQrM2VBA4… Any suggestions will be appreciated :) If you are interested in being the feature owner of the project, please reply in this thread. Advices on user's perspective from you will help us greatly and make the final deliverables more user friendly. Thanks in advance. -- Best wishes, Yilin Li Institute of Software Chinese Academy of Sciences

2 2

Re: [syslog-ng] Finding feature owner for Kafka source project
by Laszlo Budai 22 May '16

22 May '16

Sent from my HTC ----- Reply message ----- From: "Vincent Bernat" <bernat(a)luffy.cx> To: "Laszlo Budai" <laszlo.budai(a)outlook.com> Cc: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu>, "Vithulan MV" <vithulanmv.12(a)cse.mrt.ac.lk> Subject: [syslog-ng] Finding feature owner for Kafka source project Date: Sat, May 21, 2016 23:58 ❦ 21 mai 2016 11:58 -0700, Laszlo Budai <laszlo.budai(a)outlook.com> : > As you mentioned we have Kafka destination driver written in C in > syslog-ng incubator, so users can use it. It should not be a problem > if a feature is available only in the syslog-ng incubator. Well, it is because most users will never use it. I don't know why... maybe we should make it clear what is incubator.For me it contains two kind of features:1) experimental features that are not really tested, and maybe the code would also neee some polishment before merge into syslog-ng master2) well implemented features that are using some unstable dependencies> As the Kafka C client is stable enough, I don't see any problems in > integrating it to the syslog-ng master (just someone should create a > pull request). Good to know. That's not what you told me in August. I will fix the performance limitations and do a PR for master.I said that my plan is to keep the C kafka dst in incubator. Last year ;)But Kafka C lib seems to be a stable lib during the last 8-9 months which makes it possible to move forward from incubator. L.-- Make it right before you make it faster. - The Elements of Programming Style (Kernighan & Plauger)

1 0

Re: [syslog-ng] Finding feature owner for Kafka source project
by Laszlo Budai 21 May '16

21 May '16

Hi Vincent, the currently available Java features are * Kafka * Elasticsearch * HDFS (we have HTTP but we also have curl destination in C). As you mentioned we have Kafka destination driver written in C in syslog-ng incubator, so users can use it. It should not be a problem if a feature is available only in the syslog-ng incubator. As the Kafka C client is stable enough, I don't see any problems in integrating it to the syslog-ng master (just someone should create a pull request). In case of Elasticsearch we are happy that we are able to support even the latest Elasticsearch 5 but we don't have the resource to re-implement it in C after every new Elasticsearch release. If someone wants to implement a REST Elasticsearch client in C and based on that also implements an ES dest. driver in C then, we will happily integrate it into syslog-ng. HDFS: users who uses HDFS will have Java on their machines, so it should not be a problem. I think a feature implemented in Java is much-much better than an unimplemented feature... regards, Laszlo Budai Sent from my HTC ----- Reply message ----- From: "Vincent Bernat" <bernat(a)luffy.cx> To: "Vithulan MV" <vithulanmv.12(a)cse.mrt.ac.lk> Cc: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] Finding feature owner for Kafka source project Date: Sat, May 21, 2016 19:38 ❦ 21 mai 2016 20:33 +0530, Vithulan MV <vithulanmv.12(a)cse.mrt.ac.lk> : > A simple abstract idea of this project is, > The syslog-ng application can read messages from the sources. It > processes them with filters, rewrite rules, parsers and finally sends > messages to their destinations. The syslog-ng application already has > a Kafka destination that is implemented in Java. The Kafka source will > allow syslog-ng to read messages from Kafka, for an example this can > be used for example as a queue between several syslog-ng instances. It's a bit annoying that so many interesting features have to go through Java. One of the major point of syslog-ng is its light footprint. librdkafka has support for both source and destination and is officially supported by Confluent. Its development matches Kafka one and it supports every feature in Kafka 0.9, including SSL and the new consumer. It's a bit late to complain, notably if you chose this project because you were comfortable with Java (and your mentor is obviously interested in doing that in Java too). However, this has already been pointed out in March without much response from Balabit. It's not just a matter of proposing code since such code is likely to get stuck in the incubator forever. -- Make sure special cases are truly special. - The Elements of Programming Style (Kernighan & Plauger) ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

2 1

Finding feature owner for Kafka source project
by Vithulan MV 21 May '16

21 May '16

Hi all, As you may have all known already, We are going to develop a new feature, Kafka source for syslog-ng product during this GSoC 2016 under the guidance of Viktor Juhasz. A simple abstract idea of this project is, The syslog-ng application can read messages from the sources. It processes them with filters, rewrite rules, parsers and finally sends messages to their destinations. The syslog-ng application already has a Kafka destination that is implemented in Java. The Kafka source will allow syslog-ng to read messages from Kafka, for an example this can be used for example as a queue between several syslog-ng instances. Kafka source project will allow you to, -Use Kafka consumer which uses the high level Kafka group consuming API. -Continue reading from the last message from which syslog-ng reloaded/restarted. -Avoid message loosing or message duplication as much as possible whenever there is multiple syslog-ng reading the same kafka input. If you are interested to become a feature owner for Kafka source project and give us guidance on user's perspective, it will be very easy for us to build a effective component. Appreciate if you can contact us by sending mail to us or to this thread. Thanks, Best regards, Vithulan. -- Vithulan MV. Undergraduate, Department of Computer Science & Engineering, University of Moratuwa, Sri lanka.

3 2

Re: [syslog-ng] message loss using multi-line-mode
by Laszlo Budai 19 May '16

19 May '16

Hi, opening a new issue is not necessary. I hope that a GSoC student will solve this issue. BTW GSoC: we are still looking for feature owners to our GSoC[1] projects[2]. As an experienced syslog-ng user you could join to us. What do you think? [1] https://syslog-ng.org/gsoc-2016/ [2] https://lists.balabit.hu/pipermail/syslog-ng/2016-April/022862.html regards, Laszlo Budai Sent from my HTC ----- Reply message ----- From: "Patrick Hemmer" <syslogng(a)stormcloud9.net> To: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] message loss using multi-line-mode Date: Tue, May 17, 2016 00:17 Anybody? Should I open a new issue? -Patrick On 2016/3/28 12:01, Patrick Hemmer wrote: > I raised issue on an existing bug on the github tracker, but it seems > to have gone unnoticed, so I'm repeating it here to try and get some > attention on the issue. > > Using multi-line-mode without the multi-line-suffix option WILL result > in message loss. > When syslog-ng is running in multi-line-mode, it buffers multi-line > messages until it sees the start of a new message. When it sees the > start of a new message, it flushes the buffered message, and puts the > first line of the new message in the buffer. However if syslog-ng > shuts down, or receives a SIGHUP (reload), any lines currently > buffered are discarded. Given that syslog-ng can't stay running > forever, and it will get shut down or SIGHUPd eventually, using this > feature will result in messages getting lost. > > The message on the github issue where I brought this up is: > https://github.com/balabit/syslog-ng/issues/140#issuecomment-197673887 > > -Patrick > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

2 1

Re: [syslog-ng] message loss using multi-line-mode
by Laszlo Budai 19 May '16

19 May '16

Hi, it is an ideal topic for GSoC students during community bonding period (or those who want to participate in GSoC in the future). Any volunteer? L. Sent from my HTC ----- Reply message ----- From: syslogng(a)stormcloud9.net To: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] message loss using multi-line-mode Date: Wed, May 18, 2016 15:58 That was my original intent (to write a patch), but after looking at the code, I didn't think it would be a good idea (I used to code against syslog-ng back in the around the 3.1 days, but it appears the code has gotten significantly more complex). Given that there is message loss involved here, I just thought it might raise the priority of the issue. -Patrick On 2016/5/17 08:47, Scheidler, Balázs wrote: > hi, > > I don't think that would help, sorry. Everyone is working on other > stuff as it seems. Would you try to cook up a patch? I know it might > intimidating at first, but you know it's all just code and computers > :) not more complicated than a shell script. * > > [*] ok, a bit more complicated, but everyone has to start somewhere :) > > > -- > Bazsi > > On Tue, May 17, 2016 at 12:16 AM, Patrick Hemmer > <syslogng(a)stormcloud9.net <mailto:syslogng@stormcloud9.net>> wrote: > > Anybody? Should I open a new issue? > > -Patrick > > > On 2016/3/28 12:01, Patrick Hemmer wrote: >> I raised issue on an existing bug on the github tracker, but it >> seems to have gone unnoticed, so I'm repeating it here to try and >> get some attention on the issue. >> >> Using multi-line-mode without the multi-line-suffix option WILL >> result in message loss. >> When syslog-ng is running in multi-line-mode, it buffers >> multi-line messages until it sees the start of a new message. >> When it sees the start of a new message, it flushes the buffered >> message, and puts the first line of the new message in the >> buffer. However if syslog-ng shuts down, or receives a SIGHUP >> (reload), any lines currently buffered are discarded. Given that >> syslog-ng can't stay running forever, and it will get shut down >> or SIGHUPd eventually, using this feature will result in messages >> getting lost. >> >> The message on the github issue where I brought this up is: >> https://github.com/balabit/syslog-ng/issues/140#issuecomment-197673887 >> >> -Patrick >> >> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

message loss using multi-line-mode
by Patrick Hemmer 18 May '16

18 May '16

I raised issue on an existing bug on the github tracker, but it seems to have gone unnoticed, so I'm repeating it here to try and get some attention on the issue. Using multi-line-mode without the multi-line-suffix option WILL result in message loss. When syslog-ng is running in multi-line-mode, it buffers multi-line messages until it sees the start of a new message. When it sees the start of a new message, it flushes the buffered message, and puts the first line of the new message in the buffer. However if syslog-ng shuts down, or receives a SIGHUP (reload), any lines currently buffered are discarded. Given that syslog-ng can't stay running forever, and it will get shut down or SIGHUPd eventually, using this feature will result in messages getting lost. The message on the github issue where I brought this up is: https://github.com/balabit/syslog-ng/issues/140#issuecomment-197673887 -Patrick

4 4

Re: [syslog-ng] Syslog-NG RFC
by Ivan Adji - Krstev 17 May '16

17 May '16

On 05/17/2016 01:22 PM, Ivan Adji - Krstev wrote: > Hi Robert, > Everything is placed as it is. I have put the server and the client in > the networ() part the *flags* and still nothing. No template in use. > > Here is the configuration of the server: > > options { > flush_lines (0); > time_reopen (10); > log_fifo_size (1000); > chain_hostnames (off); > use_dns (no); > use_fqdn (no); > create_dirs (no); > keep_hostname (yes); > }; > > source s_sys { > system(); > internal(); > network(ip(0.0.0.0) port(6514) > flags(syslog-protocol) > transport("tls") > tls(key_file("/etc/syslog-ng/cert.d/serverkey.pem") > cert_file("/etc/syslog-ng/cert.d/servercert.pem") > ca_dir("/etc/syslog-ng/ca.d") > ) ); > > }; > > destination d_mongodb { > mongodb( > servers("localhost:27017") > database("syslog") > username("test") > password("test123") > collection("messages") > value-pairs( > scope("selected-macros" "nv-pairs" "sdata") > ) > ); > }; > > > And here it is on the client site: > > options { > flush_lines (0); > time_reopen (10); > log_fifo_size (1000); > chain_hostnames (off); > use_dns (no); > use_fqdn (no); > create_dirs (no); > keep_hostname (yes); > }; > > source s_sys { > system(); > internal(); > # udp(ip(0.0.0.0) port(514)); > }; > > > destination tls_destination { > network("x.x.x.x" port(6514) > flags(syslog-protocol) > transport("tls") > tls( ca_dir("/etc/syslog-ng/ca.d") > key_file("/etc/syslog-ng/cert.d/clientkey.pem") > cert_file("/etc/syslog-ng/cert.d/clientcert.pem") ) > ); > }; > > > And i have use the same configuration with MySQL and works perfect now > i have problems with MongoDB. > > Kind regards > Ivan > > > On 05/17/2016 01:10 PM, Fekete, Róbert wrote: >> Hi, >> >> The protocols used in the syslog-ng clients and the syslog-ng server >> should match. >> You posted a source that uses the network() driver - I take this is >> from your server. >> The destination on your client should also use the network() driver, >> and that's where you need the flags(syslog-protocol). >> >> Also check your client config to see if it uses a custom template >> that messes with the message format. >> >> Robert >> >> >> >> On Tue, May 17, 2016 at 10:49 AM, Ivan Adji - Krstev >> <akivanradix(a)gmail.com <mailto:akivanradix@gmail.com>> wrote: >> >> Any way i put it where i can and on a client but still nothing. >> >> Any other hints ? >> >> Ivan >> >> On 05/17/2016 10:31 AM, Fabien Wernli wrote: >>> Hi, >>> >>> On Tue, May 17, 2016 at 10:11:27AM +0200, Ivan Adji - Krstev wrote: >>>> some RFC model 5424. Is there an option to configure the syslog-ng to >>>> send this messages in that RFC format ? >>> Add `flags(syslog-protocol)` to the network destination >>> >>> ______________________________________________________________________________ >>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>> >> >> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> >> >> >> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >

1 0

Syslog-NG RFC
by Ivan Adji - Krstev 17 May '16

17 May '16

Hi all, I have configure the Syslog-NG with MongoDB and LogAnalyzer, and as output i have the messages but i have no Date, Facility Severity Syslogtag and other information from it. If i go on a message itself i have some additional information. I have found out that i should use some RFC model 5424. Is there an option to configure the syslog-ng to send this messages in that RFC format ? Kind regards Ivan

3 4

Building syslog-ng 3.7.3 in Ubuntu Xenial
by Vithulan MV 15 May '16

15 May '16

Hi all, I got the tarball from, https://github.com/balabit/syslog-ng/releases (version 3.7.3). When I tried to build it, - configure passed. - make passed - make check failed with the following message, /usr/bin/ld: modules/systemd-journal/tests/modules_systemd_journal_tests_test_systemd_journal-test_systemd_journal.o: undefined reference to symbol 'iv_task_registered@@IVYKIS_0.29' //usr/local/lib/libivykis.so.0: error adding symbols: DSO missing from command line collect2: error: ld returned 1 exit status Makefile:6297: recipe for target 'modules/systemd-journal/tests/test_systemd_journal' failed make[3]: *** [modules/systemd-journal/tests/test_systemd_journal] Error 1 Makefile:11738: recipe for target 'check-am' failed make[2]: *** [check-am] Error 2 Makefile:11313: recipe for target 'check-recursive' failed make[1]: *** [check-recursive] Error 1 Makefile:11741: recipe for target 'check' failed make: *** [check] Error 2 Also, getting this warning when compiling, Makefile:12423: warning: overriding recipe for target 'eclipse' Makefile:12398: warning: ignoring old recipe for target 'eclipse' What could be the cause for this problem? Thanks. Best regards, Vithulan. -- Vithulan MV. Undergraduate, Department of Computer Science & Engineering, University of Moratuwa, Sri lanka.

2 3

Syslog-ng meetup in London
by Balazs Scheidler 14 May '16

14 May '16

Hi, We are experimenting with a new syslog-ng related event format in London, 26th of May. I am travelling there and will be speaking about syslog-ng, past, present and future. Peter Czanik will also be there. The general approach is more about questions and discussions instead of presentations. Anyone in the area is welcome, beer is on @Balabit. :) RSVP at http://www.meetup.com/syslog-ng-london/events/231038819 Cheers, Bazsi

1 0

Syslog-ng with MongoDB authentication
by Ivan Adji - Krstev 13 May '16

13 May '16

Hi all, I have configure MongoDB with Username and Password, but can't find how to configure destination on Syslog-ng using MongoDB with Username and Password. For now i have just this: destination d_mongodb { mongodb( servers("localhost:27017") database("syslog") collection("messages") value-pairs( scope("selected-macros" "nv-pairs" "sdata") ) ); }; But i should add some username and passwords somewhere and don't know how. Any hints on this ? Kind regards Ivan

2 1

Re: [syslog-ng] Installing Syslog-NG 3.7 on CentOS 7
by jrhendri 12 May '16

12 May '16

This has to be something very basic.Have you tried checking if another syslog server is running? ps -aef |grep syslog Assuming this shows nothing, try a very simple syslog-ng config file and a manual start on the command line. Make sure you check all the things in your configuration that your copy should open beforehand. This should narrow down the problem I hope :-) Jim Sent from my Verizon, Samsung Galaxy smartphone -------- Original message --------From: Ivan Adji - Krstev <akivanradix(a)gmail.com> Date: 5/12/16 5:26 AM (GMT-05:00) To: syslog-ng(a)lists.balabit.hu Subject: Re: [syslog-ng] Installing Syslog-NG 3.7 on CentOS 7 So i have install EPEL and i have install syslog-ng and mongodb and when i start the syslog-ng service with syslog-ng -Fevd command i have the following error AGAIN :). Im not sure what is it and how to prevent it and what to do. But i really need this to work :(. [2016-05-12T05:21:10.739940] Error binding socket; addr='AF_INET(0.0.0.0:6514)', error='Address already in use (98)' [2016-05-12T05:21:10.739973] Error initializing message pipeline; [root@syslogserver loganalyzer]# netstat -tupl Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 localhost:27017 0.0.0.0:* LISTEN 1352/mongod tcp 0 0 0.0.0.0:syslog-tls 0.0.0.0:* LISTEN 11377/syslog-ng tcp 0 0 0.0.0.0:ssh 0.0.0.0:* LISTEN 8562/sshd tcp 0 0 localhost:smtp 0.0.0.0:* LISTEN 1778/master tcp6 0 0 [::]:http [::]:* LISTEN 11264/httpd tcp6 0 0 [::]:ssh [::]:* LISTEN 8562/sshd tcp6 0 0 localhost:smtp [::]:* LISTEN 1778/master udp 0 0 0.0.0.0:bootpc 0.0.0.0:* 638/dhclient udp 0 0 0.0.0.0:60094 0.0.0.0:* 638/dhclient udp6 0 0 [::]:3126 [::]:* 638/dhclient [root@syslogserver loganalyzer]# lsof | grep LISTEN mongod 1352 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 1393 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 2028 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 2033 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 2034 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 2138 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 2139 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 2141 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 2148 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 2404 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 2446 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 2447 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 2448 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 2449 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 2450 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 2451 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) mongod 1352 11380 mongod 6u IPv4 17057 0t0 TCP localhost:27017 (LISTEN) master 1778 root 13u IPv4 15893 0t0 TCP localhost:smtp (LISTEN) master 1778 root 14u IPv6 15894 0t0 TCP localhost:smtp (LISTEN) sshd 8562 root 3u IPv4 23963 0t0 TCP *:ssh (LISTEN) sshd 8562 root 4u IPv6 23965 0t0 TCP *:ssh (LISTEN) httpd 11264 root 4u IPv6 32697 0t0 TCP *:http (LISTEN) httpd 11265 apache 4u IPv6 32697 0t0 TCP *:http (LISTEN) httpd 11267 apache 4u IPv6 32697 0t0 TCP *:http (LISTEN) httpd 11268 apache 4u IPv6 32697 0t0 TCP *:http (LISTEN) httpd 11269 apache 4u IPv6 32697 0t0 TCP *:http (LISTEN) httpd 11270 apache 4u IPv6 32697 0t0 TCP *:http (LISTEN) httpd 11275 apache 4u IPv6 32697 0t0 TCP *:http (LISTEN) httpd 11276 apache 4u IPv6 32697 0t0 TCP *:http (LISTEN) httpd 11277 apache 4u IPv6 32697 0t0 TCP *:http (LISTEN) httpd 11278 apache 4u IPv6 32697 0t0 TCP *:http (LISTEN) syslog-ng 11377 root 14u IPv4 34906 0t0 TCP *:syslog-tls (LISTEN) syslog-ng 11377 11378 root 14u IPv4 34906 0t0 TCP *:syslog-tls (LISTEN) syslog-ng 11377 11541 root 14u IPv4 34906 0t0 TCP *:syslog-tls (LISTEN) httpd 11384 apache 4u IPv6 32697 0t0 TCP *:http (LISTEN) and the source config is as follow: source s_sys { system(); unix-stream("/dev/log"); internal(); network( port(6514) # tcp(port(5140)); # file("/proc/kmsg" log_prefix("kernel: ")); transport("tls") tls( key_file("/etc/syslog-ng/cert.d/serverkey.pem") cert_file("/etc/syslog-ng/cert.d/servercert.pem") ca_dir("/etc/syslog-ng/ca.d")) ); }; destination d_mongodb { mongodb( # servers("localhost:27017") # database("syslog") # uri('mongodb://localhost/syslog-ng') collection("messages") value-pairs( scope("selected-macros" "nv-pairs" "sdata") ) ); }; Kind regards Ivan On 05/10/2016 01:35 PM, Czanik, Péter wrote: Do you also have EPEL? The RHEL7/CentOS7 repo is built against EPEL, as some of the dependencies are missing from the base distribution: https://fedoraproject.org/wiki/EPEL Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik On Tue, May 10, 2016 at 1:29 PM, Ivan Adji - Krstev <akivanradix(a)gmail.com> wrote: Hi i note this error of mine but i try the other one: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng37/repo/epel-7/czan… And i have the similar errors when ever i try to install on new CentOS The procedure im doing is: Fresh installation of CentOS yum update yum install httpd php vim wget then install mongodb ( add repo ) then install syslog-ng ( add repo ) I'm using: CentOS Linux release 7.2.1511 (Core) And im having the following repos: [root@syslogserver ~]# yum repolist Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.switch.ch * extras: mirror.switch.ch * updates: mirror.switch.ch repo id repo name status base/7/x86_64 CentOS-7 - Base 9,007 czanik-syslog-ng37/x86_64 Copr repo for syslog-ng37 owned by czanik 59 extras/7/x86_64 CentOS-7 - Extras 266 mongodb-org-3.2/7 MongoDB Repository 35 updates/7/x86_64 CentOS-7 - Updates 1,437 repolist: 10,804 [root@syslogserver ~]# yum install syslog-ng Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.switch.ch * extras: mirror.switch.ch * updates: mirror.switch.ch Resolving Dependencies --> Running transaction check ---> Package syslog-ng.x86_64 0:3.7.3-3.el7.centos will be installed --> Processing Dependency: ivykis >= 0.36.1 for package: syslog-ng-3.7.3-3.el7.centos.x86_64 --> Processing Dependency: libivykis.so.0(IVYKIS_0.29)(64bit) for package: syslog-ng-3.7.3-3.el7.centos.x86_64 --> Processing Dependency: libivykis.so.0(IVYKIS_0.30)(64bit) for package: syslog-ng-3.7.3-3.el7.centos.x86_64 --> Processing Dependency: libevtlog.so.0()(64bit) for package: syslog-ng-3.7.3-3.el7.centos.x86_64 --> Processing Dependency: libivykis.so.0()(64bit) for package: syslog-ng-3.7.3-3.el7.centos.x86_64 --> Processing Dependency: libnet.so.1()(64bit) for package: syslog-ng-3.7.3-3.el7.centos.x86_64 --> Running transaction check ---> Package libnet.x86_64 0:1.1.6-7.el7 will be installed ---> Package syslog-ng.x86_64 0:3.7.3-3.el7.centos will be installed --> Processing Dependency: ivykis >= 0.36.1 for package: syslog-ng-3.7.3-3.el7.centos.x86_64 --> Processing Dependency: libivykis.so.0(IVYKIS_0.29)(64bit) for package: syslog-ng-3.7.3-3.el7.centos.x86_64 --> Processing Dependency: libivykis.so.0(IVYKIS_0.30)(64bit) for package: syslog-ng-3.7.3-3.el7.centos.x86_64 --> Processing Dependency: libevtlog.so.0()(64bit) for package: syslog-ng-3.7.3-3.el7.centos.x86_64 --> Processing Dependency: libivykis.so.0()(64bit) for package: syslog-ng-3.7.3-3.el7.centos.x86_64 --> Finished Dependency Resolution Error: Package: syslog-ng-3.7.3-3.el7.centos.x86_64 (czanik-syslog-ng37) Requires: libivykis.so.0(IVYKIS_0.30)(64bit) Error: Package: syslog-ng-3.7.3-3.el7.centos.x86_64 (czanik-syslog-ng37) Requires: libivykis.so.0()(64bit) Error: Package: syslog-ng-3.7.3-3.el7.centos.x86_64 (czanik-syslog-ng37) Requires: ivykis >= 0.36.1 Error: Package: syslog-ng-3.7.3-3.el7.centos.x86_64 (czanik-syslog-ng37) Requires: libevtlog.so.0()(64bit) Error: Package: syslog-ng-3.7.3-3.el7.centos.x86_64 (czanik-syslog-ng37) Requires: libivykis.so.0(IVYKIS_0.29)(64bit) You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest Any idea ? On 05/09/2016 04:09 PM, Czanik, Péter wrote: Hi, You should add the repository using the file: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng37/repo/epel-7/czan… to yum and not just download individual packages. You can use then "yum install syslog-ng" which will also download all necessary dependencies. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik On Mon, May 9, 2016 at 3:07 PM, Ivan Adji - Krstev <akivanradix(a)gmail.com> wrote: I have the following errors when i try to install Syslog-NG 3.7 on CentOS 7 I have problem when i try to install Syslog-NG 3.7 on CentOS 7. The following errors i get: --> Finished Dependency Resolution Error: Package: syslog-ng-3.7.3-1.el6.x86_64 (czanik-syslog-ng37epel6) Requires: libevtlog.so.0()(64bit) Error: Package: syslog-ng-3.7.3-1.el6.x86_64 (czanik-syslog-ng37epel6) Requires: libpcre.so.0()(64bit) You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest Any hints on this ? Kind regards Ivan ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

4 11

Installing Syslog-NG 3.7 on CentOS 7
by Ivan Adji - Krstev 12 May '16

12 May '16

I have the following errors when i try to install Syslog-NG 3.7 on CentOS 7 I have problem when i try to install Syslog-NG 3.7 on CentOS 7. The following errors i get: --> Finished Dependency Resolution Error: Package: syslog-ng-3.7.3-1.el6.x86_64 (czanik-syslog-ng37epel6) Requires: libevtlog.so.0()(64bit) Error: Package: syslog-ng-3.7.3-1.el6.x86_64 (czanik-syslog-ng37epel6) Requires: libpcre.so.0()(64bit) You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest Any hints on this ? Kind regards Ivan

3 6

Syslog-ng 3.8
by Ivan Adji - Krstev 12 May '16

12 May '16

Hi all, I have open a issue about the error i got when i update the Syslog-NG to 3.8 for "syslog-ng Address already in use (98)". I'm still working on the issue, but i would like to know if this newest version 3.8 is stabled as im building new environment and im asking myself do i use this 3.8 version or 3.7 ? Thanks Ivan

5 13

[GSOC]WebSocket for syslog-ng - Configuration file design to get feedbacks.
by Yilin Li 05 May '16

05 May '16

Hi, Very excited and thanks to the community that my proposal is accepted by syslog-ng in GSOC 2016. Here is my proposal: https://github.com/balabit/syslog-ng/wiki/GSoC-2016-Proposal:-WebSocket-for… I have some spare time recently. So I plan to begin my work early This document describes what the configuration file will look like for the project that I will accomplish. WebSocket for syslog-ng (Yilin Li) - Configure ... <https://docs.google.com/document/d/1FVc4EySiTu9RHdlsPd-NGQ9JzvV0WInQrM2VBA4…> . I post it to the community to get feed backs. I will reiterate the design on the community feedback. I hope it will be what our users exactly want at last. Any comments will be greatly apprecitated :) And there is a question for my mentors Laszlo Meszaros and Viktor Juhasz, will it be OK that I go faster than my schedule ? -- Best wishes, Yilin Li Institute of Software Chinese Academy of Sciences

1 0

Insider 2016-05: BMW; GSoC; Hadoop; grouping-by; SIEM; Java; Rust;
by Czanik, Péter 05 May '16

05 May '16

Dear syslog-ng users, This is the 49th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng in the BMW i3 ----------------------- The BMW i3 electric car uses many open source software under the hood. BMW respects the GPL and made the source code of software used available upon request. You can find syslog-ng among them: https://github.com/edent/BMW-OpenSource Google Summer of Code (GSoC) ---------------------------- The syslog-ng project will participate in GSoC this year again, this time with five students. We are looking for Feature Owners who could represent syslog-ng users in the process. Topics range from websockets through CLI to Java-based Kafka source. Read more about the topics and learn how to contact the team at https://lists.balabit.hu/pipermail/syslog-ng/2016-April/022862.html syslog-ng is Hortonworks Data Platform certified ------------------------------------------------ Both syslog-ng OSE and PE provide a Hadoop destination and are now Hortonworks Data Platform certified. You can read more about why use syslog-ng in a Hadoop environment at https://czanik.blogs.balabit.com/2016/02/filling-your-data-lake-with-log-me… or on the Hortonworks website at http://hortonworks.com/partner/balabit/ The grouping-by() parser ------------------------ Until recently, you could only correlate and aggregate information from multiple messages with the PatternDB parser. The new grouping-by() parser in the upcoming syslog-ng version 3.8 can correlate and aggregate information independently from PatternDB, using data from any of the available parsers (CSV, key=value, PatternDB, JSON): https://czanik.blogs.balabit.com/2016/04/the-grouping_by-parser-in-syslog-n… Optimize your SIEM ------------------ Join us on Wednesday, May 18, 2016, at 2:00 PM EST to hear about the benefits of front-ending your SIEM with syslog-ng. Benefits include: increased performance, higher quality data, and cost-effective scalability. Register for the live webinar at https://pages2.balabit.com/optimize-siem/ Troubleshooting Java support ---------------------------- Java based destination drivers were introduced to syslog-ng last year. The syslog-ng application uses libjvm.so to embed a Java Virtual Machine inside syslog-ng. Learn which Java to use and how to make sure, that it is found by syslog-ng: https://czanik.blogs.balabit.com/2016/03/troubleshooting-java-support-in-sy… Cybersecurity Excellence award for SSB -------------------------------------- Balabit has been named the winner of the 2016 Cybersecurity Excellence Award for Forensics for syslog-ng Store Box (SSB). Based on syslog-ng – one of the most widely adopted log management software packages – SSB is a highly reliable and high-performance log management appliance which collects, classifies, organizes, and securely stores log messages for any enterprise who operates a log management infrastructure. http://www.marketwired.com/press-release/balabits-syslog-ng-store-box-earns… Packaging Rust parsers ---------------------- Up until now, if you wanted to try the new Rust-based parsing in syslog-ng, you also had to build syslog-ng yourself from source code. Not anymore, if you use one of the RPM-based Linux distributions: https://czanik.blogs.balabit.com/2016/04/syslog-ng-3-8-preview-parsers-in-r… Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com . To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

race condition causes copytruncate log rotation problems
by Jim Segrave 04 May '16

04 May '16

On a set of very busy Usenet servers, we are seeing problems with using syslog-ng together with copytruncate in logrotation. I am not sure why, but syslog-ng opens log files without having O_APPEND set, and appends log mesages with separate syscalls - an lseek to SEEK_END, followed by a write. On a busy server- perhaps 300 logs/sec each about 120 bytes long, we sometimes find that logrotate, which is configured to do a copy-truncate, actually produces a sparse file which begins with blocks of zeroes up to the pre-rotate log file size, followed by logs written after the copytruncate completes. Looking at the Linux /proc/PID/fdinfo information, I see that the logfiles are not opened in append mode, although they are opened O_WR. I haven't begun looking through the source to see if syslog-ng ever tries to overwrite data it's already logged by seeking backwards in the file - I must confess I can't think of any reason to do so. The only other reason for not having O_APPEND set would be to avoid problems for people using NFS to collect logs centrally by having different servers all writing to one NFS file. As I see it, the sequence leading to this problem is: syslog-ng logrotate copytruncate function ------------- ------------------------------------------- ,,, begin copytruncate, reach EOF on the copy lseek(fd, SEEK_END) <------------------------------------------------------------- the end of the file before logrotate ftruncates() it ftruncate(fd, 0) write(fd, log_msg, log_msg_size) <----------------------------------------- the write takes place at the old EOF, so the kernel considers the file to have empty blocks from 0 to that point If there is a reason that some users need the logfile opened without O_APPEND mode, then making that choice a configuration option would be helpful. For those using O_APPEND, the lseek() calls are harmless and won't add any more resource usage than is already present, but file truncation will prevent a zero filled pad being prefixed to the file.

3 2