- syslog-ng - lists.balabit.hu

Switching the syslog-ng central server - client messages go missing
by Cottington-Bray, Ian 27 Apr '16

27 Apr '16

Hi I have a Solaris 10 server running version 3.0 of syslog-ng. I have built a new server running Solaris 11 with version 3.6 of syslog-ng installed. I have tested the new server by pointing another client at it and messages appear in the configured files as expected. I then shut down the Solaris 10 server - change the Solaris 11 server IP configuration to match the details for the Solaris 10 server - restart syslog-ng on the Solaris 11 server. Things seem to be working ok - except for the (Solaris and Linux) clients using syslog-ng (and referencing the central syslog-ng server by IP) their messages do not arrive at the expected files. Any suggestions ? Ian Ian Cottington-Bray | Senior Infrastructure Engineer - Linux/Unix | McLaren Technology Group Limited McLaren Technology Centre, Chertsey Road, Woking, Surrey GU21 4YH, UK T: +44 (0) 1483 261 900 D: +44 (0) 1483 262 357 E: ian.cottington-bray(a)mclaren.com W: www.mclaren.com<http://www.mclaren.com/> The contents of this e-mail are confidential and for the exclusive use of the intended recipient. If you are not the intended recipient you should not read, copy, retransmit or disclose its contents. If you have received this email in error please delete it from your system immediately and notify us either by email or telephone. The views expressed in this communication may not necessarily be the views held by McLaren Technology Group Limited. McLaren Technology Group Limited | McLaren Technology Centre | Chertsey Road | Woking | Surrey | GU21 4YH | UK | Company Number: 01967715

3 5

Re: [syslog-ng] GSoC 2016: we are looking for Featuer Owners.
by Laszlo Budai 26 Apr '16

26 Apr '16

Hi, in the past we did not have feature owners, so we don't have any experiences. This is actually a new role that I want to initiate to our GSoC projects (and later I want to share the experiences with Google). Feature owners are those community members who are highly interested in a specific feature so they can represents the users. Of course they can also check the code :) I think FOs will also keep an eye on the GSoC release timeline. >From the blogpost I posted few months ago: Feature owners:Feature owners represent the community during the development phase.Feature owners send feedback to the students.Feature owners write blogpost(s) about the feature.L. Sent from my HTC ----- Reply message ----- From: "Scheidler, Balázs" <balazs.scheidler(a)balabit.com> To: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] GSoC 2016: we are looking for Featuer Owners. Date: Tue, Apr 26, 2016 14:27 What kind of commitment do you need from these feature owners? I know it can be difficult to judge, but what are your past experiences? On Apr 26, 2016 1:18 PM, "Laszlo Budai" <laszlo.budai(a)outlook.com> wrote: > > Hi, > > > Students have sent proposals to the following five projects (other project > ideas, such as wildcard file source, were not so popular :-) ) : > > > - Automated release generation for syslog-ng > > > https://github.com/balabit/syslog-ng/wiki/GSoC2016-Idea-&-Project-list#proj… > > The selected proposal: > https://drive.google.com/open?id=0B5rOtg14cnD0UDJFcnFBLWhXMVUby Ankush > Sharma. > > > - Websocket for syslog-ng > https://github.com/balabit/syslog-ng/wiki/GSoC2016-Idea-&-Project-list#proj… > > > The selected proposal: > https://drive.google.com/open?id=0B5rOtg14cnD0MG1yX3NPNmFoMTAby Yilin Li. > > > - Kafka source > https://github.com/balabit/syslog-ng/wiki/GSoC2016-Idea-&-Project-list#proj… > > > The selected proposal: > https://drive.google.com/open?id=0B5rOtg14cnD0VXQyYncxaTliaXM by Vithulan > MV. > > > - syslog-ng as a command line tool > https://github.com/balabit/syslog-ng/wiki/GSoC2016-Idea-&-Project-list#proj… > > > The selected proposal : > https://drive.google.com/open?id=0B5rOtg14cnD0ek9TSklPSnZ4YWM by Noemi > Vanyi. > > > - Web based syslog-ng configuration editor > > > https://github.com/balabit/syslog-ng/wiki/GSoC2016-Idea-&-Project-list#proj… > > The selected proposal: > https://drive.google.com/open?id=0B5rOtg14cnD0cm9DaEtUVmZGT1U by Raymund > Acs. > > > > As we have already mentioned https://syslog-ng.org/gsoc-2016/ , we will > ask you to join us and help students in their work. > > Our goal is to build a syslog-ng-GSoC release after the program is > finished. > > > Join us! We need Feature Owners who are members of our community. > > If you are interested in one of the projects and want to be a Feature > Owner, just contact to us at ( gitter https://gitter.im/balabit/syslog-ng , > this mailing list ). > > > Regards, > Laszlo Budai > > Sent from my HTC > > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

GSoC 2016: we are looking for Featuer Owners.
by Laszlo Budai 26 Apr '16

26 Apr '16

Hi, Students have sent proposals to the following five projects (other project ideas, such as wildcard file source, were not so popular :-) ) : Automated release generation for syslog-nghttps://github.com/balabit/syslog-ng/wiki/GSoC2016-Idea-&-Project-… The selected proposal: https://drive.google.com/open?id=0B5rOtg14cnD0UDJFcnFBLWhXMVUby Ankush Sharma. Websocket for syslog-nghttps://github.com/balabit/syslog-ng/wiki/GSoC2016-Idea-&-Project-… The selected proposal: https://drive.google.com/open?id=0B5rOtg14cnD0MG1yX3NPNmFoMTAby Yilin Li. Kafka sourcehttps://github.com/balabit/syslog-ng/wiki/GSoC2016-Idea-&-Project-lis… The selected proposal: https://drive.google.com/open?id=0B5rOtg14cnD0VXQyYncxaTliaXM by Vithulan MV. syslog-ng as a command line toolhttps://github.com/balabit/syslog-ng/wiki/GSoC2016-Idea-&-Project-list#… The selected proposal : https://drive.google.com/open?id=0B5rOtg14cnD0ek9TSklPSnZ4YWM by Noemi Vanyi. Web based syslog-ng configuration editor https://github.com/balabit/syslog-ng/wiki/GSoC2016-Idea-&-Project-list#proj… The selected proposal: https://drive.google.com/open?id=0B5rOtg14cnD0cm9DaEtUVmZGT1U by Raymund Acs. As we have already mentioned https://syslog-ng.org/gsoc-2016/ , we will ask you to join us and help students in their work. Our goal is to build a syslog-ng-GSoC release after the program is finished. Join us! We need Feature Owners who are members of our community. If you are interested in one of the projects and want to be a Feature Owner, just contact to us at ( gitter https://gitter.im/balabit/syslog-ng , this mailing list ). Regards, Laszlo Budai Sent from my HTC

2 1

ELK herd to scale
by Scot Needy 25 Apr '16

25 Apr '16

Hi, Does anyone have links or care to share notes on making a syslog-ng -> ELK scale for enterprise ? I have some ideas and will gladly share my solution but also don’t want to spend days figuring these things out that have already been built. There are many ELK specific references but I also want to make sure the model fits the syslog workload. Thanks

8 18

Greeting from Yilin
by Yilin Li 23 Apr '16

23 Apr '16

Hi, I recieved the email that told me my proposal is accepted today. I'm very exited about that. I'm greatly grateful that the community and especially my mentors Laszlo Meszaros and Viktor Juhasz decide to give me the opportunity. I'll try my best to accomplish the project. I'll start the following tasks as my proposal described. * Getting familiar with syslog-ng's code base. Especially, I should understand the architecture of the destination and source modules. * Post my questions in the mailing list and IRC if I get some problems understanding the code. @my mentors Laszlo Meszaros and Viktor Juhasz Please guide me if there is anything that could help me to accomplish the project better :) -- Best wishes, Yilin Li Institute of Software Chinese Academy of Sciences

1 0

forward syslog-ng -> syslog-ng central.
by Scot Needy 22 Apr '16

22 Apr '16

What’s the best way to forward all syslog data to another syslog-ng ? simple think like this with some options? log { source(s_net); destination (d_sysmaster); };

2 1

Re: [syslog-ng] SSL Problem after update
by Ivan Adji - Krstev 22 Apr '16

22 Apr '16

Minimal conf .. i try changing the port and give me the same error on that port. And i sont know how to include them using full path. Also i have read this same problem on google and there is some syslog-ng.ctl file that seems to make this kind of problem. Also i have done without tls and agine the same error. Kind regards On 22 Apr 2016 16:27, "Scot Needy" <scotrn(a)gmail.com> wrote: As a test try to load a minimal conf with no includes and add them manually using a full path. I had to do that to troubleshoot java module. On Apr 21, 2016, at 6:39 PM, Ivan Adji - Krstev <akivanradix(a)gmail.com> wrote: in scl.conf file there is nothing except this line *@include 'scl/*/*.conf'* So i don't think so.? I get your point that tls on port 6514 may be used twice but can't find anything like that ... it is the same config file that i have used before removing the old version and putting the new one. From 3.5 to 3.8. On 04/21/2016 08:58 PM, Scot Needy wrote: All of my conf files are included in scl.conf is scl.conf and @include "/etc/syslog-ng/conf.d/*.conf loading the same module twice ? On Apr 21, 2016, at 1:58 PM, Ivan Adji - Krstev <akivanradix(a)gmail.com> wrote: Hi yes i start to put the file but i have some other problems... anyway here it's the config file ( syslog-ng.conf ) @version:3.8 @include "scl.conf" # syslog-ng configuration file. # # This should behave pretty much like the original syslog on RedHat. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # # Note: it also sources additional configuration files (*.conf) # located in /etc/syslog-ng/conf.d/ options { flush_lines (0); time_reopen (10); log_fifo_size (1000); # log_iw_size(100); log_fetch_limit(100); # chain_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source s_sys { system(); #unix-stream("/dev/log"); internal(); network( port(6514) # tcp(port(5140)); # file("/proc/kmsg" log_prefix("kernel: ")); transport("tls") tls( key_file("/etc/syslog-ng/cert.d/serverkey.pem") cert_file("/etc/syslog-ng/cert.d/servercert.pem") ca_dir("/etc/syslog-ng/ca.d")) ); }; destination d_mongodb { mongodb( servers("localhost:27017") # database("syslog") uri('mongodb://localhost/syslog-ng') collection("messages") value-pairs( scope("selected-macros" "nv-pairs" "sdata") ) ); }; destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" flush_lines(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_kern { file("/var/log/kern"); }; destination d_mlal { usertty("*"); }; filter f_kernel { facility(kern); }; filter f_default { level(info..emerg) and not (facility(mail) or facility(authpriv) or facility(cron)); }; filter f_auth { facility(authpriv); }; filter f_mail { facility(mail); }; filter f_emergency { level(emerg); }; filter f_news { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_boot { facility(local7); }; filter f_cron { facility(cron); }; #log { source(s_sys); filter(f_kernel); destination(d_cons); }; log { source(s_sys); filter(f_kernel); destination(d_mongodb); }; log { source(s_sys); filter(f_default); destination(d_mongodb); }; log { source(s_sys); filter(f_auth); destination(d_mongodb); }; log { source(s_sys); filter(f_mail); destination(d_mongodb); }; log { source(s_sys); filter(f_emergency); destination(d_mongodb); }; log { source(s_sys); filter(f_news); destination(d_mongodb); }; log { source(s_sys); filter(f_boot); destination(d_mongodb); }; log { source(s_sys); filter(f_cron); destination(d_mongodb); }; log { source(s_sys); destination(d_mongodb); }; # Source additional configuration files (.conf extension only) @include "/etc/syslog-ng/conf.d/*.conf" # vim:ft=syslog-ng:ai:si:ts=4:sw=4:et: On 04/21/2016 06:12 PM, Scot Needy wrote: Or the module does not die on stop and tries to start another. On Apr 21, 2016, at 11:20 AM, Balazs Scheidler <bazsi77(a)gmail.com> wrote: maybe you have two sources binding on the same port? On Thu, Apr 21, 2016 at 4:36 PM, Scot Needy < <scotrn(a)gmail.com> scotrn(a)gmail.com> wrote: > I don’t see your conf files but I suspect it’s trying to load the tls > module twice ? > > > On Apr 21, 2016, at 10:04 AM, Ivan Adji - Krstev < <akivanradix(a)gmail.com> > akivanradix(a)gmail.com> wrote: > > It is the same ... if i restart the syslog-ng after that ill get the same > error. If i stop the syslog-ng that port is not used anymore ... > Here is the output of the *syslog-ng -Fevd* > > [2016-04-21T16:02:41.478174] WARNING: Starting with syslog-ng 3.6, the > system() source performs JSON parsing of messages starting with the '@cim:' > prefix. No additional action is needed; > [2016-04-21T16:02:41.478635] Module loaded and initialized successfully; > module='sdjournal' > [2016-04-21T16:02:41.478776] Finishing include; content='source confgen > system', depth='1' > [2016-04-21T16:02:41.479601] Module loaded and initialized successfully; > module='afsocket' > [2016-04-21T16:02:41.481135] Module loaded and initialized successfully; > module='afmongodb' > [2016-04-21T16:02:41.481914] Module loaded and initialized successfully; > module='affile' > [2016-04-21T16:02:41.482779] Module loaded and initialized successfully; > module='afuser' > [2016-04-21T16:02:41.484211] Unable to detect fully qualified hostname for > localhost, use_fqdn() will use the short hostname; > [2016-04-21T16:02:41.484252] Compiling #unnamed sequence [log] at > [/etc/syslog-ng/syslog-ng.conf:128:7] > [2016-04-21T16:02:41.484264] Compiling s_sys reference [source] at > [/etc/syslog-ng/syslog-ng.conf:128:7] > [2016-04-21T16:02:41.484276] Compiling s_sys sequence [source] at > [/etc/syslog-ng/syslog-ng.conf:29:1] > [2016-04-21T16:02:41.484286] Compiling #unnamed junction [log] at > [/etc/syslog-ng/syslog-ng.conf:29:15] > [2016-04-21T16:02:41.484295] Compiling #unnamed sequence [log] at > [source confgen system:2:5] > [2016-04-21T16:02:41.484304] Compiling #unnamed sequence > [source] at [source confgen system:2:5] > [2016-04-21T16:02:41.484314] Compiling #unnamed junction [log] > at [source confgen system:2:13] > [2016-04-21T16:02:41.484323] Compiling #unnamed single [log] > at [source confgen system:3:1] > [2016-04-21T16:02:41.484337] Compiling #unnamed single [log] at > [/etc/syslog-ng/syslog-ng.conf:32:2] > [2016-04-21T16:02:41.484347] Compiling #unnamed single [log] at > [/etc/syslog-ng/syslog-ng.conf:33:2] > [2016-04-21T16:02:41.484363] Compiling f_auth reference [filter] at > [/etc/syslog-ng/syslog-ng.conf:128:22] > [2016-04-21T16:02:41.484374] Compiling f_auth sequence [filter] at > [/etc/syslog-ng/syslog-ng.conf:112:1] > [2016-04-21T16:02:41.484383] Compiling #unnamed single [log] at > [/etc/syslog-ng/syslog-ng.conf:112:22] > [2016-04-21T16:02:41.484393] Compiling d_mongodb reference [destination] > at [/etc/syslog-ng/syslog-ng.conf:128:38] > [2016-04-21T16:02:41.484403] Compiling d_mongodb sequence > [destination] at [/etc/syslog-ng/syslog-ng.conf:57:1] > [2016-04-21T16:02:41.484413] Compiling #unnamed junction [log] at > [/etc/syslog-ng/syslog-ng.conf:57:24] > [2016-04-21T16:02:41.484422] Compiling #unnamed single [log] at > [/etc/syslog-ng/syslog-ng.conf:58:5] > [2016-04-21T16:02:41.484434] Compiling #unnamed sequence [log] at > [/etc/syslog-ng/syslog-ng.conf:133:7] > [2016-04-21T16:02:41.484445] Compiling s_sys reference [source] at > [/etc/syslog-ng/syslog-ng.conf:133:7] > [2016-04-21T16:02:41.484456] Compiling f_emergency reference [filter] at > [/etc/syslog-ng/syslog-ng.conf:133:22] > [2016-04-21T16:02:41.484467] Compiling f_emergency sequence [filter] > at [/etc/syslog-ng/syslog-ng.conf:114:1] > [2016-04-21T16:02:41.484476] Compiling #unnamed single [log] at > [/etc/syslog-ng/syslog-ng.conf:114:22] > [2016-04-21T16:02:41.484486] Compiling d_mongodb reference [destination] > at [/etc/syslog-ng/syslog-ng.conf:133:43] > [2016-04-21T16:02:41.484497] Compiling #unnamed sequence [log] at > [/etc/syslog-ng/syslog-ng.conf:152:7] > [2016-04-21T16:02:41.484506] Compiling s_sys reference [source] at > [/etc/syslog-ng/syslog-ng.conf:152:7] > [2016-04-21T16:02:41.484517] Compiling d_mongodb reference [destination] > at [/etc/syslog-ng/syslog-ng.conf:152:22] > [2016-04-21T16:02:41.484873] Seeking the journal to the last cursor > position; > cursor='s=9383cb3eb8ee41eda3468d9841333aef;i=131c;b=0f07653f04794e14beeeebb18131926b;m=214f74fdf;t=530ff299a3a8e;x=d0f0c56745dc963a' > [2016-04-21T16:02:41.485562] Module loaded and initialized successfully; > module='syslogformat' > [2016-04-21T16:02:41.485834] Error binding socket; > addr='AF_INET(0.0.0.0:6514)', error='Address already in use (98)' > [2016-04-21T16:02:41.485874] Error initializing message pipeline; > > > On 04/21/2016 03:40 PM, Scot Needy wrote: > > kill -9 2411 > > On Apr 21, 2016, at 7:58 AM, Ivan Adji - Krstev < <akivanradix(a)gmail.com> > akivanradix(a)gmail.com> wrote: > > 2411 > > > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > > ______________________________________________________________________________ > Member info: <https://lists.balabit.hu/mailman/listinfo/syslog-ng> > https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > <http://www.balabit.com/support/documentation/?product=syslog-ng> > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: <http://www.balabit.com/wiki/syslog-ng-faq> > http://www.balabit.com/wiki/syslog-ng-faq > > > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > <http://www.balabit.com/support/documentation/?product=syslog-ng> > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > -- Bazsi ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

SSL Problem after update
by Ivan Adji - Krstev 22 Apr '16

22 Apr '16

Hi all, I have update the syslog-ng to 3.8 ( yum remove syslog-ng && yum install syslog-ng ) and change from MySQL to Mongo DB and now when i restart i have the following errors: Error binding socket; addr='AF_INET(0.0.0.0:6514)', error='Address already in use (98)' I'm using mutual self certification and the certificates are there in the directories "ca.d" "cert.d". Do i have to re-create all the certificates now or can i fix this somehow. P.S. This problem i have on the Syslog-NG Server, they all send messages to this server. So if i have to change the certificate here i will have to change on the clients too. :( Any hints on this ?

5 17

Syslog-NG with MongoDB
by Ivan Adji - Krstev 21 Apr '16

21 Apr '16

Hi all, I'm trying to change the DB from InooDB to MongoDB, and i have install the MongoDB but i can't configure/create the database and tables on it. Can some one provide with simple tutorial or the best way to create and configure the MongoDB database and tables so that can be used by syslog-ng ? Kind regards Ivan

4 4

Elastic search data loading ?
by Scot Needy 15 Apr '16

15 Apr '16

Hi, When setting up syslog-ng -> ELK the logstash portion should not be needed as syslog-ng writes directly to an ES node or remote “transport” My understanding is logstash would post parse a log for a given interval and send it to ES like lodrotated. Not a solution for a syslog-ng realtime model. So my ElK stack is built but I don’t seem to be able to make the connection or data format between syslog-ng 3.8 and ES2. I can see syslog-ng writing data to the local log file destination and syslog-ng and es2 logs indicate the TCP connection was successful on startup… But I see no data in Kibana nor do I see any activity in syslog-ng or es logs using either of the templates below. NOTE: Removing syslog-ng->es and running flat logs through logstash does seem to populate data but that is not the solution. destination d_es { elasticsearch2( index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("syslog-ng") # Description: The type of the index. For example, type("test") template("$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE})") #template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") port("9300") server("localhost") flush_limit("5000") client_mode("node") cluster("syslog-ng") custom_id("syslog") resource("/etc/elasticsearch/elasticsearch.yml") client_lib_dir("/usr/share/elasticsearch/lib") concurrent_requests("100") ); }; # sending logs to ES destination log { source(s_net); parser(pattern_db); destination(d_es); flags(flow-control); }; /etc/elasticsearch/elasticsearch.yml cluster.name: syslog node.name: node-1 path.home: /usr/share/elasticsearch

3 7