April 2016 - syslog-ng - lists.balabit.hu

GSoC 2016: we are looking for Featuer Owners.
by Laszlo Budai 26 Apr '16

26 Apr '16

Hi, Students have sent proposals to the following five projects (other project ideas, such as wildcard file source, were not so popular :-) ) : Automated release generation for syslog-nghttps://github.com/balabit/syslog-ng/wiki/GSoC2016-Idea-&-Project-… The selected proposal: https://drive.google.com/open?id=0B5rOtg14cnD0UDJFcnFBLWhXMVUby Ankush Sharma. Websocket for syslog-nghttps://github.com/balabit/syslog-ng/wiki/GSoC2016-Idea-&-Project-… The selected proposal: https://drive.google.com/open?id=0B5rOtg14cnD0MG1yX3NPNmFoMTAby Yilin Li. Kafka sourcehttps://github.com/balabit/syslog-ng/wiki/GSoC2016-Idea-&-Project-lis… The selected proposal: https://drive.google.com/open?id=0B5rOtg14cnD0VXQyYncxaTliaXM by Vithulan MV. syslog-ng as a command line toolhttps://github.com/balabit/syslog-ng/wiki/GSoC2016-Idea-&-Project-list#… The selected proposal : https://drive.google.com/open?id=0B5rOtg14cnD0ek9TSklPSnZ4YWM by Noemi Vanyi. Web based syslog-ng configuration editor https://github.com/balabit/syslog-ng/wiki/GSoC2016-Idea-&-Project-list#proj… The selected proposal: https://drive.google.com/open?id=0B5rOtg14cnD0cm9DaEtUVmZGT1U by Raymund Acs. As we have already mentioned https://syslog-ng.org/gsoc-2016/ , we will ask you to join us and help students in their work. Our goal is to build a syslog-ng-GSoC release after the program is finished. Join us! We need Feature Owners who are members of our community. If you are interested in one of the projects and want to be a Feature Owner, just contact to us at ( gitter https://gitter.im/balabit/syslog-ng , this mailing list ). Regards, Laszlo Budai Sent from my HTC

2 1

ELK herd to scale
by Scot Needy 25 Apr '16

25 Apr '16

Hi, Does anyone have links or care to share notes on making a syslog-ng -> ELK scale for enterprise ? I have some ideas and will gladly share my solution but also don’t want to spend days figuring these things out that have already been built. There are many ELK specific references but I also want to make sure the model fits the syslog workload. Thanks

8 18

Greeting from Yilin
by Yilin Li 23 Apr '16

23 Apr '16

Hi, I recieved the email that told me my proposal is accepted today. I'm very exited about that. I'm greatly grateful that the community and especially my mentors Laszlo Meszaros and Viktor Juhasz decide to give me the opportunity. I'll try my best to accomplish the project. I'll start the following tasks as my proposal described. * Getting familiar with syslog-ng's code base. Especially, I should understand the architecture of the destination and source modules. * Post my questions in the mailing list and IRC if I get some problems understanding the code. @my mentors Laszlo Meszaros and Viktor Juhasz Please guide me if there is anything that could help me to accomplish the project better :) -- Best wishes, Yilin Li Institute of Software Chinese Academy of Sciences

1 0

forward syslog-ng -> syslog-ng central.
by Scot Needy 22 Apr '16

22 Apr '16

What’s the best way to forward all syslog data to another syslog-ng ? simple think like this with some options? log { source(s_net); destination (d_sysmaster); };

2 1

Re: [syslog-ng] SSL Problem after update
by Ivan Adji - Krstev 22 Apr '16

22 Apr '16

Minimal conf .. i try changing the port and give me the same error on that port. And i sont know how to include them using full path. Also i have read this same problem on google and there is some syslog-ng.ctl file that seems to make this kind of problem. Also i have done without tls and agine the same error. Kind regards On 22 Apr 2016 16:27, "Scot Needy" <scotrn(a)gmail.com> wrote: As a test try to load a minimal conf with no includes and add them manually using a full path. I had to do that to troubleshoot java module. On Apr 21, 2016, at 6:39 PM, Ivan Adji - Krstev <akivanradix(a)gmail.com> wrote: in scl.conf file there is nothing except this line *@include 'scl/*/*.conf'* So i don't think so.? I get your point that tls on port 6514 may be used twice but can't find anything like that ... it is the same config file that i have used before removing the old version and putting the new one. From 3.5 to 3.8. On 04/21/2016 08:58 PM, Scot Needy wrote: All of my conf files are included in scl.conf is scl.conf and @include "/etc/syslog-ng/conf.d/*.conf loading the same module twice ? On Apr 21, 2016, at 1:58 PM, Ivan Adji - Krstev <akivanradix(a)gmail.com> wrote: Hi yes i start to put the file but i have some other problems... anyway here it's the config file ( syslog-ng.conf ) @version:3.8 @include "scl.conf" # syslog-ng configuration file. # # This should behave pretty much like the original syslog on RedHat. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # # Note: it also sources additional configuration files (*.conf) # located in /etc/syslog-ng/conf.d/ options { flush_lines (0); time_reopen (10); log_fifo_size (1000); # log_iw_size(100); log_fetch_limit(100); # chain_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source s_sys { system(); #unix-stream("/dev/log"); internal(); network( port(6514) # tcp(port(5140)); # file("/proc/kmsg" log_prefix("kernel: ")); transport("tls") tls( key_file("/etc/syslog-ng/cert.d/serverkey.pem") cert_file("/etc/syslog-ng/cert.d/servercert.pem") ca_dir("/etc/syslog-ng/ca.d")) ); }; destination d_mongodb { mongodb( servers("localhost:27017") # database("syslog") uri('mongodb://localhost/syslog-ng') collection("messages") value-pairs( scope("selected-macros" "nv-pairs" "sdata") ) ); }; destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" flush_lines(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_kern { file("/var/log/kern"); }; destination d_mlal { usertty("*"); }; filter f_kernel { facility(kern); }; filter f_default { level(info..emerg) and not (facility(mail) or facility(authpriv) or facility(cron)); }; filter f_auth { facility(authpriv); }; filter f_mail { facility(mail); }; filter f_emergency { level(emerg); }; filter f_news { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_boot { facility(local7); }; filter f_cron { facility(cron); }; #log { source(s_sys); filter(f_kernel); destination(d_cons); }; log { source(s_sys); filter(f_kernel); destination(d_mongodb); }; log { source(s_sys); filter(f_default); destination(d_mongodb); }; log { source(s_sys); filter(f_auth); destination(d_mongodb); }; log { source(s_sys); filter(f_mail); destination(d_mongodb); }; log { source(s_sys); filter(f_emergency); destination(d_mongodb); }; log { source(s_sys); filter(f_news); destination(d_mongodb); }; log { source(s_sys); filter(f_boot); destination(d_mongodb); }; log { source(s_sys); filter(f_cron); destination(d_mongodb); }; log { source(s_sys); destination(d_mongodb); }; # Source additional configuration files (.conf extension only) @include "/etc/syslog-ng/conf.d/*.conf" # vim:ft=syslog-ng:ai:si:ts=4:sw=4:et: On 04/21/2016 06:12 PM, Scot Needy wrote: Or the module does not die on stop and tries to start another. On Apr 21, 2016, at 11:20 AM, Balazs Scheidler <bazsi77(a)gmail.com> wrote: maybe you have two sources binding on the same port? On Thu, Apr 21, 2016 at 4:36 PM, Scot Needy < <scotrn(a)gmail.com> scotrn(a)gmail.com> wrote: > I don’t see your conf files but I suspect it’s trying to load the tls > module twice ? > > > On Apr 21, 2016, at 10:04 AM, Ivan Adji - Krstev < <akivanradix(a)gmail.com> > akivanradix(a)gmail.com> wrote: > > It is the same ... if i restart the syslog-ng after that ill get the same > error. If i stop the syslog-ng that port is not used anymore ... > Here is the output of the *syslog-ng -Fevd* > > [2016-04-21T16:02:41.478174] WARNING: Starting with syslog-ng 3.6, the > system() source performs JSON parsing of messages starting with the '@cim:' > prefix. No additional action is needed; > [2016-04-21T16:02:41.478635] Module loaded and initialized successfully; > module='sdjournal' > [2016-04-21T16:02:41.478776] Finishing include; content='source confgen > system', depth='1' > [2016-04-21T16:02:41.479601] Module loaded and initialized successfully; > module='afsocket' > [2016-04-21T16:02:41.481135] Module loaded and initialized successfully; > module='afmongodb' > [2016-04-21T16:02:41.481914] Module loaded and initialized successfully; > module='affile' > [2016-04-21T16:02:41.482779] Module loaded and initialized successfully; > module='afuser' > [2016-04-21T16:02:41.484211] Unable to detect fully qualified hostname for > localhost, use_fqdn() will use the short hostname; > [2016-04-21T16:02:41.484252] Compiling #unnamed sequence [log] at > [/etc/syslog-ng/syslog-ng.conf:128:7] > [2016-04-21T16:02:41.484264] Compiling s_sys reference [source] at > [/etc/syslog-ng/syslog-ng.conf:128:7] > [2016-04-21T16:02:41.484276] Compiling s_sys sequence [source] at > [/etc/syslog-ng/syslog-ng.conf:29:1] > [2016-04-21T16:02:41.484286] Compiling #unnamed junction [log] at > [/etc/syslog-ng/syslog-ng.conf:29:15] > [2016-04-21T16:02:41.484295] Compiling #unnamed sequence [log] at > [source confgen system:2:5] > [2016-04-21T16:02:41.484304] Compiling #unnamed sequence > [source] at [source confgen system:2:5] > [2016-04-21T16:02:41.484314] Compiling #unnamed junction [log] > at [source confgen system:2:13] > [2016-04-21T16:02:41.484323] Compiling #unnamed single [log] > at [source confgen system:3:1] > [2016-04-21T16:02:41.484337] Compiling #unnamed single [log] at > [/etc/syslog-ng/syslog-ng.conf:32:2] > [2016-04-21T16:02:41.484347] Compiling #unnamed single [log] at > [/etc/syslog-ng/syslog-ng.conf:33:2] > [2016-04-21T16:02:41.484363] Compiling f_auth reference [filter] at > [/etc/syslog-ng/syslog-ng.conf:128:22] > [2016-04-21T16:02:41.484374] Compiling f_auth sequence [filter] at > [/etc/syslog-ng/syslog-ng.conf:112:1] > [2016-04-21T16:02:41.484383] Compiling #unnamed single [log] at > [/etc/syslog-ng/syslog-ng.conf:112:22] > [2016-04-21T16:02:41.484393] Compiling d_mongodb reference [destination] > at [/etc/syslog-ng/syslog-ng.conf:128:38] > [2016-04-21T16:02:41.484403] Compiling d_mongodb sequence > [destination] at [/etc/syslog-ng/syslog-ng.conf:57:1] > [2016-04-21T16:02:41.484413] Compiling #unnamed junction [log] at > [/etc/syslog-ng/syslog-ng.conf:57:24] > [2016-04-21T16:02:41.484422] Compiling #unnamed single [log] at > [/etc/syslog-ng/syslog-ng.conf:58:5] > [2016-04-21T16:02:41.484434] Compiling #unnamed sequence [log] at > [/etc/syslog-ng/syslog-ng.conf:133:7] > [2016-04-21T16:02:41.484445] Compiling s_sys reference [source] at > [/etc/syslog-ng/syslog-ng.conf:133:7] > [2016-04-21T16:02:41.484456] Compiling f_emergency reference [filter] at > [/etc/syslog-ng/syslog-ng.conf:133:22] > [2016-04-21T16:02:41.484467] Compiling f_emergency sequence [filter] > at [/etc/syslog-ng/syslog-ng.conf:114:1] > [2016-04-21T16:02:41.484476] Compiling #unnamed single [log] at > [/etc/syslog-ng/syslog-ng.conf:114:22] > [2016-04-21T16:02:41.484486] Compiling d_mongodb reference [destination] > at [/etc/syslog-ng/syslog-ng.conf:133:43] > [2016-04-21T16:02:41.484497] Compiling #unnamed sequence [log] at > [/etc/syslog-ng/syslog-ng.conf:152:7] > [2016-04-21T16:02:41.484506] Compiling s_sys reference [source] at > [/etc/syslog-ng/syslog-ng.conf:152:7] > [2016-04-21T16:02:41.484517] Compiling d_mongodb reference [destination] > at [/etc/syslog-ng/syslog-ng.conf:152:22] > [2016-04-21T16:02:41.484873] Seeking the journal to the last cursor > position; > cursor='s=9383cb3eb8ee41eda3468d9841333aef;i=131c;b=0f07653f04794e14beeeebb18131926b;m=214f74fdf;t=530ff299a3a8e;x=d0f0c56745dc963a' > [2016-04-21T16:02:41.485562] Module loaded and initialized successfully; > module='syslogformat' > [2016-04-21T16:02:41.485834] Error binding socket; > addr='AF_INET(0.0.0.0:6514)', error='Address already in use (98)' > [2016-04-21T16:02:41.485874] Error initializing message pipeline; > > > On 04/21/2016 03:40 PM, Scot Needy wrote: > > kill -9 2411 > > On Apr 21, 2016, at 7:58 AM, Ivan Adji - Krstev < <akivanradix(a)gmail.com> > akivanradix(a)gmail.com> wrote: > > 2411 > > > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > > ______________________________________________________________________________ > Member info: <https://lists.balabit.hu/mailman/listinfo/syslog-ng> > https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > <http://www.balabit.com/support/documentation/?product=syslog-ng> > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: <http://www.balabit.com/wiki/syslog-ng-faq> > http://www.balabit.com/wiki/syslog-ng-faq > > > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > <http://www.balabit.com/support/documentation/?product=syslog-ng> > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > -- Bazsi ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

SSL Problem after update
by Ivan Adji - Krstev 22 Apr '16

22 Apr '16

Hi all, I have update the syslog-ng to 3.8 ( yum remove syslog-ng && yum install syslog-ng ) and change from MySQL to Mongo DB and now when i restart i have the following errors: Error binding socket; addr='AF_INET(0.0.0.0:6514)', error='Address already in use (98)' I'm using mutual self certification and the certificates are there in the directories "ca.d" "cert.d". Do i have to re-create all the certificates now or can i fix this somehow. P.S. This problem i have on the Syslog-NG Server, they all send messages to this server. So if i have to change the certificate here i will have to change on the clients too. :( Any hints on this ?

5 17

Syslog-NG with MongoDB
by Ivan Adji - Krstev 21 Apr '16

21 Apr '16

Hi all, I'm trying to change the DB from InooDB to MongoDB, and i have install the MongoDB but i can't configure/create the database and tables on it. Can some one provide with simple tutorial or the best way to create and configure the MongoDB database and tables so that can be used by syslog-ng ? Kind regards Ivan

4 4

Elastic search data loading ?
by Scot Needy 15 Apr '16

15 Apr '16

Hi, When setting up syslog-ng -> ELK the logstash portion should not be needed as syslog-ng writes directly to an ES node or remote “transport” My understanding is logstash would post parse a log for a given interval and send it to ES like lodrotated. Not a solution for a syslog-ng realtime model. So my ElK stack is built but I don’t seem to be able to make the connection or data format between syslog-ng 3.8 and ES2. I can see syslog-ng writing data to the local log file destination and syslog-ng and es2 logs indicate the TCP connection was successful on startup… But I see no data in Kibana nor do I see any activity in syslog-ng or es logs using either of the templates below. NOTE: Removing syslog-ng->es and running flat logs through logstash does seem to populate data but that is not the solution. destination d_es { elasticsearch2( index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("syslog-ng") # Description: The type of the index. For example, type("test") template("$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE})") #template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") port("9300") server("localhost") flush_limit("5000") client_mode("node") cluster("syslog-ng") custom_id("syslog") resource("/etc/elasticsearch/elasticsearch.yml") client_lib_dir("/usr/share/elasticsearch/lib") concurrent_requests("100") ); }; # sending logs to ES destination log { source(s_net); parser(pattern_db); destination(d_es); flags(flow-control); }; /etc/elasticsearch/elasticsearch.yml cluster.name: syslog node.name: node-1 path.home: /usr/share/elasticsearch

3 7

systemd syslog-ng does not start CentOS 7.2.1511
by Scot Needy 14 Apr '16

14 Apr '16

Hi, I need help setting up my control files for systemd I can’t figure out what I am doing wrong. If I make a SystemV start stop script it runs fine. From systemd I think there is something missing in the control files that lets syslog-ng report that is started successfully. When I add -F to SYSLOGNG_OPTS then systemctl start syslog-g.service never returns but syslog-ng does start. Compiled syslog-ng syslog-ng 3.8.0alpha0 no problems. On the command line syslog-ng will start fine. /opt/syslog-ng/sbin/syslog-ng -f /opt/syslog-ng/etc/syslog-ng.conf -p /var/run/syslogd.pid Using contrib/systemd/syslog-ng.service as an example I created /usr/lib/systemd/system/syslog-ng.service [Unit] Description=System Logger Daemon Documentation=man:syslog-ng(8) [Service] Type=notify ExecStart=/opt/syslog-ng/sbin/syslog-ng $SYSLOGNG_OPTS ExecReload=/bin/kill -HUP $MAINPID EnvironmentFile=-/opt/syslog-ng/etc/syslog-ng.env StandardOutput=journal StandardError=journal Restart=on-failure [Install] WantedBy=multi-user.target cat /opt/syslog-ng/etc/syslog-ng.env LD_LIBRARY_PATH=/usr/java/jdk1.8.0_77/jre/lib/amd64/server:$LD_LIBRARY_PATH SYSLOGNG_OPTS="-f /opt/syslog-ng/etc/syslog-ng.conf -p /var/run/syslogd.pid" systemctl daemon-reload systemctl restart systemd-journald systemctl start syslog-ng.service systemctl status syslog-ng.service ● syslog-ng.service - System Logger Daemon Loaded: loaded (/usr/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled) Active: inactive (dead) since Tue 2016-04-12 14:51:04 EDT; 29s ago Docs: man:syslog-ng(8) Process: 2494 ExecStart=/opt/syslog-ng/sbin/syslog-ng $SYSLOGNG_OPTS (code=exited, status=0/SUCCESS) Main PID: 2494 (code=exited, status=0/SUCCESS) Apr 12 14:41:08 loghost systemd[1]: Unit syslog-ng.service entered failed state. Apr 12 14:41:08 loghost systemd[1]: syslog-ng.service failed. Apr 12 14:41:08 loghost systemd[1]: syslog-ng.service holdoff time over, scheduling restart. Apr 12 14:41:08 loghost systemd[1]: Starting System Logger Daemon... I can see syslog-ng starting and stoping over and over from /var/log/messages

5 17

Configuring syslog-ng on a computer with multiple ethernet interfaces
by Dave Florek 11 Apr '16

11 Apr '16

Hi, How do I configure syslog-ng to use a specific ethernet interface on a computer with multiple interfaces enabled? Sincerely,

2 1