February 2018 - syslog-ng - lists.balabit.hu

Insider 2018-02: New website; One Identity; Docker; Error messages; Elastic 6; DevConf;
by Czanik, Péter 08 Feb '18

08 Feb '18

Dear syslog-ng users, This is the 64th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS New syslog-ng website --------------------- Our new website – http://syslog-ng.com/ – is now live. It is a central hub of information for all syslog-ng related content, including commercial products built around syslog-ng. All documentation and blogs were migrated here. The old syslog-ng.org site will go away once all open source related content found its place on the new site. Feedback is much appreciated! Balabit acquired by One Identity -------------------------------- You might have already heard that Balabit, the company behind syslog-ng, has been acquired by One Identity (http://www.oneidentity.com) and you may be wondering what would happen to syslog-ng in the future. Learn how Balázs Scheidler, original author of syslog-ng, sees the future: https://lists.balabit.hu/pipermail/syslog-ng/2018-January/024254.html Tl;dr: nothing to worry about! Logging in Docker using syslog-ng --------------------------------- Collecting logs can be especially important in a containerized environment, where you start and stop containers around the clock, and the container might not even exist anymore by the time you investigate an event. Read this white paper to find out how to: Run your central log server in Docker Collect Docker infrastructure logs using syslog-ng Collect logs from containers using Docker volumes https://syslog-ng.com/white-papers Common syslog-ng error messages and their solutions --------------------------------------------------- In this post, we explain a few common syslog-ng error and warning messages, what they mean, and how to solve them. https://syslog-ng.com/blog/common-syslog-ng-error-messages-solutions/ Getting started with Elasticsearch 6 on RHEL/CentOS --------------------------------------------------- Version 6 of the Elastic Stack has now been available for some time, packed with new features and improved performance. Compatibility of syslog-ng was checked already during the alpha phase of development, as syslog-ng is becoming popular among Elasticsearch users: it can greatly simplify logging to Elasticsearch. There are no major changes from a syslog-ng point of view but – to improve your copy & paste experience – I updated my getting started guide from Elastic Stack 5 to 6. https://syslog-ng.com/blog/syslog-ng-and-elasticsearch-6-getting-started-on… DevConf 2018 ------------ DevConf is a yearly conference for developers, administrators, and users of Linux and related technologies. It is organized by Red Hat in Brno, home to one of their major development centers. Since the majority of syslog-ng users are running their logging server on Red Hat Enterprise Linux or CentOS, it is important to know their needs and the directions where RHEL is heading. It is also a good possibility to discuss logging-related questions with Red Hat engineers in person. https://syslog-ng.com/blog/devconf-long-live-containerization/ UPCOMING EVENTS SCALE: Logging Docker using syslog-ng: https://www.socallinuxexpo.org/scale/16x/presentations/logging-docker-using… Pasadena Convention Center March 8 - 11, 2018 Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://syslog-ng.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

problem elasticsearch2 creating index per HOST
by Abe Lebo 07 Feb '18

07 Feb '18

LS, i want to create an Elasticsearch index per host i receive logs from (also syslog-ng) , somehow that fails. in have a setup that creates an index per day (see below), that works. I can build a file output per host per day by adding the $HOST to the filename, that works too.(see below) but if add $HOST to the indexname it no longer creates indices. Anybody know what's wrong? Thanks in advance, Abel ...... destination d_testfile { file( "/tmp/testlogabel-$HOST$YEAR$MONTH" flags(syslog-protocol) template("$(format-json @timestamp=$ISODATE --scope nv-pairs)\n") ); }; ################################################################################ # Elasticsearch destination ################################################################################ destination d_es_per_day { elasticsearch2( client-lib-dir("/usr/share/elasticsearch/lib/:/usr/lib/syslog-ng/3.13/java-modules/") index("test-abel_$YEAR.$MONTH.$DAY") type("test") cluster-url("http://node01:9400 http://node02:9400 http://node03:9400") client-mode("http") template("$(format-json @timestamp=$ISODATE --scope nv-pairs)\n") ); }; ###### # sending logs to ES destination log { source(s_net); source(s_src); filter(f_abel); parser(p_abel); destination(d_es_per_day); flags(flow-control); };

2 2

OSE, Shared Storage, & Load Balanced syslog Messages
by Pierce, Stevan - PHX 07 Feb '18

07 Feb '18

I am working on setting up OSE as a relay and would like to have storage set aside in case a destination upstream from the relay goes down. How does load balancing syslog messages work when clustering at the OS level? If I am sharing storage across multiple relays, how would I go about setting it up so that messages from one server does not overwrite another server? swp

2 1

Rsyslog relay or syslog-ng ?
by Scot 06 Feb '18

06 Feb '18

Hi, Hoping someone has seen an easy fix for this. Sorry is it's specifically referenced somewhere I'm not seeing. Dealing with a vendor who is not able to leverage the RFC headers or TCP input. We have rsyslog relays in remote sites sending TCP/514 to syslog-ng and others locally sending directly to syslog-ng TCP/UDP 514. The devices sending directly to syslog-ng are reporting to the IDS correctly. Hosts relaying through rsyslog are showing a source address of the relay. */etc/rsyslog.d/forward.conf* *$ActionQueueFileName fwdRule1 # unique name prefix for spool files* *$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)* *$ActionQueueSaveOnShutdown on # save messages to disk on shutdown* *$ActionQueueType LinkedList # run asynchronously* *$ActionResumeRetryCount -1 # infinite retries if host is down* *# remote host is: name/ip:port, e.g. 192.168.0.1:514 <http://192.168.0.1:514>, port optional* **.* @@syslog-ngIP:514* */etc/syslog-ng/conf.d* *source s_net_tcp {tcp(ip(0.0.0.0) port(514) max-connections(300) keep_hostname(yes) so_rcvbuf(262142) log_iw_size(25000)); };* *source s_net_udp {syslog(ip(0.0.0.0) keep_hostname(yes) port(514) transport("udp") flags(no-hostname) so_rcvbuf(262142));};* *destination d_ids {network("IDSHOSTNAME" spoof_source(yes) transport(udp) port(514) flags(syslog-protocol)); };* *log { source(s_net_udp);* * channel {filter(f_ids); destination (d_ids);};* * channel {parser(pattern_db); destination (d_es);};* *};* *log { source(s_net_tcp);* * channel {filter(f_ids); destination (d_ids);};* * channel {parser(pattern_db); destination (d_es);};* *};*

3 4

RFC5424 sending - dest=unix-stream. Framing?
by Declan White 05 Feb '18

05 Feb '18

I have syslog-ng pumping data into a unix-stream destination socketfilething. Data going into this domain socket has lots of adventures, and then emerges as a source unix-stream on another host. Both have flags(syslog-protocol), but the data gets framing errors at the receiver. I think it's become a bytestream (instead of a message stream) enroute, and the recv() is relying on only getting one message at a time, because it can't know the message length any other way, as unix-stream gets messages without framing(?). Is there a way to salvage this? (I noticed TLS might add framing, but since the tunnel is already TLS ...) -- Declan White

2 5

Re: [syslog-ng] Help with ES destination error
by Marco Mignone 01 Feb '18

01 Feb '18

Hi Fabien, I have tried the following: - emptying all index/docs in ES - create the test/test index with the CURL in my email - pointing syslog ES destination to the test/test index This resulted in the same error again. I have tried to change the template to just output all nv-pairs and use a complete new index - same error. Grabbing some packet capture now to see if I can spot anything wrong. Marco > On 28 Jan 2018, at 14:19, Fabien Wernli <wernli(a)in2p3.fr> wrote: > > Hi, > > The reason I asked you to configure syslogng to index to "test" was to make > sure you are in the same conditions as your curl command. > You might for instance have a mapping template matching fw-* but not test. > > Please either configure syslogng to index to test, or use the same fw- index > on the curl cmdline. > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq >

2 5