- syslog-ng - lists.balabit.hu

Insider 2017-09: 3.12; survey; geoip2; docker; AWS;
by Czanik, Péter 21 Sep '17

21 Sep '17

Dear syslog-ng users, This is the 61th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng 3.12 is coming ------------------------ The latest version of syslog-ng, 3.12 is expected to arrive soon. The most important new feature is the XML parser, which creates name-value pairs from XML-based log messages. You will also be able to configure JVM options for Java destinations, use PKCS-12, and utilize macros in HDFS file names. There are many more smaller features and bug fixes. For a complete list, check the git commits: https://github.com/balabit/syslog-ng/commits/master Test RPM packages from git head: https://www.balabit.com/blog/rpm-packages-from-syslog-ng-git-head/ New syslog-ng website --------------------- We will introduce a brand new website for all that is syslog-ng, and with that will come an official syslog-ng OSE repository with binaries for some of the most popular Linux distributions. We expect to launch syslog-ng.com in November, moving content for both OSE and its commercial versions to this new site so that it can serve as a single hub for all syslog-ng users. Learn more about it and help our work by answering a brief survey: https://www.balabit.com/blog/new-website-and-official-repo-coming-up-tell-u… Collecting logs from containers ------------------------------- There are many software products that log to files or pipes instead of their stdout, the place where Docker expects them. Fortunately by using Docker volumes, you can share data among containers, and syslog-ng can collect these logs as well. Learn how: https://www.balabit.com/blog/collecting-logs-containers-using-docker-volume… Heat maps using the new geoip2 parser ------------------------------------- The new geoip2 parser of syslog-ng 3.11 is not only faster than its predecessor, but can also provide a lot more detailed geographical information about IP addresses. Next to the usual country name and longitude/latitude information, it also provides the continent, time zone, postal code and even county name. Some of these are available in multiple languages. Learn how you can utilize all this information by parsing logs from iptables using syslog-ng, storing them to Elasticsearch, and displaying the results in Kibana! https://www.balabit.com/blog/creating-heat-maps-using-new-syslog-ng-geoip2-… Learn how you can impress your C-level leader(s) even more by animating these heat maps: https://www.balabit.com/blog/creating-time-lapse-videos-log-messages-using-… Installing syslog-ng on AWS Linux AMI ------------------------------------- You do not have to live without your favorite syslog implementation even in Amazon Web Services (AWS) Linux AMI. This Linux distribution is based on Red Hat Enterprise Linux version 6 and it takes minimal extra work to install syslog-ng on it. https://www.balabit.com/blog/installing-syslog-ng-aws-linux-ami/ EVENTS Virtualization day in Budapest, November 7th: https://vday.hu/2017/ Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

New website and official repo coming up, tell us what you think!
by Czanik, Péter 19 Sep '17

19 Sep '17

Hi, We at Balabit are and have always been both very excited and humbled by the community that formed around syslog-ng over the past two decades. Many of you have been a part of this long journey and helped syslog-ng to get to where it is today, and we thank you for that. Throughout the years, we provided a best-in-class documentation not just for our commercial offerings but for syslog-ng Open Source Edition as well. However, for the past half decade official binaries existed only for syslog-ng Premium Edition. Soon, this will change. We will introduce a brand new website for all that is syslog-ng, and with that will come an official syslog-ng OSE repository with binaries for some of the most popular Linux distributions. We expect to launch syslog-ng.com in November, moving content for both OSE and its commercial versions to the site to become a single hub for all syslog-ng users. The official syslog-ng OSE repository will be just a part (although a major one) of a comprehensive overhaul of our OSE related content providing platform. We will provide regular technical webinars and trainings for syslog-ng OSE users, more blogs on practical tips&tricks, common use cases, etc. We plan to channel all syslog-ng related discussion into a forum on the new site, although this will take some more time. And we already started putting in more resources into the development of the Open Source Edition. This is a major commitment on our side to the community, and we know that it is the right move. However, we haven't quite decided on the implementation yet, and we are looking for your feedback on this. This is a serious business decision and ultimately all stakeholders (we as a company and you as a member of the community) have to benefit from this move, otherwise there is no reason to do it. We are listing all these so you can see the bigger picture before we present you the question: Would you register on the upcoming syslog-ng website to get access to the official repository and the aforementioned content? What would you gain if you registered? - Access to a repository containing the latest syslog-ng packages (technically, you would receive a key that you could include in FIXME) - Alerts, announcements and news about the syslog-ng releases - Regular news about syslog-ng related content to help you better solve problems with syslog-ng (blog posts, tutorials, interesting use cases about problems that others solved with syslog-ng, and so on) What would we gain if you registered? - First and foremost, we would get information immensely useful for us to optimize our development resources. How many of you use the repositories, which versions of syslog-ng you use, which platforms are the most important, which ones you do not use anymore, and so on. Unfortunately, we cannot get such data if you use external repositories (like the current unofficial ones), or if you install syslog-ng from your distribution. - Information about in which countries are hubs of syslog-ng users, for example, where could we organize meetups most successfully, or participate in local conferences to strengthen the community. - The ability to send you information about syslog-ng, both to help you learn about the possibilities of syslog-ng, and also to let you know about our commercial offerings. We know that some of you won't be interested in anything commercial, and that's fine. But to keep syslog-ng alive, we also need the support of those who do. We believe that registering on the new website and receiving the occasional email from us is a small favor to ask in return of the convenience of having access to a official repository with up-to-date packages, and we will strive to send you emails with useful content. Please let us know what you think using this short Google Form. <https://goo.gl/forms/TmTJY8CkRAzUwtEA2> Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

3 6

CentOS cant start syslog-ng via systemd but runs fine manually
by Andrew 14 Sep '17

14 Sep '17

I recently updated my CentOS server so there were new versions of Java and syslog-ng and I have been having issues ever since. If I run it from systemd then it fails to load, yet if I run it manually it works fine and pushes the logs into elasticsearch. When trying to run from systemd it complains about Java plugin not being found. systemctl start syslog-ng Error parsing destination, destination plugin java not found in destination block elasticsearch2 at line 2, column 3 If I run it manually by typing syslog-ng then it works just fine.

4 6

3.11 Multiple elastic destinations.
by Scot 14 Sep '17

14 Sep '17

Has anyone had success/failure using multiple ES destinations in syslog-ng. I am want to direct traffic to different indexes based on syslog-ng filters destination d_es { elasticsearch2( client-mode("http") index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("syslog") # Description: The type of the index. For example, type("test") template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") cluster-url("http://elasticsearch:9200/") concurrent-requests("5") # Number of concurrrent batches flush_limit("5000") # The number of messages in a single batch skip-cluster-health-check("yes") cluster("clustername") client_lib_dir("/usr/share/elasticsearch/lib") ); }; destination d_es_network { elasticsearch2( client-mode("http") index("network_${YEAR}.${MONTH}.${DAY}") type("syslog") # Description: The type of the index. For example, type("test") template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") cluster-url("http://elasticsearch:9200/") concurrent-requests("5") # Number of concurrrent batches flush_limit("5000") # The number of messages in a single batch skip-cluster-health-check("yes") cluster("clustername") client_lib_dir("/usr/share/elasticsearch/lib") ); };

2 1

Re: [syslog-ng] Syslog-ng on IBM System i
by Balazs Scheidler 13 Sep '17

13 Sep '17

This was a commercial offering from balabit which has been discontinued. The balabit sales team can possibly refer to an alternative, and i can connect you if you want. Just email me in private. On Sep 13, 2017 12:09 AM, "Lucas Ihara Alves" <lucas.alves(a)trtsp.jus.br> wrote: Dear Sir or Madam, I would like to know if Syslog-ng Open Source has a compiled version for IBM System i. If so, please, I would be grateful if anyone could send me the link to download. -- Yours Truly, ---- Lucas Alves ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

Syslog-ng on IBM System i
by Lucas Ihara Alves 12 Sep '17

12 Sep '17

Dear Sir or Madam, I would like to know if Syslog-ng Open Source has a compiled version for IBM System i. If so, please, I would be grateful if anyone could send me the link to download. -- Yours Truly, ---- Lucas Alves

1 0

URGENT: http://download.opensuse.org/repositories/home:/laszlo_budai:/ site is down
by Clayton Dukes 08 Sep '17

08 Sep '17

Hey Guys, http://download.opensuse.org/repositories/home:/laszlo_budai:/ returns a 404, which means our customers can't install syslog-ng. Can someone look into it? To avoid this in the future, we're happy to host for you if needed. [id:image001.png@01D306E3.0FEBC990] Clayton Dukes Founder & CEO LogZilla Corporation 2900 N. Quinlan Park Rd, B240-341 Austin, TX, 78732 Tel: 936-4NetOps (463-8677) Web: www.logzilla.net<http://www.logzilla.net/> [id:image002.png@01D306E3.0FEBC990]<https://twitter.com/logzilla>[id:image003.png@01D306E3.0FEBC990]<https://youtu.be/drg5wv_mgfA>[id:image004.png@01D306E3.0FEBC990]<https://www.linkedin.com/in/lzcdukes/> For NetOps, By NetOps!

4 7

syslog-ng not writing to specified path
by vijay amruth 01 Sep '17

01 Sep '17

Hello all, hope you are all doing great. I have set up syslog-ng on a host and I am able to see packets on tcpdump but syslog-ng is not wriiting to the specified paths. I have checked firewall rules, filters, write permissions at the path. I have another host on the same VLAN with the same config that is able to receive packets and write to the destination. what I am I missing ? *version used:* syslog-ng 3.9.1 Installer-Version: 3.9.1 Revision: Module-Directory: /usr/local/lib/syslog-ng Module-Path: /usr/local/lib/syslog-ng Available-Modules: syslogformat,afsocket,affile,afprog,afuser,afamqp,afmongodb,csvparser,confgen,system-source,linux-kmsg-format,basicfuncs,cryptofuncs,dbparser,json-plugin,afstomp,pseudofile,graphite,sdjournal,kvformat,date,cef,disk-buffer,add-contextual-data Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off syslog-ng -Fvde shows : [2017-08-03T13:57:20.214552] Module loaded and initialized successfully; module='syslogformat' Any help is appreciated. Thanks, Vijay Amrut.

2 1

Creating time lapse videos from log messages
by Czanik, Péter 31 Aug '17

31 Aug '17

Hi, A couple of you asked how I created the time lapse videos visualizing connection attempts to my router. I used the new geoip2 parser of syslog-ng 3.11, Elasticsearch for storage, Kibana for showing the map and OpenShot to convert screenshots into a movie. Read the details from my blog at https://www.balabit.com/blog/creating-time-lapse-videos-log-messages-using-… I'm not really sure if it has any practical use, but definitely a nice eye candy which can raise attention towards security of your network. Do you have any other suggestions for log data to be visualized in a similar way? Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

host related macros
by Fabien Wernli 28 Aug '17

28 Aug '17

Hi, I have a hard time understanding the behaviour of the multiple host based macros with respect to the value of the use-fqdn(), and keep-hostname() params. First of all, when setting use-fqdn(yes), isn't the network destination supposed to send the fqdn? In our case, tcpdump shows that the short hostname is sent instead. Additionally, what does the following paragraph from the documentation mean? The hostname-related macros (${FULLHOST}, ${FULLHOST_FROM}, ${HOST}, and ${HOST_FROM}) do not have any effect if the keep-hostname() option is disabled Does that mean they are empty if keep-hostname(no) ? Cheers

2 2