- syslog-ng - lists.balabit.hu

syslog-ng parsing Error
by David Campeau 25 Jun '18

25 Jun '18

Hello, I have a syslog source node sending syslogs, and they are being generated via a python script, and is using Python Rfc5426SysLogHandler. So, these syslog messages should be RFC compliant. However, syslog-ng does prepend an error message before sending it on to be put into storage. Example error message from syslog-ng = <43>Jun 21 10:27:38 syslog-ng-Server syslog-ng[2559]: Error processing log message: xxxxx timestamp, source hostname and payload follows. I've done some googling, but haven't been able to find out what error 2559 means. Any thoughts of what I might do to determine what syslog-ng isn't liking about the syslog it is receiving? I need to relay this information to a developer so they can make adjustments to the python script. Best regards, David

2 4

getting error destination plugin java not found while using mod-java
by Pratik Sharma 25 Jun '18

25 Jun '18

Hi Team, I have a setup of syslog-ng in Centos 7 with jdk 1.8.To use a custom java program i have installed syslog-ng-java plugin using yum install. I have added a custom syslog-ng file in conf.d directory.On starting the syslog-ng service i am reciving error destination java plugin not found. *ERROR:* > Jun 24 14:16:01 localhost syslog-ng: Error parsing destination, > destination plugin java not found in > /etc/syslog-ng/conf.d/syslog-ng-custom.conf at line 15, column 3: > Jun 24 14:16:01 localhost syslog-ng: included from > /etc/syslog-ng/syslog-ng.conf line 68, column 1 > Jun 24 14:16:01 localhost syslog-ng: java( > Jun 24 14:16:01 localhost syslog-ng: ^^^^ > Syslog-ng is detecting the java plugin(verified by syslog-ng -V) *syslog-ng -V:* > [root@localhost server]# syslog-ng -V >>>>> syslog-ng 3 (3.14.1) >>>>> Config version: 3.14 >>>>> Installer-Version: 3.14.1 >>>>> Revision: >>>>> Compile-Date: Feb 28 2018 07:34:20 >>>>> Module-Directory: //usr/lib64/syslog-ng >>>>> Module-Path: //usr/lib64/syslog-ng >>>>> Available-Modules: >>>>> add-contextual-data,afamqp,affile,afprog,afsocket,afstomp,afuser,appmodel,basicfuncs,cef,confgen,cryptofuncs,csvparser,date,dbparser,disk-buffer,graphite,json-plugin,kvformat,linux-kmsg-format,map-value-pairs,pseudofile,sdjournal,snmptrapd-parser,stardate,syslogformat,system-source,tags-parser,tfgetent,xml, >>>>> *mod-java* >>>>> Enable-Debug: off >>>>> Enable-GProf: off >>>>> Enable-Memtrace: off >>>>> Enable-IPv6: on >>>>> Enable-Spoof-Source: on >>>>> Enable-TCP-Wrapper: on >>>>> Enable-Linux-Caps: on >>>>> Enable-Systemd: on >>>>> >>>> Java Version: > [root@localhost server]# java -version > java version "1.8.0_171" > Java(TM) SE Runtime Environment (build 1.8.0_171-b11) > Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode) > *conf.d/syslog-ng-custom.conf* source s_apache { > system(); # Check which OS & collect system logs > internal(); > }; > > destination d_remote { > tcp("1.2.3.4" port(514)); > }; > destination d_device_file { > file("/var/log/$HOST.$MIN.log"); > }; > > destination java_to_file{ > java( > class_name("SampleJavaDestination") > class_path("/root") > option("name", "value") > option("filepath", "/var/log/example.txt") > ); > }; > > log { > source(s_apache); > destination(d_device_file); > destination(java_to_file); > > }; > Can someone please tell me what i am missing?

3 2

Fwd: SIGBUS error patch version
by som shukla 22 Jun '18

22 Jun '18

---------- Forwarded message ---------- From: som shukla <somj.shukla(a)gmail.com> Date: Fri, Jun 22, 2018 at 12:04 PM Subject: SIGBUS error patch version To: syslog-ng(a)lists.balabit.hu Hi , I'm using OSE version of the syslog-ng. There is a bug with persist file which has been solved recently in Feb. Following is the link for the githubpage https://github.com/balabit/syslog-ng/pull/1785/files. Was it ever included as patch in the standard versions. If yes then what is the version number. Thanks Somji

2 1

Poll: how quickly you update syslog-ng
by Czanik, Péter 20 Jun '18

20 Jun '18

Hi, I just posted a syslog-ng update related poll on Twitter: https://twitter.com/PCzanik/status/1009429698353553408 The reason is, that I'd like to learn how quickly you start to use a new version of syslog-ng after release. I see some rough download statistics for my Fedora / CentOS packages, but that is only a slice of syslog-ng users. Please answer the poll, and If you have any comments, alternative answers, explanations, you can leave it here or on Twitter. Also: retweet to reach more syslog-ng users! Thanks, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://syslog-ng.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

Re: [syslog-ng] hostname not appearing correctly when receiving logs from switches
by Clayton Dukes 19 Jun '18

19 Jun '18

Was anyone able to figure out why this is happening? From: "Scheidler, Balázs" <balazs.scheidler(a)balabit.com> Date: Tuesday, May 15, 2018 at 10:58 AM To: "cdukes(a)cdukes.com" <cdukes(a)cdukes.com> Cc: Clayton Dukes <cdukes(a)logzilla.net> Subject: Re: Fwd: [syslog-ng] hostname not appearing correctly when receiving logs from switches Thanks On May 15, 2018 16:52, "Clayton Dukes" <cdukes(a)gmail.com<mailto:cdukes@gmail.com>> wrote: Making sure you got this (in case the other went to spam) ______________________________________________________________ Clayton Dukes ______________________________________________________________ ---------- Forwarded message ---------- From: Clayton Dukes <cdukes(a)logzilla.net<mailto:cdukes@logzilla.net>> Date: Mon, May 7, 2018 at 4:46 PM Subject: Re: [syslog-ng] hostname not appearing correctly when receiving logs from switches To: "Scheidler, Balázs" <balazs.scheidler(a)balabit.com<mailto:balazs.scheidler@balabit.com>> Cc: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Hi Balazs, Sorry for the delay, I don’t get a lot of free time these days :) I have attached a pcap as well as a raw log. The log is prior to any manipulation of LogZilla rules, etc. This is easily reproduceable. Also, if I add show-timezone to the device config, the host field shows up. The problem, of course, is that we can’t tell all of our customers to re-configure all of their cisco devices. We have documented the work-around here (search the page for “hostname missing”): http://demo.logzilla.net/help/receiving_data/cisco_ios_configuration This problem never existed before, but I am not certain which syslog-ng version it started occurring in. From: "Scheidler, Balázs" <balazs.scheidler(a)balabit.com<mailto:balazs.scheidler@balabit.com>> Date: Tuesday, May 1, 2018 at 8:45 PM To: Clayton Dukes <cdukes(a)logzilla.net<mailto:cdukes@logzilla.net>> Cc: Joshua <aces621(a)yahoo.com<mailto:aces621@yahoo.com>>, Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: Re: [syslog-ng] hostname not appearing correctly when receiving logs from switches Interesting that I saw this message the first time in your response, and not the original one. Anyhow, to understand the problem we would need an exact byte-by-byte representation of what syslog-ng is receiving from the switch together with the configuration that is used to process it. A tcpdump or an "Incoming message" from syslog debug outout should work. We haven't intentionally changed the syslog parser as far as I remember. On May 1, 2018 22:50, "Clayton Dukes" <cdukes(a)logzilla.net<mailto:cdukes@logzilla.net>> wrote: Interesting! We’ve been getting a lot of support tickets for this very problem. I can easily recreate the issue. Balabit Team: is this a new bug? [cid:image001.png@01D306E3.0FEBC990] Clayton Dukes Founder & CEO LogZilla Corporation 2900 N. Quinlan Park Rd<https://maps.google.com/?q=2900+N.+Quinlan+Park+Rd&entry=gmail&source=g>, B240-341 Austin, TX, 78732 Tel: 936-4NetOps (463-8677) Web: www.logzilla.net<http://www.logzilla.net/> [cid:image002.png@01D306E3.0FEBC990]<https://twitter.com/logzilla>[cid:image003.png@01D306E3.0FEBC990]<https://youtu.be/drg5wv_mgfA>[cid:image004.png@01D306E3.0FEBC990]<https://www.linkedin.com/in/lzcdukes/> For NetOps, By NetOps! From: syslog-ng <syslog-ng-bounces(a)lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Joshua <aces621(a)yahoo.com<mailto:aces621@yahoo.com>> Reply-To: Joshua <aces621(a)yahoo.com<mailto:aces621@yahoo.com>>, Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Date: Monday, April 30, 2018 at 7:09 PM To: "syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>" <syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: [Suspected Spam] [syslog-ng] hostname not appearing correctly when receiving logs from switches Hi All, I am pretty new to syslog-ng but do have some basic knowledge. I have deployed syslog-ng v3.14 on a newly deployed Linux server because syslog-ng v3.5 is working very well on another syslog server. On this new deployment, the syslogs received from most of the servers are able to show IP/host, however, the syslogs from our switches contains IP/host showing as ":" (colons). I copied the current working custom build .conf from another syslog server into our new server. Can someone help me figure out what I am missing? It is working for some components but not for switches. I tested the same switch by sending syslog to another syslog server and the hostname is appearing but just not appearing on the new syslog server. The only difference between the two server is that one uses v3.5 (the working one) and the other uses syslog-ng v3.14. I have set: "keep_hostname (yes)" but it still doesn't work. Can someone please help? Am I missing something here? Thanks Joshua Lai ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

2 3

OSE 3.16
by Budai, László 19 Jun '18

19 Jun '18

3.16.1 Features - Telegram destination and $(urlencode) template function (#2085 <https://github.com/balabit/syslog-ng/pull/2085>) - Error reporting on misspelled block args (#1952 <https://github.com/balabit/syslog-ng/pull/1952>) - New ignore_tns_config Oracle SQL destination config option (#2054 <https://github.com/balabit/syslog-ng/pull/2054>) - Per-source "src.host" and "src.sender" counters (#2013 <https://github.com/balabit/syslog-ng/pull/2013>) Bugfixes - Fix possible loss of log messages in the systemd-journal() source ( #1570 <https://github.com/balabit/syslog-ng/issues/1570>, #1587 <https://github.com/balabit/syslog-ng/issues/1587>, #1612 <https://github.com/balabit/syslog-ng/pull/1612>) - Fix file source location information in internal logs (#2028 <https://github.com/balabit/syslog-ng/pull/2028>) - Fix SDATA deserialization (disk-buffer crash) (#1919 <https://github.com/balabit/syslog-ng/issues/1919>, #2067 <https://github.com/balabit/syslog-ng/pull/2067>) - Fix unaccepted embedded 'file' keyword (file source and destination) ( #2076 <https://github.com/balabit/syslog-ng/pull/2076>) - Fix memory leaks in appmodel and varargs (#2086 <https://github.com/balabit/syslog-ng/pull/2086>) - Fix a bug in the old LogMessage deserialization (#2103 <https://github.com/balabit/syslog-ng/pull/2103>) - Fix reading the output of the confgen program (#1780 <https://github.com/balabit/syslog-ng/issues/1780>, #2108 <https://github.com/balabit/syslog-ng/pull/2108>) - Add safer mem_zero() to secret-storage (#2049 <https://github.com/balabit/syslog-ng/pull/2049>) - Fix undefined behavior in syslog-ng-ctl query (#2043 <https://github.com/balabit/syslog-ng/pull/2043>) - Fix lloc tracking for multi line blockrefs (#2035 <https://github.com/balabit/syslog-ng/pull/2035>) - Added missing 'else {};' to default-network-drivers() to forward unparsable messages (#2027 <https://github.com/balabit/syslog-ng/pull/2027>) - Fix mixed linking (#2020 <https://github.com/balabit/syslog-ng/issues/2020>, #2022 <https://github.com/balabit/syslog-ng/pull/2022>) - Fix compilation of evtlog on FreeBSD (#2014 <https://github.com/balabit/syslog-ng/issues/2014>, #2015 <https://github.com/balabit/syslog-ng/pull/2015>) - Fix thread_id allocation for more than 32 CPUs (crash) (#2008 <https://github.com/balabit/syslog-ng/pull/2008>) - Add safe logging of errno (#1990 <https://github.com/balabit/syslog-ng/issues/1990>, #1999 <https://github.com/balabit/syslog-ng/pull/1999>) - Fix warnings related to floating point operations (#1959 <https://github.com/balabit/syslog-ng/issues/1959>, #1996 <https://github.com/balabit/syslog-ng/pull/1996>) - Partial revert of plugin discovery to bring back valgrind (#1953 <https://github.com/balabit/syslog-ng/issues/1953>, #1995 <https://github.com/balabit/syslog-ng/pull/1995>) - Fix connection close in network sources (#1991 <https://github.com/balabit/syslog-ng/pull/1991>) - Fix file deletion in the wildcard-file() source (#1974 <https://github.com/balabit/syslog-ng/pull/1974>) - Disable the DNS cache if use-dns(no) is used (#1923 <https://github.com/balabit/syslog-ng/pull/1923>) - Fix compiler error for gcc 4.4 (#2044 <https://github.com/balabit/syslog-ng/pull/2044>) - Fix emitted warnings due to -no-pie detection for gcc 4.4 (#2037 <https://github.com/balabit/syslog-ng/pull/2037>) - Fix date format in functional tests (#2036 <https://github.com/balabit/syslog-ng/pull/2036>) - Dbld fixes (#2034 <https://github.com/balabit/syslog-ng/pull/2034>) - Rename PAGESIZE variables to pagesize in secret-storage (compilation fix) (#2111 <https://github.com/balabit/syslog-ng/pull/2111>) - Fix the lifetime of TLSContext to prevent crash on reload (#2080 <https://github.com/balabit/syslog-ng/issues/2080>, #2109 <https://github.com/balabit/syslog-ng/pull/2109>) - Fix reaping program() source and destination when a Java-based destination is used (#2099 <https://github.com/balabit/syslog-ng/pull/2099>) Other changes - Add debug message to program source/destination about successful start (#2046 <https://github.com/balabit/syslog-ng/pull/2046>) - Report memory exhaustion errors during config parsing (#2033 <https://github.com/balabit/syslog-ng/pull/2033>) - Improved debug logs (#2032 <https://github.com/balabit/syslog-ng/pull/2032>) - Dbld coverage (#2031 <https://github.com/balabit/syslog-ng/pull/2031>) - LogTransportMock enhancement (#2017 <https://github.com/balabit/syslog-ng/pull/2017>) - Modify the license of loggen from GPL to LGPL (#2006 <https://github.com/balabit/syslog-ng/pull/2006>) - Loggen refactor (#1987 <https://github.com/balabit/syslog-ng/pull/1987> ) - Update RPM generation (#1980 <https://github.com/balabit/syslog-ng/pull/1980>, #2113 <https://github.com/balabit/syslog-ng/pull/2113>) - Support ENABLE_EXTRA_WARNINGS with CMake (#2072 <https://github.com/balabit/syslog-ng/pull/2072>) - Rewrite unit tests based on Criterion (#2026 <https://github.com/balabit/syslog-ng/pull/2026>, #2058 <https://github.com/balabit/syslog-ng/pull/2058>, #2039 <https://github.com/balabit/syslog-ng/pull/2039>, #2018 <https://github.com/balabit/syslog-ng/pull/2018>, #2003 <https://github.com/balabit/syslog-ng/pull/2003>) - Lexer test coverage improvements (#2062 <https://github.com/balabit/syslog-ng/pull/2062>) - preparation for 3.16 OSE rhel/packaging (#2113 <https://github.com/balabit/syslog-ng/pull/2113>) Notes to the developers - Do not ship rabbitmq-c with syslog-ng (remove as submodule) (#2052 <https://github.com/balabit/syslog-ng/pull/2052>) - Multitransport (#2057 <https://github.com/balabit/syslog-ng/pull/2057>) - Timeout support for LogReader and LogWriter (#2056 <https://github.com/balabit/syslog-ng/pull/2056>) - Update ivykis to 0.42.3 (submodule) (#2012 <https://github.com/balabit/syslog-ng/pull/2012>) - Drop explicit version numbers from requirements.txt (#2050 <https://github.com/balabit/syslog-ng/pull/2050>) - CMake modernization pt. 1 (#2007 <https://github.com/balabit/syslog-ng/pull/2007>) - Assert when log_pipe_clone() is not implemented (#2019 <https://github.com/balabit/syslog-ng/pull/2019>) - Small Java code refactor (#2066 <https://github.com/balabit/syslog-ng/pull/2066>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Andrej Valek, Antal Nemes, Balazs Scheidler, Gabor Nagy, Gergely Nagy, German Service Network, Jakub Wilk, Laszlo Budai, Laszlo Szemere, Laszlo Varady, Mehul Prajapati, Norbert Takacs, Peter Czanik, Peter Kokai, Tomasz Kazimierczak, Viktor Juhasz — You are receiving this because you are subscribed to this thread. View it on GitHub <https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.16.1> or mute the thread <https://github.com/notifications/unsubscribe-auth/AGRToMeewA9tzQTqybl0VI2qG…> .

1 0

Unofficial syslog-ng 3.16 Debian/Ubuntu packages
by Laszlo Budai 18 Jun '18

18 Jun '18

Hi, syslog-ng 3.16.1[1] packages are available in OBS repo[2]. List of supported OSs: * Debian 8.0 * Debian 9.0 * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 16.10 * Ubuntu 17.04 * Ubuntu 17.10 * Ubuntu 18.04 In case of Debian 9.0, armv7l architecture also supported, so you can install it to your Raspberry Pi. (Note: as of writing this mail, armv7l is still building.) Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.16.1 [2] https://build.opensuse.org/project/show/home:laszlo_budai:syslog-ng regards, Laszlo Budai

1 0

java 10+ support for elasticsearch2
by Nik Ambrosch 15 Jun '18

15 Jun '18

Oracle is pushing to deprecate Java 8/9 - I was wondering if anyone has gotten syslog-ng 3.15 working with Oracle JDK 10.0.1? I'm seeing this on startup which seems to prevent the loading of the elasticsearch2 destination: Jun 15 01:51:39 centos3 syslog-ng[55366]: WARNING: An illegal reflective access operation has occurred Jun 15 01:51:39 centos3 syslog-ng[55366]: WARNING: Illegal reflective access by org.syslog_ng.SyslogNgClassLoader (file:/usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar) to method java.net.URLClassLoader.addURL(java.net.URL) Jun 15 01:51:39 centos3 syslog-ng[55366]: WARNING: Please consider reporting this to the maintainers of org.syslog_ng.SyslogNgClassLoader Jun 15 01:51:39 centos3 syslog-ng[55366]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations Jun 15 01:51:39 centos3 syslog-ng[55366]: WARNING: All illegal access operations will be denied in a future release Jun 15 01:51:39 centos3 syslog-ng[55366]: [2018-06-15T01:51:39.333858] Error while expanding path list: object is not an instance of declaring class; Jun 15 01:51:39 centos3 syslog-ng[55366]: java.lang.IllegalArgumentException: object is not an instance of declaring class Jun 15 01:51:39 centos3 syslog-ng[55366]: at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) Jun 15 01:51:39 centos3 syslog-ng[55366]: at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) Jun 15 01:51:39 centos3 syslog-ng[55366]: at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) Jun 15 01:51:39 centos3 syslog-ng[55366]: at java.base/java.lang.reflect.Method.invoke(Method.java:564) Jun 15 01:51:39 centos3 syslog-ng[55366]: at org.syslog_ng.SyslogNgClassLoader.expandClassPath(SyslogNgClassLoader.java:144) Jun 15 01:51:39 centos3 syslog-ng[55366]: at org.syslog_ng.SyslogNgClassLoader.loadClass(SyslogNgClassLoader.java:65) Jun 15 01:51:39 centos3 syslog-ng[55366]: [2018-06-15T01:51:39.335187] Can't find class; class_name='org.syslog_ng.elasticsearch_v2.ElasticSearchDestination' Jun 15 01:51:39 centos3 syslog-ng[55366]: [2018-06-15T01:51:39.337627] Error initializing message pipeline; plugin_name='java', location='#buffer:2:3'

1 0

Insider 2018-06: 3.15; ARM; Big Data; journald; Elasticsearch; conferences;
by Czanik, Péter 14 Jun '18

14 Jun '18

Dear syslog-ng users, This is the 67th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng 3.15 is released -------------------------- Version 3.15 of syslog-ng is now available. Support was added for if / elif / else blocks, which can simplify syslog-ng configuration considerably. Improved debug and error messages make finding configuration problems easier. Support for basic client-side failover was added. For a complete list of changes, check https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.15.1. For binary packages, check https://syslog-ng.com/3rd-party-binaries. Debian ARM builds ----------------- Starting with version 3.15, the unofficial syslog-ng Debian build is also available for ARM v7. It has all of the features of the x86 edition. The build is made for Debian 9.0 and tested on a Raspberry Pi, but it most likely works on recent editions of Raspbian as well. It is available at https://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/De…. Big Data: save all or save costs? --------------------------------- When starting a new project, Big Data vendors usually recommend a “save all” and “save raw” approach, as you never know what data might come in handy later on and in what format. Companies starting those projects also often have the same approach, as they still have their infrastructure under heavy development. Both go on the assumption that storage is practically free compared to the value of data. But is it really so? Learn more at https://syslog-ng.com/blog/big-data-save-save-costs/. syslog-ng vs. systemd’s journald -------------------------------- People often ask what to use: systemd’s journald or syslog-ng? The quick answer is that most likely both, but it depends on how you use your computer(s). If you have a single standalone machine, journald is probably enough. There is even a nice desktop application to view the logs in the journal. But once you have multiple machines to manage, using syslog-ng has many advantages. Read more at https://news.opensuse.org/2018/04/30/syslog-ng-vs-systemds-journald/. Storing logs in Elasticsearch using syslog-ng --------------------------------------------- Get started with sending logs directly to Elasticsearch using syslog-ng. Find out how to parse data with syslog-ng, store it in Elasticsearch, and analyze it with Kibana. Learn about the basics of using syslog-ng and Elasticsearch 6 on Red Hat Enterprise Linux / CentOS. Discover how to send netdata metrics through syslog-ng to Elasticsearch. Read our technical white paper at https://pages.balabit.com/storing-logs-in-elasticsearch-using-syslog-ng.html Containers and automation: five conferences in two words -------------------------------------------------------- In the past couple of weeks I visited five different conferences in four different countries either as a speaker or as booth staff. Altogether I had a chance to present syslog-ng to thousands of people and discuss syslog-ng and logging in general with hundreds. Except for a Big Data event in Budapest, I could summarize the conferences in two words: containers and automation. Of course, all events covered other diverse topics as well, but the main themes were these two. Let me give you a quick overview of the events: https://syslog-ng.com/blog/containers-automation-five-conferences-two-words/. Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://syslog-ng.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

Issue with Juniper SRX structured stream logs getting parsed by syslog-ng
by Brad Riemann 13 Jun '18

13 Jun '18

Hello, I've been banging my head against the wall for a while with this issue, I have a Juniper SRX that sends traffic logs to syslog-ng (centos 6, 6, version 3.13.2-2), the key being they are structured so the look like the following. [2018-06-12T19:03:54.384014] Incoming log entry; line='<14>1 2018-06-12T19:03:54.125Z corefw-site-0 RT_FLOW - RT_FLOW_SESSION_CLOSE [junos(a)2636.1.1.1.2.40 reason="idle Timeout" source-address="10.1.1.1" source-port="56998" destination-address="0.0.0.0" destination-port="443" service-name="junos-https" nat-source-address="10.100.1.1" nat-source-port="39155" nat-destination-address="0.0.0.0" nat-destination-port="443" src-nat-rule-type="source rule" src-nat-rule-name="test-servers" dst-nat-rule-type="N/A" dst-nat-rule-name="N/A" protocol-id="6" policy-name="allow-outbound" source-zone-name="test-zone" destination-zone-name="untrust" session-id-32="499236" packets-from-client="3" bytes-from-client="180" packets-from-server="0" bytes-from-server="0" elapsed-time="19" application="UNKNOWN" nested-application="UNKNOWN" username="N/A" roles="N/A" packet-incoming-interface="reth0.1" encrypted="UNKNOWN"]' I can confirm that syslog-ng is seeing the whole message as part of the incoming log entry, however it's just not showing up. I've dug through RFC 5424 and these logs DO conform but my confusion still remains. When I check the logs that were written the line looks like the following, which seems to just trunk everything after the RT_FLOW of the message header. 2018-06-12T19:03:54.125+00:00 corefw-site-0 RT_FLOW: I've run syslog-ng in debug mode, which was how I was able to confirm the outgoing message matches the prior line as well as the incoming message matching the expected data from my srx. [2018-06-12T19:03:54.385505] Outgoing message; message='2018-06-12T19:03:54.125+00:00 corefw-site-0 RT_FLOW: \x0a' Here is a snippet of my syslog config, it isn't that complex (at least I think so..) and i'm running out of hair to pull, any input would be GREATLY appreciated. https://gist.github.com/robotman321/217a69d456e543831936b1209b8bf501 -Brad

3 5