- syslog-ng - lists.balabit.hu

Listen backlog issues?
by Claus Albøge 19 Sep '16

19 Sep '16

Hi, I have ~6000 hosts sending syslog-TLS to a syslog-ng server. I’m seeing a lot of reconnects from the clients, and on the syslog-ng servers I get the following messages: TCP: request_sock_TCP: Possible SYN flooding on port 6514. Sending cookies. Check SNMP counters. netstat -s show the following: $ netstat -s | grep -i listen 10480620 times the listen queue of a socket overflowed 11144791 SYNs to LISTEN sockets dropped A few resources suggest this could be due to the applications socket listen backlog. Starting syslog-ng with strace reveals the following: 30067 listen(5, 255) = 0 30067 listen(11, 255) = 0 30067 listen(12, 255) = 0 30067 listen(17, 255) = 0 It seems like the backlog is hardcoded to 255 in the source. A bit more info: Saving the output from: “netstat -n | grep -c SYN_RECV” to a file a few times per second, shows the following: 2016-09-19 09:54:14.767469015: 0 2016-09-19 09:54:14.906638364: 0 2016-09-19 09:54:15.048591252: 0 2016-09-19 09:54:15.184325070: 0 2016-09-19 09:54:15.324150368: 0 2016-09-19 09:54:15.459749187: 0 2016-09-19 09:54:15.587983284: 0 2016-09-19 09:54:15.722558975: 42 2016-09-19 09:54:15.873601766: 256 2016-09-19 09:54:16.020145083: 247 2016-09-19 09:54:16.177231109: 178 2016-09-19 09:54:16.340875439: 178 2016-09-19 09:54:16.488506916: 178 2016-09-19 09:54:16.637815500: 176 2016-09-19 09:54:16.790781389: 149 2016-09-19 09:54:16.950254517: 141 2016-09-19 09:54:17.125316109: 131 2016-09-19 09:54:17.286839687: 131 2016-09-19 09:54:17.445583267: 127 2016-09-19 09:54:17.598144758: 127 2016-09-19 09:54:17.747967473: 124 2016-09-19 09:54:17.905359412: 2 2016-09-19 09:54:18.062665358: 2 2016-09-19 09:54:18.239973822: 0 2016-09-19 09:54:18.398021514: 0 2016-09-19 09:54:18.548074304: 0 This matches the entry in /var/log/messages Sep 19 09:54:15 log03 kernel: [1116811.081666] TCP: request_sock_TCP: Possible SYN flooding on port 6514. Sending cookies. Check SNMP counters. Are there any tunings in the syslog-ng configuration I need to implement, or do I have to compile syslog-ng myself with a larger listen backlog? Perhaps this should be configurable? My source definition for syslog-TLS is like this: source s_syslog_tls { syslog( ip(0.0.0.0) port(6514) log_fetch_limit(100) log_iw_size(11000000) max-connections(10000) transport("tls") tls( key-file("/etc/pki/tls/certs/logserver.key") cert-file("/etc/pki/tls/certs/logserver.crt") peer-verify(optional-untrusted) ) ); }; $ syslog-ng -V syslog-ng 3.8.1 Installer-Version: 3.8.1 Revision: Module-Directory: //usr/lib64/syslog-ng Module-Path: //usr/lib64/syslog-ng Available-Modules: disk-buffer,sdjournal,afstomp,json-plugin,cryptofuncs,graphite,cef,kvformat,add-contextual-data,dbparser,pseudofile,curl,csvparser,syslogformat,confgen,afsocket,afuser,date,linux-kmsg-format,system-source,basicfuncs,afamqp,affile,afprog Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: off syslog-ng-3.8.1-1.el7.centos.x86_64 from https://copr-be.cloud.fedoraproject.org/results/czanik/syslog-ng38/epel-7-x… Please let me know if more info is needed. /Claus A

3 2

syslog-ng in ubuntu 16.10
by Gil Bahat 15 Sep '16

15 Sep '16

Ubuntu 16.10 is now in feature freeze and there is (apparently) one bug on s390x platform that prevents v3.7.3 from making it into the release in lieu of the older 3.5.6. that's a real shame, I hope someone from this list can help out. here is the failing build log on ubuntu 16.10: https://launchpadlibrarian.net/281969183/buildlog_ubuntu- yakkety-s390x.syslog-ng-incubator_0.5.0-1_BUILDING.txt.gz this does build correctly on debian sid though: https://buildd.debian.org/status/fetch.php?pkg=syslog- ng-incubator&arch=s390x&ver=0.5.0-1&stamp=1468967484 Regards, Gil Bahat

2 1

syslog-ng 3.8.1 rpm packages
by Czanik, Péter 15 Sep '16

15 Sep '16

Hi, This week my packaging focus was to make sure, that 3.8.1 will be available as part of the next openSUSE release. My initial unofficial 3.8.1 RPM packages with Java modules, etc. are available but still need some polishing: - move curl (HTTP) module into a separate sub package - update rust modules and on Fedora / EPEL: - build Java modules without Internet access - re-enable missing features, like riemann (which were turned off to make local test builds easier) Fedora/EPEL7: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng38/ EPEL6: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng38epel6/ openSUSE/SLES: https://build.opensuse.org/project/show/home:czanik:syslog-ng38 I hope to have more time working on this by the end of next week. Any feedback, suggestions, patches are welcome Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

4 6

Insider 2016-09: 3.8.1 released; GSoC; Turris Omnia; meetups;
by Czanik, Péter 15 Sep '16

15 Sep '16

Dear syslog-ng users, This is the 51st issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng 3.8.1 released ------------------------ I am happy to announce that after almost a year of development, syslog-ng 3.8.1 has been released. This is the first stable release from the syslog-ng 3.8 branch. Version 3.8.1 adds some long-awaited features to syslog-ng and improves the performance of name-value-pair related operations. Announcement: https://www.balabit.com/blog/syslog-ng-hits-major-milestone-with-version-3-… Development, packaging related changes: https://czanik.blogs.balabit.com/2016/08/syslog-ng-3-8-what-changed/ Google Summer of Code 2016 -------------------------- The syslog-ng project has a long tradition of participating in the Google Summer of Code program. Each year some interesting new features originate from GSoC students. Here are some of the blogs, where students explain what they developed during the Summer: Websocket destination: https://litterbear.github.io/2016-08-26-GSOC2016/ Release automation: http://black-perl.in/gsoc16-syslog-ng-autorel-summary/ Kafka source: https://vithulanmv.wordpress.com/2016/08/21/syslog-ng-kafka-source-in-depth… As command line tool: https://kvch.github.io/syslog-ng-as-a-command-line-tool.html If you are at the GSoC mentor summit (Sunnyvale, California, October 28-30, https://sites.google.com/site/2016gsocmentorsummit/home) you can even meet one of our mentors, Viktor Juhász, at the event. As a syslog-ng developer, he can talk to you about syslog-ng, even get you started developing syslog-ng. Coming up: All Things Open -------------------------- My next syslog-ng presentation will be in Raleigh, NC, at the All Things Open conference. I will talk about scaling a logging infrastructure using syslog-ng: https://allthingsopen.org/talk/scaling-your-logging-infrastructure/ syslog-ng in the Turris Omnia Linux router ------------------------------------------ The Turris Omnia is a Linux-based router running on an ARM CPU. As it turned out during a conference discussion, this popular little device is running syslog-ng: https://czanik.blogs.balabit.com/2016/06/turris-omnia-a-linux-router-powere… If you know of anything that could be added to https://syslog-ng.org/powered-by-syslog-ng/, let me know! syslog-ng at meetups -------------------- There were two meetups this summer where a member of Balabit’s crew talked about syslog-ng. Both were quite a positive experience. If you know of a meetup where participants might be interested in syslog-ng, please let me know! I can be reached by e-mail or on Twitter as @PCzanik San Francisco: https://bazsi.blogs.balabit.com/2016/07/syslog-ng-in-san-francisco/ London: https://www.balabit.com/blog/linuxing-in-london/ NEW RELEASES syslog-ng 3.8.1: https://www.balabit.com/blog/syslog-ng-hits-major-milestone-with-version-3-… SSB 4F4: https://istvanszabo.blogs.balabit.com/2016/07/improve-the-efficiency-of-sec… Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

pattern_db ES use case.
by Scot Needy 12 Sep '16

12 Sep '16

Hello List, I’m trying to understand the use case of pattern_db when the destination will be ES. My initial understanding was that I could use patterndb as an engine to tag my log message data with attributes, but it doesn’t seem to work that way. I have a json output like this in Kibana. In a loghost deployment, It looks like I would need to manually align a patterndb filter with each host_message type even before patterned comes into play. Q) What is the right solution for enriching message data into ES ? Example JSON from Kibana MESSAGE is not parsed. ======================= { "_index": "syslog-ng_2016.09.12", "_type": "syslog-ng", "_id": "AVcdnzJla9VjMdxDYo8Z", "_score": null, "_source": { "PROGRAM": “###-asa11", "PRIORITY": "warning", "MESSAGE": "%ASA-4-106023: Deny tcp src outside:###.###.31.2/33553 dst public:###.###.7.191/443 by access-group \"outside_access_in\" [0x2c1c6a65, 0x0]", "ISODATE": "2016-09-12T13:57:03-04:00", "HOST": “###.###.###.###", "FACILITY": "local5", "@timestamp": "2016-09-12T13:57:03-04:00" }, "fields": { "ISODATE": [ 1473703023000 ], "@timestamp": [ 1473703023000 ] }, "sort": [ 1473703023000 ] }

2 5

Archive module ?
by Scot Needy 12 Sep '16

12 Sep '16

Hi All, Is there a better way to archive my syslog data once it’s been received and injected into elk,spunk or whatever ? I see how I can log my data to flat files using date macros then compress and archive as normal data but is there a better way to write out the data in the same format it was received to facilitate re-injesting ? For example: My syslog data on file does not include the original syslog header data . Syslog is just one example.

2 6

Re: [syslog-ng] syslog-ng Digest, Vol 137, Issue 11
by Szabó, István 09 Sep '16

09 Sep '16

Hi Scot, syslog-ng in its premium edition offers logstore: https://www.balabit.com/documents/syslog-ng-pe-6.0-guides/en/syslog-ng-pe-g… Disclaimer: I'm employed by Balabit (being the product manager of syslog-ng there) I don't want to push anything here, the only reason I wrote this is because I felt there is an exact match with what you required and by checking the above documentation you can at least see whether this is indeed the functionality what you are after. /Istvan Date: Thu, 8 Sep 2016 21:56:55 -0400 > From: Scot Needy <scotrn(a)gmail.com> > Subject: Re: [syslog-ng] Archive module ? > To: Syslog-ng users' and developers' mailing list > <syslog-ng(a)lists.balabit.hu> > Message-ID: <3E22C1C9-9275-41F4-A583-F140CAF611A6(a)gmail.com> > Content-Type: text/plain; charset="utf-8" > > > That looks like it would work but if the source isn?t syslog data ? > > I pondered this question when looking at the disk buffer settings in 3.8. > If my data is in ES, MySQL or some other parsed presentation platform then > all I really need to write out blocks of raw data to disk and archive for > SOX compliance. > > Let the storage archive, compress, encrypt and de-dup. > > Just > > > > On Sep 8, 2016, at 5:48 PM, Evan Rempel <erempel(a)uvic.ca> wrote: > > > > You would have to use a syslog-ng template defined as > > > > "<$PRI>$ISO_DATE $FULLHOST $MSGHDR$MESSAGE\n" > > > > and that would be the "on the wire" format. The you can netcat this to > the syslog port which would replay the log. > > > > Evan. > > > > On 09/08/2016 02:39 PM, Scot Needy wrote: > >> Thanks Even, > >> > >> Maybe I?m doing something wrong. > >> I see some of the headers but not all. > >> > >> 2016-09-08T17:31:17-04:00 <IPADDR> <HOST> %ASA-3-305006: regular > translation creation failed for imp src ?. > >> > >> I guess what I was looking for is something that I could write blocks > of data to and read from irregardless of the type of data. > >> Something almost like tcpdump or buffer that could rotate every hour or > so and then be used to re-play directly back into syslog-ng. > >> > >> Log > YYYY.MM.DD.HH_compressed_data. > >> > >> > >> > >> > >> > >>> On Sep 8, 2016, at 5:13 PM, Evan Rempel <erempel(a)uvic.ca> <mailto: > erempel(a)uvic.ca> wrote: > >>> > >>> If you write the syslog data to a file with all of the data, then it > can be ingested again. > >>> > >>> We save the data to the files like > >>> > >>> 2016-09-08T00:00:01-07:00 host.name.here mail.info sm-mta[9521]: > u887019I009521: milter=mimedefang, action=rcpt, continue > >>> > >>> and then we have a syslog-ng pattern database that will turn this back > into an "on the wiire" format that can be sent to any syslog port. > >>> > >>> Alternatively, you could save it in the file as the original format, > and then it can just be sent back to syslog when you need to. Human > readablility is not quite as good, but if you have the data in splunk or > whatever, then humans don't need to read the raw files. > >>> > >>> Evan. > >>> > >>> > >>> On 09/08/2016 01:32 PM, Scot Needy wrote: > >>>> Hi All, > >>>> > >>>> Is there a better way to archive my syslog data once it?s been > received and injected into elk,spunk or whatever ? > >>>> > >>>> I see how I can log my data to flat files using date macros then > compress and archive as normal data but is there a better way to write out > the data in the same format it was received to facilitate re-injesting ? > >>>> > >>>> For example: My syslog data on file does not include the original > syslog header data . > >>>> > >>>> Syslog is just one example. > >>>> > >>>> > >>> ____________________________________________________________ > __________________ > >>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng < > https://lists.balabit.hu/mailman/listinfo/syslog-ng> > >>> Documentation: http://www.balabit.com/support/documentation/? > product=syslog-ng <http://www.balabit.com/support/documentation/? > product=syslog-ng> > >>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq < > http://www.balabit.com/wiki/syslog-ng-faq> > >>> > >> ____________________________________________________________ > __________________ > >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng < > https://lists.balabit.hu/mailman/listinfo/syslog-ng> > >> Documentation: http://www.balabit.com/support/documentation/? > product=syslog-ng <http://www.balabit.com/support/documentation/? > product=syslog-ng> > >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq < > http://www.balabit.com/wiki/syslog-ng-faq> > >> > > > > > > -- > > Evan Rempel erempel(a)uvic.ca > <mailto:erempel@uvic.ca> > > Senior Systems Administrator 250.721.7691 > > Data Centre Services, University Systems, University of Victoria > > ____________________________________________________________ > __________________ > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > Documentation: http://www.balabit.com/support/documentation/? > product=syslog-ng > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > >

1 0

Help! CentOS 7 ELK Stack from repos no index data.
by Scot Needy 09 Sep '16

09 Sep '16

So, I rebuilt my home stand alone system and only used the supported 3.8 and ES 2.4 current repos. Everything seems to be up and running with no complaints but I am not getting any data in Kibana, syslog-ng_* index and I can’t figure out if it’s something in syslog-ng.conf. Getting a bunch of stuff in /var/log/network.log from my cable modem but nothing is showing in discovery HELP! I’ve looked at this for hours and tried so many variants on destination d_es client-mode, all supported template options… Only thing weird I can see is this in the es logs. data=false [2016-09-08 00:00:01,977][INFO ][cluster.metadata ] [node-1] [syslog-ng_2016.09.08] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [] [2016-09-08 00:00:02,108][INFO ][cluster.routing.allocation] [node-1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[syslog-ng_2016.09.08][0], [syslog-ng_2016.09.08][0]] ...]). [2016-09-08 00:00:02,127][INFO ][cluster.metadata ] [node-1] [syslog-ng_2016.09.08] create_mapping [syslog-ng] [2016-09-08 00:15:28,886][INFO ][cluster.service ] [node-1] removed {{node-1}{kSmxkMoZQu6ZXvWWb70L5g}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false},}, reason: zen-disco-node-left({node-1}{kSmxkMoZQu6ZXvWWb70L5g}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false}), reason(left) [2016-09-08 00:15:39,358][INFO ][cluster.service ] [node-1] added {{node-1}{Df73oV_kQCubR5MeHiJnJA}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false},}, reason: zen-disco-join(join from node[{node-1}{Df73oV_kQCubR5MeHiJnJA}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false}]) [2016-09-08 00:15:44,409][INFO ][cluster.metadata ] [node-1] [syslog-ng_2016.09.08] update_mapping [syslog-ng] [2016-09-08 00:31:14,268][INFO ][cluster.service ] [node-1] removed {{node-1}{Df73oV_kQCubR5MeHiJnJA}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false},}, reason: zen-disco-node-left({node-1}{Df73oV_kQCubR5MeHiJnJA}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false}), reason(left) [2016-09-08 00:31:19,823][INFO ][cluster.service ] [node-1] added {{node-1}{4h_70582RoWKL-jsAPzF4g}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false},}, reason: zen-disco-join(join from node[{node-1}{4h_70582RoWKL-jsAPzF4g}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false}]) [2016-09-08 00:38:29,163][INFO ][cluster.service ] [node-1] removed {{node-1}{4h_70582RoWKL-jsAPzF4g}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false},}, reason: zen-disco-node-left({node-1}{4h_70582RoWKL-jsAPzF4g}{127.0.0.1}{127.0.0.1:9301}{client=true, data=false}), reason(left) https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-repos… <https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-repos…> https://www.elastic.co/guide/en/kibana/4.6/setup-repositories.html <https://www.elastic.co/guide/en/kibana/4.6/setup-repositories.html> https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng38/repo/epel-7/czan… <https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng38/repo/epel-7/czan…> [root@meo syslog-ng]# cat /etc/syslog-ng/syslog-ng.conf @version:3.8 @include "scl.conf" @module mod-java options { flush_lines (0); time_reopen (10); log_fifo_size (1000); chain_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (yes); ts_format(iso); }; source s_netsyslog { udp(); tcp(); syslog(); }; source s_sys { system(); internal(); }; destination d_es { elasticsearch2( index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("syslog-ng") # Description: The type of the index. For example, type("test") port("9300") server("127.0.0.1") concurrent-requests("5") flush_limit("1") client-mode("transport") skip-cluster-health-check("yes") cluster("meo") custom_id("syslog-ng") resource("/etc/elasticsearch/elasticsearch.yml") client_lib_dir("/usr/share/elasticsearch/lib") concurrent_requests("100") ); }; destination d_netsyslog { file("/var/log/network.log" owner("root") group("root") perm(0644)); }; log { source(s_netsyslog); destination(d_es); }; log { source(s_sys); source(s_netsyslog); destination(d_netsyslog); }; # Source additional configuration files (.conf extension only) @include "/etc/syslog-ng/conf.d/*.conf"

3 8

Tiered storage for syslog archives.
by Scot Needy 07 Sep '16

07 Sep '16

Hi, Has anyone here looked or would like to collaborate, using tiered storage solutions with their log solution? Specifically, I am looking to use elastic search bucket retention. I know this is a syslog-ng forum but the solution is for syslog archives with Kibana as the presentation layer. Welcome any open feedback or PM. Thanks

1 0

Re: [syslog-ng] Web Visualize logs (Ivan Adji - Krstev)
by Szabó, István 07 Sep '16

07 Sep '16

Hi, For using syslog-ng together with elasticsearch and Kibana this is a pretty good description, also giving you a good understanding of what it enables you to do: https://czanik.blogs.balabit.com/2015/10/how-to-parse-data-with-syslog-ng-s… https://czanik.blogs.balabit.com/2015/12/elasticsearch-and-syslog-ng-fast-a… /Istvan

9 24