- syslog-ng - lists.balabit.hu

Beat to syslog-ng
by Czanik, Péter 07 Nov '17

07 Nov '17

Hi, Somehow related to my previous e-mail: I was asked if Filebeat can send logs directly into syslog-ng. None of the network destinations of Beat are supported by syslog-ng, but I did a quick check and we can read and parse the file destination of Beat. Filebeat module functionality can be practically replaced by syslog-ng and one of its parsers. But there are quite a few other Beat applications. Question: is there a practical use case for syslog-ng reading the file output of a Beat module? Do you use it anywhere? Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

2 1

Question about using multiple amqp destination
by Alicia Smith 06 Nov '17

06 Nov '17

I am trying to configure syslog-ng to send multiple json formatted logs to a rabbitmq endpoint. The consumer for the logs checks for a particular field to determine the type of log it is. However, I can't seem to figure out how to send all the different logs as the field it checks is "source" that I add using --pair within the format-json template destination d_amqp_ssh { amqp( vhost("/") host("mozdefqa2.private.scl3.mozilla.com") port(5672) exchange("eventtask") exchange-type("direct") routing-key("eventtask") body("$(format-json --scope nv_pairs --pair category=\"bro\" --pair source=\"ssh\" --pair customendpoint=\" \" --pair tags=\"bro\")") persistent(no) username("guest") password("guest") ); }; So I added another destination with the name d_amqp_conn and it's source field value is "conn" and the amqp endpoint is the same host as the ssh log. But syslog-ng fails to start with this particular setup. It seems to not like having more than one amqp destination. How would something like this typically be solved? -- Alicia Smith @phrozyn Information Security Engineer asmith(a)mozilla.com

3 10

Syslog-NG Statistics
by Schoonover, Mark E HHHH 01 Nov '17

01 Nov '17

Hello, I'm working on a Splunk dashboard for our NG PE servers. Looking at the stats from the output of /opt/syslog-ng/sbin/syslog-ng-ctl stats, this one has me puzzled: src.program,,49587758,d,processed,1 I don't know what "program" means as well as the 3rd column. I understand the remaining columns. What do they mean? Thanks! Regards, Mark Schoonover - KA6WKE Infrastructure Engineering Manager, Splunk Architect ENE : Tools, Instrumentation and Common Services Team Office: 32.8697° N, 116.9711° W Phone : 770-261-7934 Email : mark.schoonover(a)cigna.com<mailto:mark.schoonover@cigna.com> HPSM Team: ENE NMS Engineering Confidential, unpublished property of CIGNA. Do not duplicate or distribute. Use and distribution limited solely to authorized personnel. ¬© Copyright 2017 CIGNA ------------------------------------------------------------------------------ CONFIDENTIALITY NOTICE: If you have received this email in error, please immediately notify the sender by e-mail at the address shown. This email transmission may contain confidential information. This information is intended only for the use of the individual(s) or entity to whom it is intended even if addressed incorrectly. Please delete it from your files if you are not the intended recipient. Thank you for your compliance. Copyright (c) 2017 Cigna ==============================================================================

2 1

Re: [syslog-ng] Need to parse named query log entries and get just the domain portion
by Robert Webb 31 Oct '17

31 Oct '17

Evan, Thanks you for pointing me in the right direction. I was so focused on the matching starting @ "queries: info:" that I totally forgot that the first part of the actual MSG has the date and time sent with it. My pattern now takes that into account. Message: 27-Oct-2017 14:19:42.359 queries: info: client 137.75.76.138#38643 (sr.symcb.com) query: sr.symcb.com IN A +E (137.75.75.75) <pattern>@ESTRING:: @@ESTRING:: @queries: info: client @IPvANY:named.client.ip@#@NUMBER:named.client.port:@ @ESTRING:named.domain.name::@ query: @HOSTNAME:named.host.name:@ @STRING:@ @STRING:named.record.type:@ @ESTRING:named.record.flags: @(@IPvANY:named.server.ip@)</pattern> Robert From: Evan Rempel [mailto:erempel@uvic.ca] Sent: Friday, October 27, 2017 9:52 AM To: Robert Webb <rwebb(a)ropeguru.com> Subject: Re: [syslog-ng] Need to parse named query log entries and get just the domain portion In the past when I have had this problem it was in the patterndb portion just outside if what you have included. essentially the program does not match, so the rule is never attempted. This results in none of the tag-value pairs being populated. The pdbtool shows a match because the pattern is always in the correct program. Your patterndb portion is inside of a set of a config such as <patterndb version="4" pub_date="2009-09-01"> <ruleset name="named" id="some-unique-value"> <pattern>named</pattern> <rules> [ your included pattern db data ] </ruleset> </pattendb> It is the line <pattern>named</pattern> that matches the parsed program name from the syslog line to allow the rule to be parsed against the syslog message. For testing purposes I make a destination to write a json object. That way I can see everything that populated. template t_json { template("$(format-json --scope everything )\n"); template_escape(no); }; destination test.log { file("/var/syslog/test.log.$S_YEAR$S_MONTH$S_DAY.000000" template(t_json)); }; Evan. On 10/27/2017 05:22 AM, Robert Webb wrote: So I have a situation where I have a syslog feed coming in that is sending named (BIND9) query logs. On the DNS servers themselves, we are doing sink hole via a local zone file and sending the dns response back as 127.0.0.1. Since there is no way for BIND9 to give me logging of the DNS query response back to the user, I need to be able to take a query log entry, parse it to get just the domain portion, and then match the domain to a list. I am trying to do all this inside syslog-ng. I have been given some guidance to use a pattern database. I worked with this and am successfully matching my incoming query log data. Using the pdbtool, I get a full match and a list of the values I expect. But where I run into an issue after searching through all the documentation is, where can these values be used? I tried building a template against a destination, and none of the values populate. So I am stuck at this point. I have also been told that I may have to do a custom python parser instead. Any help or guidance is appreciated. Here is my rule: <rule provider="balabit" id="b0ca071d-309e-454a-8e31-721b01ef55ee" class="system"> <patterns> <pattern>queries: info: client @IPvANY:named.client.ip@#@NUMBER:named.client.port:@ @ESTRING:named.domain.name::@ query: @HOSTNAME:named.host.name:@ @STRING:@ @STRING:named.record.type:@ @ESTRING:named.record.flags: @(@IPvANY:named.server.ip@)</pattern> </patterns> <examples> <example> <test_message program="named">client 213.46.246.51#1890: query: tumtali.fegade.com IN A - (103.5.198.211)</test_message> <test_values> <test_value name="named.client.ip">213.46.246.51</test_value> <test_value name="named.client.port">1890</test_value> <test_value name="named.domain.name">tumtali.fegade.com</test_value> <test_value name="named.record.type">A</test_value> <test_value name="named.record.flags">-</test_value> <test_value name="named.server.ip">103.5.198.211</test_value> </test_values> </example> <example> <test_message program="named">client 10.187.195.186#40441: query: banister.paars.tld IN A -EDC (10.226.134.25)</test_message> <test_values> <test_value name="named.client.ip">10.187.195.186</test_value> <test_value name="named.client.port">40441</test_value> <test_value name="named.domain.name">banister.paars.tld</test_value> <test_value name="named.record.type">A</test_value> <test_value name="named.record.flags">-EDC</test_value> <test_value name="named.server.ip">10.226.134.25</test_value> </test_values> </example> <example> <test_message program="named">queries: client 192.168.102.189#58454: query: maps.googleapis.com IN A + (192.168.102.2)</test_message> <test_values> <test_value name="named.client.ip">192.168.102.189</test_value> <test_value name="named.client.port">58454</test_value> <test_value name="named.domain.name">maps.googleapis.com</test_value> <test_value name="named.record.type">A</test_value> <test_value name="named.record.flags">+</test_value> <test_value name="named.server.ip">192.168.102.2</test_value> </test_values> </example> </examples> <values> </values> <tags> <tag>named</tag> <tag>queries</tag> </tags> </rule> Here are test results: Pattern matching part: queries: info: client @IP:named.client.ip=137.75.76.138@#@NUMBER:named.client.port=38643@ @ESTRING:named.domain.name=(sr.symcb.com)@ query: @HOSTNAME:named.host.name=sr.symcb.com@ @STRING:None=IN@ @STRING:named.record.type=A@ @ESTRING:named.record.flags=+E@(@IP:named.server.ip=137.75.75.75@) Matching part: queries: info: client 137.75.76.138#38643 (sr.symcb.com) query: sr.symcb.com IN A +E (137.75.75.75) Values: MESSAGE=queries: info: client 137.75.76.138#38643 (sr.symcb.com) query: sr.symcb.com IN A +E (137.75.75.75) PROGRAM=named .classifier.class=system .classifier.rule_id=b0ca071d-309e-454a-8e31-721b01ef55ee named.client.ip=137.75.76.138 named.client.port=38643 named.domain.name=(sr.symcb.com) named.record.type=A named.server.ip=137.75.75.75 named.host.name=sr.symcb.com named.record.flags=+E TAGS=.classifier.system,named,queries

2 2

Syslog 3.5 (pipe file issue)
by Amin, Jitesh CTR (US) 29 Oct '17

29 Oct '17

Hi, We have the syslog-ng 3.5 installed on Linux OS. We are trying to use the pipe function (to pipe data to the log file). This pipe file is FIFO. One of the issues we are seeing is that, the file size of this pipe file is always ZERO and never grows and second the timestampe on the file never updates or changes. Though when we do a "cat pipefilename" command we do see contents of the file. Can you please let us know what we could be doing wrong or any feedback on how to resolve the issue? Thanks Jitesh

4 5

Need to parse named query log entries and get just the domain portion
by Robert Webb 27 Oct '17

27 Oct '17

So I have a situation where I have a syslog feed coming in that is sending named (BIND9) query logs. On the DNS servers themselves, we are doing sink hole via a local zone file and sending the dns response back as 127.0.0.1. Since there is no way for BIND9 to give me logging of the DNS query response back to the user, I need to be able to take a query log entry, parse it to get just the domain portion, and then match the domain to a list. I am trying to do all this inside syslog-ng. I have been given some guidance to use a pattern database. I worked with this and am successfully matching my incoming query log data. Using the pdbtool, I get a full match and a list of the values I expect. But where I run into an issue after searching through all the documentation is, where can these values be used? I tried building a template against a destination, and none of the values populate. So I am stuck at this point. I have also been told that I may have to do a custom python parser instead. Any help or guidance is appreciated. Here is my rule: <rule provider="balabit" id="b0ca071d-309e-454a-8e31-721b01ef55ee" class="system"> <patterns> <pattern>queries: info: client @IPvANY:named.client.ip@#@NUMBER:named.client.port:@ @ESTRING:named.domain.name::@ query: @HOSTNAME:named.host.name:@ @STRING:@ @STRING:named.record.type:@ @ESTRING:named.record.flags: @(@IPvANY:named.server.ip@)</pattern> </patterns> <examples> <example> <test_message program="named">client 213.46.246.51#1890: query: tumtali.fegade.com IN A - (103.5.198.211)</test_message> <test_values> <test_value name="named.client.ip">213.46.246.51</test_value> <test_value name="named.client.port">1890</test_value> <test_value name="named.domain.name">tumtali.fegade.com</test_value> <test_value name="named.record.type">A</test_value> <test_value name="named.record.flags">-</test_value> <test_value name="named.server.ip">103.5.198.211</test_value> </test_values> </example> <example> <test_message program="named">client 10.187.195.186#40441: query: banister.paars.tld IN A -EDC (10.226.134.25)</test_message> <test_values> <test_value name="named.client.ip">10.187.195.186</test_value> <test_value name="named.client.port">40441</test_value> <test_value name="named.domain.name">banister.paars.tld</test_value> <test_value name="named.record.type">A</test_value> <test_value name="named.record.flags">-EDC</test_value> <test_value name="named.server.ip">10.226.134.25</test_value> </test_values> </example> <example> <test_message program="named">queries: client 192.168.102.189#58454: query: maps.googleapis.com IN A + (192.168.102.2)</test_message> <test_values> <test_value name="named.client.ip">192.168.102.189</test_value> <test_value name="named.client.port">58454</test_value> <test_value name="named.domain.name">maps.googleapis.com</test_value> <test_value name="named.record.type">A</test_value> <test_value name="named.record.flags">+</test_value> <test_value name="named.server.ip">192.168.102.2</test_value> </test_values> </example> </examples> <values> </values> <tags> <tag>named</tag> <tag>queries</tag> </tags> </rule> Here are test results: Pattern matching part: queries: info: client @IP:named.client.ip=137.75.76.138@#@NUMBER:named.client.port=38643@ @ESTRING:named.domain.name=(sr.symcb.com)@ query: @HOSTNAME:named.host.name=sr.symcb.com@ @STRING:None=IN@ @STRING:named.record.type=A@ @ESTRING:named.record.flags=+E@(@IP:named.server.ip=137.75.75.75@) Matching part: queries: info: client 137.75.76.138#38643 (sr.symcb.com) query: sr.symcb.com IN A +E (137.75.75.75) Values: MESSAGE=queries: info: client 137.75.76.138#38643 (sr.symcb.com) query: sr.symcb.com IN A +E (137.75.75.75) PROGRAM=named .classifier.class=system .classifier.rule_id=b0ca071d-309e-454a-8e31-721b01ef55ee named.client.ip=137.75.76.138 named.client.port=38643 named.domain.name=(sr.symcb.com) named.record.type=A named.server.ip=137.75.75.75 named.host.name=sr.symcb.com named.record.flags=+E TAGS=.classifier.system,named,queries

1 0

test email
by Scheidler, Balázs 27 Oct '17

27 Oct '17

Hi, This is just to test if email sending to a new subscriber works as intended. -- Bazsi

1 0

syslog-ng stats to ES ?
by Scot 26 Oct '17

26 Oct '17

Hi, Looked around for a few hours and didn't see anything. Has anyone worked on sending syslog-ng stats to ES ? I see several ways I could but wondering if anyone has already. A push method directly from syslog-ng would be awesome. Scot

4 13

Insider 2017-10: Hadoop; web interfaces; OSE to PE; vDay;
by Czanik, Péter 24 Oct '17

24 Oct '17

Dear syslog-ng users, This is the 62nd issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Hadoop destination ------------------ Petabytes of data are now collected into huge data lakes around the world. Hadoop is the technology enabling this. While syslog-ng was able write logs to Hadoop using some workarounds (mounting HDFS through FUSE) for quite some time, the Java-based HDFS destination driver in syslog-ng provides both better performance and more reliability. Learn about HDFS support in syslog-ng and about features introduced in recent versions: https://www.balabit.com/blog/filling-your-data-lake-with-log-messages-the-s… Web interfaces for syslog-ng ---------------------------- This is the 2017 edition of syslog-ng web-based Graphical User Interfaces (web GUIs) blog. Many things have changed in the past few years. In 2011 only a single Logging as a service (LaaS) solution was available, now we regularly run into yet another one. Also, the number of logging-related GUIs is growing, so we will mostly focus on syslog-ng based solutions. https://www.balabit.com/blog/web-interfaces-for-your-syslog-server-an-overv… Upgrading from syslog-ng OSE to PE ---------------------------------- The syslog-ng application has two different editions. Most of my readers use syslog-ng Open Source Edition (OSE). There is also a commercial version of syslog-ng, called syslog-ng Premium Edition (PE). It comes with a few extra features, as well as commercial support. Upgrading from OSE to PE or the other way around is not automated and not even always possible. This is due to feature set differences, OSE and PE share a common core but have a different focus. Learn about some of the limitations and some tips about upgrading. https://www.balabit.com/blog/upgrading-from-syslog-ng-open-source-to-premiu… EVENTS Virtualization day in Budapest, November 7th: https://vday.hu/2017/ NEW RELEASES: syslog-ng OSE 3.12.1: https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.12.1 syslog-ng PE 7.0.5: https://www.balabit.com/documents/syslog-ng-pe-7.0-guides/en/syslog-ng-pe-g… Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

collecting SAR data
by Jim Hendrick 17 Oct '17

17 Oct '17

Hi, Does anyone have experience pulling performance data from sysstat (sar) with syslog-ng (or even *cough* rsyslog...) Looking to automate the performance metrics and it seems like it should be straightforward. What I would like is to pull the text output that is generated every ~10 minutes by default without having to run a different cron across all the systems. (that is - I get that I could probably modify the "sa1" script or write another one to dump the data to a socket or something for syslog to pick up. I am hoping to just do something to monitor the data that is automatically generated. - something like using any file in /var/log/sa/sar?? as a source) The end system (for me at the moment) would be splunk so parsing the different format data *could* be pushed to that end or could be done with patterndb or something else on the syslog-ng side. Thoughts? Thanks in advance!! Jim

1 0