October 2017 - syslog-ng - lists.balabit.hu

Timezone
by Sebastian Roland 01 Dec '17

01 Dec '17

Hi, after reading the admin guide and playing around with different setups several times I'm still struggling to fully understand the timezone functionality of syslog-ng. How are time-zone() / recv-time-zone() and send-time-zone() related? Some notes I made during investigation: * Logging through syslog() function logs in old BSD syslog format which does not contain a timezone. recv-time-zone() is utilized to assign a timezone. If no value has been specified the local time zone is used. * According to the admin guide send-time-zone() is only used when the timezone is not specified otherwise. This didn't turned out to be true. Example: Syslog server a sends via syslog protocol over tcp (timezone is part of the message) to server b. setting send-time-zone(x) on server b changes the timezone (and timestamp) in the destination file to the time in timezone x. If send-time-zone() is not set at all nothing happens although the admin guide states that the default is to use the local timezone. IMHO no change should be applied to the message. Note that keep- timestamp(yes) is set on server b. * If both time-zone() and send-time-zone() are set globally time-zone() overrides send-time-zone() * time-zone() can be set globally and on drivers. Specific settings overrides global config. The confusing part is the behavior when a timestamp is already set within an incoming message and send-time-zone() is explicitly set (with keep-timestamp(yes)). Is it actually intended that send-time-zone() changes a timestamp? Shouldn't the logic be that recv-time-zone() and send-time-zone() are only relevant when there is no tz offset available and a default one needs to be set for receiving and sending respectively and time-zone() is used to actually convert to a different timezone? If I'm getting something fundamentally wrong please advice. Kind regards Sebastian

2 1

Install Issue
by Elphick, Matthew Thomas 08 Nov '17

08 Nov '17

Hello, I am trying to install syslog-ng OSE on RHEL 7 but I get the following error. Transaction check error: file /etc/logrotate.d/syslog from install of syslog-ng-3.12.1-2.el7.centos.x86_64 conflicts with file from package rsyslog-8.24.0-12.el7.x86_64 I can't remove rsyslog from the system as other people us it. What is the best way to resolve this issue? Thanks for your help. Matthew Elphick, Network Engineering Marsh & McLennan Companies Global Technology Infrastructure (MGTI) | Enterprise Networking Services Tower Place, London, EC3R 5BU O +44 (0)20 73573667 | M +44 (0)7392 122399 | matthew.elphick(a)mmc.com<mailto:matthew.elphick@mmc.com> ________________________________ Marsh Ltd. Registered in England and Wales Number: 1507274 Registered office 1 Tower Place West, Tower Place, London, EC3R 5BU. Marsh Ltd is authorised and regulated by the Financial Conduct Authority. This message and any attachments are confidential. If you have received this message in error please delete it from your system. If you require any assistance please notify the sender. Thank you.

4 6

any limitation with in-list filter
by Gopi Joshi 07 Nov '17

07 Nov '17

I am using in-list filter to match logs for specific users , these specific users are saved in a plain text file . is there any restriction or limitation on number of lines it can match in one file ? I currently have less than 10 users to compare with but this list is going to increase . Regards Gopi Joshi

3 2

Syslog-NG Statistics
by Schoonover, Mark E HHHH 01 Nov '17

01 Nov '17

Hello, I'm working on a Splunk dashboard for our NG PE servers. Looking at the stats from the output of /opt/syslog-ng/sbin/syslog-ng-ctl stats, this one has me puzzled: src.program,,49587758,d,processed,1 I don't know what "program" means as well as the 3rd column. I understand the remaining columns. What do they mean? Thanks! Regards, Mark Schoonover - KA6WKE Infrastructure Engineering Manager, Splunk Architect ENE : Tools, Instrumentation and Common Services Team Office: 32.8697° N, 116.9711° W Phone : 770-261-7934 Email : mark.schoonover(a)cigna.com<mailto:mark.schoonover@cigna.com> HPSM Team: ENE NMS Engineering Confidential, unpublished property of CIGNA. Do not duplicate or distribute. Use and distribution limited solely to authorized personnel. ¬© Copyright 2017 CIGNA ------------------------------------------------------------------------------ CONFIDENTIALITY NOTICE: If you have received this email in error, please immediately notify the sender by e-mail at the address shown. This email transmission may contain confidential information. This information is intended only for the use of the individual(s) or entity to whom it is intended even if addressed incorrectly. Please delete it from your files if you are not the intended recipient. Thank you for your compliance. Copyright (c) 2017 Cigna ==============================================================================

2 1

Re: [syslog-ng] Need to parse named query log entries and get just the domain portion
by Robert Webb 31 Oct '17

31 Oct '17

Evan, Thanks you for pointing me in the right direction. I was so focused on the matching starting @ "queries: info:" that I totally forgot that the first part of the actual MSG has the date and time sent with it. My pattern now takes that into account. Message: 27-Oct-2017 14:19:42.359 queries: info: client 137.75.76.138#38643 (sr.symcb.com) query: sr.symcb.com IN A +E (137.75.75.75) <pattern>@ESTRING:: @@ESTRING:: @queries: info: client @IPvANY:named.client.ip@#@NUMBER:named.client.port:@ @ESTRING:named.domain.name::@ query: @HOSTNAME:named.host.name:@ @STRING:@ @STRING:named.record.type:@ @ESTRING:named.record.flags: @(@IPvANY:named.server.ip@)</pattern> Robert From: Evan Rempel [mailto:erempel@uvic.ca] Sent: Friday, October 27, 2017 9:52 AM To: Robert Webb <rwebb(a)ropeguru.com> Subject: Re: [syslog-ng] Need to parse named query log entries and get just the domain portion In the past when I have had this problem it was in the patterndb portion just outside if what you have included. essentially the program does not match, so the rule is never attempted. This results in none of the tag-value pairs being populated. The pdbtool shows a match because the pattern is always in the correct program. Your patterndb portion is inside of a set of a config such as <patterndb version="4" pub_date="2009-09-01"> <ruleset name="named" id="some-unique-value"> <pattern>named</pattern> <rules> [ your included pattern db data ] </ruleset> </pattendb> It is the line <pattern>named</pattern> that matches the parsed program name from the syslog line to allow the rule to be parsed against the syslog message. For testing purposes I make a destination to write a json object. That way I can see everything that populated. template t_json { template("$(format-json --scope everything )\n"); template_escape(no); }; destination test.log { file("/var/syslog/test.log.$S_YEAR$S_MONTH$S_DAY.000000" template(t_json)); }; Evan. On 10/27/2017 05:22 AM, Robert Webb wrote: So I have a situation where I have a syslog feed coming in that is sending named (BIND9) query logs. On the DNS servers themselves, we are doing sink hole via a local zone file and sending the dns response back as 127.0.0.1. Since there is no way for BIND9 to give me logging of the DNS query response back to the user, I need to be able to take a query log entry, parse it to get just the domain portion, and then match the domain to a list. I am trying to do all this inside syslog-ng. I have been given some guidance to use a pattern database. I worked with this and am successfully matching my incoming query log data. Using the pdbtool, I get a full match and a list of the values I expect. But where I run into an issue after searching through all the documentation is, where can these values be used? I tried building a template against a destination, and none of the values populate. So I am stuck at this point. I have also been told that I may have to do a custom python parser instead. Any help or guidance is appreciated. Here is my rule: <rule provider="balabit" id="b0ca071d-309e-454a-8e31-721b01ef55ee" class="system"> <patterns> <pattern>queries: info: client @IPvANY:named.client.ip@#@NUMBER:named.client.port:@ @ESTRING:named.domain.name::@ query: @HOSTNAME:named.host.name:@ @STRING:@ @STRING:named.record.type:@ @ESTRING:named.record.flags: @(@IPvANY:named.server.ip@)</pattern> </patterns> <examples> <example> <test_message program="named">client 213.46.246.51#1890: query: tumtali.fegade.com IN A - (103.5.198.211)</test_message> <test_values> <test_value name="named.client.ip">213.46.246.51</test_value> <test_value name="named.client.port">1890</test_value> <test_value name="named.domain.name">tumtali.fegade.com</test_value> <test_value name="named.record.type">A</test_value> <test_value name="named.record.flags">-</test_value> <test_value name="named.server.ip">103.5.198.211</test_value> </test_values> </example> <example> <test_message program="named">client 10.187.195.186#40441: query: banister.paars.tld IN A -EDC (10.226.134.25)</test_message> <test_values> <test_value name="named.client.ip">10.187.195.186</test_value> <test_value name="named.client.port">40441</test_value> <test_value name="named.domain.name">banister.paars.tld</test_value> <test_value name="named.record.type">A</test_value> <test_value name="named.record.flags">-EDC</test_value> <test_value name="named.server.ip">10.226.134.25</test_value> </test_values> </example> <example> <test_message program="named">queries: client 192.168.102.189#58454: query: maps.googleapis.com IN A + (192.168.102.2)</test_message> <test_values> <test_value name="named.client.ip">192.168.102.189</test_value> <test_value name="named.client.port">58454</test_value> <test_value name="named.domain.name">maps.googleapis.com</test_value> <test_value name="named.record.type">A</test_value> <test_value name="named.record.flags">+</test_value> <test_value name="named.server.ip">192.168.102.2</test_value> </test_values> </example> </examples> <values> </values> <tags> <tag>named</tag> <tag>queries</tag> </tags> </rule> Here are test results: Pattern matching part: queries: info: client @IP:named.client.ip=137.75.76.138@#@NUMBER:named.client.port=38643@ @ESTRING:named.domain.name=(sr.symcb.com)@ query: @HOSTNAME:named.host.name=sr.symcb.com@ @STRING:None=IN@ @STRING:named.record.type=A@ @ESTRING:named.record.flags=+E@(@IP:named.server.ip=137.75.75.75@) Matching part: queries: info: client 137.75.76.138#38643 (sr.symcb.com) query: sr.symcb.com IN A +E (137.75.75.75) Values: MESSAGE=queries: info: client 137.75.76.138#38643 (sr.symcb.com) query: sr.symcb.com IN A +E (137.75.75.75) PROGRAM=named .classifier.class=system .classifier.rule_id=b0ca071d-309e-454a-8e31-721b01ef55ee named.client.ip=137.75.76.138 named.client.port=38643 named.domain.name=(sr.symcb.com) named.record.type=A named.server.ip=137.75.75.75 named.host.name=sr.symcb.com named.record.flags=+E TAGS=.classifier.system,named,queries

2 2

Syslog 3.5 (pipe file issue)
by Amin, Jitesh CTR (US) 29 Oct '17

29 Oct '17

Hi, We have the syslog-ng 3.5 installed on Linux OS. We are trying to use the pipe function (to pipe data to the log file). This pipe file is FIFO. One of the issues we are seeing is that, the file size of this pipe file is always ZERO and never grows and second the timestampe on the file never updates or changes. Though when we do a "cat pipefilename" command we do see contents of the file. Can you please let us know what we could be doing wrong or any feedback on how to resolve the issue? Thanks Jitesh

4 5

Need to parse named query log entries and get just the domain portion
by Robert Webb 27 Oct '17

27 Oct '17

So I have a situation where I have a syslog feed coming in that is sending named (BIND9) query logs. On the DNS servers themselves, we are doing sink hole via a local zone file and sending the dns response back as 127.0.0.1. Since there is no way for BIND9 to give me logging of the DNS query response back to the user, I need to be able to take a query log entry, parse it to get just the domain portion, and then match the domain to a list. I am trying to do all this inside syslog-ng. I have been given some guidance to use a pattern database. I worked with this and am successfully matching my incoming query log data. Using the pdbtool, I get a full match and a list of the values I expect. But where I run into an issue after searching through all the documentation is, where can these values be used? I tried building a template against a destination, and none of the values populate. So I am stuck at this point. I have also been told that I may have to do a custom python parser instead. Any help or guidance is appreciated. Here is my rule: <rule provider="balabit" id="b0ca071d-309e-454a-8e31-721b01ef55ee" class="system"> <patterns> <pattern>queries: info: client @IPvANY:named.client.ip@#@NUMBER:named.client.port:@ @ESTRING:named.domain.name::@ query: @HOSTNAME:named.host.name:@ @STRING:@ @STRING:named.record.type:@ @ESTRING:named.record.flags: @(@IPvANY:named.server.ip@)</pattern> </patterns> <examples> <example> <test_message program="named">client 213.46.246.51#1890: query: tumtali.fegade.com IN A - (103.5.198.211)</test_message> <test_values> <test_value name="named.client.ip">213.46.246.51</test_value> <test_value name="named.client.port">1890</test_value> <test_value name="named.domain.name">tumtali.fegade.com</test_value> <test_value name="named.record.type">A</test_value> <test_value name="named.record.flags">-</test_value> <test_value name="named.server.ip">103.5.198.211</test_value> </test_values> </example> <example> <test_message program="named">client 10.187.195.186#40441: query: banister.paars.tld IN A -EDC (10.226.134.25)</test_message> <test_values> <test_value name="named.client.ip">10.187.195.186</test_value> <test_value name="named.client.port">40441</test_value> <test_value name="named.domain.name">banister.paars.tld</test_value> <test_value name="named.record.type">A</test_value> <test_value name="named.record.flags">-EDC</test_value> <test_value name="named.server.ip">10.226.134.25</test_value> </test_values> </example> <example> <test_message program="named">queries: client 192.168.102.189#58454: query: maps.googleapis.com IN A + (192.168.102.2)</test_message> <test_values> <test_value name="named.client.ip">192.168.102.189</test_value> <test_value name="named.client.port">58454</test_value> <test_value name="named.domain.name">maps.googleapis.com</test_value> <test_value name="named.record.type">A</test_value> <test_value name="named.record.flags">+</test_value> <test_value name="named.server.ip">192.168.102.2</test_value> </test_values> </example> </examples> <values> </values> <tags> <tag>named</tag> <tag>queries</tag> </tags> </rule> Here are test results: Pattern matching part: queries: info: client @IP:named.client.ip=137.75.76.138@#@NUMBER:named.client.port=38643@ @ESTRING:named.domain.name=(sr.symcb.com)@ query: @HOSTNAME:named.host.name=sr.symcb.com@ @STRING:None=IN@ @STRING:named.record.type=A@ @ESTRING:named.record.flags=+E@(@IP:named.server.ip=137.75.75.75@) Matching part: queries: info: client 137.75.76.138#38643 (sr.symcb.com) query: sr.symcb.com IN A +E (137.75.75.75) Values: MESSAGE=queries: info: client 137.75.76.138#38643 (sr.symcb.com) query: sr.symcb.com IN A +E (137.75.75.75) PROGRAM=named .classifier.class=system .classifier.rule_id=b0ca071d-309e-454a-8e31-721b01ef55ee named.client.ip=137.75.76.138 named.client.port=38643 named.domain.name=(sr.symcb.com) named.record.type=A named.server.ip=137.75.75.75 named.host.name=sr.symcb.com named.record.flags=+E TAGS=.classifier.system,named,queries

1 0

test email
by Scheidler, Balázs 27 Oct '17

27 Oct '17

Hi, This is just to test if email sending to a new subscriber works as intended. -- Bazsi

1 0

syslog-ng stats to ES ?
by Scot 26 Oct '17

26 Oct '17

Hi, Looked around for a few hours and didn't see anything. Has anyone worked on sending syslog-ng stats to ES ? I see several ways I could but wondering if anyone has already. A push method directly from syslog-ng would be awesome. Scot

4 13

Insider 2017-10: Hadoop; web interfaces; OSE to PE; vDay;
by Czanik, Péter 24 Oct '17

24 Oct '17

Dear syslog-ng users, This is the 62nd issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Hadoop destination ------------------ Petabytes of data are now collected into huge data lakes around the world. Hadoop is the technology enabling this. While syslog-ng was able write logs to Hadoop using some workarounds (mounting HDFS through FUSE) for quite some time, the Java-based HDFS destination driver in syslog-ng provides both better performance and more reliability. Learn about HDFS support in syslog-ng and about features introduced in recent versions: https://www.balabit.com/blog/filling-your-data-lake-with-log-messages-the-s… Web interfaces for syslog-ng ---------------------------- This is the 2017 edition of syslog-ng web-based Graphical User Interfaces (web GUIs) blog. Many things have changed in the past few years. In 2011 only a single Logging as a service (LaaS) solution was available, now we regularly run into yet another one. Also, the number of logging-related GUIs is growing, so we will mostly focus on syslog-ng based solutions. https://www.balabit.com/blog/web-interfaces-for-your-syslog-server-an-overv… Upgrading from syslog-ng OSE to PE ---------------------------------- The syslog-ng application has two different editions. Most of my readers use syslog-ng Open Source Edition (OSE). There is also a commercial version of syslog-ng, called syslog-ng Premium Edition (PE). It comes with a few extra features, as well as commercial support. Upgrading from OSE to PE or the other way around is not automated and not even always possible. This is due to feature set differences, OSE and PE share a common core but have a different focus. Learn about some of the limitations and some tips about upgrading. https://www.balabit.com/blog/upgrading-from-syslog-ng-open-source-to-premiu… EVENTS Virtualization day in Budapest, November 7th: https://vday.hu/2017/ NEW RELEASES: syslog-ng OSE 3.12.1: https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.12.1 syslog-ng PE 7.0.5: https://www.balabit.com/documents/syslog-ng-pe-7.0-guides/en/syslog-ng-pe-g… Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0