- syslog-ng - lists.balabit.hu

Need to parse named query log entries and get just the domain portion
by Robert Webb 27 Oct '17

27 Oct '17

So I have a situation where I have a syslog feed coming in that is sending named (BIND9) query logs. On the DNS servers themselves, we are doing sink hole via a local zone file and sending the dns response back as 127.0.0.1. Since there is no way for BIND9 to give me logging of the DNS query response back to the user, I need to be able to take a query log entry, parse it to get just the domain portion, and then match the domain to a list. I am trying to do all this inside syslog-ng. I have been given some guidance to use a pattern database. I worked with this and am successfully matching my incoming query log data. Using the pdbtool, I get a full match and a list of the values I expect. But where I run into an issue after searching through all the documentation is, where can these values be used? I tried building a template against a destination, and none of the values populate. So I am stuck at this point. I have also been told that I may have to do a custom python parser instead. Any help or guidance is appreciated. Here is my rule: <rule provider="balabit" id="b0ca071d-309e-454a-8e31-721b01ef55ee" class="system"> <patterns> <pattern>queries: info: client @IPvANY:named.client.ip@#@NUMBER:named.client.port:@ @ESTRING:named.domain.name::@ query: @HOSTNAME:named.host.name:@ @STRING:@ @STRING:named.record.type:@ @ESTRING:named.record.flags: @(@IPvANY:named.server.ip@)</pattern> </patterns> <examples> <example> <test_message program="named">client 213.46.246.51#1890: query: tumtali.fegade.com IN A - (103.5.198.211)</test_message> <test_values> <test_value name="named.client.ip">213.46.246.51</test_value> <test_value name="named.client.port">1890</test_value> <test_value name="named.domain.name">tumtali.fegade.com</test_value> <test_value name="named.record.type">A</test_value> <test_value name="named.record.flags">-</test_value> <test_value name="named.server.ip">103.5.198.211</test_value> </test_values> </example> <example> <test_message program="named">client 10.187.195.186#40441: query: banister.paars.tld IN A -EDC (10.226.134.25)</test_message> <test_values> <test_value name="named.client.ip">10.187.195.186</test_value> <test_value name="named.client.port">40441</test_value> <test_value name="named.domain.name">banister.paars.tld</test_value> <test_value name="named.record.type">A</test_value> <test_value name="named.record.flags">-EDC</test_value> <test_value name="named.server.ip">10.226.134.25</test_value> </test_values> </example> <example> <test_message program="named">queries: client 192.168.102.189#58454: query: maps.googleapis.com IN A + (192.168.102.2)</test_message> <test_values> <test_value name="named.client.ip">192.168.102.189</test_value> <test_value name="named.client.port">58454</test_value> <test_value name="named.domain.name">maps.googleapis.com</test_value> <test_value name="named.record.type">A</test_value> <test_value name="named.record.flags">+</test_value> <test_value name="named.server.ip">192.168.102.2</test_value> </test_values> </example> </examples> <values> </values> <tags> <tag>named</tag> <tag>queries</tag> </tags> </rule> Here are test results: Pattern matching part: queries: info: client @IP:named.client.ip=137.75.76.138@#@NUMBER:named.client.port=38643@ @ESTRING:named.domain.name=(sr.symcb.com)@ query: @HOSTNAME:named.host.name=sr.symcb.com@ @STRING:None=IN@ @STRING:named.record.type=A@ @ESTRING:named.record.flags=+E@(@IP:named.server.ip=137.75.75.75@) Matching part: queries: info: client 137.75.76.138#38643 (sr.symcb.com) query: sr.symcb.com IN A +E (137.75.75.75) Values: MESSAGE=queries: info: client 137.75.76.138#38643 (sr.symcb.com) query: sr.symcb.com IN A +E (137.75.75.75) PROGRAM=named .classifier.class=system .classifier.rule_id=b0ca071d-309e-454a-8e31-721b01ef55ee named.client.ip=137.75.76.138 named.client.port=38643 named.domain.name=(sr.symcb.com) named.record.type=A named.server.ip=137.75.75.75 named.host.name=sr.symcb.com named.record.flags=+E TAGS=.classifier.system,named,queries

1 0

test email
by Scheidler, Balázs 27 Oct '17

27 Oct '17

Hi, This is just to test if email sending to a new subscriber works as intended. -- Bazsi

1 0

syslog-ng stats to ES ?
by Scot 26 Oct '17

26 Oct '17

Hi, Looked around for a few hours and didn't see anything. Has anyone worked on sending syslog-ng stats to ES ? I see several ways I could but wondering if anyone has already. A push method directly from syslog-ng would be awesome. Scot

4 13

Insider 2017-10: Hadoop; web interfaces; OSE to PE; vDay;
by Czanik, Péter 24 Oct '17

24 Oct '17

Dear syslog-ng users, This is the 62nd issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Hadoop destination ------------------ Petabytes of data are now collected into huge data lakes around the world. Hadoop is the technology enabling this. While syslog-ng was able write logs to Hadoop using some workarounds (mounting HDFS through FUSE) for quite some time, the Java-based HDFS destination driver in syslog-ng provides both better performance and more reliability. Learn about HDFS support in syslog-ng and about features introduced in recent versions: https://www.balabit.com/blog/filling-your-data-lake-with-log-messages-the-s… Web interfaces for syslog-ng ---------------------------- This is the 2017 edition of syslog-ng web-based Graphical User Interfaces (web GUIs) blog. Many things have changed in the past few years. In 2011 only a single Logging as a service (LaaS) solution was available, now we regularly run into yet another one. Also, the number of logging-related GUIs is growing, so we will mostly focus on syslog-ng based solutions. https://www.balabit.com/blog/web-interfaces-for-your-syslog-server-an-overv… Upgrading from syslog-ng OSE to PE ---------------------------------- The syslog-ng application has two different editions. Most of my readers use syslog-ng Open Source Edition (OSE). There is also a commercial version of syslog-ng, called syslog-ng Premium Edition (PE). It comes with a few extra features, as well as commercial support. Upgrading from OSE to PE or the other way around is not automated and not even always possible. This is due to feature set differences, OSE and PE share a common core but have a different focus. Learn about some of the limitations and some tips about upgrading. https://www.balabit.com/blog/upgrading-from-syslog-ng-open-source-to-premiu… EVENTS Virtualization day in Budapest, November 7th: https://vday.hu/2017/ NEW RELEASES: syslog-ng OSE 3.12.1: https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.12.1 syslog-ng PE 7.0.5: https://www.balabit.com/documents/syslog-ng-pe-7.0-guides/en/syslog-ng-pe-g… Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

collecting SAR data
by Jim Hendrick 17 Oct '17

17 Oct '17

Hi, Does anyone have experience pulling performance data from sysstat (sar) with syslog-ng (or even *cough* rsyslog...) Looking to automate the performance metrics and it seems like it should be straightforward. What I would like is to pull the text output that is generated every ~10 minutes by default without having to run a different cron across all the systems. (that is - I get that I could probably modify the "sa1" script or write another one to dump the data to a socket or something for syslog to pick up. I am hoping to just do something to monitor the data that is automatically generated. - something like using any file in /var/log/sa/sar?? as a source) The end system (for me at the moment) would be splunk so parsing the different format data *could* be pushed to that end or could be done with patterndb or something else on the syslog-ng side. Thoughts? Thanks in advance!! Jim

1 0

Re: [syslog-ng] [syslog
by james.r.hendrick 17 Oct '17

17 Oct '17

Do you mean the internal stats facility?I would look into the elasticsearch native destination if you are not already. Jim Sent from my Verizon, Samsung Galaxy smartphone -------- Original message --------From: Evan Rempel <erempel(a)uvic.ca> Date: 10/16/17 11:01 AM (GMT-05:00) To: syslog-ng(a)lists.balabit.hu Subject: Re: [syslog I have a perl script that collects some stats and logs them to syslog again. The syslog stream gets sent to ES, so they end up there, but as a syslog line, not a specific statistic item for things like grafana. On 10/15/2017 05:57 PM, Scot wrote: > Hi, > > Looked around for a few hours and didn't see anything. > > Has anyone worked on sending syslog-ng stats to ES ? > I see several ways I could but wondering if anyone has already. A push method directly from syslog-ng would be awesome. > > Scot > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

3 2

Re: [syslog-ng] [syslog
by james.r.hendrick 17 Oct '17

17 Oct '17

Yeah, probably separate source / dest and maybe parsed for better formatting.Send it in json perhaps. Sent from my Verizon, Samsung Galaxy smartphone -------- Original message --------From: Scot <scotrn(a)gmail.com> Date: 10/16/17 6:24 PM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] [syslog Yep but how do you get that to a d_es destination without adding an external tool ? My understanding of the purpose is that stats may stop getting updated if the pipeline fails so It needs to be a separate "thread" but seems like that would be handled in separate src and destination definitions ? i.e. Keep your stats/monitoring separate from the data stream. On Mon, Oct 16, 2017 at 11:43 AM, james.r.hendrick <james.r.hendrick(a)gmail.com> wrote: Do you mean the internal stats facility?I would look into the elasticsearch native destination if you are not already. Jim Sent from my Verizon, Samsung Galaxy smartphone -------- Original message --------From: Evan Rempel <erempel(a)uvic.ca> Date: 10/16/17 11:01 AM (GMT-05:00) To: syslog-ng(a)lists.balabit.hu Subject: Re: [syslog I have a perl script that collects some stats and logs them to syslog again. The syslog stream gets sent to ES, so they end up there, but as a syslog line, not a specific statistic item for things like grafana. On 10/15/2017 05:57 PM, Scot wrote: > Hi, > > Looked around for a few hours and didn't see anything. > > Has anyone worked on sending syslog-ng stats to ES ? > I see several ways I could but wondering if anyone has already. A push method directly from syslog-ng would be awesome. > > Scot > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

$FULLHOST doesn't work
by Thomas Haupt 16 Oct '17

16 Oct '17

Hi, I'm trying to use the hostname from inside the syslog message as filename but it doesn't work for me: https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-… my config: @version: 3.9 options { keep_hostname(yes); use-dns(no); }; source s_system { udp(ip("0.0.0.0") port(514)); }; destination d_all { file("/tmp/${FULLHOST}.log"); }; log { source(s_system); destination(d_all); }; The result is my ip address instead of "my-hostname", but I expect "my-hostname": syslog-ng -Fevd .... [2017-10-16T10:32:15.188058] Incoming log entry; line='<14>1 2017-10-16T10:32:15.151157+02:00 my-hostname lampelogtest 54321 111111 - Test syslog message' [2017-10-16T10:32:15.188327] Initializing destination file writer; template='/tmp/${FULLHOST}.log', filename='/tmp/10.3.25.4.log' Some more information: syslog-ng --version syslog-ng 3.9.1 Installer-Version: 3.9.1 Revision: Module-Directory: /usr/lib64/syslog-ng Module-Path: /usr/lib64/syslog-ng Available-Modules: add-contextual-data,afamqp,affile,afprog,afsocket,afstomp,afuser,basicfuncs,cef,confgen,cryptofuncs,csvparser,date,dbparser,disk-buffer,graphite,json-plugin,kvformat,linux-kmsg-format,pseudofile,syslogformat,system-source,tfgetent Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: off The system is a SLES11SP4 Regards, Thomas

4 4

in-list filter with MESSAGE
by Gopi Joshi 15 Oct '17

15 Oct '17

I am trying to filter messages matching text stored in a txt file (plain txt , exact match , one word each line). but its not working filter f_userlist { in-list("/etc/syslog-ng/userlist.list", value("MESSAGE")); }; ---> NOT WORKING however it works with value(“PROGRAM”) filter f_whitelist { in-list("/etc/syslog-ng/programlist.list", value("PROGRAM")); }; --->WORKING List ( userlist.list ) is not long and has less than 10 words to match. anything missing ? or in-list filter doenot work with message contents . any troubleshooting tips will e helpful.

4 4

unofficial syslog-ng packages: OBS status
by Laszlo Budai 14 Oct '17

14 Oct '17

Hi, if you get 'connection failed' error while you are trying to access the OBS repository, please check status of OBS : https://status.opensuse.org/ openSUSE status page<https://status.opensuse.org/> Stay up to date with the latest service updates from openSUSE status page. status.opensuse.org regards, Laszlo Budai

1 0