- syslog-ng - lists.balabit.hu

Cisco ACS logs truncated
by Scot 15 Nov '17

15 Nov '17

Hi, Has anyone worked with ACS logs and solved the message header limit ? We can get syslog working but as expected the message gets truncated. Local logs on the ACS have the entire payload. Thinking there may be a way to script a log fetch or something. Thanks Scot

4 8

(no subject)
by craig bowser 15 Nov '17

15 Nov '17

I installed syslog-ng on an offline CentOS 6.x machine by first downloading all required rpms using this command: yumdownloader --resolve syslog-ng There were two files needed, that did not download and I downloaded them separately using: yumdownloader eventlog-0.2.13-1.el6.x86_64.rpm and: yumdownloader libnet-1.1.6-7.el6.x86_64.rpm These are the files I moved to the offline server: 17,684 eventlog-0.2.13-1.el6.i686.rpm 17,804 eventlog-0.2.13-1.el6.x86_64.rpm 122,940 gamin-0.1.10-9.el6.i686.rpm 1,738,964 glib2-2.28.8-9.el6.i686.rpm 4,570,040 glibc-2.12-1.209.el6_9.2.i686.rpm 117,400 libgcc-4.4.7-18.el6.i686.rpm 60,464 libnet-1.1.6-7.el6.i686.rpm 59,196 libnet-1.1.6-7.el6.x86_64.rpm 111,688 libselinux-2.0.94-7.el6.i686.rpm 309,920 libstdc++-4.4.7-18.el6.i686.rpm 161,168 nss-softokn-freebl-3.14.3-23.3.el6_8.i686.rpm 201,096 pcre-7.8-7.el6.i686.rpm 452,548 syslog-ng-3.2.5-4.el6.i686.rpm 451,092 syslog-ng-3.2.5-4.el6.x86_64.rpm 63,536 tcp_wrappers-libs-7.6-58.el6.i686.rpm 74,284 zlib-1.2.3-29.el6.i686.rpm Once I moved all these files to the offline server into their own folder, I ran: rpm –ivh *.rpm to install. I then checked the config file, made a couple of tweaks, then started syslog-ng using: service syslog-ng start I got the following errors: Plugin module not found in 'module-path'; module-path='/lib64/syslog-ng', module='afsql' Plugin module not found in 'module-path'; module-path='/lib64/syslog-ng', module='afsql' [ OK ] Syslog-ng was now running, just with these two errors. A quick google pointed to the problem, syslog-ng is looking for the sql plugin and the solution: yum install syslog-ng-libdbi But when I tried to run yumdownloader syslog-ng-libdbi I got "Nothing to download" And when I tried yum install --downloadonly syslog-ng-libdbi I got "Nothing to do" So here are my questions: 1. Will syslog-ng run just fine receiving udp and sending udp with those errors and without that module? If, so, I don’t care if I don’t get that module. 2. Is it possible to run syslog-ng and disable that module in the run command? 3. Can I remove syslog-ng and re-install disable that module on install? 4. I’d rather not recompile if I don’t have to, but I found instructions for how to do that. Any tips would be appreciated. Thanks. Craig L Bowser ____________________________ This email is measured by size. Bits and bytes may have settled during transport.

2 2

Install Issue
by Elphick, Matthew Thomas 08 Nov '17

08 Nov '17

Hello, I am trying to install syslog-ng OSE on RHEL 7 but I get the following error. Transaction check error: file /etc/logrotate.d/syslog from install of syslog-ng-3.12.1-2.el7.centos.x86_64 conflicts with file from package rsyslog-8.24.0-12.el7.x86_64 I can't remove rsyslog from the system as other people us it. What is the best way to resolve this issue? Thanks for your help. Matthew Elphick, Network Engineering Marsh & McLennan Companies Global Technology Infrastructure (MGTI) | Enterprise Networking Services Tower Place, London, EC3R 5BU O +44 (0)20 73573667 | M +44 (0)7392 122399 | matthew.elphick(a)mmc.com<mailto:matthew.elphick@mmc.com> ________________________________ Marsh Ltd. Registered in England and Wales Number: 1507274 Registered office 1 Tower Place West, Tower Place, London, EC3R 5BU. Marsh Ltd is authorised and regulated by the Financial Conduct Authority. This message and any attachments are confidential. If you have received this message in error please delete it from your system. If you require any assistance please notify the sender. Thank you.

4 6

logstash to syslog-ng
by Czanik, Péter 08 Nov '17

08 Nov '17

Hi, In the past few weeks I have seen questions about sending logs from Logstash to syslog-ng, so I tried it myself and put together a blog with simple configurations to help you: https://www.balabit.com/blog/sending-logs-logstash-syslog-ng/ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

3 3

any limitation with in-list filter
by Gopi Joshi 07 Nov '17

07 Nov '17

I am using in-list filter to match logs for specific users , these specific users are saved in a plain text file . is there any restriction or limitation on number of lines it can match in one file ? I currently have less than 10 users to compare with but this list is going to increase . Regards Gopi Joshi

3 2

Beat to syslog-ng
by Czanik, Péter 07 Nov '17

07 Nov '17

Hi, Somehow related to my previous e-mail: I was asked if Filebeat can send logs directly into syslog-ng. None of the network destinations of Beat are supported by syslog-ng, but I did a quick check and we can read and parse the file destination of Beat. Filebeat module functionality can be practically replaced by syslog-ng and one of its parsers. But there are quite a few other Beat applications. Question: is there a practical use case for syslog-ng reading the file output of a Beat module? Do you use it anywhere? Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

2 1

Question about using multiple amqp destination
by Alicia Smith 06 Nov '17

06 Nov '17

I am trying to configure syslog-ng to send multiple json formatted logs to a rabbitmq endpoint. The consumer for the logs checks for a particular field to determine the type of log it is. However, I can't seem to figure out how to send all the different logs as the field it checks is "source" that I add using --pair within the format-json template destination d_amqp_ssh { amqp( vhost("/") host("mozdefqa2.private.scl3.mozilla.com") port(5672) exchange("eventtask") exchange-type("direct") routing-key("eventtask") body("$(format-json --scope nv_pairs --pair category=\"bro\" --pair source=\"ssh\" --pair customendpoint=\" \" --pair tags=\"bro\")") persistent(no) username("guest") password("guest") ); }; So I added another destination with the name d_amqp_conn and it's source field value is "conn" and the amqp endpoint is the same host as the ssh log. But syslog-ng fails to start with this particular setup. It seems to not like having more than one amqp destination. How would something like this typically be solved? -- Alicia Smith @phrozyn Information Security Engineer asmith(a)mozilla.com

3 10

Syslog-NG Statistics
by Schoonover, Mark E HHHH 01 Nov '17

01 Nov '17

Hello, I'm working on a Splunk dashboard for our NG PE servers. Looking at the stats from the output of /opt/syslog-ng/sbin/syslog-ng-ctl stats, this one has me puzzled: src.program,,49587758,d,processed,1 I don't know what "program" means as well as the 3rd column. I understand the remaining columns. What do they mean? Thanks! Regards, Mark Schoonover - KA6WKE Infrastructure Engineering Manager, Splunk Architect ENE : Tools, Instrumentation and Common Services Team Office: 32.8697° N, 116.9711° W Phone : 770-261-7934 Email : mark.schoonover(a)cigna.com<mailto:mark.schoonover@cigna.com> HPSM Team: ENE NMS Engineering Confidential, unpublished property of CIGNA. Do not duplicate or distribute. Use and distribution limited solely to authorized personnel. ¬© Copyright 2017 CIGNA ------------------------------------------------------------------------------ CONFIDENTIALITY NOTICE: If you have received this email in error, please immediately notify the sender by e-mail at the address shown. This email transmission may contain confidential information. This information is intended only for the use of the individual(s) or entity to whom it is intended even if addressed incorrectly. Please delete it from your files if you are not the intended recipient. Thank you for your compliance. Copyright (c) 2017 Cigna ==============================================================================

2 1

Re: [syslog-ng] Need to parse named query log entries and get just the domain portion
by Robert Webb 31 Oct '17

31 Oct '17

Evan, Thanks you for pointing me in the right direction. I was so focused on the matching starting @ "queries: info:" that I totally forgot that the first part of the actual MSG has the date and time sent with it. My pattern now takes that into account. Message: 27-Oct-2017 14:19:42.359 queries: info: client 137.75.76.138#38643 (sr.symcb.com) query: sr.symcb.com IN A +E (137.75.75.75) <pattern>@ESTRING:: @@ESTRING:: @queries: info: client @IPvANY:named.client.ip@#@NUMBER:named.client.port:@ @ESTRING:named.domain.name::@ query: @HOSTNAME:named.host.name:@ @STRING:@ @STRING:named.record.type:@ @ESTRING:named.record.flags: @(@IPvANY:named.server.ip@)</pattern> Robert From: Evan Rempel [mailto:erempel@uvic.ca] Sent: Friday, October 27, 2017 9:52 AM To: Robert Webb <rwebb(a)ropeguru.com> Subject: Re: [syslog-ng] Need to parse named query log entries and get just the domain portion In the past when I have had this problem it was in the patterndb portion just outside if what you have included. essentially the program does not match, so the rule is never attempted. This results in none of the tag-value pairs being populated. The pdbtool shows a match because the pattern is always in the correct program. Your patterndb portion is inside of a set of a config such as <patterndb version="4" pub_date="2009-09-01"> <ruleset name="named" id="some-unique-value"> <pattern>named</pattern> <rules> [ your included pattern db data ] </ruleset> </pattendb> It is the line <pattern>named</pattern> that matches the parsed program name from the syslog line to allow the rule to be parsed against the syslog message. For testing purposes I make a destination to write a json object. That way I can see everything that populated. template t_json { template("$(format-json --scope everything )\n"); template_escape(no); }; destination test.log { file("/var/syslog/test.log.$S_YEAR$S_MONTH$S_DAY.000000" template(t_json)); }; Evan. On 10/27/2017 05:22 AM, Robert Webb wrote: So I have a situation where I have a syslog feed coming in that is sending named (BIND9) query logs. On the DNS servers themselves, we are doing sink hole via a local zone file and sending the dns response back as 127.0.0.1. Since there is no way for BIND9 to give me logging of the DNS query response back to the user, I need to be able to take a query log entry, parse it to get just the domain portion, and then match the domain to a list. I am trying to do all this inside syslog-ng. I have been given some guidance to use a pattern database. I worked with this and am successfully matching my incoming query log data. Using the pdbtool, I get a full match and a list of the values I expect. But where I run into an issue after searching through all the documentation is, where can these values be used? I tried building a template against a destination, and none of the values populate. So I am stuck at this point. I have also been told that I may have to do a custom python parser instead. Any help or guidance is appreciated. Here is my rule: <rule provider="balabit" id="b0ca071d-309e-454a-8e31-721b01ef55ee" class="system"> <patterns> <pattern>queries: info: client @IPvANY:named.client.ip@#@NUMBER:named.client.port:@ @ESTRING:named.domain.name::@ query: @HOSTNAME:named.host.name:@ @STRING:@ @STRING:named.record.type:@ @ESTRING:named.record.flags: @(@IPvANY:named.server.ip@)</pattern> </patterns> <examples> <example> <test_message program="named">client 213.46.246.51#1890: query: tumtali.fegade.com IN A - (103.5.198.211)</test_message> <test_values> <test_value name="named.client.ip">213.46.246.51</test_value> <test_value name="named.client.port">1890</test_value> <test_value name="named.domain.name">tumtali.fegade.com</test_value> <test_value name="named.record.type">A</test_value> <test_value name="named.record.flags">-</test_value> <test_value name="named.server.ip">103.5.198.211</test_value> </test_values> </example> <example> <test_message program="named">client 10.187.195.186#40441: query: banister.paars.tld IN A -EDC (10.226.134.25)</test_message> <test_values> <test_value name="named.client.ip">10.187.195.186</test_value> <test_value name="named.client.port">40441</test_value> <test_value name="named.domain.name">banister.paars.tld</test_value> <test_value name="named.record.type">A</test_value> <test_value name="named.record.flags">-EDC</test_value> <test_value name="named.server.ip">10.226.134.25</test_value> </test_values> </example> <example> <test_message program="named">queries: client 192.168.102.189#58454: query: maps.googleapis.com IN A + (192.168.102.2)</test_message> <test_values> <test_value name="named.client.ip">192.168.102.189</test_value> <test_value name="named.client.port">58454</test_value> <test_value name="named.domain.name">maps.googleapis.com</test_value> <test_value name="named.record.type">A</test_value> <test_value name="named.record.flags">+</test_value> <test_value name="named.server.ip">192.168.102.2</test_value> </test_values> </example> </examples> <values> </values> <tags> <tag>named</tag> <tag>queries</tag> </tags> </rule> Here are test results: Pattern matching part: queries: info: client @IP:named.client.ip=137.75.76.138@#@NUMBER:named.client.port=38643@ @ESTRING:named.domain.name=(sr.symcb.com)@ query: @HOSTNAME:named.host.name=sr.symcb.com@ @STRING:None=IN@ @STRING:named.record.type=A@ @ESTRING:named.record.flags=+E@(@IP:named.server.ip=137.75.75.75@) Matching part: queries: info: client 137.75.76.138#38643 (sr.symcb.com) query: sr.symcb.com IN A +E (137.75.75.75) Values: MESSAGE=queries: info: client 137.75.76.138#38643 (sr.symcb.com) query: sr.symcb.com IN A +E (137.75.75.75) PROGRAM=named .classifier.class=system .classifier.rule_id=b0ca071d-309e-454a-8e31-721b01ef55ee named.client.ip=137.75.76.138 named.client.port=38643 named.domain.name=(sr.symcb.com) named.record.type=A named.server.ip=137.75.75.75 named.host.name=sr.symcb.com named.record.flags=+E TAGS=.classifier.system,named,queries

2 2

Syslog 3.5 (pipe file issue)
by Amin, Jitesh CTR (US) 29 Oct '17

29 Oct '17

Hi, We have the syslog-ng 3.5 installed on Linux OS. We are trying to use the pipe function (to pipe data to the log file). This pipe file is FIFO. One of the issues we are seeing is that, the file size of this pipe file is always ZERO and never grows and second the timestampe on the file never updates or changes. Though when we do a "cat pipefilename" command we do see contents of the file. Can you please let us know what we could be doing wrong or any feedback on how to resolve the issue? Thanks Jitesh

4 5