- syslog-ng - lists.balabit.hu

Re: [syslog-ng] can syslog-ng process email attachments?
by james.r.hendrick 12 Feb '17

12 Feb '17

Thanks, I'll take a look Jim Sent from my Verizon, Samsung Galaxy smartphone -------- Original message --------From: "Fekete, Róbert" <robert.fekete(a)balabit.com> Date: 2/11/17 10:41 AM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] can syslog-ng process email attachments? Hi, Fabien created an mbox source a while back, maybe you can use it: https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-… Regards, Robert On Fri, Feb 10, 2017 at 9:03 PM, Jim Hendrick <james.r.hendrick(a)gmail.com> wrote: We have security appliances that generate CSV files and automatically send them (nightly) I am looking for a way to get them into a log stream and trying to make it as simple as possible. I suppose I could do something with procmail or even openssl and an IMAP connection, but I wanted to ask this esteemed company if any of you have solved this already. even something radical like "source s_mail ( smtp() );" and have config options to tweak what it does with different recipients / mail formats Any (appropriate) suggestions? Thanks!Jim ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

can syslog-ng process email attachments?
by Jim Hendrick 11 Feb '17

11 Feb '17

We have security appliances that generate CSV files and automatically send them (nightly) I am looking for a way to get them into a log stream and trying to make it as simple as possible. I suppose I could do something with procmail or even openssl and an IMAP connection, but I wanted to ask this esteemed company if any of you have solved this already. even something radical like "source s_mail ( smtp() );" and have config options to tweak what it does with different recipients / mail formats Any (appropriate) suggestions? Thanks! Jim

2 1

Unable to install ubuntu 16.04
by Simon Mousey Smith 11 Feb '17

11 Feb '17

Hi All Im struggling to install the syslog-ng-mod-sql on ubuntu 16.04 It keeps saying 'syslog-ng-mod-sql : Depends: syslog-ng-abi-3.9-0 but it is not installable' root@syslog:~# apt-get install syslog-ng-core syslog-ng-mod-sql Reading package lists... Done Building dependency tree Reading state information... Done Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: syslog-ng-mod-sql : Depends: syslog-ng-abi-3.9-0 but it is not installable E: Unable to correct problems, you have held broken packages. root@syslog:~# apt-get install syslog-ng-core Reading package lists... Done Building dependency tree Reading state information... Done Suggested packages: syslog-ng-mod-sql syslog-ng-mod-mongodb syslog-ng-mod-json syslog-ng-mod-smtp syslog-ng-mod-amqp syslog-ng-mod-geoip syslog-ng-mod-redis syslog-ng-mod-stomp syslog-ng-mod-riemann syslog-ng-mod-graphite syslog-ng-mod-python syslog-ng-mod-java syslog-ng-mod-java-common-lib syslog-ng-mod-elastic syslog-ng-mod-hdfs syslog-ng-mod-kafka syslog-ng-mod-http syslog-ng-mod-curl syslog-ng-mod-kvformat syslog-ng-mod-add-contextual-data The following NEW packages will be installed: syslog-ng-core 0 upgraded, 1 newly installed, 0 to remove and 5 not upgraded. Need to get 0 B/408 kB of archives. After this operation, 1,839 kB of additional disk space will be used. Selecting previously unselected package syslog-ng-core. (Reading database ... 107008 files and directories currently installed.) Preparing to unpack .../syslog-ng-core_3.9.1-1_amd64.deb ... Unpacking syslog-ng-core (3.9.1-1) ... Processing triggers for systemd (229-4ubuntu16) ... Processing triggers for ureadahead (0.100.0-19) ... Setting up syslog-ng-core (3.9.1-1) ... Processing triggers for systemd (229-4ubuntu16) ... Processing triggers for ureadahead (0.100.0-19) … Success... root@syslog:~# apt-get install syslog-ng-mod-sql Reading package lists... Done Building dependency tree Reading state information... Done Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: syslog-ng-mod-sql : Depends: syslog-ng-abi-3.9-0 but it is not installable E: Unable to correct problems, you have held broken packages. root@syslog:~# Any ideas? Regards Simon

3 7

syslog-ng logging only as foreground process and not as daemon on centos7
by Jordan Ladora 10 Feb '17

10 Feb '17

I've got syslog-ng v3.7.3_7 running on FreeBSD 11-REL as the log server with both FreeBSD 11-REL and CentOS7 clients (the CentOS clients are running v3.5.6-3.el7 as it's the latest version installed by yum epel repo). It's always worked perfectly on the FreeBSD systems, but a few months ago I had the same problem described here when setting up a CentOS client- https://serverfault.com/questions/760383/syslog-ng-working-as-foreground-pr… ...I followed the fix of uncommenting the "ForwardToSyslog=yes" line in /etc/systemd/journald.conf and it started logging perfectly on CentOS clients, too. However, in setting up another CentOS7 client yesterday, this fix no longer worked. If I start syslog-ng via- chkconfig syslog-ng on systemctl restart syslog-ng.service ..it starts and runs and is shown in ps, starts automatically after reboot, but doesn't log to the server. If I start it manually from the terminal with- nohup /usr/sbin/syslog-ng -F -p /var/run/syslogd.pid & ...then it does work and starts logging to the server immediately. In that case there are two identical (except for pid) lines shown by ps, e.g.- 482 /usr/sbin/syslog-ng -F -p /var/run/syslogd.pid 904 /usr/sbin/syslog-ng -F -p /var/run/syslogd.pid ...but only the one I started from the terminal (nohup...) actually logs. Any help would be much appreciated. -jl

3 3

Insider 2017-02: securing Elasticsearch; Troubleshooting; FOSDEM & SCALE;
by Czanik, Péter 09 Feb '17

09 Feb '17

Dear syslog-ng users, This is the 55th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Securing connections to Elasticsearch ------------------------------------- Recently, news have come out that unprotected MongoDB databases are being actively compromised: content is copied and replaced by a message asking for a ransom to get it back. As “The Register” reports: Elasticsearch is next. Read our latest blog post to learn more about how to secure your Elasticsearch cluster and avoid a ransomware: https://www.balabit.com/blog/secure-your-elasticsearch-cluster-and-avoid-ra… Troubleshooting syslog-ng to syslog-ng connections -------------------------------------------------- syslog-ng users often face the challenge of not being able to send logs over the network from one syslog-ng instance to another. There can be many reasons for this; some are independent from syslog-ng, while others are related to the syslog-ng configuration. Here are a few troubleshooting tips: https://www.balabit.com/blog/troubleshooting-syslog-ng-syslog-ng-connection… Load balancing HTTP connections to Elasticsearch ------------------------------------------------ In addition to developing the HTTPS driver for syslog-ng, Fabien Wernli also contributed HTTP load balancing support to the syslog-ng Elasticsearch driver, which is also used by HTTPS support for Elasticsearch: https://github.com/balabit/syslog-ng/pull/1319 FOSDEM 2017 ----------- This year Peter Czanik, community manager at Balabit, participated with two syslog-ng presentations at the annual FOSDEM conference in Brussels. Read about his experiences at the event: https://www.balabit.com/blog/syslog-ng-fosdem-2017/ How to send JSON log messages to RabbitMQ from syslog-ng -------------------------------------------------------- In this post, you can read about how to configure syslog-ng to send logs in JSON format in the body of an AMQP message: https://sharknet.us/2017/02/04/how-to-send-json-log-messages-to-rabbitmq-fr… UPCOMING EVENTS Balabit will participate at the Southern California Linux Expo this year again: http://www.socallinuxexpo.org/scale/15x You can talk to our engineers at the Balabit booth and listen to Peter Czanik present about parsing security logs using syslog-ng in room 106: http://www.socallinuxexpo.org/scale/15x/presentations/get-most-out-your-sec… Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

[request for feedback] syslog fundamentals post
by Radu Gheorghe 01 Feb '17

01 Feb '17

Hello, I just wrote a blog post that tries to disambiguate the word "syslog" and clarify the myths I heard around it (as I'm sure you know, there's a lot of vagueness): https://sematext.com/blog/2017/01/30/what-is-syslog-daemons-message-formats… Obviously, syslog-ng is an important chunk of it, so I wanted to ask for any feedback you may have (did I miss something important regarding syslog-ng? is there something wrong or something else you'd want to see there?) whichever way you prefer to send it (here, on the blog post comments, personal Email or Twitter...). Thanks in advance and best regards, Radu -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/

2 2

Stupid E-S-K Question
by Scot 31 Jan '17

31 Jan '17

*E*lastic, *S*yslog-ng *K*ibana Upgraded to latest of ES Stack, Kibana 5 and syslog-ng 3.9.1 I had a Kibana dashboard with a bar chart of unique count of systems that had sent a syslog heartbeat. So I could see any missed heartbeats for any host in the last 24 hours. Post upgrade of syslog-ng the host_from, host fields do not seem to come into ES as usable fields because they are not indexed. So visualizations "bar charts by unique 'host" is broken. Has anyone seen this? client-mode("http") index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("syslog") # Description: The type of the index. For example, type("test") template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n")

3 4

Convert rewrite rules (regex) to patterndb
by Clayton Dukes 27 Jan '17

27 Jan '17

Hi All, Are there any tools/scripts anyone has written that will convert regex to patterndb types of matches? So, given some rewrite rule from the "old" way of doing it, convert that rule to a pdb file. Like: filter f_asa { match( '(.*?):?(\d{1,3}+\.\d{1,3}+\.\d{1,3}+\.\d{1,3}+)\/(\d+)(.*?)(\).*?)?:?(\d{1,3}+\.\d{1,3}+\.\d{1,3}+\.\d{1,3}+)\/(\d+)(.*?)(\d{1,3}+\.\d{1,3}+\.\d{1,3}+\.\d{1,3}+)\/(\d+)(.*)' value("MESSAGE") type("pcre") flags("store-matches" "ignore-case") ); rewrite rw_cisco { set( "$1 SourceIP: $2 SourcePort:$3 $4 $5DestIP: $6 DestPort:$7 $8 SourceIP: $9 SourcePort:$10 $11" , value("MESSAGE") condition(filter(f_asa))); }; Convert to something like: <pattern>%ASA-6-305011: Built dynamic TCP translation from inside:@IPv4:src_ip@/@NUMBER:src_port@ to Outside:@IPv4:dst_ip@/@NUMBER:dst_port@</pattern> Note: the above is just an example, the regex may not be the same as the actual <pattern> - I just copy pasted from different files to show an example. [cid:image001.png@01D27730.889CAAF0]

3 3

RPM packages from git HEAD
by Czanik, Péter 23 Jan '17

23 Jan '17

Hi, Recently I started to create RPM packages from git HEAD. Obviously this is not for production, but helps to check, if a problem reported for 3.8.1 or earlier still exists. It is available for openSUSE, Fedora, RHEL 7 and SLES 11/12. I don't create RHEL 6 packages, as that would require maintaining a third spec file and repo, but if there are too many people trying to convince me, I can do it next year :) And the good news is, that I learned right after publishing my blog, that 3.8.2 is nearing and expected to arrive this year! Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

5 6

https to elasticsearch
by Czanik, Péter 20 Jan '17

20 Jan '17

Hi, Secure connection to Elasticsearch was already supported through Shield (now X-pack) for a while and SearchGuard support was also added recently. Both used transport mode for connection. Today encrypted connection using HTTPS became available thanks to Faxm0dem: https://github.com/balabit/syslog-ng/pull/1321 Before leaving for the weekend I prepared RPM packages from this PR in my 'githead' repositories. You can read about how to use those in my blog at https://www.balabit.com/blog/rpm-packages-from-syslog-ng-git-head/ Would be great if someone tested this against X-pack! Have a nice weekend! Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0