- syslog-ng - lists.balabit.hu

insider 2015-12: Elasticsearch & Kibana; SCL; Kafka; syslog-ng on Windows; conferences
by Czanik, Péter 03 Dec '15

03 Dec '15

Dear syslog-ng users, This is the 45th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS ESK: Elasticsearch + syslog-ng + Kibana --------------------------------------- An official Elasticsearch destination was a long awaited feature of syslog-ng. The Elasticsearch syslog-ng Kibana (ESK) stack provides high performance log collection, message parsing and filtering on the syslog-ng side and storage and analysis on the Elastic side. Read more about the benefits at https://czanik.blogs.balabit.com/2015/10/how-to-parse-data-with-syslog-ng-s… and https://czanik.blogs.balabit.com/2015/12/elasticsearch-and-syslog-ng-fast-a… The power of SCL ---------------- SCL, the syslog-ng configuration library is a lesser known feature of syslog-ng. It can provide reusable configuration blocks, which help to format messages to a specific format, hide complex configurations, This blog shows how to create a simple SCL through the example of a Loggly (logging as a service provider) destination: https://bazsi.blogs.balabit.com/2015/11/the-power-of-scl-integrating-with-l… Kafka and syslog-ng ------------------- Kafka is a high-throughput distributed messaging system. It was originally developed by LinkedIn as the backbone of a website activity tracking infrastructure and is now used by Netflix, Spotify, Twitter and many other companies for message queuing, log aggregation and stream processing. Read more about how syslog-ng can collect messages from multiple sources and process as well as filter them before forwarding them to Kafka: https://czanik.blogs.balabit.com/2015/11/kafka-and-syslog-ng/ syslog-ng on Windows -------------------- Most Linux admins only consider Windows as a client machine and use syslog-ng Agent for Windows if they need to collect logs from Windows. The following use cases demonstrate why you would want to use syslog-ng Premium Edition server on Windows instead of the syslog-ng Agent for Windows: https://czanik.blogs.balabit.com/2015/11/syslog-ng-on-windows/ syslog-ng @ conferences ----------------------- This autumn, syslog-ng was present at many conferences. I gave presentations about syslog-ng at the annual FSF Hungary and Virtualization & DevOps day conferences (in Hungarian), and participated SUSECon, LISA and Crunch Big Data Conference as an exhibitor. Most questions I received were related to the Elasticsearch destination and also many people were interested in commercial support for syslog-ng. In January I'll present “syslog-ng: from raw data to Big Data” at the Southern California Linux Expo: https://www.socallinuxexpo.org/scale/14x/presentations/syslog-ng-raw-data-b… See you there! NEW RELEASES syslog-ng PE 5F5: https://jluby.blogs.balabit.com/2015/11/16/syslog-ng-premium-edition-5f5-re… syslog-ng OSE 3.7.2: https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.7.2 Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

(no subject)
by Harsha S Aryan 01 Dec '15

01 Dec '15

hi, can any one let me know how to configure syslog-ng -- Rehards, Harsha

2 1

need help to configure
by Harsha S Aryan 01 Dec '15

01 Dec '15

hi, i am looking to configure syslog-ng in my system can any one let me know how to configure and add client (some linux servers and router) -- Rehards, Harsha

2 4

unit test valgrind coverage
by Scheidler, Balázs 24 Nov '15

24 Nov '15

Hi, I've wanted to make our unit tests valgrind clean for a long time, here's where I am at right now, the branch fixes a number of production related bugs as well. Feedback, help appreciated. https://github.com/balabit/syslog-ng/tree/f/unit-test-cleanups I am at 57 successful and 18 failing unit tests so far. -- Bazsi

2 1

log_fetch_limit
by Al Itchon 20 Nov '15

20 Nov '15

I have about 1000 servers sending tcp messages through a call to /dev/tcp to a port on my syslog-ng server. I'm assuming each call to /dev/tcp is a single connection and a single message, so am I correct that the log_fetch_limit should be set to 1 since it applies to each connection? Also, should I bother adjusting the so-rcvbuf? I know the manual mentions that if using UDP, the receive buffer should be increased. I'm using TCP and my message rate is about 250 messages/sec.

2 1

rabbitmq source
by Czanik, Péter 18 Nov '15

18 Nov '15

Hi, Someone asked me about the RabitMQ source. As I don't use RabitMQ I don't have much personal experience, so I ask you about it: What are your experiences with the RabitMQ source? What is your message rate? What is your use case? Any problems, missing features? If for any reason, you don't want to share your experiences publicly, please e-mail me directly! Any feedback is very welcome! Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

RFC: groupingby parser
by Scheidler, Balázs 13 Nov '15

13 Nov '15

Hi, I have just created a pull request for the groupingby() parser that I am seeking feedback for: https://github.com/balabit/syslog-ng/pull/785 The commit message is pretty detailed on what it does, but let me reproduce it here for simplicity. Any feedback is appreciated. Thanks. dbparser: add groupingby() parser This patch adds a new parser that can perform simple correllation on log messages, e.g. when multiple input log messages describe the same event. In a way it is similar to the SQL GROUP BY operation, where an aggregate of a set of input records can be calculated. The major difference between SQL GROUP BY and groupingby() is that the first _always_ operates on a enumerable list of records, whereas groupingby() works on a stream of data. groupingby() produces related groups by using a sliding window on time, e.g. it can be specified how much time we need to look back to group related messages together. As a specific use-case, let's see Linux audit logs. Linux audit logs tend to be broken to several lines generated as a list of lines. These tend to be pretty close in time, however there might be multiple events logged at around the same time, which get mixed up in the output. The example below is the audit log for an ntpdate execution: type=SYSCALL msg=audit(1440927434.124:40347): arch=c000003e syscall=59 success=yes exit=0 a0=7f121cef0b88 a1=7f121cef0c00 a2=7f121e690d98 a3=2 items=2 ppid=4312 pid=4347 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ntpdate" exe="/usr/sbin/ntpdate" key=(null) type=EXECVE msg=audit(1440927434.124:40347): argc=3 a0="/usr/sbin/ntpdate" a1="-s" a2="ntp.ubuntu.com" type=CWD msg=audit(1440927434.124:40347): cwd="/" type=PATH msg=audit(1440927434.124:40347): item=0 name="/usr/sbin/ntpdate" inode=2006003 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL type=PATH msg=audit(1440927434.124:40347): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=5243184 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL type=PROCTITLE msg=audit(1440927434.124:40347): proctitle=2F62696E2F7368002F7573722F7362696E2F6E7470646174652D64656269616E002D73 These lines are connected by their 2nd field, msg equals to audit(1440927434.124:40347). This can be processed by the groupingby() parser in a similar way that db-parser() could do correllation. These are the options for groupingby(): * key(): specifies the key for the grouping, e.g. the value that must be the same for all messages in the group * scope(): specifies one of three values: "global", "host", "process", meaning the same as in db-parser; whether to apply grouping for all messages received by syslog-ng (global), only messages coming from the same host (host), or the same process/pid combination. * where(): specifies a filter condition, messages not matching the filter will NOT be added to the group. where() only has access to a single message, the current one being processed. * having(): specifies a filter condition that must match in order for the group to generate an aggregate message. having() has access to the entire group through the "context". * timeout(): specifies the maximum time to wait for all messages in the group to arrive. After this time, the group is assumed to be complete and is aggregation is triggered. * aggregate(): this specifies the aggregate message that's going to be generated when the group is complete. - tags(): - value(): - inherit-mode(): * inject-mode(): how the aggregate message is injected into the syslog-ng message routing, can be one of: "pass-through", "internal". * trigger(): trigger the closure of the group by matching an incoming message. If the filter condition specified here matches the incoming message, it will cause the aggregate message to be calculated, emitted and the group be discarded from the state table. A few use-cases where this can be useful: * Linux audit logs * postfix logs Signed-off-by: Balazs Scheidler <balazs.scheidler(a)balabit.com> -- Bazsi

1 0

OSE 3.7.2 questions
by Fekete, Róbert 13 Nov '15

13 Nov '15

Hi, I've seen that you had a few patches merged to 3.7.2, and I'm not sure what to include in the docs about them: - There were some csv-parser changes, is any of them user-visible? - What does linux-audit-parser do? Does it require any configuration, or it just works? Thanks! Robert

2 1

Re: [syslog-ng] Having trouble with java-modules
by Laszlo Budai 13 Nov '15

13 Nov '15

You shouldn't run autogen.sh for a release tarball. Could you copy-paste the download link for the syslog-ng tarball that doesn't contain the configure script? Sent from my HTC ----- Reply message ----- From: "Rory Toma" <rory(a)ooma.com> To: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] Having trouble with java-modules Date: Thu, Nov 12, 2015 21:17 This one doesn't have configure, and when I run autogen.sh, it errors with: Running autogen in 'lib/ivykis'... Don't know how to bootstrap submodule 'lib/ivykis' If I remove it, then later I get: configure: error: Internal ivykis sources not found in lib/ivykis. This is a hard dependency, unable to build syslog-ng without them. On 11/10/15 11:45 PM, Budai, László wrote: > You can download releases from here: > https://github.com/balabit/syslog-ng/releases > > L. > > On Wednesday, November 11, 2015, Rory Toma <rory(a)ooma.com > <mailto:rory@ooma.com>> wrote: > > I can't locate the 3.7 source release tarballs, only the git, > which isn't running autogen.sh correctly for me. > > On 11/10/15 9:39 PM, Budai, László wrote: >> >> Hi, >> >> Java is not supported in 3.6.x ( only in syslog-ng incubator ). >> For 3.7 it is ported(reworked) from the incubator, so you should >> give a try to use the java support with syslog-ng-3.7.2. >> >> L. >> >> On Nov 11, 2015 1:44 AM, "Rory Toma" <rory(a)ooma.com >> <javascript:_e(%7B%7D,'cvml','rory(a)ooma.com');>> wrote: >> >> I've downloaded syslog-ng-3.6.4 (I'm not using master because >> I can't >> get the configure script to generate correctly) >> >> I'm following the steps here: >> https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-… >> >> and have even added "--enable-java" >> >> I notice that even with JAVA_HOME set on the command line, >> config.log >> picks up the wrong java, and passing --with-java-home has no >> effect. >> >> In any case, I cannot get java modules installed. Is there an >> additional >> command to run to create/populate the java modules directory? >> >> This is on CentOS-6.7 x86_64 >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> >> >> ______________________________________________________________________________ >> Member info:https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation:http://www.balabit.com/support/documentation/?product=syslog-… >> FAQ:http://www.balabit.com/wiki/syslog-ng-faq >> > > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

2 1

Having trouble with java-modules
by Rory Toma 12 Nov '15

12 Nov '15

I've downloaded syslog-ng-3.6.4 (I'm not using master because I can't get the configure script to generate correctly) I'm following the steps here: https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-… and have even added "--enable-java" I notice that even with JAVA_HOME set on the command line, config.log picks up the wrong java, and passing --with-java-home has no effect. In any case, I cannot get java modules installed. Is there an additional command to run to create/populate the java modules directory? This is on CentOS-6.7 x86_64

3 5