- syslog-ng - lists.balabit.hu

insider 2015-12: Elasticsearch & Kibana; SCL; Kafka; syslog-ng on Windows; conferences
by Czanik, Péter 03 Dec '15

03 Dec '15

Dear syslog-ng users, This is the 45th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS ESK: Elasticsearch + syslog-ng + Kibana --------------------------------------- An official Elasticsearch destination was a long awaited feature of syslog-ng. The Elasticsearch syslog-ng Kibana (ESK) stack provides high performance log collection, message parsing and filtering on the syslog-ng side and storage and analysis on the Elastic side. Read more about the benefits at https://czanik.blogs.balabit.com/2015/10/how-to-parse-data-with-syslog-ng-s… and https://czanik.blogs.balabit.com/2015/12/elasticsearch-and-syslog-ng-fast-a… The power of SCL ---------------- SCL, the syslog-ng configuration library is a lesser known feature of syslog-ng. It can provide reusable configuration blocks, which help to format messages to a specific format, hide complex configurations, This blog shows how to create a simple SCL through the example of a Loggly (logging as a service provider) destination: https://bazsi.blogs.balabit.com/2015/11/the-power-of-scl-integrating-with-l… Kafka and syslog-ng ------------------- Kafka is a high-throughput distributed messaging system. It was originally developed by LinkedIn as the backbone of a website activity tracking infrastructure and is now used by Netflix, Spotify, Twitter and many other companies for message queuing, log aggregation and stream processing. Read more about how syslog-ng can collect messages from multiple sources and process as well as filter them before forwarding them to Kafka: https://czanik.blogs.balabit.com/2015/11/kafka-and-syslog-ng/ syslog-ng on Windows -------------------- Most Linux admins only consider Windows as a client machine and use syslog-ng Agent for Windows if they need to collect logs from Windows. The following use cases demonstrate why you would want to use syslog-ng Premium Edition server on Windows instead of the syslog-ng Agent for Windows: https://czanik.blogs.balabit.com/2015/11/syslog-ng-on-windows/ syslog-ng @ conferences ----------------------- This autumn, syslog-ng was present at many conferences. I gave presentations about syslog-ng at the annual FSF Hungary and Virtualization & DevOps day conferences (in Hungarian), and participated SUSECon, LISA and Crunch Big Data Conference as an exhibitor. Most questions I received were related to the Elasticsearch destination and also many people were interested in commercial support for syslog-ng. In January I'll present “syslog-ng: from raw data to Big Data” at the Southern California Linux Expo: https://www.socallinuxexpo.org/scale/14x/presentations/syslog-ng-raw-data-b… See you there! NEW RELEASES syslog-ng PE 5F5: https://jluby.blogs.balabit.com/2015/11/16/syslog-ng-premium-edition-5f5-re… syslog-ng OSE 3.7.2: https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.7.2 Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

(no subject)
by Harsha S Aryan 01 Dec '15

01 Dec '15

hi, can any one let me know how to configure syslog-ng -- Rehards, Harsha

2 1

need help to configure
by Harsha S Aryan 01 Dec '15

01 Dec '15

hi, i am looking to configure syslog-ng in my system can any one let me know how to configure and add client (some linux servers and router) -- Rehards, Harsha

2 4

unit test valgrind coverage
by Scheidler, Balázs 24 Nov '15

24 Nov '15

Hi, I've wanted to make our unit tests valgrind clean for a long time, here's where I am at right now, the branch fixes a number of production related bugs as well. Feedback, help appreciated. https://github.com/balabit/syslog-ng/tree/f/unit-test-cleanups I am at 57 successful and 18 failing unit tests so far. -- Bazsi

2 1

log_fetch_limit
by Al Itchon 20 Nov '15

20 Nov '15

I have about 1000 servers sending tcp messages through a call to /dev/tcp to a port on my syslog-ng server. I'm assuming each call to /dev/tcp is a single connection and a single message, so am I correct that the log_fetch_limit should be set to 1 since it applies to each connection? Also, should I bother adjusting the so-rcvbuf? I know the manual mentions that if using UDP, the receive buffer should be increased. I'm using TCP and my message rate is about 250 messages/sec.

2 1

rabbitmq source
by Czanik, Péter 18 Nov '15

18 Nov '15

Hi, Someone asked me about the RabitMQ source. As I don't use RabitMQ I don't have much personal experience, so I ask you about it: What are your experiences with the RabitMQ source? What is your message rate? What is your use case? Any problems, missing features? If for any reason, you don't want to share your experiences publicly, please e-mail me directly! Any feedback is very welcome! Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

RFC: groupingby parser
by Scheidler, Balázs 13 Nov '15

13 Nov '15

Hi, I have just created a pull request for the groupingby() parser that I am seeking feedback for: https://github.com/balabit/syslog-ng/pull/785 The commit message is pretty detailed on what it does, but let me reproduce it here for simplicity. Any feedback is appreciated. Thanks. dbparser: add groupingby() parser This patch adds a new parser that can perform simple correllation on log messages, e.g. when multiple input log messages describe the same event. In a way it is similar to the SQL GROUP BY operation, where an aggregate of a set of input records can be calculated. The major difference between SQL GROUP BY and groupingby() is that the first _always_ operates on a enumerable list of records, whereas groupingby() works on a stream of data. groupingby() produces related groups by using a sliding window on time, e.g. it can be specified how much time we need to look back to group related messages together. As a specific use-case, let's see Linux audit logs. Linux audit logs tend to be broken to several lines generated as a list of lines. These tend to be pretty close in time, however there might be multiple events logged at around the same time, which get mixed up in the output. The example below is the audit log for an ntpdate execution: type=SYSCALL msg=audit(1440927434.124:40347): arch=c000003e syscall=59 success=yes exit=0 a0=7f121cef0b88 a1=7f121cef0c00 a2=7f121e690d98 a3=2 items=2 ppid=4312 pid=4347 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ntpdate" exe="/usr/sbin/ntpdate" key=(null) type=EXECVE msg=audit(1440927434.124:40347): argc=3 a0="/usr/sbin/ntpdate" a1="-s" a2="ntp.ubuntu.com" type=CWD msg=audit(1440927434.124:40347): cwd="/" type=PATH msg=audit(1440927434.124:40347): item=0 name="/usr/sbin/ntpdate" inode=2006003 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL type=PATH msg=audit(1440927434.124:40347): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=5243184 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL type=PROCTITLE msg=audit(1440927434.124:40347): proctitle=2F62696E2F7368002F7573722F7362696E2F6E7470646174652D64656269616E002D73 These lines are connected by their 2nd field, msg equals to audit(1440927434.124:40347). This can be processed by the groupingby() parser in a similar way that db-parser() could do correllation. These are the options for groupingby(): * key(): specifies the key for the grouping, e.g. the value that must be the same for all messages in the group * scope(): specifies one of three values: "global", "host", "process", meaning the same as in db-parser; whether to apply grouping for all messages received by syslog-ng (global), only messages coming from the same host (host), or the same process/pid combination. * where(): specifies a filter condition, messages not matching the filter will NOT be added to the group. where() only has access to a single message, the current one being processed. * having(): specifies a filter condition that must match in order for the group to generate an aggregate message. having() has access to the entire group through the "context". * timeout(): specifies the maximum time to wait for all messages in the group to arrive. After this time, the group is assumed to be complete and is aggregation is triggered. * aggregate(): this specifies the aggregate message that's going to be generated when the group is complete. - tags(): - value(): - inherit-mode(): * inject-mode(): how the aggregate message is injected into the syslog-ng message routing, can be one of: "pass-through", "internal". * trigger(): trigger the closure of the group by matching an incoming message. If the filter condition specified here matches the incoming message, it will cause the aggregate message to be calculated, emitted and the group be discarded from the state table. A few use-cases where this can be useful: * Linux audit logs * postfix logs Signed-off-by: Balazs Scheidler <balazs.scheidler(a)balabit.com> -- Bazsi

1 0

OSE 3.7.2 questions
by Fekete, Róbert 13 Nov '15

13 Nov '15

Hi, I've seen that you had a few patches merged to 3.7.2, and I'm not sure what to include in the docs about them: - There were some csv-parser changes, is any of them user-visible? - What does linux-audit-parser do? Does it require any configuration, or it just works? Thanks! Robert

2 1

Re: [syslog-ng] Having trouble with java-modules
by Laszlo Budai 13 Nov '15

13 Nov '15

You shouldn't run autogen.sh for a release tarball. Could you copy-paste the download link for the syslog-ng tarball that doesn't contain the configure script? Sent from my HTC ----- Reply message ----- From: "Rory Toma" <rory(a)ooma.com> To: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] Having trouble with java-modules Date: Thu, Nov 12, 2015 21:17 This one doesn't have configure, and when I run autogen.sh, it errors with: Running autogen in 'lib/ivykis'... Don't know how to bootstrap submodule 'lib/ivykis' If I remove it, then later I get: configure: error: Internal ivykis sources not found in lib/ivykis. This is a hard dependency, unable to build syslog-ng without them. On 11/10/15 11:45 PM, Budai, László wrote: > You can download releases from here: > https://github.com/balabit/syslog-ng/releases > > L. > > On Wednesday, November 11, 2015, Rory Toma <rory(a)ooma.com > <mailto:rory@ooma.com>> wrote: > > I can't locate the 3.7 source release tarballs, only the git, > which isn't running autogen.sh correctly for me. > > On 11/10/15 9:39 PM, Budai, László wrote: >> >> Hi, >> >> Java is not supported in 3.6.x ( only in syslog-ng incubator ). >> For 3.7 it is ported(reworked) from the incubator, so you should >> give a try to use the java support with syslog-ng-3.7.2. >> >> L. >> >> On Nov 11, 2015 1:44 AM, "Rory Toma" <rory(a)ooma.com >> <javascript:_e(%7B%7D,'cvml','rory(a)ooma.com');>> wrote: >> >> I've downloaded syslog-ng-3.6.4 (I'm not using master because >> I can't >> get the configure script to generate correctly) >> >> I'm following the steps here: >> https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-… >> >> and have even added "--enable-java" >> >> I notice that even with JAVA_HOME set on the command line, >> config.log >> picks up the wrong java, and passing --with-java-home has no >> effect. >> >> In any case, I cannot get java modules installed. Is there an >> additional >> command to run to create/populate the java modules directory? >> >> This is on CentOS-6.7 x86_64 >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> >> >> ______________________________________________________________________________ >> Member info:https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation:http://www.balabit.com/support/documentation/?product=syslog-… >> FAQ:http://www.balabit.com/wiki/syslog-ng-faq >> > > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

2 1

Having trouble with java-modules
by Rory Toma 12 Nov '15

12 Nov '15

I've downloaded syslog-ng-3.6.4 (I'm not using master because I can't get the configure script to generate correctly) I'm following the steps here: https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-… and have even added "--enable-java" I notice that even with JAVA_HOME set on the command line, config.log picks up the wrong java, and passing --with-java-home has no effect. In any case, I cannot get java modules installed. Is there an additional command to run to create/populate the java modules directory? This is on CentOS-6.7 x86_64

3 5

Remote server not keeping message intact
by Gareth Allen 12 Nov '15

12 Nov '15

Hi all I'm sending Apache logs to a remote syslog-ng server, but the remote server isn't keeping the message intact. Source: My Apache log format: LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined What the log looks like: 172.27.15.149 - - [12/Nov/2015:08:30:59 +0000] "GET / HTTP/1.1" 200 3594 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36" My syslog-ng configuration: source s_apache { file("/var/log/apache2/access.log" follow_freq(1) flags(no-parse)); }; destination d_apache_tcp { tcp("x.x.x.x" port(514)); }; log { source(s_apache); destination(d_apache_tcp); }; Log server: source s_net { udp(port(514)); tcp(port(514)); }; template apache { template("${MESSAGE}\n"); template-escape(no); }; destination apache { file("/var/log/apachetest" template(apache)); }; What I see in /var/log/apachetest: - - [12/Nov/2015:08:30:59 +0000] "GET / HTTP/1.1" 200 3594 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36" As you can see the IP at the beginning of the log entry is being removed. I've tried using $MSG and $MSGONLY. Any ideas would be greatly appreciated. Gareth

5 6

Regex Solaris from Linux hosts in Syslog-ng config file
by vijay amruth 04 Nov '15

04 Nov '15

Hello All, We are drawing logs from several hosts which include solaris(10,11) , linux (centos, ubuntu, rhel) into syslog servers, I want to be able to separate solaris logs, is there any pattern we can match for solaris logs that you may know ? Thanks, Vijay Amrut.

9 20

format-json reverse order?
by Scheidler, Balázs 03 Nov '15

03 Nov '15

Hi, I've encountered a case where format-json orders keys not alphabetically, but rather in the other direction. Can you remember any reason for that? This is the only patch needed to fix the order, but I've figured there may have been some reason behind the ordering. static gint vp_walk_cmp(const gchar *s1, const gchar *s2) { - return strcmp(s2, s1); + return strcmp(s1, s2); } Thanks in advance, -- Bazsi

3 4

Syslog-ng 3.5 , "destination plugin python not found ..."
by Alexander Urcioli 02 Nov '15

02 Nov '15

Trying to use a custom python destination I wrote. I wrote the python class, did the necessary PYTHONPATH stuff and wrote a separate .conf file for syslog-ng. When testing this file with the command "syslog-ng -f seclog.conf -Fevd" I get the following error: Error parsing destination, destination plugin python not found in seclog.conf at line 9, column 31: destination d_SECLOG_SERVER { python( ^^^^^^ Is the python destination not supported in 3.5? Alex

2 1

unofficial openSUSE/fedora/RHEL syslog-ng 3.7.2 rpm packages
by Czanik, Péter 28 Oct '15

28 Oct '15

Hi, My unofficial syslog-ng 3.7.3 RPM packages for openSUSE/SLES/Fedora/EPEL are now available at the usual locations: Fedora/RHEL7: https://copr.fedoraproject.org/coprs/czanik/syslog-ng37/ RHEL6: https://copr.fedoraproject.org/coprs/czanik/syslog-ng37epel6/ openSUSE/SLES: https://build.opensuse.org/project/show/home:czanik:syslog-ng37 If you want to use the new Java based destination drivers, check my blog at https://czanik.blogs.balabit.com/2015/08/getting-started-with-syslog-ng-3-7… how to get started! Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

syslog-ng OSE: wildcards in source file path status
by Wout Peeters 27 Oct '15

27 Oct '15

Hi list, After some searching, I've noticed this question has been asked a few times before. The most recent answer I saw was from October 2014. Are there any updates regarding the implementation of wildcards in source file paths in syslog-ng OSE? Thanks!

3 2

unofficial syslog-ng 3.7.2 Debian packages
by Budai, László 27 Oct '15

27 Oct '15

Hi, I've just created my unofficial OBS repo[1] for syslog-ng 3.7.2 (latest revision: 3.7.2-2). List of supported OSs (i386/amd64): * Debian 7.0 * Debian 8.0 * Ubuntu 12.04 * Ubuntu 14.04 * Ubuntu 14.10 * Ubuntu 15.04 Install ------- example: Debian 8.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core=3.7.2-2` You can replace Debian_8.0 to * Debian_7.0 * xUbuntu_12.04, * xUbuntu_14.04 * xUbuntu_14.10 * xUbuntu_15.04 Available packages: * syslog-ng-core * syslog-ng-dbg * syslog-ng-dev * syslog-ng-mod-stomp * syslog-ng-mod-amqp * syslog-ng-mod-elastic * syslog-ng-mod-geoip * syslog-ng-mod-graphite * syslog-ng-mod-hdfs * syslog-ng-mod-http * syslog-ng-mod-java * syslog-ng-mod-java-common-lib * syslog-ng-mod-journal * syslog-ng-mod-json * syslog-ng-mod-kafka * syslog-ng-mod-mongodb * syslog-ng-mod-python * syslog-ng-mod-redis * syslog-ng-mod-riemann * syslog-ng-mod-smtp * syslog-ng-mod-sql * syslog-ng-mod-stomp Notes to the java destinations ------------------------------ About the ``` Error opening plugin module; module='mod-java', error='libjvm.so: cannot open shared object file: No such file or directory' ``` problem. libmod-java.so is linked to the libjvm.so library. If your java installation doesn't create a symlink in /usr/lib directory to libjvm.so then you have to modify your ld.so.conf or expand your LD_LIBRARY_PATH with the containing library before starting syslog-ng. eg.: LD_LIBRARY_PATH=/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:$LD_LIBRARY_PATH Links -------- [1] https://build.opensuse.org/project/show/home:laszlo_budai:syslog-ng [2] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.7.2 regards, Laszlo Budai

2 1

Re: [syslog-ng] date parser
by Scheidler, Balázs 25 Oct '15

25 Oct '15

Hi, I am basically done with cleaning up the stuff. You can find my latest version at https://github.com/balabit/syslog-ng/tree/f/date-parser I still have to clean up the branch, it is currently just a dump of patches, but the final state should be pretty close that I'd try to push for integration. There's one issue that is still open: strptime() at least as implemented in various libc versions is unusable, so we should probably roll our own implementation. Reasons: 1) tm_gmtoff is not portable 2) %z as a parsing format is not portable 3) %z doesn't accept ISO8601 standard zone offsets when there's a colon between hours and minutes Also, it can be affected by locale settings (which is not good for the syslog-ng case) and is probably very slow. (at one point syslog-ng used strptime for normal log traffic and I remember replacing it with hand-coded routines increased performance a lot). I'll try to get an strptime implementation from somewhere and tailor it to our needs. Any feedback is appreciated. Bazsi -- Bazsi On Sun, Oct 25, 2015 at 4:44 PM, Scheidler, Balázs < balazs.scheidler(a)balabit.com> wrote: > Hi, > > As I see you assume a date is from UTC if there's no timezone information. > I would think that using the local timezone is more intuitive in this case: > > testcase("Tue, 27 Jan 2015 11:48:46", NULL, "%a, %d %b %Y %T", > "2015-01-27T11:48:46+00:00"); > > What do you think? > > > -- > Bazsi > > On Sun, Oct 25, 2015 at 12:36 PM, Scheidler, Balázs < > balazs.scheidler(a)balabit.com> wrote: > >> thanks for the quick reply. >> >> >> -- >> Bazsi >> >> On Sun, Oct 25, 2015 at 12:35 PM, Vincent Bernat <vincent(a)bernat.im> >> wrote: >> >>> ❦ 25 octobre 2015 12:32 +0100, "Scheidler, Balázs" < >>> balazs.scheidler(a)balabit.com> : >>> >>> > I'd have a question though: I can see the date-offset() option for >>> > date parser, which seems to skip the specified number of characters in >>> > the input. >>> > >>> > Can you please describe the usecase behind that? I'd be reluctant to >>> > add such an option to a parser (as none of the other have such an >>> > option), and it should be possible to do the same using template >>> > functions, e.g. something like this: >>> > >>> > date-parser(template("$(substr $MSG 5)")); >>> > >>> > This would similarly skip the first 5 characters. >>> > >>> > If this addresses your original use-case, I'd drop the date-offset() >>> > option. >>> >>> Yes, this would cover my use case. >>> -- >>> Use library functions. >>> - The Elements of Programming Style (Kernighan & Plauger) >>> >> >> >

2 1

date parser
by Scheidler, Balázs 25 Oct '15

25 Oct '15

Hi Vincent, I am working on your date-parser() stuff that was integrated to syslog-ng-incubator recently. It is great stuff, so I'd like to make it part of the main syslog-ng codebase. I am doing some cleanups, which you can follow at the f/date-parser branch in github.com/balabit/syslog-ng I'd have a question though: I can see the date-offset() option for date parser, which seems to skip the specified number of characters in the input. Can you please describe the usecase behind that? I'd be reluctant to add such an option to a parser (as none of the other have such an option), and it should be possible to do the same using template functions, e.g. something like this: date-parser(template("$(substr $MSG 5)")); This would similarly skip the first 5 characters. If this addresses your original use-case, I'd drop the date-offset() option. Thanks in advance, -- Bazsi

1 0

Python external program destination and line buffer
by Giovanni Mancuso 21 Oct '15

21 Oct '15

Hi, i wrote a program destination (in python) that parse a message and insert data in redis. My code is: import re import fileinput import redis r = redis.Redis(host="127.0.0.1", port=6379, db=0) for line in fileinput.input(): tid = re.search('X-TransactionId: (.+?) ', line) #print tid.group(1) regexp_result = re.findall(" Recipient: ([A-z0-9._%+-]+(a)[A-z0-9.-]+)", line) for email in regexp_result: r.sadd(tid.group(1)+"_recs", email) I configure my destination in this way: destination python_to_file { program("/usr/bin/python /etc/syslog-ng/python/Ric_acc_dest.py" flush-lines(1)); }; It works only if i send many message. If i send only one message it doesn't insert nothing in redis. Any ideas? --

2 2

Re: [syslog-ng] syslog-ng 3.7.1 OSE compile error
by Ho, Ming 21 Oct '15

21 Oct '15

Hi Tibor, Below is the output. It seems only the last few lines related to eventlog # pkg-config --debug eventlog Option --debug seen Error printing disabled by default, value of --print-errors: 0 Error printing disabled Adding virtual 'pkg-config' package to list of known packages Scanning directory '/usr/lib64/pkgconfig' File 'xpyb.pc' appears to be a .pc file Will find package 'xpyb' in file '/usr/lib64/pkgconfig/xpyb.pc' File 'pangox.pc' appears to be a .pc file Will find package 'pangox' in file '/usr/lib64/pkgconfig/pangox.pc' File 'atk.pc' appears to be a .pc file Will find package 'atk' in file '/usr/lib64/pkgconfig/atk.pc' File 'gdk-pixbuf-xlib-2.0.pc' appears to be a .pc file Will find package 'gdk-pixbuf-xlib-2.0' in file '/usr/lib64/pkgconfig/gdk-pixbuf-xlib-2.0.pc' File 'gtk+-x11-2.0.pc' appears to be a .pc file Will find package 'gtk+-x11-2.0' in file '/usr/lib64/pkgconfig/gtk+-x11-2.0.pc' File 'xcomposite.pc' appears to be a .pc file Will find package 'xcomposite' in file '/usr/lib64/pkgconfig/xcomposite.pc' File 'xcb-shm.pc' appears to be a .pc file Will find package 'xcb-shm' in file '/usr/lib64/pkgconfig/xcb-shm.pc' File 'gail.pc' appears to be a .pc file Will find package 'gail' in file '/usr/lib64/pkgconfig/gail.pc' File 'cairo-ft.pc' appears to be a .pc file Will find package 'cairo-ft' in file '/usr/lib64/pkgconfig/cairo-ft.pc' File 'xcb-xprint.pc' appears to be a .pc file Will find package 'xcb-xprint' in file '/usr/lib64/pkgconfig/xcb-xprint.pc' File 'gmodule-export-2.0.pc' appears to be a .pc file Will find package 'gmodule-export-2.0' in file '/usr/lib64/pkgconfig/gmodule-export-2.0.pc' File 'cairo-xlib-xrender.pc' appears to be a .pc file Will find package 'cairo-xlib-xrender' in file '/usr/lib64/pkgconfig/cairo-xlib-xrender.pc' File 'xcb-xvmc.pc' appears to be a .pc file Will find package 'xcb-xvmc' in file '/usr/lib64/pkgconfig/xcb-xvmc.pc' File 'gio-unix-2.0.pc' appears to be a .pc file Will find package 'gio-unix-2.0' in file '/usr/lib64/pkgconfig/gio-unix-2.0.pc' File 'gmodule-no-export-2.0.pc' appears to be a .pc file Will find package 'gmodule-no-export-2.0' in file '/usr/lib64/pkgconfig/gmodule-no-export-2.0.pc' File 'xcb-xfixes.pc' appears to be a .pc file Will find package 'xcb-xfixes' in file '/usr/lib64/pkgconfig/xcb-xfixes.pc' File 'xft.pc' appears to be a .pc file Will find package 'xft' in file '/usr/lib64/pkgconfig/xft.pc' File 'gtk+-unix-print-2.0.pc' appears to be a .pc file Will find package 'gtk+-unix-print-2.0' in file '/usr/lib64/pkgconfig/gtk+-unix-print-2.0.pc' File 'xau.pc' appears to be a .pc file Will find package 'xau' in file '/usr/lib64/pkgconfig/xau.pc' File 'xcb-screensaver.pc' appears to be a .pc file Will find package 'xcb-screensaver' in file '/usr/lib64/pkgconfig/xcb-screensaver.pc' File 'libpng12.pc' appears to be a .pc file Will find package 'libpng12' in file '/usr/lib64/pkgconfig/libpng12.pc' File 'fontconfig.pc' appears to be a .pc file Will find package 'fontconfig' in file '/usr/lib64/pkgconfig/fontconfig.pc' File 'freetype2.pc' appears to be a .pc file Will find package 'freetype2' in file '/usr/lib64/pkgconfig/freetype2.pc' File 'xcb-damage.pc' appears to be a .pc file Will find package 'xcb-damage' in file '/usr/lib64/pkgconfig/xcb-damage.pc' File 'x11-xcb.pc' appears to be a .pc file Will find package 'x11-xcb' in file '/usr/lib64/pkgconfig/x11-xcb.pc' File 'xrandr.pc' appears to be a .pc file Will find package 'xrandr' in file '/usr/lib64/pkgconfig/xrandr.pc' File 'xcb-xevie.pc' appears to be a .pc file Will find package 'xcb-xevie' in file '/usr/lib64/pkgconfig/xcb-xevie.pc' File 'gtk+-2.0.pc' appears to be a .pc file Will find package 'gtk+-2.0' in file '/usr/lib64/pkgconfig/gtk+-2.0.pc' File 'xcb-shape.pc' appears to be a .pc file Will find package 'xcb-shape' in file '/usr/lib64/pkgconfig/xcb-shape.pc' File 'cairo-png.pc' appears to be a .pc file Will find package 'cairo-png' in file '/usr/lib64/pkgconfig/cairo-png.pc' File 'xcb-randr.pc' appears to be a .pc file Will find package 'xcb-randr' in file '/usr/lib64/pkgconfig/xcb-randr.pc' File 'xcb-dri2.pc' appears to be a .pc file Will find package 'xcb-dri2' in file '/usr/lib64/pkgconfig/xcb-dri2.pc' File 'xfixes.pc' appears to be a .pc file Will find package 'xfixes' in file '/usr/lib64/pkgconfig/xfixes.pc' File 'pangocairo.pc' appears to be a .pc file Will find package 'pangocairo' in file '/usr/lib64/pkgconfig/pangocairo.pc' File 'xcb-render.pc' appears to be a .pc file Will find package 'xcb-render' in file '/usr/lib64/pkgconfig/xcb-render.pc' File 'xcb-sync.pc' appears to be a .pc file Will find package 'xcb-sync' in file '/usr/lib64/pkgconfig/xcb-sync.pc' File 'xcb-xtest.pc' appears to be a .pc file Will find package 'xcb-xtest' in file '/usr/lib64/pkgconfig/xcb-xtest.pc' File 'xinerama.pc' appears to be a .pc file Will find package 'xinerama' in file '/usr/lib64/pkgconfig/xinerama.pc' File 'xcb-record.pc' appears to be a .pc file Will find package 'xcb-record' in file '/usr/lib64/pkgconfig/xcb-record.pc' File 'gdk-2.0.pc' appears to be a .pc file Will find package 'gdk-2.0' in file '/usr/lib64/pkgconfig/gdk-2.0.pc' File 'xcb.pc' appears to be a .pc file Will find package 'xcb' in file '/usr/lib64/pkgconfig/xcb.pc' File 'pixman-1.pc' appears to be a .pc file Will find package 'pixman-1' in file '/usr/lib64/pkgconfig/pixman-1.pc' File 'pangoxft.pc' appears to be a .pc file Will find package 'pangoxft' in file '/usr/lib64/pkgconfig/pangoxft.pc' File 'gmodule-2.0.pc' appears to be a .pc file Will find package 'gmodule-2.0' in file '/usr/lib64/pkgconfig/gmodule-2.0.pc' File 'xcursor.pc' appears to be a .pc file Will find package 'xcursor' in file '/usr/lib64/pkgconfig/xcursor.pc' File 'xcb-res.pc' appears to be a .pc file Will find package 'xcb-res' in file '/usr/lib64/pkgconfig/xcb-res.pc' File 'xcb-glx.pc' appears to be a .pc file Will find package 'xcb-glx' in file '/usr/lib64/pkgconfig/xcb-glx.pc' File 'libpng.pc' appears to be a .pc file Will find package 'libpng' in file '/usr/lib64/pkgconfig/libpng.pc' File 'cairo-svg.pc' appears to be a .pc file Will find package 'cairo-svg' in file '/usr/lib64/pkgconfig/cairo-svg.pc' File 'zlib.pc' appears to be a .pc file Will find package 'zlib' in file '/usr/lib64/pkgconfig/zlib.pc' File 'gobject-2.0.pc' appears to be a .pc file Will find package 'gobject-2.0' in file '/usr/lib64/pkgconfig/gobject-2.0.pc' File 'xcb-xv.pc' appears to be a .pc file Will find package 'xcb-xv' in file '/usr/lib64/pkgconfig/xcb-xv.pc' File 'xcb-composite.pc' appears to be a .pc file Will find package 'xcb-composite' in file '/usr/lib64/pkgconfig/xcb-composite.pc' Ignoring file '..' in search directory; not a .pc file File 'xcb-xf86dri.pc' appears to be a .pc file Will find package 'xcb-xf86dri' in file '/usr/lib64/pkgconfig/xcb-xf86dri.pc' File 'cairo-xlib.pc' appears to be a .pc file Will find package 'cairo-xlib' in file '/usr/lib64/pkgconfig/cairo-xlib.pc' File 'glib-2.0.pc' appears to be a .pc file Will find package 'glib-2.0' in file '/usr/lib64/pkgconfig/glib-2.0.pc' Ignoring file '.' in search directory; not a .pc file File 'gdk-x11-2.0.pc' appears to be a .pc file Will find package 'gdk-x11-2.0' in file '/usr/lib64/pkgconfig/gdk-x11-2.0.pc' File 'pthread-stubs.pc' appears to be a .pc file Will find package 'pthread-stubs' in file '/usr/lib64/pkgconfig/pthread-stubs.pc' File 'xcb-dpms.pc' appears to be a .pc file Will find package 'xcb-dpms' in file '/usr/lib64/pkgconfig/xcb-dpms.pc' File 'gio-2.0.pc' appears to be a .pc file Will find package 'gio-2.0' in file '/usr/lib64/pkgconfig/gio-2.0.pc' File 'gdk-pixbuf-2.0.pc' appears to be a .pc file Will find package 'gdk-pixbuf-2.0' in file '/usr/lib64/pkgconfig/gdk-pixbuf-2.0.pc' File 'xext.pc' appears to be a .pc file Will find package 'xext' in file '/usr/lib64/pkgconfig/xext.pc' File 'xrender.pc' appears to be a .pc file Will find package 'xrender' in file '/usr/lib64/pkgconfig/xrender.pc' File 'xcb-xselinux.pc' appears to be a .pc file Will find package 'xcb-xselinux' in file '/usr/lib64/pkgconfig/xcb-xselinux.pc' File 'pangoft2.pc' appears to be a .pc file Will find package 'pangoft2' in file '/usr/lib64/pkgconfig/pangoft2.pc' File 'pango.pc' appears to be a .pc file Will find package 'pango' in file '/usr/lib64/pkgconfig/pango.pc' File 'cairo-pdf.pc' appears to be a .pc file Will find package 'cairo-pdf' in file '/usr/lib64/pkgconfig/cairo-pdf.pc' File 'cairo-ps.pc' appears to be a .pc file Will find package 'cairo-ps' in file '/usr/lib64/pkgconfig/cairo-ps.pc' File 'xcb-xinerama.pc' appears to be a .pc file Will find package 'xcb-xinerama' in file '/usr/lib64/pkgconfig/xcb-xinerama.pc' File 'gthread-2.0.pc' appears to be a .pc file Will find package 'gthread-2.0' in file '/usr/lib64/pkgconfig/gthread-2.0.pc' File 'x11.pc' appears to be a .pc file Will find package 'x11' in file '/usr/lib64/pkgconfig/x11.pc' File 'cairo.pc' appears to be a .pc file Will find package 'cairo' in file '/usr/lib64/pkgconfig/cairo.pc' File 'xi.pc' appears to be a .pc file Will find package 'xi' in file '/usr/lib64/pkgconfig/xi.pc' Scanning directory '/usr/share/pkgconfig' File 'fontsproto.pc' appears to be a .pc file Will find package 'fontsproto' in file '/usr/share/pkgconfig/fontsproto.pc' File 'bigreqsproto.pc' appears to be a .pc file Will find package 'bigreqsproto' in file '/usr/share/pkgconfig/bigreqsproto.pc' File 'xcmiscproto.pc' appears to be a .pc file Will find package 'xcmiscproto' in file '/usr/share/pkgconfig/xcmiscproto.pc' File 'compositeproto.pc' appears to be a .pc file Will find package 'compositeproto' in file '/usr/share/pkgconfig/compositeproto.pc' File 'xf86vidmodeproto.pc' appears to be a .pc file Will find package 'xf86vidmodeproto' in file '/usr/share/pkgconfig/xf86vidmodeproto.pc' File 'glproto.pc' appears to be a .pc file Will find package 'glproto' in file '/usr/share/pkgconfig/glproto.pc' File 'xf86miscproto.pc' appears to be a .pc file Will find package 'xf86miscproto' in file '/usr/share/pkgconfig/xf86miscproto.pc' File 'randrproto.pc' appears to be a .pc file Will find package 'randrproto' in file '/usr/share/pkgconfig/randrproto.pc' File 'resourceproto.pc' appears to be a .pc file Will find package 'resourceproto' in file '/usr/share/pkgconfig/resourceproto.pc' File 'dri3proto.pc' appears to be a .pc file Will find package 'dri3proto' in file '/usr/share/pkgconfig/dri3proto.pc' File 'kbproto.pc' appears to be a .pc file Will find package 'kbproto' in file '/usr/share/pkgconfig/kbproto.pc' File 'renderproto.pc' appears to be a .pc file Will find package 'renderproto' in file '/usr/share/pkgconfig/renderproto.pc' File 'dmxproto.pc' appears to be a .pc file Will find package 'dmxproto' in file '/usr/share/pkgconfig/dmxproto.pc' File 'xextproto.pc' appears to be a .pc file Will find package 'xextproto' in file '/usr/share/pkgconfig/xextproto.pc' File 'xf86driproto.pc' appears to be a .pc file Will find package 'xf86driproto' in file '/usr/share/pkgconfig/xf86driproto.pc' File 'inputproto.pc' appears to be a .pc file Will find package 'inputproto' in file '/usr/share/pkgconfig/inputproto.pc' File 'xineramaproto.pc' appears to be a .pc file Will find package 'xineramaproto' in file '/usr/share/pkgconfig/xineramaproto.pc' File 'recordproto.pc' appears to be a .pc file Will find package 'recordproto' in file '/usr/share/pkgconfig/recordproto.pc' File 'xf86bigfontproto.pc' appears to be a .pc file Will find package 'xf86bigfontproto' in file '/usr/share/pkgconfig/xf86bigfontproto.pc' Ignoring file '..' in search directory; not a .pc file File 'fixesproto.pc' appears to be a .pc file Will find package 'fixesproto' in file '/usr/share/pkgconfig/fixesproto.pc' File 'videoproto.pc' appears to be a .pc file Will find package 'videoproto' in file '/usr/share/pkgconfig/videoproto.pc' Ignoring file '.' in search directory; not a .pc file File 'xproxymngproto.pc' appears to be a .pc file Will find package 'xproxymngproto' in file '/usr/share/pkgconfig/xproxymngproto.pc' File 'presentproto.pc' appears to be a .pc file Will find package 'presentproto' in file '/usr/share/pkgconfig/presentproto.pc' File 'shared-mime-info.pc' appears to be a .pc file Will find package 'shared-mime-info' in file '/usr/share/pkgconfig/shared-mime-info.pc' File 'scrnsaverproto.pc' appears to be a .pc file Will find package 'scrnsaverproto' in file '/usr/share/pkgconfig/scrnsaverproto.pc' File 'xf86dgaproto.pc' appears to be a .pc file Will find package 'xf86dgaproto' in file '/usr/share/pkgconfig/xf86dgaproto.pc' File 'evieproto.pc' appears to be a .pc file Will find package 'evieproto' in file '/usr/share/pkgconfig/evieproto.pc' File 'xproto.pc' appears to be a .pc file Will find package 'xproto' in file '/usr/share/pkgconfig/xproto.pc' File 'usbutils.pc' appears to be a .pc file Will find package 'usbutils' in file '/usr/share/pkgconfig/usbutils.pc' File 'damageproto.pc' appears to be a .pc file Will find package 'damageproto' in file '/usr/share/pkgconfig/damageproto.pc' File 'dri2proto.pc' appears to be a .pc file Will find package 'dri2proto' in file '/usr/share/pkgconfig/dri2proto.pc' Looking for package 'eventlog' Looking for package 'eventlog-uninstalled' Looking for 'eventlog' using legacy -config scripts Calling gnome-config Thanks. Ming From: Benke, Tibor [mailto:tibor.benke@balabit.com] Sent: Wednesday, October 21, 2015 7:19 AM To: Syslog-ng users' and developers' mailing list; Ho, Ming Subject: Re: [syslog-ng] syslog-ng 3.7.1 OSE compile error Hi, What's the output of the following command? pkg-config --debug eventlog Once I saw that autotools didn't find a package but the command line pkg-config did. Tibor On Tue, Oct 20, 2015 at 11:48 PM, Ho, Ming <Mingho(a)visa.com<mailto:Mingho@visa.com>> wrote: Hello, I followed syslog-ng 3.7 admin guide to compile eventlog-0.2.12+20120504+1700 on a Redhat 6.5 server and it created the following directory: # pwd /usr/local/lib # ll total 152 -rw-r–r– 1 root root 91250 Oct 16 21:55 libevtlog.a -rwxr-xr-x 1 root root 957 Oct 16 21:55 libevtlog.la<http://libevtlog.la> lrwxrwxrwx 1 root root 18 Oct 16 21:55 libevtlog.so -> libevtlog.so.0.0.0 lrwxrwxrwx 1 root root 18 Oct 16 21:55 libevtlog.so.0 -> libevtlog.so.0.0.0 -rwxr-xr-x 1 root root 51128 Oct 16 21:55 libevtlog.so.0.0.0 drwxr-x— 2 root root 4096 Oct 16 21:55 pkgconfig I also set PKG_CONFIG_PATH # echo $PKG_CONFIG_PATH /usr/local/lib/pkgconfig But when I ran “./configure” in the syslog-ng-3.7.1 directory, I got the following error: checking for clock_gettime… yes checking for EVTLOG… no configure: error: Package requirements (eventlog >= 0.2.12) were not met: No package ‘eventlog’ found Consider adjusting the PKG_CONFIG_PATH environment variable if you installed software in a non-standard prefix. Alternatively, you may set the environment variables EVTLOG_CFLAGS and EVTLOG_LIBS to avoid the need to call pkg-config. See the pkg-config man page for more details. # pwd /usr/local/lib/pkgconfig # ll total 4 -rw-r–r– 1 root root 238 Oct 16 21:55 eventlog.pc # cat eventlog.pc prefix=/usr/local exec_prefix=${prefix} libdir=${exec_prefix}/lib includedir=${prefix}/include/eventlog Name: EventLog Description: General system logging format library Version: 0.2.12 Cflags: -I${includedir} Libs: -L${libdir} -levtlog Appreciate anyone who can tell me what’s wrong Ming ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

Parse client IP out of Proxy Protocol Line in TCP syslog->ELB->syslog-ng
by Nadine Miller 21 Oct '15

21 Oct '15

I've searched through the archives and spent some time trying to find possible answers on the web, but haven't found a definitive answer. I'm in a situation where I need to parse syslog streams being forwarded through an AWS ELB. The normal configuration of the ELB resets the source IP to be the ELB's IP address. Logs are coming from multiple AWS VPCs, and we've already discovered duplicate hostnames across different VPCs, which has mingled logs from different hosts into one receiving log file. The ELB has another mode, referred to as "Proxy Protocol" which adds a single line to the TCP stream in the form: PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP + single space + PROXY_IP + single space + CLIENT_PORT + single space + PROXY_PORT + "\r\n" Example: PROXY TCP4 198.51.100.22 203.0.113.7 35646 80\r\n Is it possible to use this proxy line in syslog-ng to properly segregate the log messages? If so, what would be the best method to use? I've done a lot of filtering/templating with normal UDP syslog and syslog-ng, but this is the first time I've had to consider something crazy like this. Currently there is no option at this time to change configurations at endpoints sending the syslog messages, nor can we remove the ELB. For reference: http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enabl… Thanks in advance-- =N=

6 12

syslog-ng 3.7.1 OSE compile error
by Ho, Ming 21 Oct '15

21 Oct '15

Hello, I followed syslog-ng 3.7 admin guide to compile eventlog-0.2.12+20120504+1700 on a Redhat 6.5 server and it created the following directory: # pwd /usr/local/lib # ll total 152 -rw-r-r- 1 root root 91250 Oct 16 21:55 libevtlog.a -rwxr-xr-x 1 root root 957 Oct 16 21:55 libevtlog.la lrwxrwxrwx 1 root root 18 Oct 16 21:55 libevtlog.so -> libevtlog.so.0.0.0 lrwxrwxrwx 1 root root 18 Oct 16 21:55 libevtlog.so.0 -> libevtlog.so.0.0.0 -rwxr-xr-x 1 root root 51128 Oct 16 21:55 libevtlog.so.0.0.0 drwxr-x- 2 root root 4096 Oct 16 21:55 pkgconfig I also set PKG_CONFIG_PATH # echo $PKG_CONFIG_PATH /usr/local/lib/pkgconfig But when I ran "./configure" in the syslog-ng-3.7.1 directory, I got the following error: checking for clock_gettime... yes checking for EVTLOG... no configure: error: Package requirements (eventlog >= 0.2.12) were not met: No package 'eventlog' found Consider adjusting the PKG_CONFIG_PATH environment variable if you installed software in a non-standard prefix. Alternatively, you may set the environment variables EVTLOG_CFLAGS and EVTLOG_LIBS to avoid the need to call pkg-config. See the pkg-config man page for more details. # pwd /usr/local/lib/pkgconfig # ll total 4 -rw-r-r- 1 root root 238 Oct 16 21:55 eventlog.pc # cat eventlog.pc prefix=/usr/local exec_prefix=${prefix} libdir=${exec_prefix}/lib includedir=${prefix}/include/eventlog Name: EventLog Description: General system logging format library Version: 0.2.12 Cflags: -I${includedir} Libs: -L${libdir} -levtlog Appreciate anyone who can tell me what's wrong Ming

2 1

Create Pattern-DB rules
by Justin Kala 20 Oct '15

20 Oct '15

Hello I am trying to create Pattern-DB for the following Authorization messages coming from O/S. Can you help on creating a matching rule . Note: I do not have PATTERN-DB parser utility.I am going to create the db-parser.xml manually and put this rules inside the file. 2014-09-28T14:12:44-04:00 abcdef01-app/abcdef01-app sshd[11019]: [ID 800047 auth.notice] Failed password for root from 100.200.255.01 port 54438 ssh2 2014-09-28T14:03:46-04:00 abcdef01-app/abcdef01-app sshd[27420]: [ID 800047 auth.notice] Failed publickey for root from 100.200.255.02 port 59219 ssh2 2014-09-28T14:08:28-04:00 abcdef01-app/abcdef01-app sshd[3954]: [ID 800047 auth.notice] Failed keyboard-interactive for root from 100.200.255.03 port 65410 ssh2 2014-09-28T14:10:11-04:00 abcdef01-app/abcdef01-app sshd[5222]: [ID 293258 auth.error] libsldap: Status: 49 Mesg: openConnection: simple bind failed - Invalid credentials Thanks & Regards Kaladhar

4 16

Parsing multifield lines
by Giovanni Mancuso 20 Oct '15

20 Oct '15

Hi, i'm writing a parser for lines having a repeated field on the same line eg(Recipient is repeated 3 times): ID: 123AABB456 Sender: user1(a)example.com Recipient: user2(a)example.com Recipient: user3(a)example.com Recipient: user3(a)example.com Status: Sent I would like to have a CUSTOM MACRO with all recipients. Is there a way to do this? Thanks --

3 5