- syslog-ng - lists.balabit.hu

Filtering binary data fields and catch all
by Xuri Nagarin 22 May '13

22 May '13

Hi, My log source sends data over syslog in CEF (Arcsight format). I have a simple parser written in Syslog-NG that takes advantage of the CEF format (that is all messages are in eight fields separated by a pipe char). -----------xxxxxxxxxxxxxxxxx------------------- parser p_cef { csv-parser(columns("cef.ff","cef.vendor","cef.product","cef.c4","cef.c5","cef.c6","cef.c7","cef.c8") delimiters("|") flags(drop-invalid) ); }; destination d_file { file("/var/log/net/${cef.vendor}/${cef.product}/logfile"); }; log { source(s_tcp); parser(p_cef); destination(d_file); }; -----------xxxxxxxxxxxxxxxxx------------------- Some events coming in contain binary data that get translated into control characters and create thousands of directories with garbage in the names. How do I handle binary data in the message? Should I use the sanitize function in the destination/file definition or is there a better way to do it? Also, how do I create a catch-all destination for all the messages that do not match my parser? Thanks!

2 2

Extremely slow receive on TCP
by Xuri Nagarin 18 May '13

18 May '13

Hi, I have a syslog-ng server that does not seem to receive well over TCP. Instead of an expected ~150 Mbits/s, I get a trickle of few events/sec. The setup is: Server L is an ArcSight Logger that streams out events over TCP or UDP. Server X is a test Syslog-ng VM that is running 3.2.5 (from EPEL). Server P is running Syslog-NG 3.2.5 built from EPEL srpms with TLS enabled. L -> X over UDP Port 514, I get ~100 Mbit/sec L -> X over TCP Port 514, I get ~150 Mbit/sec L-> P over UDP Port 514, I get ~100 Mbit/sec L-> P over TCP Port 514, I get almost nothing. Just a few tens of events make it through over a half hour period. From L, I relay the same data over and over so there isn't an issue with what's being streamed. Config on X and P is identical and very basic: ----------------------------------------------------- options { time_reopen (10); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (yes); keep_timestamp(yes); log_fifo_size (1000); stats_freq(60); flush_timeout(10000); log_fetch_limit(100); flush_lines(100); log_iw_size(100); }; source s_tcp { tcp( port(514) max-connections(100) ); }; source s_udp { udp( port(514)); }; destination d_tcp { file("/var/log/tcp/$HOST-$YEAR-$MONTH-$DAY-$HOUR"); }; destination d_udp { file("/var/log/udp/$HOST-$YEAR-$MONTH-$DAY-$HOUR"); }; ----------------------------------------------------- L -> X over TCP, L opens one TCP connections and everything comes over it. L -> P over TCP, L keeps opening up connections with P but no data comes over those connections or very few events come over those connections. I am going to investigate if there are any network QoS rules in place between L and P but wondering if there might be something in Syslog-NG that might be broken. I ran syslog-ng in debug mode but saw no errors. I looked at the pcap captures between L and P and saw no errors. Any other clues/ideas for troubleshooting are welcome. TIA!

2 3

Re: [syslog-ng] weird filter problem
by Evan Rempel 17 May '13

17 May '13

Wait a second. Version 3.2.x ... really? That's quite old. There was a bug with the .classifier.X tags some time in the past, and it might have been in those old versions. Certainly version 3.3 would be recommended, and all of y work is done with 3.4.x My advice my be specific to version 3.4 :-( Evan Rempel 250.271.7691 University Systems, University of Victoria Evan Rempel <erempel(a)uvic.ca> wrote: This definitely works. I'm using it right now. If it isn't working, then your pattern in the patterndb is not matching. We literally run millions of messages per hour through this exact filter ... I copied and pasted it from our pattern database. Evan Rempel 250.271.7691 University Systems, University of Victoria Russell Fulton <r.fulton(a)auckland.ac.nz> wrote: On 11/05/2013, at 2:26 PM, Evan Rempel <erempel(a)uvic.ca> wrote: > Try this filter > > > filter f_unknown { > tags(".classifier.unknown"); > }; > This always appears to return true. I.e. this filter includes everything. Negating it includes nothing. I have tried to install 3.2.5 as this is the last version that ELSA is confirmed to work with but that does not start: Starting syslog-ng /usr/local/syslog-ng/sbin/syslog-ng: error while loading shared libraries: libsyslog-ng.so.0: cannot open shared object file: No such file or directory So far as I can tell all the lib files are present and correct and in the same place as the previous version? I have syslog-ng installed in /usr/local/syslog-ng-<version> and a symlink /usr/local/syslog-ng pointing to the version to use. Russell ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

4 5

[PATCH (3.5)] system-source: Make /proc/kmsg optional
by Gergely Nagy 16 May '13

16 May '13

In certain environments (vserver and OpenVZ come to mind), /proc/kmsg is not readable, not even for root. On these systems, emit a warning, and skip it, instead of aborting the startup. Reported-by: Tamas Pal <folti(a)balabit.hu> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/system-source/system-source.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/modules/system-source/system-source.c b/modules/system-source/system-source.c index 3c72859..955ade7 100644 --- a/modules/system-source/system-source.c +++ b/modules/system-source/system-source.c @@ -167,8 +167,17 @@ system_sysblock_add_linux_kmsg(GString *sysblock) close (fd); } - system_sysblock_add_file(sysblock, kmsg, -1, - "kernel", "kernel", format); + if (access(kmsg, R_OK) == -1) + { + msg_warning("system(): The kernel message buffer is not readable, " + "please check permissions if this is unintentional.", + evt_tag_str("device", kmsg), + evt_tag_errno("error", errno), + NULL); + } + else + system_sysblock_add_file(sysblock, kmsg, -1, + "kernel", "kernel", format); } gboolean -- 1.7.10.4

2 1

insider 2013-05: syslog-ng configurator on Android; Using syslog-ng with Splunk; EU data protection and logging
by Peter Czanik 16 May '13

16 May '13

Dear syslog-ng users, This is the 24th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. FEATURED NEWS syslog-ng configurator app for Android -------------------------------------- There is now a new configurator app available for Android, developed as a hobby project by one of the syslog-ng team members. The focus of the application is to create a syslog-ng.conf wich provides optimum performance based on a number of questions. For more details and download locations read the authors blog at http://pzolee.blogs.balabit.com/2013/04/little-syslog-ng-configurator-appli… syslog-ng and Splunk -------------------- We often receive questions, how to use syslog-ng and Splunk together in a logging infrastructure. We collected the most popular usage scenarios into a white paper, together with example configurations to make testing and integration even more easy. The WP is available at http://www.balabit.com/support/documentation/pdf/syslog-ng_splunk_deploymen… syslog-ng is participating GSoC again ------------------------------------- Just as last year, syslog-ng is participating Google Summer of Code under the umbrella of the openSUSE project. We have candidates for developing a native mysql destination, a redis destination, a XMPP (jabber) destination and log signing, which is a big improvement from last year, where we only had a single candidate for a similar number of development projects. The application process is closed now and there are still a couple of weeks to go, before the final list of approved students is announced. http://news.opensuse.org/2013/04/25/opensuse-hedgewars-and-owncloud-are-mov… Big changes ahead for EU data protection regulation --------------------------------------------------- This summer will most likely bring big changes in the regulation of Data Protection in the European Union. We collected these proposed changes and also how syslog-ng and proper central log management can help to comply with these regulations: http://jluby.blogs.balabit.com/2013/04/22/big-changes-ahead-for-eu- <http://jluby.blogs.balabit.com/2013/04/22/big-changes-ahead-for-eu-data-pro…>data-protection-regulation/ <http://jluby.blogs.balabit.com/2013/04/22/big-changes-ahead-for-eu-data-pro…> Compiling syslog-ng with MS SQL support on RHEL / CentOS & Co. -------------------------------------------------------------- During the past few months many people asked, how to log from syslog-ng to MS SQL on RHEL or CentOS. If you cannot buy syslog-ng PE (http://www.balabit.com/network-security/syslog-ng/central-syslog-server) follow these steps to compile all the necessary components yourselfand configure the MS SQL part: https://czanik.blogs.balabit.com/2013/04/compiling-syslog-ng-with-ms-sql-su… SHORT NEWS: * RFC5424 syslog between syslog-ng and rsyslog: https://czanik.blogs.balabit.com/2013/04/using-the-rfc5424-syslog-protocol-… * Linuxwochen: syslog-ng, AMQP, and 3D printers: http://czanik.blogs.balabit.com/2013/05/czp-linuxwochen-syslog-ng-amqp-and-… Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

Problems with failed connections and time_reopen()?
by Matt Wise 13 May '13

13 May '13

We're running Syslog-NG 3.3.4 in our mixed Ubuntu 10/12 environment. We use SSL for all of our syslog-to-syslog connections, and have logging going to two different data pipelines. Data Dest #1: SyslogNG Client ----(SSL)----> SyslogNG Server ------> Logstash File-read-in-service Data Dest #2: SyslogNG Client ----(SSL)----> Stunnel Service ------> Flume Syslog Service The data streams work fine most of the time, but if we restart either the remote syslog-ng server, or the stunnel service, it seems that the syslog ng clients don't try to reconnect for a LONG time (or ever) to the endpoints again. I end up seeing the connection on the client go into a CLOSE_WAIT state, and syslog-ng keeps thinking that its sending log events through the connection, so it seems to never try to reconnect. I've tried setting time_reopen() to 0, 1 and 5... no luck or change in behavior. Any thoughts? --Matt

3 9

Question on custom log writer message output
by Dylan Kulesza 12 May '13

12 May '13

I'm working on a custom module to integrate with a third parties native log format. My intent is to have messages come into syslog-ng and processed as usual and then sent out a custom destination driver. Right now I've hacked together different code to make it work (tcp socket connection per log source) and now I'm at the point of actually sending a custom message. I've tried to stay as "true" to the syslog-ng as possible and have leveraged the log_forward_msg method to send my LogMessage. I was hoping I could just prepend data to the LogMessage but realized after doing all the other leg work that it wasn't a simple string :) So, my question is - what would the easiest way to leverage the existing queue->log_forward_msg (doesn't require the socket to be open vs examples such as spoof_source in afsocket) to write a custom message? I see that LogTemplate may have what I need, but after submerging myself in syslog-ng for the past week I'm not seeing clearly... Can anyone lend a hint/helping hand? What I'm trying to do: Open Socket Send Magic/StartPacket Prepend all log messages with a byte message - for example: 040404040400010MESSAGE (Of course Message would be in bytes/hex). It seems I would create an NVENTRY for my prepend message and then override log_writer_format_log? to do this? Not 100% clear how I would accomplish this. I also don't want to change any of the core/lib syslog-ng to accomplish this. Should be implemented purely as a module. Thanks!

2 2

weird filter problem
by Russell Fulton 11 May '13

11 May '13

Currently on 3.2.4. I am using patterndb to classify my incoming logs and want to send those that match pattern rules do a particular destination. filter f_unknown { "${.classifier.class}" ne 'unknown' ; }; but this appears to always fail (if I change ne to eq then I get everything). I have also tried using match with the same result. Russell

2 3

rewrite part of the message...
by Russell Fulton 11 May '13

11 May '13

Apologies to those of you who have already seen this on the ELSA list. I want to get rid of the explanatory essay that accompanies some eventlog messages from windows. I tried this: rewrite r_snarex { subst("\s+This event is generated when[^|]+\|", "|", value("MSGONLY") type("pcre")); }; and added it to log section, but it did not work. I have tried various variations on the theme too. Russell

2 2

syslog-ng and logrotate
by Rakesh Rajasekharan 10 May '13

10 May '13

I am using the syslog-ng 3.3.7 version. My question is , can we manage log rotation within the syslog-ng conf file ? Please advice

4 5