April 2011 - syslog-ng - lists.balabit.hu

syslog-ng Insider - April 2011
by Peter Czanik 14 Apr '11

14 Apr '11

Dear syslog-ng users, This is the second issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> FEATURED NEWS syslog-ng OSE 3.3 development ----------------------------- The next syslog-ng release is developed at rocket speed. The second alpha release brought in support for MongoDB as destination, and threading for increased performance and scalability. Community involvement in the development process is increasing, with many patches and detailed bug reports, which helped to fix correlation problems. Direct SMTP support is also available as a not yet merged patch. There were also long discussions about JSON and name value pairs with value-pairs() and tfjson() available as patches. One can download the alpha2 release, or if impatient also check it out from git, which has regular updates at http://git.balabit.hu/?p=bazsi/syslog-ng-3.3.git;a=summary Alpha2 release: https://bazsi.blogs.balabit.com/2011/03/the-2nd-alpha-for-syslog-ng-3-3-is-… Mailing list archive with many patches: https://lists.balabit.hu/pipermail/syslog-ng/ A blog, how to get started with coding: http://algernon.blogs.balabit.com/2011/01/howto-write-a-simple-template-fun… syslog-ng called open source project of the year ------------------------------------------------ IDG hold its first Open Source conference on the 24th of February in Hungary. A new award was established for open source projects, and it was first given to syslog-ng OSE. Blog series on web GUIs for syslog-ng ------------------------------------- Web based GUIs for syslog-ng is a hot topic recently, so a new blog series was started. Loganalyzer, Loggly and Logzilla are already covered and a few more reviews (ELSA, Octopussy) are already under way. If you have any suggestions what else to cover, please let us know! Logzilla (formerly php-syslog-ng): http://czanik.blogs.balabit.com/2011/04/logzilla-brings-syslog-ng-and-cisco… Loganalyzer: https://czanik.blogs.balabit.com/2011/02/loganalyzer/ Loggly: https://czanik.blogs.balabit.com/2011/03/logging-to-the-cloud/ Support the syslog-ng project with your testimonial! ---------------------------------------------------- We send syslog-ng T-Shirts for the best use-cases and testimonials. We are especially interested in how you use syslog-ng: your environment, performance metrics, the UNIX platforms that you use and anything that you deem interesting about your setup. This is your possibility to influence the syslog-ng development process and to help the syslog-ng project. http://www.balabit.com/testimonials/send OTHER SHORT NEWS * openSUSE 11.4 comes with syslog-ng 3.2: https://czanik.blogs.balabit.com/2011/03/opensuse-11-4-with-syslog-ng-and-s… * FreeBSD 8.2 comes with syslog-ng 3.2: https://czanik.blogs.balabit.com/2011/02/freebsd-8-2-released/ * Intrusion prevention based on syslog-ng and patterndb: https://bazsi.blogs.balabit.com/2011/02/another-nice-hack-with-syslog-ng/ * ELSA (Enterprise Log Search and Archive Project) had its first public release: http://code.google.com/p/enterprise-log-search-and-archive RECENT WHITEPAPERS * http://www.balabit.com/whitepaper?wp=8988187171444164966624785 This document aims at briefly demonstrating the technological and business processes that led to the revaluation of log information to such an extent that its collection and procession has become one of the most essential tasks of contemporary corporate IT. Furthermore, we attempt to identify those critical points of a central logging management project that determine its success the most. We hope that taking our six pieces of advice relating to these six points will substantially contribute to the successful professional activities of our readers. ARCHIVE http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

Openssl SSL3_WRITE_PENDING error correction
by Juhasz Viktor 12 Apr '11

12 Apr '11

Hi All! We've found a bug in the log_proto_framed_client. The bug is the following: - using ssl communication between the server and client. - using syslog protocol - If client send messages faster than the server can process messages, then the client go into the SSL3_WRITE_PENDING openssl error and close the connection. partial message handling correction: the flush is called only if proto have partially sent message after flush is called log_proto_text_client_post have to return because the post send one message per call it is important in case of framed messages diff --git a/lib/logproto.c b/lib/logproto.c index d821275..7f2f897 100644 --- a/lib/logproto.c +++ b/lib/logproto.c @@ -145,9 +145,13 @@ log_proto_text_client_post(LogProto *s, guchar *msg, gsize msg_len, gboolean *co g_assert(self->super.convert == (GIConv) -1); *consumed = FALSE; - rc = log_proto_flush(s); - if (rc == LPS_ERROR) - goto write_error; + if(self->partial) + { + rc = log_proto_flush(s); + if (rc == LPS_ERROR) + goto write_error; + return rc; + } /* OK, partial buffer empty, now flush msg that we just got */ @@ -1774,6 +1778,7 @@ log_proto_framed_client_new(LogTransport *transport) self->super.super.prepare = log_proto_text_client_prepare; self->super.super.post = log_proto_framed_client_post; + self->super.super.flush = log_proto_text_client_flush; self->super.super.transport = transport; self->super.super.convert = (GIConv) -1; return &self->super.super; Best Regards, Viktor

3 4

Current stable version
by Yu Watanabe 12 Apr '11

12 Apr '11

Hi all I would like to ask question about open source edition. Which version will be the current stable version? Thanks, Yu Watanabe

1 0

syslog-ng IPv6 problem ver 3.01
by Arya, Manish Kumar 10 Apr '11

10 Apr '11

I am getting syntax error on line udp6 or tcp6 in destination same works well as source in version 3.2.1 deb:/var/log# netstat -na | grep 514 tcp6 0 0 fd00:abcd:4321:2:20:514 :::* LISTEN udp6 0 0 fd00:abcd:4321:2:20:514 :::* here is my config for syslog client using syslog-ng 3.01 source s_local { internal(); unix-stream("/dev/log"); file("/proc/kmsg" log_prefix("kernel: ")); udp(); }; destination ipv6_host { # udp6(ip(fd00:abcd:4321:2:20c:29ff:fea8:9671) port(514)); tcp6(ip(fd00:abcd:4321:2:20c:29ff:fea8:9671) port(514)); }; destination d_local { file("/var/log/messages"); }; log { source(s_local); destination(ipv6_host); destination(d_local); };

4 7

syslog-ng -s errors with catchall flag (1.6 to 3.1)
by Mishou Michael 09 Apr '11

09 Apr '11

Folks, I've done a lot of reading of the list archives through Google, but this is the first question that's really stumped me and also stumped the archives as well. I'm currently converting a 1.6 config to version 3.1, eliminating the warnings generated by -s and whatnot. One of the warnings I get is the following: Unknown log statement flag; flag='catchall' The configuration line this warning/error is referencing looks like this: log { source(network); destination(finalcatchall); destination(catchall_relay); flags(final,catchall); }; For some reason, the syslog-ng version I'm using doesn't like that flag option, but according to the 3.1 documentation, it is valid, as far as I can see. http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1- guide-admin-en.html/index.html-single.html#logpath I tried removing the final flag (thinking perhaps it was redundant) and just leaving the catchall in, but the error persists. Here is the version information for the syslog-ng I'm using, perhaps it is a bug? I'm not quite sure how to search the bug database, if anyone could point me in the right direction it would be appreciated. # /usr/local/sbin/syslog-ng -V syslog-ng 3.1.2 Installer-Version: 3.1.2 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainli ne--3.1#master#8bf13c304b6ab5fc1a372b49d55c78370efe14ca Compile-Date: Oct 25 2010 23:56:18 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: on Enable-Sun-Door: on Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: on The version of syslog-ng I'm using is on Solaris 10 SPARC, pre-compiled packages I got from here: http://www.sunfreeware.com/programlistsparc10.html#syslogng If anyone has any ideas, please let me know. Thank you very much, --Mike

4 6

client -> server logging.. filters?
by Matt Wise 08 Apr '11

08 Apr '11

I'm doing some logging from clients -> servers via syslog-ng over a TCP session with stunnel. On my client, I'm seeing some log messages come in that are not making it to the server.. reliably. I don't have any filtering setup I don't think — is there some default setting somewhere that I don't know about that only passes certain types of messages to the remote dest? Do I need to specify somewhere 'send all'? Whats quite strange is that I have the hosts logging to 2 systems right now.. old syslog-ng server, and new one. The old one is getting all of the messages, but the new one is ignoring messages like these: Apr 8 18:21:43 xxx stunnel: LOG5[5459:1079724352]: Connection reset: 0 bytes sent to SSL, 0 bytes sent to socket It seems to get all other messages.. but messages that have that LOG5 or LOG3 or LOG7 appended to them seem to get lost in translation... any ideas? —Matt

2 2

Using macro substring as another macro
by Jakub Jankowski 08 Apr '11

08 Apr '11

Hi, I'd like to implement simple hashing in file() destination, mainly to avoid ext3 32k directories limit, but also for convenience. Is it possible to do something simliar what this shell snippet does: $ NAME=value $ echo ${NAME[*]:0:1} v $ but for macros (especially those coming from parsers)? What I want is logs written to /var/log/h/ho/hos/hostname/service.log (with configurable level of subdirs) using a single destination(). Regards, -- Jakub Jankowski|shasta@toxcorp.com|http://toxcorp.com/ GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D

3 6

syslog-ng and IPv6 Interface Binding
by Reuben Farrelly 08 Apr '11

08 Apr '11

Hi, I've recently upgraded from syslog-ng 3.1 to syslog-ng 3.2.2 and I'm having limited success with my modified config in getting syslog-ng to bind to only specific IPv6 IP addresses. At the moment I have this in my config: source net { tcp6( port(601) ); udp6( port(601) ); }; However I'd like to configure this slightly differently - namely to increase the maximum connections from 10 to 25 and also to bind to only certain IPv6 addresses, as the box has multiple. I was hoping for a config like this (IPv6 address truncated somewhat): source net { tcp6( ip(2001::20) port(601) max_connections(25) ); udp6( ip(2001::20) port(601) max_connections(25) ); }; But seems to be a no-go: ----- Error parsing afsocket, syntax error, unexpected LL_NUMBER, expecting LL_IDENTIFIER or LL_STRING in /etc/syslog-ng/syslog-ng.conf at line 32, column 18: tcp6( ip(2001::20) port(601) max_connections(25) ); ^^^^ ----- Neither the max_connections() NOR the ip() options are accepted nor is the use of localip() for IPv6. http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-gui… does not specifically mention if this command is meant to work under IPv6 or not, but it's reasonable I think to assume it would given the title of the page. Can someone please provide me with the relevant part of a working config in which I can specify the IPv6 interface address(es) I want syslog-ng to listen on, or have I run into a bug? The sample config files that ship with the package also seem a bit out of date, and the example at the bottom of s6.1.7 in the manual would be 100x more useful if it also included some IPv6 examples. Thanks, Reuben

4 5

stunnel through elb.. need packets sent semi-frequently
by Matt Wise 08 Apr '11

08 Apr '11

I'm using stunnel to pass some data through an Amazon ELB. Unfortunately, there are times when the session times out due to packets not passing for 60 seconds (an amazon ELB timeout). I'm wondering whether theres some way I can configure stunnel itself to pass some whitespaces or something back and forth every few seconds to make sure the session stays open and live? (and yes, I'm posing the same question to the app developer who's ap we're passing through.. but I'm looking at both avenues.. )

2 2

diskqueue framework patch
by Nagy Attila 08 Apr '11

08 Apr '11

Hi all, due to the need for disk queue usage in PE 4.1 we had to extend the syslog-ng sources, in order to be able write plugins that implement this feature. This patch contains these modifications. Basically it extends the 'class' hierarchy (and 'virtual' functions, see the new LogDestDriver) so that plugins can define their own queue mechanisms (as some do this in PE modules) and these special queues can be assigned to destinations. -Attila NAGY (syslog-ng-dev)

3 5