March 2012 - syslog-ng - lists.balabit.hu

unable to use multiple application patterns per ruleset
by Kurt Yoder 13 Mar '12

13 Mar '12

Hi list, Using patterndb, has anyone successfully used multiple application patterns per ruleset? I have the following: <?xml version='1.0' encoding='UTF-8'?> <patterndb version='4' pub_date='2012-03-08'> <ruleset name='rdc-pam' id='rdc-pam'> <patterns> <pattern>blahblah</pattern> <pattern>sshd</pattern> </patterns> <rules> <rule class='system' id='pam_session_opened_for_user_by_uid' provider='rdc'> <description></description> <patterns> <pattern>pam_unix(sshd:session): session opened for user kurt by (uid=0)</pattern> </patterns> <values> </values> <tags> <tag>rdc.alert.ok</tag> </tags> </rule> </rules> </ruleset> </patterndb> As soon as I remove "<pattern>blahblah</pattern>", or move it below "<pattern>sshd</pattern>", this rule starts matching events. When I add it back, it stops matching events. The documentation at http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-gui… says (incidentally, the closing "<patterns>" in this example should instead read "</patterns>"): " Specifying multiple patterns is useful if two or more applications have different names (that is, different $PROGRAM fields), but otherwise send identical log messages. <patterns> <pattern>firstapplication</pattern> <pattern>otherapplication</pattern> <patterns> This is exactly what I am trying to do, but it does not appear to work as documented. I manually backported my syslog-ng from Ubuntu Oneiric: $ syslog-ng --version syslog-ng 3.2.4 Installer-Version: 3.2.4 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#ef7b91e4a1b1f9628c66138b4ae83de7e4c697c6 Compile-Date: Jan 19 2012 02:57:58 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: on Enable-Linux-Caps: on Enable-Pcre: on Enable-Pacct: off Can anyone offer a suggestion toward making this work? Thanks

1 1

[Bug 160] New: memory leak in message buffering
by bugzilla＠bugzilla.balabit.com 12 Mar '12

12 Mar '12

https://bugzilla.balabit.com/show_bug.cgi?id=160 Summary: memory leak in message buffering Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 Created an attachment (id=47) --> (https://bugzilla.balabit.com/attachment.cgi?id=47) tunable read rate program destination When writing to a destination that requires syslog-ng to buffer messages, the memory footprint of syslog-ng grows endlessly, even when there is sufficient time to flush the buffers. I am including a read rate tunable perl program destination that I use to slow the destination down. I am including a syslog-ng.conf file I am including the output of valgrind. I run syslog-ng, cat a /var/messages file (100,000 lines) to the input pipe. valgrind now shows definitely lost: 2,620,649 bytes in 4,439 blocks -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 8

[Bug 162] New: Invalid frame header; header=''
by bugzilla＠bugzilla.balabit.com 12 Mar '12

12 Mar '12

https://bugzilla.balabit.com/show_bug.cgi?id=162 Summary: Invalid frame header; header='' Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: yuri(a)fused.com Type of the Report: bug Estimated Hours: 0.0 Got always reproducible bug when sending huge logs between machines. Here are the steps: * setup default OSE installs of client & server on rhel5 * configure client to read file("/tmp/input" flags(no-parse)) and send it to syslog(transport("tcp")) * configure server to listen on syslog(transport(tcp)) and store messages in file() * Now let client send big chunk of data: ruby -e 'loop { puts (97+rand(26)).chr*250 }' | head -1000 >> /tmp/input * See on server the following message: syslog-ng[xxx]: Invalid frame header; header='' server's RSTing the connection afterwards It doesn't happen when you start with a couple of lines and then raise the "chunk" to 100-1000 messages per insertion. If you need more info, ask. Thanks. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 6

max recursion depth on filters?
by Patrick Hemmer 11 Mar '12

11 Mar '12

Is there a maximum depth on recursive filter statements? I seem to have an issue which would indicate so. For example, in the config below, i have f_discard, which calls f_iptables_discard, which calls f_iptables. I have it like this so that its easy to read, configure, and add other filters. Its supposed to result in lines matching this to not be logged, but its not working. However if I take the contents of `f_iptables_discard` and put them directly into `f_discard` it works fine. filter f_iptables { program('^kernel') and message('^\s*\[\s*[\d\.]+\] iptables/' type(pcre)) }; filter f_iptables_discard { # ignore iptables broadcast messages filter(f_iptables) and message('MAC=ff:ff:ff:ff:ff:ff') ; }; filter f_discard { not ( filter(f_iptables_discard) ); }; rewrite r_tag { set('iptables', value('DBTAG') condition(filter(f_iptables))); }; log { source(s_local); source(s_net); rewrite(r_tag); filter(f_discard); destination(d_sqlite); };

2 2

Parser not parsing :-)
by T. A. Smooth 11 Mar '12

11 Mar '12

I can only assume I am not implementing this correctly. :-) But I have a parser I am trying to use so I can take a subset of the information of a message and send that subset to another receiver. This is the whole message: <13>Feb 4 18:40:17 myhost syslogng: 2012-02-04T18:40:17-08:00 > myhostserver-http /tmp/logs/access_log Hi Mom > What I want to do is send out the message as : <13>Feb 4 18:40:17 myhost syslogng: Hi Mom Notice how I dropped the middle part out. >From what I have read, the parser acts on the message body alone. Is this correct? So I set it up to look for four(4) columns of data and to be "greedy" on the last column. I have played around with the number of columns and even used a rewrite function instead. But the Parser continues to produce empty variables. And my template just echos out my default value. Any thoughts? > > > parser p_et_logmessage { > csv-parser( > #columns("ETMSG") > #columns("ETMSG.ISODATE") > columns("ETMSG.ISODATE", "ETMSG.EASI", "ETMSG.SOURCE", > "ETMSG.BODY") > delimiters(" ") > #template("${MSG}") > flags(greedy) > ); > }; > > rewrite r_rewrite_set{set('${ETMSG.BODY:-nothing}', value("MESSAGE"));}; > > template t_et_basic_logmessage { > template("${ETMSG.BODY:-nothing}\n"); template_escape(no); }; > > > destination destination_info { > tcp("host2" port(8080) > template(t_et_basic_logmessage) > log_disk_fifo_size(32212254720) > ); > }; > > log { > source(INTAKE); > parser(p_et_logmessage); > destination(destination_info); > }; > My latest Post: Givenchy Fall/Winter 2012 Collection – Runway | Highsnobiety.com<http://www.webeclubbin.com/blog/2012/01/givenchy-fallwinter-2012-collection…> Get a signature like this. <http://r1.wisestamp.com/r/landing?promo=19&dest=http%3A%2F%2Fwww.wisestamp.…> CLICK HERE.<http://r1.wisestamp.com/r/landing?promo=19&dest=http%3A%2F%2Fwww.wisestamp.…>

4 9

syslog-ng 3.4.0alpha1 has been released
by devel＠balabit.hu 11 Mar '12

11 Mar '12

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.4.0alpha1 SUMMARY : new stable release DATE : Mar 11, 2012 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.4.0alpha1) has been released. This is an alpha release aiming at collecting feedback and testing on the upcoming 3.4 version. Use in production environments is not recommended. CHANGES: 3.4.0alpha1 Features: ========= * Support for junctions & channels were added, which improve the flexibility of the syslog-ng configuration language. This allows combining sources with their closely tied processing functionality (like parser, rewrite and filter statements). Read this blog post for more information: http://bazsi.blogs.balabit.com/2012/01/syslog-ng-flexibility-improvements/ In the final form of the functionality the "log" keyword as described in the blog post above was replaced with "channel". * The functionality to query and manipulate sets of name-value pairs (often referenced as value-pairs and used in the mongodb() destination driver and the $(format-json) template function). got significantly improved. It is now possible to change the name of the keys when creating the output. See this commit for more information: https://github.com/bazsi/syslog-ng-3.4/commit/ddc7c2539bd66fa35e8df441e4baf… * Plugins & modules are now demand-loaded automatically if the "autoload-compiled-modules" global variable is set to 1, which is the default. Any shared libraries found on the module search path is considered for loading if the configuration file contains a reference to a functionality it provides. To disable this functionality simply set the referenced variable to 0 with a "@define" statement and load modules explicitly via "@module" statements. To list the available plugins & modules, use the --module-registry command line option for syslog-ng, which results in a detailed listing. * Added a new parser named json-parser() to parse incoming JSON formatted messages. See this commit for more information: https://github.com/bazsi/syslog-ng-3.4/commit/e5569687bba2551c89a78faee55bc… * Added a number of template functions: $(length ARG) - length of a template expression $(substr ARG START [LEN]) - substring of a string $(strip ARG) - remove white space from the start and end $(sanitize ARG1 ARG2) - join args to form a filename while removing special characters like '/' $(+), $(-), $(*), $(/), $(%) - perform numeric operations * Added support for replicasets to the mongodb() destination driver. https://github.com/bazsi/syslog-ng-3.4/commit/a980b9d268efa54879d3deeb4a53f… * Added support for safe-mode() to the mongodb() destination driver that ensures inserts were properly executed at the cost of some performance penalties. https://github.com/bazsi/syslog-ng-3.4/commit/768f0c6ec8eba2ad51531f2331fb5… * Added support for SMTP destination to send out emails triggered by log messages. https://github.com/bazsi/syslog-ng-3.4/commit/404ceb959efe9715ce7437d7dcdc2… * Added support for generating UUIDs via the $(uuid) template function. https://github.com/bazsi/syslog-ng-3.4/commit/a16798c653c057239236945034114… * Added the @SET@ parser to db-parser(). https://github.com/bazsi/syslog-ng-3.4/commit/ab08c84abbeda5a200b8150a6af7b… * Added support for dbd-option() in the sql() destination driver that makes it possible to supply driver specific options to the DBI driver. https://github.com/bazsi/syslog-ng-3.4/commit/9254fe9e6236746bf1f955f0cac12… * Reload of the configuration can now be triggered using "syslog-ng-ctl reload". * A new macro named $LOGHOST was added, which expands to the local hostname running syslog-ng. * A set of time macros were added prefixed with "C_" that use the current time instead of the reception time (prefixed R_) and and the time that was included in the message (prefixed S_). This means that C_DATE expands to the current date, whereas R_DATE would expand to the date the current message was received at. https://github.com/bazsi/syslog-ng-3.4/commit/c2d17009e2ce14960acb519750fe2… * Improved error reporting by including the configuration-file location of the object associated with the error. This makes it easier to diagnose errors even in the case of otherwise unnamed objects. * This release also includes all fixes of the 3.3 branch, which are not listed here for brevity's sake. The merged commit ID is: bf742b0, which is a couple of patches ahead of "3.3.4". Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Andreas Piesk Balazs Scheidler (BalaBit) Balint Kovacs (BalaBit) Evan Rempel (University of Victoria) Gergely Nagy (BalaBit) Heiko Gerstung Hendrik Völker (Verizon) Jakub Jankowski (superhost.pl) Martin Grauel (BalaBit) Matthias Runge (Fedora) Patrick Hemmer Russ Milne (Seccuris) DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

Strong Cipher for syslogng
by Pramod Pillai 07 Mar '12

07 Mar '12

Hi We are using Syslog NG version is 3.2.4 We haave a requirement to use strong cipher for syslogng listening ports. Is it possible in syslogng version 3.2.4. Regards Pramod

2 1

[Bug 167] New: "configure" error on Solaris 10 SPARC: Glib headers inconsistent....
by bugzilla＠bugzilla.balabit.com 06 Mar '12

06 Mar '12

https://bugzilla.balabit.com/show_bug.cgi?id=167 Summary: "configure" error on Solaris 10 SPARC: Glib headers inconsistent.... Product: syslog-ng Version: 3.3.x Platform: Other OS/Version: Solaris Status: NEW Severity: blocker Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: david.j.fenske(a)usps.gov Type of the Report: bug Estimated Hours: 0.0 1. Downloaded syslog-ng-3.3.4 source 2. gunzip/de-tar'ed source 3. Updated eventlog, glib, gcc packages from Sunfreeware.com 4. set PKG_CONFIG_PATH=/usr/local/lib/pkgconfig 5. Ran: ./configure 6. Receive error: configure: error: Glib headers inconsistent with current compiler setting. You might be using 32 bit Glib with a 64 bit compiler, check PKG_CONFIG_PATH 7. Have tried setting CFLAGS to -m64 and -m32 resulting in same error. I know this problem was reported in 2008 or so but no "solution" was ever given. (other than to download from sunfreeware.com, which I would do if the latest version was was available). -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 8

start problems
by Jenei Gábor 02 Mar '12

02 Mar '12

Hello, I've already adopted the recommended init script for debian, but still my syslog-ng 3.4 doesn't seem to work. At the moment actually I can run my compiled program just by entering ./syslog-ng, however it does not seem to have access to files, because I don't see any trace in my log files of starting. If I try to start the program by its init script it waits for about a minute, and then says: failed. However after init script there are still running instances. Can you tell me what my problem is? Gabor

1 0

Bazsi's blog: Project Lumberjack to improve Linux logging
by Balazs Scheidler 01 Mar '12

01 Mar '12

Project Lumberjack to improve Linux logging In a lively discussion at the RedHat offices two weeks ago in Brno, a number of well respected individuals were discussing how logging in general, and Linux logging in particular could be improved. As you may have guessed I was invited because of syslog-ng, but representatives of other logging related projects were also in nice numbers: Steve Gibbs (auditd), Lennart Poettering (systemd, journald), Rainer Gerhards (rsyslog), William Heinbockel (CEE, Mitre) and a number of nice people from the RedHat team. We discussed a couple of pain points for logging, logging is usually an afterthought during development, computer based processing, correllation of application logs is nearly impossible. We roughly agreed that the key to improve the situation is to involve the community at large, initiate a momentum and try to get application developers on board and have them create structured logs. We also agreed that this will not happen overnight and we need to take a gradual approach. To move into that direction, the benefits of good logging needs to be communicated and delivered to both application developers and their users. We also talked about what kind of building blocks are needed to deliver a solution fast, and concluded that we basically have everything available, and even better they are open source. The key is to tie these components together, document best practices and perhaps provide better integration. Thus project Lumberjack was born, hosted as a Fedora project at https://fedorahosted.org/lumberjack/. The building blocks that need some care are: * some applications already produce logs in structured format, those should be integrated (auditd for instance) * we need to define a mechanism to submit structured logs to local logging services for further processing (ELAPI and some enhanced syslog) * we need to make sure that local logging services cope with structured data (already available for a long time now) * we need to define a mechanism to store messages in a structured form and a way query them * last, but not least we need to define a naming scheme for event data which CEE can bring to the table Most of these is already possible by using a combination of tools and proper configuration, however learning how to do this is not a trivial undertaking for those who only want to develop or use applications. Changing that is the primary aim of Project Lumberjack. If you are interested in logging, make sure to check that out.

1 0