October 2008 - syslog-ng - lists.balabit.hu

Inline Compression
by Aaron Wiebe 06 Oct '08

06 Oct '08

Greetings, I've noticed in 1.6.x that there are tags already built into the configuration file to support inline compression of file output (but they complain if used). Is there any plan to support this? Are there reasons this hasn't been implemented yet? Given my knowledge of the libgzip, I can't imagine its terribly hard. The ability to write logs compressed would be -very- useful. -Aaron

2 3

Re: [syslog-ng] Reload does not free up RAM
by Evan Rempel 06 Oct '08

06 Oct '08

Balazs Scheidler wrote: >>>>>> We have cases where a syslog-ng buffers quite a bit of data (2GB or so) before >>>>>> it is able to flush the data to the destination. Currently, the only way to "free up" that RAM >>>>>> is to restart syslog-ng, which means that some log messages are lost during the restart >>>>>> window. > > Both PE and OSE frees up disk/memory when possible. If they don't, then > that's probably a leak. > > So if your syslog-ng process is 2GB in size, and you didn't specify an > overly large log_fifo_size(), it's probably because of a leak. I have a large log_fifo_size() but the process starts small, grows when it needs to buffer, and does NOT get small again when the destination is available. >> 2. The buffer space should be thrown away only after the message >> has been delivered to the destination. > > I see, this is what should happen now. This does not appear to be happening. I am using OSE 2.0.8 Have there been memory fixes/changes since then? Is there a way to get buffer count statistics out? How many messages are in each fifo buffer? Evan.

2 1

Re: [syslog-ng] Reload does not free up RAM
by Evan Rempel 05 Oct '08

05 Oct '08

Balazs Scheidler wrote: > On Thu, 2008-10-02 at 08:43 -0700, Evan Rempel wrote: >> Balazs Scheidler wrote: >>> On Wed, 2008-10-01 at 09:54 -0700, Evan Rempel wrote: >>>> We have cases where a syslog-ng buffers quite a bit of data (2GB or so) before >>>> it is able to flush the data to the destination. Currently, the only way to "free up" that RAM >>>> is to restart syslog-ng, which means that some log messages are lost during the restart >>>> window. >>>> >>>> I open for discussion the idea that on a reload that syslog-ng flush and free all buffer >>>> space and essentially start from a "just started" state again. >>> Well, this is currently a PE feature, and BalaBit decided not to open >>> source it this time with 3.0 as that's one of the primary incentives to >>> buy the commercial version. >>> >>> PE with its persistent disk buffers is saving queue contents upon >>> restart right into the disk buffer file. And also, syslog-ng does not >>> buffer so much data in memory as it first uses the disk. >> The disk buffering offers a lot more than this. >> >> Persistent buffers around a syslog restart >> Persistent buffers around a system restart (think power outage) >> Much larger buffering (potentially hundreds of GBytes) with minimal >> system impact. >> >> I would hope that PE version would shrink its disk buffers once they >> are flushed, just like I am hoping that OSE would shrink its RAM buffers >> once they are flushed. >> >> I don't think that RAM usage AFTER buffers are flushed is much of an >> incentive to purchase the PE. The persistence and size of the buffers >> is the incentive. >> > > E.g. you want to throw away the buffer contents but without restarts? > that should be doable. Just to verify the points. 1. I don't want to drop/loose messages with this change. 2. The buffer space should be thrown away only after the message has been delivered to the destination. 3. This should be done at least on a reload as a minimum. Since all of the sources are closed during a reload, I thought that the buffers could be flushed and all buffer space thrown away at that time. This is really an implementation decision that is really your choice, and does not matter to me. 4. Freeing and allocating memory is an expensive operation, so I don't want it done for each message, but doing it without the need for a reload would be ideal. Algorithms that allocate memory in blocks, and free one block when two are unused always keeps 1 block available for use and you don't get into a busy allocate/drop that repeats because one message comes and goes. Thanks for entertaining this discussion. Evan.

2 1

Re: [syslog-ng] escaping \[ not respected
by Evan Rempel 05 Oct '08

05 Oct '08

Balazs Scheidler wrote: > On Thu, 2008-10-02 at 09:12 +1300, chris packham wrote: >> One option would be to use the kind quotes ' vs " to determine whether to >> escape or not. I believe bash (and maybe perl) does this kind of thing when >> handling strings. > > Yeah, the problem is with compatibility, ' and " style quoting has been > working this way for about a decade, changing that would break a lot of > configurations. > > I should have made this decision back then, but changing this now is not > really possible I think. Perhaps adding a new config keyword match_pcre(/no syslog-ng escape codes, all escapes are handled by the pcre evaluation/) The down side to this is that you really need to have program_pcre and any other additions along those lines. Things get a little messy. Hopefully in a later version the normal match, program directives could be made to work this way, and the match_pcre etc would become backwards compatible aliases to the non_pcre commands. I don't really like that solution, but I thought I would "throw it out there". Another option is to an something like is already present for the templates where there is a template_escape(no) option. If you add a regex_escape(yes/no) directive and default to yes, then all of the current expressions would work. If you do not need to have quotes in your expression, then you can use regex_escape(no) and do away with the escapes for the escapes. Just a couple of thoughts. -- Evan Rempel

3 2

Re: [syslog-ng] Reload does not free up RAM
by Evan Rempel 04 Oct '08

04 Oct '08

Balazs Scheidler wrote: > On Wed, 2008-10-01 at 09:54 -0700, Evan Rempel wrote: >> We have cases where a syslog-ng buffers quite a bit of data (2GB or so) before >> it is able to flush the data to the destination. Currently, the only way to "free up" that RAM >> is to restart syslog-ng, which means that some log messages are lost during the restart >> window. >> >> I open for discussion the idea that on a reload that syslog-ng flush and free all buffer >> space and essentially start from a "just started" state again. > > Well, this is currently a PE feature, and BalaBit decided not to open > source it this time with 3.0 as that's one of the primary incentives to > buy the commercial version. > > PE with its persistent disk buffers is saving queue contents upon > restart right into the disk buffer file. And also, syslog-ng does not > buffer so much data in memory as it first uses the disk. The disk buffering offers a lot more than this. Persistent buffers around a syslog restart Persistent buffers around a system restart (think power outage) Much larger buffering (potentially hundreds of GBytes) with minimal system impact. I would hope that PE version would shrink its disk buffers once they are flushed, just like I am hoping that OSE would shrink its RAM buffers once they are flushed. I don't think that RAM usage AFTER buffers are flushed is much of an incentive to purchase the PE. The persistence and size of the buffers is the incentive. Thanks for the background. -- Evan Rempel

2 1

Re: [syslog-ng] Message suppression not working
by Balazs Scheidler 02 Oct '08

02 Oct '08

On Thu, 2008-10-02 at 10:57 -0400, Kevin Cruse wrote: > That's what I thought. I downloaded 2.1.1 for evaluation but cannot > figure out how to load the license.txt file. hmm, the open source version does not need a license file. only the premium edition does. to evaluate the latter, ask for an eval license on mybalabit. > > -----Original Message----- > From: Balazs Scheidler [mailto:bazsi@balabit.hu] > Sent: Thursday, October 02, 2008 10:45 AM > To: Kevin Cruse > Cc: Syslog-ng users' and developers' mailing list > Subject: RE: Message suppression not working > > On Thu, 2008-10-02 at 10:11 -0400, Kevin Cruse wrote: > > Bazsi, > > > > Thanks for the quick response, I should have been more clear. When I > > use the suppress() option I get a syntax error when running the > > syslog-ng binary. My apologies. > > > > it is only supported by syslog-ng 2.1.1 and later. > -- Bazsi

1 0

Re: [syslog-ng] Message suppression not working
by Balazs Scheidler 02 Oct '08

02 Oct '08

On Thu, 2008-10-02 at 10:11 -0400, Kevin Cruse wrote: > Bazsi, > > Thanks for the quick response, I should have been more clear. When I > use the suppress() option I get a syntax error when running the > syslog-ng binary. My apologies. > it is only supported by syslog-ng 2.1.1 and later. -- Bazsi

1 0

Re: [syslog-ng] Message suppression not working
by Balazs Scheidler 02 Oct '08

02 Oct '08

On Thu, 2008-10-02 at 08:39 -0400, Kevin Cruse wrote: > Bazsi, > > I am trying to suppress messages sent to the database and sendmail. I > have configured suppress(900) globally and within my destinations and it > doesn't seem to work. Does anyone have this working? I am getting > flooded with emails when there are problems on my firewalls or routers. > > destination d_mysql { > pipe("/var/log/mysql.pipe" > template("INSERT INTO logs > (host, facility, priority, level, tag, datetime, program, msg) > VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', > '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC', > '$PROGRAM', '$MSG' );\n") template-escape(yes) suppress(900)); > }; > > Or > > destination mail-alert-perl { program("/usr/local/bin/mail_logger.pl" > suppress(900)); }; > > also tried > > destination mail-alert-perl { program("/usr/local/bin/mail_logger.pl") > suppress(900); }; > > Any help would be appreciated. Thanks. suppress() only filters duplicates, and is not a general ratelimit. -- Bazsi

1 0

syslog-ng craches when sending from syslog-ng to another loghost (that have syslog-ng)
by Mattias Ryrlén 02 Oct '08

02 Oct '08

Hi, Im running syslog-ng-2.1beta2-1 on 2 of my servers. i have the same configuration on both the servers except that i want server A to forward messages to server B All works fine for a couple of minutes, then server A:s syslog-ng service craches without any information. running CentOS 5.0 on both servers with syslog-ng installed as rpm packages from balabit. i have also tried with udp() instead of tcp() with the same result. options { sync (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (yes); dns_cache(yes); use_fqdn (no); create_dirs (no); keep_hostname (yes); log_msg_size(8192); }; source s_sys { file ("/proc/kmsg" log_prefix("kernel: ")); unix-stream ("/dev/log"); internal(); # udp(ip(0.0.0.0) port(514)); }; source s_udp { udp(); }; source s_tcp { tcp(max-connections(200)); }; destination d_loghost { tcp("192.168.1.150" port(514)); }; log { source(s_sys); source(s_tcp); source(s_udp); destination(d_loghost); }; Any sugestions? -- Vänliga hälsningar / Best Regards Mattias Ryrlén __________________________ op5 AB Första Långgatan 19 SE-413 27 Göteborg Mobil: +46 735-17 70 99 Support: +46 31-774 09 24 www.op5.com

2 2

syslog-ng OSE git tree opened
by Balazs Scheidler 02 Oct '08

02 Oct '08

Dear syslog-ng users, I'm proud to announce that I've opened the syslog-ng OSE 3.0 branch and pushed a backport of the result of the latest Premium Edition development cycle. The size of the patch is enourmous, it took me a while to integrate it back to OSE 3.0, that's why this announcement is coming so late. The diffstat is as follows: 175 files changed, 21537 insertions(+), 7377 deletions(-) The list of changes of interest are: * various performance improvements, syslog-ng can do about 120-130k messages/sec in case no destination is written to (not very useful though), and about 75k msg/sec when writing to a plain text file * TLS encrypted channels, works the same as the PE version, documentation is: http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch03s09.html * support for the new IETF syslog protocols, in order to use this, you need to use the syslog() driver either as a source or a destination. It behaves quiet closely to tcp/udp, except it uses the new protocol * log message classification using a database, scales up to tens of thousands of messages per second, the pattern database also serves as a means to parse variable parts of messages. No real documentation so far, there's an XML Schema file in the tarball, it can be used using the db_parser() parser. * rewrite parts of the message using regexps, and simple string search and replace. Here's a post on the syslog-ng mailing list that has some info on usage: http://marc.info/?l=syslog-ng&m=122051372205555&w=2 * parse parts of the syslog message, makes it possible to parse fields in an Apache log for instance. Also the parsed values can be used as macros. Again, the post above has some info: http://marc.info/?l=syslog-ng&m=122051372205555&w=2 * generalize the log message paths, instead of a flat list of source->destination connections, log statements can be organized into a tree, making the configuration simpler and the performance better, here is a description: http://bazsi.blogs.balabit.com/2008/02/redesigning-syslog-ng-internals.html * PCRE and glob filters (in addition to POSIX regexps), use the type("pcre") option in the filter, like this: host("something" type("pcre")) * support for capabilities under Linux, this running syslog-ng as non-root is possible, also with reload support, see the documentation of Linux capabilities in capabilities(7), for the syntax of the --caps option, see cap_from_text(3) * program sources, simply use program("cat /dev/xconsole") in your configuration, the program will be restarted automatically if that exits. * syslog-ng got the ability to restart itself if it crashes, see the --process-mode command line option * new statistics framework that can be queried using UNIX domain sockets, provides per-source-host and per-destination-host statistics. In order to use it, possibly set the stats_level() option and write "STATS\n" to /var/run/syslog-ng.ctl which is a UNIX domain socket. This mass amount of changes also mean some incompatibilities between 2.1 and 3.0, however 3.0 will work in compatibility mode unless it is told to use the 3.0 configuration file format. It'll also report warnings against your configuration file you should change in order to use all the features and best performance. The most important incompatibilities include: * the $MSG macro will not include the "program[pid]:" string anymore, you should prepent an $MSGHDR macro in order to have that, in 2.1 mode the MSG will expand as before, but please note that this value is calculated everytime you reference the macro, thus it affects performance. * some defaults were changed (chain_hostname, template-escape, follow-freq for file sources), in 2.1 mode the old defaults are used but a warning is emitted * the input path does not remove NL characters anymore by default, but you can ask for this behaviour by specifying the "no-multi-line" flag. The documentation for the new features is not yet completely finished, I hope to get a snapshot of the current state to the website soon. In the meanwhile either use the source :) or ask direct questions on the mailing list. I'll also try to make some posts on my blog about various features in more detail and configuration examples. -- Bazsi

1 0