- syslog-ng - lists.balabit.hu

Re: [syslog-ng] Reload does not free up RAM
by Evan Rempel 07 Oct '08

07 Oct '08

Balazs Scheidler wrote: > On Sun, 2008-10-05 at 13:49 -0700, Evan Rempel wrote: >> Balazs Scheidler wrote: >>>>>>>> We have cases where a syslog-ng buffers quite a bit of data (2GB or so) before >>>>>>>> it is able to flush the data to the destination. Currently, the only way to "free up" that RAM >>>>>>>> is to restart syslog-ng, which means that some log messages are lost during the restart >>>>>>>> window. >>> Both PE and OSE frees up disk/memory when possible. If they don't, then >>> that's probably a leak. No, it isn't a memory leak as the process can be months old and only grows when it buffers messages. >>> >>> So if your syslog-ng process is 2GB in size, and you didn't specify an >>> overly large log_fifo_size(), it's probably because of a leak. >> >> I have a large log_fifo_size() but the process starts small, grows when it needs >> to buffer, and does NOT get small again when the destination is available. > > Hmm... that might be related to the malloc implementation. Which > platform are you using? > >>>> 2. The buffer space should be thrown away only after the message >>>> has been delivered to the destination. >>> I see, this is what should happen now. >> >> This does not appear to be happening. >> I am using OSE 2.0.8 Have there been memory fixes/changes since then? > > Not that I know of. > >> Is there a way to get buffer count statistics out? How many messages are >> in each fifo buffer? > > In the experimental 3.0 branch, yes there is. In current 2.0, I'm afraid > there's none. > > Is there a limit on the growth of syslog-ng? I mean if it gets large > once and then empties its buffers, it stays large, but does it start > growing again, or it stays steadily at the size it was before? I have not specifically tested this scenario, but we do track processes and memory usage, and it appears that once the space is allocated, it does not grow unless a larger set of messages gets buffered. > Again, the platform matters a lot, how the local malloc implementation > takes care about fragmentation, whether it shrinks the process once > memory is freed. This is running on Redhat AS 3 (2.4 kernel) and Redhat AS 4 (2.6 kernel) On the Redhat 4 system the glibc packages installed are glibc-common-2.3.4-2.36 glibc-devel-2.3.4-2.36 glibc-headers-2.3.4-2.36 glib2-2.4.7-1 glibc-kernheaders-2.4-9.1.100.EL > The problem is, that if the local malloc implementation is wrong the > only thing I could do is to try to use a different allocator (e.g. use > GNU malloc on your platform), but that's not a simple change. Understood. -- Evan Rempel

2 1

syslog-ng 2.1: upstream libdbi does NOT provide pkgconfig .pc file
by Wolfram Schlich 07 Oct '08

07 Oct '08

Hi, syslog-ng 2.1 depends on libdbi >= 0.8.0 with --enable-sql. Unfortunately, it assumes that a pkg-config .pc file for libdbi is in place which the upstream libdbi just does not include. It seems that there is a balabit-modified version of libdbi available (libdbi8_0.8.2bb2-2.tar.gz) which was patched to include a dbi-0.8.pc file. What's the reason for that? Was upstream asked to include that .pc file in their regular releases? What other changes were made to the upstream libdbi to form the balabit version? This is kinda maintenance and dependency hell so far :( Please clear up the situation so I can take care of building a Gentoo package for 2.1 *including* SQL support :) Thanks! -- Regards, Wolfram Schlich <wschlich(a)gentoo.org> Gentoo Linux * http://dev.gentoo.org/~wschlich/

2 4

Regexp question
by Mártha Csaba 06 Oct '08

06 Oct '08

Hello I use this regexp: match ="FlashLog: ([0-9a-zA-Z]+)-([0-9a-zA-Z]+)-([0-9]{4})-([0-9]{2})- ([0-9]{2})-([0-9]{2})-([0-9]{2})-([0-9a-zA-Z]+)"); and this df_out: destination df_out { file("/var/log/client/flashlog/${1}/${8}_.txt"); }; I have /var/log/client/flashlog/_.txt file. What is the problem? Best Regards: csaba

1 0

Inline Compression
by Aaron Wiebe 06 Oct '08

06 Oct '08

Greetings, I've noticed in 1.6.x that there are tags already built into the configuration file to support inline compression of file output (but they complain if used). Is there any plan to support this? Are there reasons this hasn't been implemented yet? Given my knowledge of the libgzip, I can't imagine its terribly hard. The ability to write logs compressed would be -very- useful. -Aaron

2 3

Re: [syslog-ng] Reload does not free up RAM
by Evan Rempel 06 Oct '08

06 Oct '08

Balazs Scheidler wrote: >>>>>> We have cases where a syslog-ng buffers quite a bit of data (2GB or so) before >>>>>> it is able to flush the data to the destination. Currently, the only way to "free up" that RAM >>>>>> is to restart syslog-ng, which means that some log messages are lost during the restart >>>>>> window. > > Both PE and OSE frees up disk/memory when possible. If they don't, then > that's probably a leak. > > So if your syslog-ng process is 2GB in size, and you didn't specify an > overly large log_fifo_size(), it's probably because of a leak. I have a large log_fifo_size() but the process starts small, grows when it needs to buffer, and does NOT get small again when the destination is available. >> 2. The buffer space should be thrown away only after the message >> has been delivered to the destination. > > I see, this is what should happen now. This does not appear to be happening. I am using OSE 2.0.8 Have there been memory fixes/changes since then? Is there a way to get buffer count statistics out? How many messages are in each fifo buffer? Evan.

2 1

Re: [syslog-ng] Reload does not free up RAM
by Evan Rempel 05 Oct '08

05 Oct '08

Balazs Scheidler wrote: > On Thu, 2008-10-02 at 08:43 -0700, Evan Rempel wrote: >> Balazs Scheidler wrote: >>> On Wed, 2008-10-01 at 09:54 -0700, Evan Rempel wrote: >>>> We have cases where a syslog-ng buffers quite a bit of data (2GB or so) before >>>> it is able to flush the data to the destination. Currently, the only way to "free up" that RAM >>>> is to restart syslog-ng, which means that some log messages are lost during the restart >>>> window. >>>> >>>> I open for discussion the idea that on a reload that syslog-ng flush and free all buffer >>>> space and essentially start from a "just started" state again. >>> Well, this is currently a PE feature, and BalaBit decided not to open >>> source it this time with 3.0 as that's one of the primary incentives to >>> buy the commercial version. >>> >>> PE with its persistent disk buffers is saving queue contents upon >>> restart right into the disk buffer file. And also, syslog-ng does not >>> buffer so much data in memory as it first uses the disk. >> The disk buffering offers a lot more than this. >> >> Persistent buffers around a syslog restart >> Persistent buffers around a system restart (think power outage) >> Much larger buffering (potentially hundreds of GBytes) with minimal >> system impact. >> >> I would hope that PE version would shrink its disk buffers once they >> are flushed, just like I am hoping that OSE would shrink its RAM buffers >> once they are flushed. >> >> I don't think that RAM usage AFTER buffers are flushed is much of an >> incentive to purchase the PE. The persistence and size of the buffers >> is the incentive. >> > > E.g. you want to throw away the buffer contents but without restarts? > that should be doable. Just to verify the points. 1. I don't want to drop/loose messages with this change. 2. The buffer space should be thrown away only after the message has been delivered to the destination. 3. This should be done at least on a reload as a minimum. Since all of the sources are closed during a reload, I thought that the buffers could be flushed and all buffer space thrown away at that time. This is really an implementation decision that is really your choice, and does not matter to me. 4. Freeing and allocating memory is an expensive operation, so I don't want it done for each message, but doing it without the need for a reload would be ideal. Algorithms that allocate memory in blocks, and free one block when two are unused always keeps 1 block available for use and you don't get into a busy allocate/drop that repeats because one message comes and goes. Thanks for entertaining this discussion. Evan.

2 1

Re: [syslog-ng] escaping \[ not respected
by Evan Rempel 05 Oct '08

05 Oct '08

Balazs Scheidler wrote: > On Thu, 2008-10-02 at 09:12 +1300, chris packham wrote: >> One option would be to use the kind quotes ' vs " to determine whether to >> escape or not. I believe bash (and maybe perl) does this kind of thing when >> handling strings. > > Yeah, the problem is with compatibility, ' and " style quoting has been > working this way for about a decade, changing that would break a lot of > configurations. > > I should have made this decision back then, but changing this now is not > really possible I think. Perhaps adding a new config keyword match_pcre(/no syslog-ng escape codes, all escapes are handled by the pcre evaluation/) The down side to this is that you really need to have program_pcre and any other additions along those lines. Things get a little messy. Hopefully in a later version the normal match, program directives could be made to work this way, and the match_pcre etc would become backwards compatible aliases to the non_pcre commands. I don't really like that solution, but I thought I would "throw it out there". Another option is to an something like is already present for the templates where there is a template_escape(no) option. If you add a regex_escape(yes/no) directive and default to yes, then all of the current expressions would work. If you do not need to have quotes in your expression, then you can use regex_escape(no) and do away with the escapes for the escapes. Just a couple of thoughts. -- Evan Rempel

3 2

Re: [syslog-ng] Reload does not free up RAM
by Evan Rempel 04 Oct '08

04 Oct '08

Balazs Scheidler wrote: > On Wed, 2008-10-01 at 09:54 -0700, Evan Rempel wrote: >> We have cases where a syslog-ng buffers quite a bit of data (2GB or so) before >> it is able to flush the data to the destination. Currently, the only way to "free up" that RAM >> is to restart syslog-ng, which means that some log messages are lost during the restart >> window. >> >> I open for discussion the idea that on a reload that syslog-ng flush and free all buffer >> space and essentially start from a "just started" state again. > > Well, this is currently a PE feature, and BalaBit decided not to open > source it this time with 3.0 as that's one of the primary incentives to > buy the commercial version. > > PE with its persistent disk buffers is saving queue contents upon > restart right into the disk buffer file. And also, syslog-ng does not > buffer so much data in memory as it first uses the disk. The disk buffering offers a lot more than this. Persistent buffers around a syslog restart Persistent buffers around a system restart (think power outage) Much larger buffering (potentially hundreds of GBytes) with minimal system impact. I would hope that PE version would shrink its disk buffers once they are flushed, just like I am hoping that OSE would shrink its RAM buffers once they are flushed. I don't think that RAM usage AFTER buffers are flushed is much of an incentive to purchase the PE. The persistence and size of the buffers is the incentive. Thanks for the background. -- Evan Rempel

2 1

Re: [syslog-ng] Message suppression not working
by Balazs Scheidler 02 Oct '08

02 Oct '08

On Thu, 2008-10-02 at 10:57 -0400, Kevin Cruse wrote: > That's what I thought. I downloaded 2.1.1 for evaluation but cannot > figure out how to load the license.txt file. hmm, the open source version does not need a license file. only the premium edition does. to evaluate the latter, ask for an eval license on mybalabit. > > -----Original Message----- > From: Balazs Scheidler [mailto:bazsi@balabit.hu] > Sent: Thursday, October 02, 2008 10:45 AM > To: Kevin Cruse > Cc: Syslog-ng users' and developers' mailing list > Subject: RE: Message suppression not working > > On Thu, 2008-10-02 at 10:11 -0400, Kevin Cruse wrote: > > Bazsi, > > > > Thanks for the quick response, I should have been more clear. When I > > use the suppress() option I get a syntax error when running the > > syslog-ng binary. My apologies. > > > > it is only supported by syslog-ng 2.1.1 and later. > -- Bazsi

1 0

Re: [syslog-ng] Message suppression not working
by Balazs Scheidler 02 Oct '08

02 Oct '08

On Thu, 2008-10-02 at 10:11 -0400, Kevin Cruse wrote: > Bazsi, > > Thanks for the quick response, I should have been more clear. When I > use the suppress() option I get a syntax error when running the > syslog-ng binary. My apologies. > it is only supported by syslog-ng 2.1.1 and later. -- Bazsi

1 0