[syslog-ng] Ubuntu Precise -ng filling out buffer, dropping messages
Chaman Chakalaka
chebannedmeagain at hotmail.com
Tue Apr 29 18:43:06 CEST 2014
Thanks for responding!
The exact version is of syslog-ng is 3.3.4. Here's something you may find of interest.
1K Messages
root at badbox:/proc/net# nc -lku 514 > /tmp/testing123.txt
root at goodbox:/proc/sys/net# loggen --inet --dgram --size 500 --rate 1000 --interval 30 badbox.cbf 514
average rate = 996.43 msg/sec, count=29893 <--------------- sent
root at badbox:/proc/net# cat /tmp/testing123.txt | wc -l
24081 <------------ received/processed
Lower the rate to 150 messages per second:
root at goodbox:/proc/sys/net# loggen --inet --dgram --size 500 --rate 150 --interval 30 badbox.cbf 514
average rate = 149.03 msg/sec, count=4471 < -------------------- sent
root at badbox:/proc/net# cat /tmp/testing123.txt | wc -l
4471 < -------------------------- received/processed
At this point, given the test above, I don't know if this is a system issue or a syslog-ng issue. It seems to be system, but I really can't tell what.
> From: syslog-ng-request at lists.balabit.hu
> Subject: syslog-ng Digest, Vol 108, Issue 24
> To: syslog-ng at lists.balabit.hu
> Date: Tue, 29 Apr 2014 13:25:01 +0200
>
> Send syslog-ng mailing list submissions to
> syslog-ng at lists.balabit.hu
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> or, via email, send a message with subject or body 'help' to
> syslog-ng-request at lists.balabit.hu
>
> You can reach the person managing the list at
> syslog-ng-owner at lists.balabit.hu
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of syslog-ng digest..."
>
>
> Today's Topics:
>
> 1. Re: Ubuntu Precise -ng filling out buffer, dropping messages
> (Gergely Nagy)
> 2. [Bug 279] Syslog-ng central loging server seg fault gentoo
> (bugzilla at bugzilla.balabit.com)
> 3. [Bug 279] Syslog-ng central loging server seg fault gentoo
> (bugzilla at bugzilla.balabit.com)
> 4. Re: Pattern DB Parser "Default Values" (Gergely Nagy)
> 5. Re: syslog-ng does not start if destination host not found
> (Gergely Nagy)
> 6. [Bug 275] lib/filter/filter-in-list.c does not compile under
> Solaris 10 (bugzilla at bugzilla.balabit.com)
> 7. [Bug 279] Syslog-ng central loging server seg fault gentoo
> (bugzilla at bugzilla.balabit.com)
> 8. Re: Basic (?) multi line question (Jim Hendrick)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 29 Apr 2014 12:24:09 +0200
> From: Gergely Nagy <algernon at balabit.hu>
> Subject: Re: [syslog-ng] Ubuntu Precise -ng filling out buffer,
> dropping messages
> To: syslog-ng at lists.balabit.hu
> Message-ID: <87mwf4xwl2.fsf at balabit.hu>
> Content-Type: text/plain
>
> Hi!
>
> Chaman Chakalaka <chebannedmeagain at hotmail.com> writes:
>
> > I'm trying to process ~800 UDP messages second, which I don't think is
> > much. The current setup worked fine in Ubuntu 10.04 (Lucid) and
> > syslog-ng 2.6 (I believe). I'm running into what I believe is receive
> > buffer problems on Ubuntu Server 12.04 (Precise) w/ ng 3.XX
>
> First of all, what's the exact version of your syslog-ng? Precise has a
> fairly old version, one that's... not exactly the best release. I'd
> suggest you give a try to the packages at:
> http://asylum.madhouse-project.org/projects/debian/
>
> I'd suggest the syslog-ng 3.5 branch from there, and see if the problem
> persists with an upgraded syslog-ng. If it persists, let us know, and
> we'll help debug the issue further.
>
> --
> |8]
>
>
>
> ------------------------------
>
> Message: 2
> Date: Tue, 29 Apr 2014 12:25:38 +0200 (CEST)
> From: bugzilla at bugzilla.balabit.com
> Subject: [syslog-ng] [Bug 279] Syslog-ng central loging server seg
> fault gentoo
> To: syslog-ng at lists.balabit.hu
> Message-ID: <20140429102538.E2BAC39DC88 at lists.balabit.hu>
> Content-Type: text/plain; charset="UTF-8"
>
> https://bugzilla.balabit.com/show_bug.cgi?id=279
>
>
> Gergely Nagy <algernon at balabit.hu> changed:
>
> What |Removed |Added
> ----------------------------------------------------------------------------
> CC| |algernon at balabit.hu
> AssignedTo|bazsi at balabit.hu |algernon at balabit.hu
>
>
>
>
> --
> Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email
> ------- You are receiving this mail because: -------
> You are watching all bug changes.
>
>
> ------------------------------
>
> Message: 3
> Date: Tue, 29 Apr 2014 12:29:08 +0200 (CEST)
> From: bugzilla at bugzilla.balabit.com
> Subject: [syslog-ng] [Bug 279] Syslog-ng central loging server seg
> fault gentoo
> To: syslog-ng at lists.balabit.hu
> Message-ID: <20140429102908.849C539DC78 at lists.balabit.hu>
> Content-Type: text/plain; charset="UTF-8"
>
> https://bugzilla.balabit.com/show_bug.cgi?id=279
>
>
>
>
>
> --- Comment #1 from Gergely Nagy <algernon at balabit.hu> 2014-04-29 12:29:08 ---
> Without debug symbols, the backtrace is fairly useless for debugging purposes, unfortunately. It would help tremendously, if you could reproduce the problem
> with a non-stripped binary, so we see the functions in the backtrace.
>
> Meanwhile, can I ask what config you use on the host where the segmentation fault happened? Maybe we can figure something out from that...
>
> Thanks!
>
>
> --
> Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email
> ------- You are receiving this mail because: -------
> You are watching all bug changes.
>
>
> ------------------------------
>
> Message: 4
> Date: Tue, 29 Apr 2014 12:41:44 +0200
> From: Gergely Nagy <algernon at balabit.hu>
> Subject: Re: [syslog-ng] Pattern DB Parser "Default Values"
> To: syslog-ng at lists.balabit.hu
> Cc: Balazs Scheidler <bazsi at balabit.com>
> Message-ID: <87iopsxvrr.fsf at balabit.hu>
> Content-Type: text/plain
>
> David Hauck <davidh at netacquire.com> writes:
>
> > I was wondering if there was a way to specify default values for
> > pattern DB parsers that include a value, but where the parsed value is
> > <null>[/empty]?
> >
> > In particular if I have something like the following:
> >
> > <pattern>test message; field1=@ESTRING:field1: @field2=@ESTRING:field2: @field3=@ESTRING:: @field4=@ESTRING:field4: @</pattern>
> >
> > I'd like to be able to do something like either, 1:
> >
> > <pattern>test message; field1=@ESTRING:field1<foo>: @field2=@ESTRING:field2<bar>: @field3=@ESTRING:: @field4=@ESTRING:field4<beef>: @</pattern>
> >
> > Or 2:
> >
> > <pattern>test message; field1=@ESTRING:field1: @field2=@ESTRING:field2: @field3=@ESTRING:: @field4=@ESTRING:field4: @</pattern>
> > <values>
> > <value name="field1.default">foo</value>
> > <value name="field2.default">bar</value>
> > <value name="field4.default">beef</value>
> >
> > Just curious...
>
> You can use ${field1:-foo} in templates, to set a default if none is
> set. It doesn't work for empty fields, though, but that can be worked
> around with an $(if $(length $field1) eq 0 "default" $field1) template,
> possibly in a rewrite rule.
>
> Though, maybe ${field1:-foo} should work for empty values too, not just
> unset ones (to mimic shell better, which does just that). I can make it
> do so, if that'd be desired, would make it unnecessary to use the $(if)
> hack.
>
> @Bazsi: What do you think?
>
> --
> |8]
>
>
>
> ------------------------------
>
> Message: 5
> Date: Tue, 29 Apr 2014 12:59:56 +0200
> From: Gergely Nagy <algernon at balabit.hu>
> Subject: Re: [syslog-ng] syslog-ng does not start if destination host
> not found
> To: syslog-ng at lists.balabit.hu
> Message-ID: <87eh0gxuxf.fsf at balabit.hu>
> Content-Type: text/plain
>
> "Bendler, Ehren" <ebendler at ciena.com> writes:
>
> [...]
> > If this is the intended behavior, that's fine too. We can deploy our
> > own patch to the afsocket module if it isn't going to be changed in a
> > release.
>
> No, this is definitely not the intended behaviour. Some change between
> 3.3.5 and 3.5.7 broke the fix, I'll go ahead and restore the intended
> behaviour. Thanks for reporting the issue!
>
> Unfortunately, I can't help with the other issue at the moment, but I'll
> try to revisit it later.
>
> --
> |8]
>
>
>
> ------------------------------
>
> Message: 6
> Date: Tue, 29 Apr 2014 13:09:22 +0200 (CEST)
> From: bugzilla at bugzilla.balabit.com
> Subject: [syslog-ng] [Bug 275] lib/filter/filter-in-list.c does not
> compile under Solaris 10
> To: syslog-ng at lists.balabit.hu
> Message-ID: <20140429110922.7AD9939DC99 at lists.balabit.hu>
> Content-Type: text/plain; charset="UTF-8"
>
> https://bugzilla.balabit.com/show_bug.cgi?id=275
>
>
>
>
>
> --- Comment #2 from Gergely Nagy <algernon at balabit.hu> 2014-04-29 13:09:22 ---
> I think we can change the code to use find_cr_or_lf(), instead of using getline(), or reimplement something like getline() in terms of find_cr_or_lf() + fgets
> (or mmap or something). That would solve the problem without having to add much to misc.c. I'll see what I can do.
>
>
> --
> Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email
> ------- You are receiving this mail because: -------
> You are watching all bug changes.
>
>
> ------------------------------
>
> Message: 7
> Date: Tue, 29 Apr 2014 13:21:01 +0200 (CEST)
> From: bugzilla at bugzilla.balabit.com
> Subject: [syslog-ng] [Bug 279] Syslog-ng central loging server seg
> fault gentoo
> To: syslog-ng at lists.balabit.hu
> Message-ID: <20140429112101.BD6F339DCA1 at lists.balabit.hu>
> Content-Type: text/plain; charset="UTF-8"
>
> https://bugzilla.balabit.com/show_bug.cgi?id=279
>
>
>
>
>
> --- Comment #2 from Martin <hlavacek at gmx.com> 2014-04-29 13:21:02 ---
> I thought that I have recompiled syslog with debug symbols because I have added --enable-debug to my ebuild:
>
> syslog1 ~ # syslog-ng -V
> syslog-ng 3.4.7
> Installer-Version: 3.4.7
> Revision: ssh+git://algernon@git.balabit/var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.4#detached_from_v3.4.3#999a7a6102d40da44b75a2acf78e54244164771f
> Compile-Date: Apr 29 2014 13:06:40
> Available-Modules:
> affile,afprog,afsocket-notls,afsocket-tls,afuser,basicfuncs,confgen,csvparser,dbparser,syslogformat,cryptofuncs,system-source,afamqp,afsocket
> Enable-Debug: on
> Enable-GProf: off
> Enable-Memtrace: off
> Enable-IPv6: on
> Enable-Spoof-Source: off
> Enable-TCP-Wrapper: on
> Enable-Linux-Caps: off
> Enable-Pcre: on
>
> You can see that opt "Enable-Debug:" is ON. It is not enought? If not can you please give me any advice how should I recompile this binary in proper way in
> gentoo?
>
> Size of binary is:
> syslog1 ~ # ls -lah /usr/sbin/syslog-ng
> 16K -rwxr-xr-x 1 root root 15K Apr 29 13:18 /usr/sbin/syslog-ng*
>
> Configure options by emerge:
> ./configure --prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share
> --sysconfdir=/etc --localstatedir=/var/lib --libdir=/usr/lib64 --disable-silent-rules --disable-dependency-tracking --with-ivykis=internal
> --with-libmongo-client=internal --sysconfdir=/etc/syslog-ng --localstatedir=/var/lib/syslog-ng --with-pidfile-dir=/var/run
> --with-module-dir=/usr/lib64/syslog-ng --enable-debug --with-systemdsystemunitdir=/usr/lib/systemd/system --disable-systemd --disable-linux-caps
> --disable-geoip --enable-ipv6 --disable-json --disable-mongodb --enable-pcre --disable-smtp --disable-spoof-source --disable-sql --enable-ssl
> --enable-tcp-wrapper
>
>
> --
> Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email
> ------- You are receiving this mail because: -------
> You are watching all bug changes.
>
>
> ------------------------------
>
> Message: 8
> Date: Tue, 29 Apr 2014 07:24:58 -0400
> From: Jim Hendrick <jrhendri at roadrunner.com>
> Subject: Re: [syslog-ng] Basic (?) multi line question
> To: Syslog-ng users' and developers' mailing list
> <syslog-ng at lists.balabit.hu>
> Message-ID: <535F8C0A.5060104 at roadrunner.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> Thanks all for the thoughts -
>
> I will try to write up some of the patterns and correlations, starting
> with the most simple.
>
> This would (I think) be a valuable addition to track different logs that
> have some dynamic id as a key.
>
> (ultimately I am hoping to parse specific data out of these multi-line
> beasties and be able to populate a database directly from syslog-ng)
>
> I will work on writing this up this week.
>
> Thanks again!
> Jim
>
>
> On 04/29/2014 04:53 AM, Tusa Viktor wrote:
> > Hi!
> >
> > If you know the format of all the messages which possibly contains a
> > MID, you can write patterns for them and then you can use correlation
> > to extract information from these messages. But it only works with
> > special conditions, I think it wouldn't work in your case. But it
> > wouldn't be so hard to create such functionality in syslog-ng, so if
> > you open a github issue in http://github.com/balabit/syslog-ng, some
> > of us will try to make it work.
> >
> > Best Regards,
> > Viktor
> >
> >
> > On Tue, Apr 29, 2014 at 8:14 AM, C. L. Martinez <carlopmart at gmail.com
> > <mailto:carlopmart at gmail.com>> wrote:
> >
> > Hi Jim,
> >
> > Some time ago, I have tried the same: correlate logs for Ironport
> > devices. And my conclusion was: impossible. I loose a lot info and
> > some correlated logs are wrong ...
> >
> > The only approach that maybe should work with opensource tools, IMO,
> > is rsyslog+sec.pl <http://sec.pl>. But, as a Orangepeel says,
> > logstash can be an
> > option.
> >
> > Bye.
> >
> > On Mon, Apr 28, 2014 at 2:44 PM, <jrhendri at roadrunner.com
> > <mailto:jrhendri at roadrunner.com>> wrote:
> > > Hmmm - crickets :-)
> > >
> > > I have some examples like this:
> > > <date> <host> <program>: Info: New SMTP ICID [0-9]{9} <rest of
> > message>
> > > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9}
> > <rest of message>
> > > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9}
> > <rest of message>
> > > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9}
> > <rest of message>
> > > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9}
> > <rest of message>
> > > <date> <host> <program>: Info: New SMTP DCID [0-9]{9} <rest of
> > message>
> > > <date> <host> <program>: Info: Message done DCID [0-9]{9} MID
> > [0-9]{9} <rest of message>
> > > <date> <host> <program>: Info: ICID [0-9]{9} close
> > >
> > > this is only an example to illustrate the different message
> > elements that contain different kinds of IDs.
> > >
> > > The issue is there will be interleaving with *different* ICID
> > (inbound connections from different SMTP servers) each sending
> > multiple MIDs (message IDs) and also different DCID (destination
> > connections *to* different mail relays).
> > >
> > > I have been looking at multi-line-mode(regexp) but that seems to
> > imply all consecutive lines until the next regex match are assumed
> > to be part of the same message.
> > >
> > > I hope I can do something where all matching ICIDs are treated
> > as part of one line, that can be parsed separately.
> > >
> > > Not sure if this is possible with multi-line-mode *or* with some
> > patterndb wizardry.
> > >
> > > Has anyone addressed this?
> > >
> > > Thanks for any working-examples/guidance/sympathy (in roughly
> > that order :-)
> > >
> > > Jim
> > >
> > >
> > >
> > >
> > > ---- jrhendri at roadrunner.com <mailto:jrhendri at roadrunner.com> wrote:
> > >> Hi,
> > >>
> > >> I am trying to parse data elements out of a variable number
> > of log lines that all are associated by a single unique key.
> > >>
> > >> Specifically - they are Cisco IronPort email logs that have
> > various "ID" fields (MID - message ID is the most common)
> > >>
> > >>
> > >> Essentially I want to pull the MID out of the line marked marked:
> > >>
> > >> "Start MID (\d+) <other stuff>"
> > >>
> > >> and then process every line that matches that specific MID
> > value as part of the message.
> > >>
> > >> Note: they all have this string included somewhere:
> > >>
> > >> "MID (\d+) "
> > >>
> > >> Up to a reasonable timeout - or ended by:
> > >>
> > >> "Message finished mid (\d+) done" with the matching ID.
> > >>
> > >> Is this possible with syslog-ng? (OSE or PE?)
> > >>
> > >> I thought I had seen something using patterndb but I cannot
> > seem to find the reference
> > >>
> > >> Clearly there will be interleaved lines with *different* MIDs
> > that need to be processed independently.
> > >>
> > >> Thanks in advance!
> > >> Jim
> > >
> > >
> > ______________________________________________________________________________
> > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > > Documentation:
> > http://www.balabit.com/support/documentation/?product=syslog-ng
> > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> > >
> > ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation:
> > http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >
> >
> >
> >
> > ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20140429/ebda655f/attachment.htm
>
> ------------------------------
>
> _______________________________________________
> syslog-ng maillist - syslog-ng at lists.balabit.hu
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
>
>
> End of syslog-ng Digest, Vol 108, Issue 24
> ******************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20140429/8ff31961/attachment-0001.htm
More information about the syslog-ng
mailing list