Re: [syslog-ng] Ubuntu Precise -ng filling out buffer, dropping messages

29 Apr 2014

      I fixed the issue with udp being dropped at the system level by changing the linux core files but this time restarting the system, now I know they are missing somewhere between getting to the system and syslog-ng :(

I'm lost once again...

From: chebannedmeagain@hotmail.com
To: syslog-ng@lists.balabit.hu
Subject: Re:  Ubuntu Precise -ng filling out buffer,	dropping messages
Date: Tue, 29 Apr 2014 09:43:06 -0700

Thanks for responding!

The exact version is of syslog-ng is 3.3.4.  Here's something you may find of interest.

1K Messages
root@badbox:/proc/net# nc -lku 514 > /tmp/testing123.txt
root@goodbox:/proc/sys/net# loggen --inet --dgram --size 500 --rate 1000 --interval 30 badbox.cbf 514
average rate = 996.43 msg/sec, count=29893 <--------------- sent
root@badbox:/proc/net# cat /tmp/testing123.txt | wc -l
24081  <------------ received/processed

Lower the rate to 150 messages per second:
root@goodbox:/proc/sys/net# loggen --inet --dgram --size 500 --rate 150 --interval 30 badbox.cbf 514
average rate = 149.03 msg/sec, count=4471 < -------------------- sent
root@badbox:/proc/net# cat /tmp/testing123.txt | wc -l
4471 < -------------------------- received/processed

At this point, given the test above, I don't know if this is a system issue or a syslog-ng issue.  It seems to be system, but I really can't tell what.
...
From: syslog-ng-request@lists.balabit.hu
Subject: syslog-ng Digest, Vol 108, Issue 24
To: syslog-ng@lists.balabit.hu
Date: Tue, 29 Apr 2014 13:25:01 +0200
Send syslog-ng mailing list submissions to
  syslog-ng@lists.balabit.hu
To subscribe or unsubscribe via the World Wide Web, visit
  https://lists.balabit.hu/mailman/listinfo/syslog-ng
or, via email, send a message with subject or body 'help' to
  syslog-ng-request@lists.balabit.hu
You can reach the person managing the list at
  syslog-ng-owner@lists.balabit.hu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of syslog-ng digest..."
Today's Topics:
1. Re:  Ubuntu Precise -ng filling out buffer,	dropping messages
      (Gergely Nagy)
   2.  [Bug 279] Syslog-ng central loging server seg fault	gentoo
      (bugzilla@bugzilla.balabit.com)
   3.  [Bug 279] Syslog-ng central loging server seg fault	gentoo
      (bugzilla@bugzilla.balabit.com)
   4. Re:  Pattern DB Parser "Default Values" (Gergely Nagy)
   5. Re:  syslog-ng does not start if destination host not	found
      (Gergely Nagy)
   6.  [Bug 275] lib/filter/filter-in-list.c does not compile under
      Solaris 10 (bugzilla@bugzilla.balabit.com)
   7.  [Bug 279] Syslog-ng central loging server seg fault	gentoo
      (bugzilla@bugzilla.balabit.com)
   8. Re:  Basic (?) multi line question (Jim Hendrick)
----------------------------------------------------------------------
Message: 1
Date: Tue, 29 Apr 2014 12:24:09 +0200
From: Gergely Nagy <algernon@balabit.hu>
Subject: Re: [syslog-ng] Ubuntu Precise -ng filling out buffer,
  dropping messages
To: syslog-ng@lists.balabit.hu
Message-ID: <87mwf4xwl2.fsf@balabit.hu>
Content-Type: text/plain
Hi!
Chaman Chakalaka <chebannedmeagain@hotmail.com> writes:
...
I'm trying to process ~800 UDP messages second, which I don't think is
much. The current setup worked fine in Ubuntu 10.04 (Lucid) and
syslog-ng 2.6 (I believe). I'm running into what I believe is receive
buffer problems on Ubuntu Server 12.04 (Precise) w/ ng 3.XX
First of all, what's the exact version of your syslog-ng? Precise has a
fairly old version, one that's... not exactly the best release. I'd
suggest you give a try to the packages at:
  http://asylum.madhouse-project.org/projects/debian/
I'd suggest the syslog-ng 3.5 branch from there, and see if the problem
persists with an upgraded syslog-ng. If it persists, let us know, and
we'll help debug the issue further.
-- 
|8]
------------------------------
Message: 2
Date: Tue, 29 Apr 2014 12:25:38 +0200 (CEST)
From: bugzilla@bugzilla.balabit.com
Subject: [syslog-ng] [Bug 279] Syslog-ng central loging server seg
  fault	gentoo
To: syslog-ng@lists.balabit.hu
Message-ID: <20140429102538.E2BAC39DC88@lists.balabit.hu>
Content-Type: text/plain; charset="UTF-8"
https://bugzilla.balabit.com/show_bug.cgi?id=279
Gergely Nagy <algernon@balabit.hu> changed:
What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |algernon@balabit.hu
         AssignedTo|bazsi@balabit.hu            |algernon@balabit.hu
-- 
Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching all bug changes.
------------------------------
Message: 3
Date: Tue, 29 Apr 2014 12:29:08 +0200 (CEST)
From: bugzilla@bugzilla.balabit.com
Subject: [syslog-ng] [Bug 279] Syslog-ng central loging server seg
  fault	gentoo
To: syslog-ng@lists.balabit.hu
Message-ID: <20140429102908.849C539DC78@lists.balabit.hu>
Content-Type: text/plain; charset="UTF-8"
https://bugzilla.balabit.com/show_bug.cgi?id=279
--- Comment #1 from Gergely Nagy <algernon@balabit.hu>  2014-04-29 12:29:08 ---
Without debug symbols, the backtrace is fairly useless for debugging purposes, unfortunately. It would help tremendously, if you could reproduce the problem
with a non-stripped binary, so we see the functions in the backtrace.
Meanwhile, can I ask what config you use on the host where the segmentation fault happened? Maybe we can figure something out from that...
Thanks!
-- 
Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching all bug changes.
------------------------------
Message: 4
Date: Tue, 29 Apr 2014 12:41:44 +0200
From: Gergely Nagy <algernon@balabit.hu>
Subject: Re: [syslog-ng] Pattern DB Parser "Default Values"
To: syslog-ng@lists.balabit.hu
Cc: Balazs Scheidler <bazsi@balabit.com>
Message-ID: <87iopsxvrr.fsf@balabit.hu>
Content-Type: text/plain
David Hauck <davidh@netacquire.com> writes:
...
I was wondering if there was a way to specify default values for
pattern DB parsers that include a value, but where the parsed value is
<null>[/empty]?
In particular if I have something like the following:
<pattern>test message; field1=@ESTRING:field1: @field2=@ESTRING:field2: @field3=@ESTRING:: @field4=@ESTRING:field4: @</pattern>
I'd like to be able to do something like either, 1:
<pattern>test message; field1=@ESTRING:field1<foo>: @field2=@ESTRING:field2<bar>: @field3=@ESTRING:: @field4=@ESTRING:field4<beef>: @</pattern>
Or 2:
<pattern>test message; field1=@ESTRING:field1: @field2=@ESTRING:field2: @field3=@ESTRING:: @field4=@ESTRING:field4: @</pattern>
        <values>
          <value name="field1.default">foo</value>
          <value name="field2.default">bar</value>
          <value name="field4.default">beef</value>
Just curious...
You can use ${field1:-foo} in templates, to set a default if none is
set. It doesn't work for empty fields, though, but that can be worked
around with an $(if $(length $field1) eq 0 "default" $field1) template,
possibly in a rewrite rule.
Though, maybe ${field1:-foo} should work for empty values too, not just
unset ones (to mimic shell better, which does just that). I can make it
do so, if that'd be desired, would make it unnecessary to use the $(if)
hack.
@Bazsi: What do you think?
-- 
|8]
------------------------------
Message: 5
Date: Tue, 29 Apr 2014 12:59:56 +0200
From: Gergely Nagy <algernon@balabit.hu>
Subject: Re: [syslog-ng] syslog-ng does not start if destination host
  not	found
To: syslog-ng@lists.balabit.hu
Message-ID: <87eh0gxuxf.fsf@balabit.hu>
Content-Type: text/plain
"Bendler, Ehren" <ebendler@ciena.com> writes:
[...]
...
If this is the intended behavior, that's fine too. We can deploy our
own patch to the afsocket module if it isn't going to be changed in a
release.
No, this is definitely not the intended behaviour. Some change between
3.3.5 and 3.5.7 broke the fix, I'll go ahead and restore the intended
behaviour. Thanks for reporting the issue!
Unfortunately, I can't help with the other issue at the moment, but I'll
try to revisit it later.
-- 
|8]
------------------------------
Message: 6
Date: Tue, 29 Apr 2014 13:09:22 +0200 (CEST)
From: bugzilla@bugzilla.balabit.com
Subject: [syslog-ng] [Bug 275] lib/filter/filter-in-list.c does not
  compile under Solaris 10
To: syslog-ng@lists.balabit.hu
Message-ID: <20140429110922.7AD9939DC99@lists.balabit.hu>
Content-Type: text/plain; charset="UTF-8"
https://bugzilla.balabit.com/show_bug.cgi?id=275
--- Comment #2 from Gergely Nagy <algernon@balabit.hu>  2014-04-29 13:09:22 ---
I think we can change the code to use find_cr_or_lf(), instead of using getline(), or reimplement something like getline() in terms of find_cr_or_lf() + fgets
(or mmap or something). That would solve the problem without having to add much to misc.c. I'll see what I can do.
-- 
Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching all bug changes.
------------------------------
Message: 7
Date: Tue, 29 Apr 2014 13:21:01 +0200 (CEST)
From: bugzilla@bugzilla.balabit.com
Subject: [syslog-ng] [Bug 279] Syslog-ng central loging server seg
  fault	gentoo
To: syslog-ng@lists.balabit.hu
Message-ID: <20140429112101.BD6F339DCA1@lists.balabit.hu>
Content-Type: text/plain; charset="UTF-8"
https://bugzilla.balabit.com/show_bug.cgi?id=279
--- Comment #2 from Martin <hlavacek@gmx.com>  2014-04-29 13:21:02 ---
I thought that I have recompiled syslog with debug symbols because I have added --enable-debug to my ebuild:
syslog1 ~ # syslog-ng -V
syslog-ng 3.4.7
Installer-Version: 3.4.7
Revision: ssh+git://algernon@git.balabit/var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.4#detached_from_v3.4.3#999a7a6102d40da44b75a2acf78e54244164771f
Compile-Date: Apr 29 2014 13:06:40
Available-Modules:
affile,afprog,afsocket-notls,afsocket-tls,afuser,basicfuncs,confgen,csvparser,dbparser,syslogformat,cryptofuncs,system-source,afamqp,afsocket
Enable-Debug: on  
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: off
Enable-TCP-Wrapper: on
Enable-Linux-Caps: off
Enable-Pcre: on
You can see that opt "Enable-Debug:" is ON. It is not enought? If not can you please give me any advice how should I recompile this binary in proper way in
gentoo?
Size of binary is:
syslog1 ~ # ls -lah /usr/sbin/syslog-ng
16K -rwxr-xr-x 1 root root 15K Apr 29 13:18 /usr/sbin/syslog-ng*
Configure options by emerge:
./configure --prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share
--sysconfdir=/etc --localstatedir=/var/lib --libdir=/usr/lib64 --disable-silent-rules --disable-dependency-tracking --with-ivykis=internal
--with-libmongo-client=internal --sysconfdir=/etc/syslog-ng --localstatedir=/var/lib/syslog-ng --with-pidfile-dir=/var/run
--with-module-dir=/usr/lib64/syslog-ng --enable-debug --with-systemdsystemunitdir=/usr/lib/systemd/system --disable-systemd --disable-linux-caps
--disable-geoip --enable-ipv6 --disable-json --disable-mongodb --enable-pcre --disable-smtp --disable-spoof-source --disable-sql --enable-ssl
--enable-tcp-wrapper
-- 
Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching all bug changes.
------------------------------
Message: 8
Date: Tue, 29 Apr 2014 07:24:58 -0400
From: Jim Hendrick <jrhendri@roadrunner.com>
Subject: Re: [syslog-ng] Basic (?) multi line question
To: Syslog-ng users' and developers' mailing list
  <syslog-ng@lists.balabit.hu>
Message-ID: <535F8C0A.5060104@roadrunner.com>
Content-Type: text/plain; charset="iso-8859-1"
Thanks all for the thoughts -
I will try to write up some of the patterns and correlations, starting
with the most simple.
This would (I think) be a valuable addition to track different logs that
have some dynamic id as a key.
(ultimately I am hoping to parse specific data out of these multi-line
beasties and be able to populate a database directly from syslog-ng)
I will work on writing this up this week.
Thanks again!
Jim
On 04/29/2014 04:53 AM, Tusa Viktor wrote:
...
Hi!
If you know the format of all the messages which possibly contains a
MID, you can write patterns for them and then you can use correlation
to extract information from these messages. But it only works with
special conditions, I think it wouldn't work in your case. But it
wouldn't be so hard to create such functionality in syslog-ng, so if
you open a github issue in http://github.com/balabit/syslog-ng, some
of us will try to make it work.
Best Regards,
Viktor
On Tue, Apr 29, 2014 at 8:14 AM, C. L. Martinez <carlopmart@gmail.com
<mailto:carlopmart@gmail.com>> wrote:
Hi Jim,
Some time ago, I have tried the same: correlate logs for Ironport
    devices. And my conclusion was: impossible. I loose a lot info and
    some correlated logs are wrong ...
The only approach that maybe should work with opensource tools, IMO,
    is rsyslog+sec.pl <http://sec.pl>. But, as a Orangepeel says,
    logstash can be an
    option.
Bye.
On Mon, Apr 28, 2014 at 2:44 PM,  <jrhendri@roadrunner.com
    <mailto:jrhendri@roadrunner.com>> wrote:
    > Hmmm - crickets :-)
    >
    > I have some examples like this:
    > <date> <host> <program>: Info: New SMTP ICID [0-9]{9} <rest of
    message>
    > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9}
    <rest of message>
    > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9}
    <rest of message>
    > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9}
    <rest of message>
    > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9}
    <rest of message>
    > <date> <host> <program>: Info: New SMTP DCID [0-9]{9} <rest of
    message>
    > <date> <host> <program>: Info: Message done DCID [0-9]{9} MID
    [0-9]{9} <rest of message>
    > <date> <host> <program>: Info: ICID [0-9]{9} close
    >
    > this is only an example to illustrate the different message
    elements that contain different kinds of IDs.
    >
    > The issue is there will be interleaving with *different* ICID
    (inbound connections from different SMTP servers) each sending
    multiple MIDs (message IDs) and also different DCID (destination
    connections *to* different mail relays).
    >
    > I have been looking at multi-line-mode(regexp) but that seems to
    imply all consecutive lines until the next regex match are assumed
    to be part of the same message.
    >
    > I hope I can do something where all matching ICIDs are treated
    as part of one line, that can be parsed separately.
    >
    > Not sure if this is possible with multi-line-mode *or* with some
    patterndb wizardry.
    >
    > Has anyone addressed this?
    >
    > Thanks for any working-examples/guidance/sympathy (in roughly
    that order :-)
    >
    > Jim
    >
    >
    >
    >
    > ---- jrhendri@roadrunner.com <mailto:jrhendri@roadrunner.com> wrote:
    >> Hi,
    >>
    >>   I am trying to parse data elements out of a variable number
    of log lines that all are associated by a single unique key.
    >>
    >> Specifically - they are Cisco IronPort email logs that have
    various "ID" fields (MID - message ID is the most common)
    >>
    >>
    >> Essentially I want to pull the MID out of the line marked marked:
    >>
    >> "Start MID (\d+) <other stuff>"
    >>
    >>  and then process every line that matches that specific MID
    value as part of the message.
    >>
    >> Note: they all have this string included somewhere:
    >>
    >> "MID (\d+) "
    >>
    >> Up to a reasonable timeout - or ended by:
    >>
    >>  "Message finished mid (\d+) done" with the matching ID.
    >>
    >> Is this possible with syslog-ng? (OSE or PE?)
    >>
    >> I thought I had seen something using patterndb but I cannot
    seem to find the reference
    >>
    >> Clearly there will be interleaved lines with *different* MIDs
    that need to be processed independently.
    >>
    >> Thanks in advance!
    >> Jim
    >
    >
    ______________________________________________________________________________
    > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
    > Documentation:
    http://www.balabit.com/support/documentation/?product=syslog-ng
    > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
    >
    ______________________________________________________________________________
    Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
    Documentation:
    http://www.balabit.com/support/documentation/?product=syslog-ng
    FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq