Re: [syslog-ng] Ubuntu Precise -ng filling out buffer, dropping messages
Thanks for responding! The exact version is of syslog-ng is 3.3.4. Here's something you may find of interest. 1K Messages root@badbox:/proc/net# nc -lku 514 > /tmp/testing123.txt root@goodbox:/proc/sys/net# loggen --inet --dgram --size 500 --rate 1000 --interval 30 badbox.cbf 514 average rate = 996.43 msg/sec, count=29893 <--------------- sent root@badbox:/proc/net# cat /tmp/testing123.txt | wc -l 24081 <------------ received/processed Lower the rate to 150 messages per second: root@goodbox:/proc/sys/net# loggen --inet --dgram --size 500 --rate 150 --interval 30 badbox.cbf 514 average rate = 149.03 msg/sec, count=4471 < -------------------- sent root@badbox:/proc/net# cat /tmp/testing123.txt | wc -l 4471 < -------------------------- received/processed At this point, given the test above, I don't know if this is a system issue or a syslog-ng issue. It seems to be system, but I really can't tell what.
From: syslog-ng-request@lists.balabit.hu Subject: syslog-ng Digest, Vol 108, Issue 24 To: syslog-ng@lists.balabit.hu Date: Tue, 29 Apr 2014 13:25:01 +0200
Send syslog-ng mailing list submissions to syslog-ng@lists.balabit.hu
To subscribe or unsubscribe via the World Wide Web, visit https://lists.balabit.hu/mailman/listinfo/syslog-ng or, via email, send a message with subject or body 'help' to syslog-ng-request@lists.balabit.hu
You can reach the person managing the list at syslog-ng-owner@lists.balabit.hu
When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..."
Today's Topics:
1. Re: Ubuntu Precise -ng filling out buffer, dropping messages (Gergely Nagy) 2. [Bug 279] Syslog-ng central loging server seg fault gentoo (bugzilla@bugzilla.balabit.com) 3. [Bug 279] Syslog-ng central loging server seg fault gentoo (bugzilla@bugzilla.balabit.com) 4. Re: Pattern DB Parser "Default Values" (Gergely Nagy) 5. Re: syslog-ng does not start if destination host not found (Gergely Nagy) 6. [Bug 275] lib/filter/filter-in-list.c does not compile under Solaris 10 (bugzilla@bugzilla.balabit.com) 7. [Bug 279] Syslog-ng central loging server seg fault gentoo (bugzilla@bugzilla.balabit.com) 8. Re: Basic (?) multi line question (Jim Hendrick)
----------------------------------------------------------------------
Message: 1 Date: Tue, 29 Apr 2014 12:24:09 +0200 From: Gergely Nagy <algernon@balabit.hu> Subject: Re: [syslog-ng] Ubuntu Precise -ng filling out buffer, dropping messages To: syslog-ng@lists.balabit.hu Message-ID: <87mwf4xwl2.fsf@balabit.hu> Content-Type: text/plain
Hi!
Chaman Chakalaka <chebannedmeagain@hotmail.com> writes:
I'm trying to process ~800 UDP messages second, which I don't think is much. The current setup worked fine in Ubuntu 10.04 (Lucid) and syslog-ng 2.6 (I believe). I'm running into what I believe is receive buffer problems on Ubuntu Server 12.04 (Precise) w/ ng 3.XX
First of all, what's the exact version of your syslog-ng? Precise has a fairly old version, one that's... not exactly the best release. I'd suggest you give a try to the packages at: http://asylum.madhouse-project.org/projects/debian/
I'd suggest the syslog-ng 3.5 branch from there, and see if the problem persists with an upgraded syslog-ng. If it persists, let us know, and we'll help debug the issue further.
-- |8]
------------------------------
Message: 2 Date: Tue, 29 Apr 2014 12:25:38 +0200 (CEST) From: bugzilla@bugzilla.balabit.com Subject: [syslog-ng] [Bug 279] Syslog-ng central loging server seg fault gentoo To: syslog-ng@lists.balabit.hu Message-ID: <20140429102538.E2BAC39DC88@lists.balabit.hu> Content-Type: text/plain; charset="UTF-8"
https://bugzilla.balabit.com/show_bug.cgi?id=279
Gergely Nagy <algernon@balabit.hu> changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |algernon@balabit.hu AssignedTo|bazsi@balabit.hu |algernon@balabit.hu
-- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
------------------------------
Message: 3 Date: Tue, 29 Apr 2014 12:29:08 +0200 (CEST) From: bugzilla@bugzilla.balabit.com Subject: [syslog-ng] [Bug 279] Syslog-ng central loging server seg fault gentoo To: syslog-ng@lists.balabit.hu Message-ID: <20140429102908.849C539DC78@lists.balabit.hu> Content-Type: text/plain; charset="UTF-8"
https://bugzilla.balabit.com/show_bug.cgi?id=279
--- Comment #1 from Gergely Nagy <algernon@balabit.hu> 2014-04-29 12:29:08 --- Without debug symbols, the backtrace is fairly useless for debugging purposes, unfortunately. It would help tremendously, if you could reproduce the problem with a non-stripped binary, so we see the functions in the backtrace.
Meanwhile, can I ask what config you use on the host where the segmentation fault happened? Maybe we can figure something out from that...
Thanks!
-- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
------------------------------
Message: 4 Date: Tue, 29 Apr 2014 12:41:44 +0200 From: Gergely Nagy <algernon@balabit.hu> Subject: Re: [syslog-ng] Pattern DB Parser "Default Values" To: syslog-ng@lists.balabit.hu Cc: Balazs Scheidler <bazsi@balabit.com> Message-ID: <87iopsxvrr.fsf@balabit.hu> Content-Type: text/plain
David Hauck <davidh@netacquire.com> writes:
I was wondering if there was a way to specify default values for pattern DB parsers that include a value, but where the parsed value is <null>[/empty]?
In particular if I have something like the following:
<pattern>test message; field1=@ESTRING:field1: @field2=@ESTRING:field2: @field3=@ESTRING:: @field4=@ESTRING:field4: @</pattern>
I'd like to be able to do something like either, 1:
<pattern>test message; field1=@ESTRING:field1<foo>: @field2=@ESTRING:field2<bar>: @field3=@ESTRING:: @field4=@ESTRING:field4<beef>: @</pattern>
Or 2:
<pattern>test message; field1=@ESTRING:field1: @field2=@ESTRING:field2: @field3=@ESTRING:: @field4=@ESTRING:field4: @</pattern> <values> <value name="field1.default">foo</value> <value name="field2.default">bar</value> <value name="field4.default">beef</value>
Just curious...
You can use ${field1:-foo} in templates, to set a default if none is set. It doesn't work for empty fields, though, but that can be worked around with an $(if $(length $field1) eq 0 "default" $field1) template, possibly in a rewrite rule.
Though, maybe ${field1:-foo} should work for empty values too, not just unset ones (to mimic shell better, which does just that). I can make it do so, if that'd be desired, would make it unnecessary to use the $(if) hack.
@Bazsi: What do you think?
-- |8]
------------------------------
Message: 5 Date: Tue, 29 Apr 2014 12:59:56 +0200 From: Gergely Nagy <algernon@balabit.hu> Subject: Re: [syslog-ng] syslog-ng does not start if destination host not found To: syslog-ng@lists.balabit.hu Message-ID: <87eh0gxuxf.fsf@balabit.hu> Content-Type: text/plain
"Bendler, Ehren" <ebendler@ciena.com> writes:
[...]
If this is the intended behavior, that's fine too. We can deploy our own patch to the afsocket module if it isn't going to be changed in a release.
No, this is definitely not the intended behaviour. Some change between 3.3.5 and 3.5.7 broke the fix, I'll go ahead and restore the intended behaviour. Thanks for reporting the issue!
Unfortunately, I can't help with the other issue at the moment, but I'll try to revisit it later.
-- |8]
------------------------------
Message: 6 Date: Tue, 29 Apr 2014 13:09:22 +0200 (CEST) From: bugzilla@bugzilla.balabit.com Subject: [syslog-ng] [Bug 275] lib/filter/filter-in-list.c does not compile under Solaris 10 To: syslog-ng@lists.balabit.hu Message-ID: <20140429110922.7AD9939DC99@lists.balabit.hu> Content-Type: text/plain; charset="UTF-8"
https://bugzilla.balabit.com/show_bug.cgi?id=275
--- Comment #2 from Gergely Nagy <algernon@balabit.hu> 2014-04-29 13:09:22 --- I think we can change the code to use find_cr_or_lf(), instead of using getline(), or reimplement something like getline() in terms of find_cr_or_lf() + fgets (or mmap or something). That would solve the problem without having to add much to misc.c. I'll see what I can do.
-- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
------------------------------
Message: 7 Date: Tue, 29 Apr 2014 13:21:01 +0200 (CEST) From: bugzilla@bugzilla.balabit.com Subject: [syslog-ng] [Bug 279] Syslog-ng central loging server seg fault gentoo To: syslog-ng@lists.balabit.hu Message-ID: <20140429112101.BD6F339DCA1@lists.balabit.hu> Content-Type: text/plain; charset="UTF-8"
https://bugzilla.balabit.com/show_bug.cgi?id=279
--- Comment #2 from Martin <hlavacek@gmx.com> 2014-04-29 13:21:02 --- I thought that I have recompiled syslog with debug symbols because I have added --enable-debug to my ebuild:
syslog1 ~ # syslog-ng -V syslog-ng 3.4.7 Installer-Version: 3.4.7 Revision: ssh+git://algernon@git.balabit/var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.4#detached_from_v3.4.3#999a7a6102d40da44b75a2acf78e54244164771f Compile-Date: Apr 29 2014 13:06:40 Available-Modules: affile,afprog,afsocket-notls,afsocket-tls,afuser,basicfuncs,confgen,csvparser,dbparser,syslogformat,cryptofuncs,system-source,afamqp,afsocket Enable-Debug: on Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: off Enable-Pcre: on
You can see that opt "Enable-Debug:" is ON. It is not enought? If not can you please give me any advice how should I recompile this binary in proper way in gentoo?
Size of binary is: syslog1 ~ # ls -lah /usr/sbin/syslog-ng 16K -rwxr-xr-x 1 root root 15K Apr 29 13:18 /usr/sbin/syslog-ng*
Configure options by emerge: ./configure --prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --libdir=/usr/lib64 --disable-silent-rules --disable-dependency-tracking --with-ivykis=internal --with-libmongo-client=internal --sysconfdir=/etc/syslog-ng --localstatedir=/var/lib/syslog-ng --with-pidfile-dir=/var/run --with-module-dir=/usr/lib64/syslog-ng --enable-debug --with-systemdsystemunitdir=/usr/lib/systemd/system --disable-systemd --disable-linux-caps --disable-geoip --enable-ipv6 --disable-json --disable-mongodb --enable-pcre --disable-smtp --disable-spoof-source --disable-sql --enable-ssl --enable-tcp-wrapper
-- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
------------------------------
Message: 8 Date: Tue, 29 Apr 2014 07:24:58 -0400 From: Jim Hendrick <jrhendri@roadrunner.com> Subject: Re: [syslog-ng] Basic (?) multi line question To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Message-ID: <535F8C0A.5060104@roadrunner.com> Content-Type: text/plain; charset="iso-8859-1"
Thanks all for the thoughts -
I will try to write up some of the patterns and correlations, starting with the most simple.
This would (I think) be a valuable addition to track different logs that have some dynamic id as a key.
(ultimately I am hoping to parse specific data out of these multi-line beasties and be able to populate a database directly from syslog-ng)
I will work on writing this up this week.
Thanks again! Jim
On 04/29/2014 04:53 AM, Tusa Viktor wrote:
Hi!
If you know the format of all the messages which possibly contains a MID, you can write patterns for them and then you can use correlation to extract information from these messages. But it only works with special conditions, I think it wouldn't work in your case. But it wouldn't be so hard to create such functionality in syslog-ng, so if you open a github issue in http://github.com/balabit/syslog-ng, some of us will try to make it work.
Best Regards, Viktor
On Tue, Apr 29, 2014 at 8:14 AM, C. L. Martinez <carlopmart@gmail.com <mailto:carlopmart@gmail.com>> wrote:
Hi Jim,
Some time ago, I have tried the same: correlate logs for Ironport devices. And my conclusion was: impossible. I loose a lot info and some correlated logs are wrong ...
The only approach that maybe should work with opensource tools, IMO, is rsyslog+sec.pl <http://sec.pl>. But, as a Orangepeel says, logstash can be an option.
Bye.
On Mon, Apr 28, 2014 at 2:44 PM, <jrhendri@roadrunner.com <mailto:jrhendri@roadrunner.com>> wrote: > Hmmm - crickets :-) > > I have some examples like this: > <date> <host> <program>: Info: New SMTP ICID [0-9]{9} <rest of message> > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> > <date> <host> <program>: Info: New SMTP DCID [0-9]{9} <rest of message> > <date> <host> <program>: Info: Message done DCID [0-9]{9} MID [0-9]{9} <rest of message> > <date> <host> <program>: Info: ICID [0-9]{9} close > > this is only an example to illustrate the different message elements that contain different kinds of IDs. > > The issue is there will be interleaving with *different* ICID (inbound connections from different SMTP servers) each sending multiple MIDs (message IDs) and also different DCID (destination connections *to* different mail relays). > > I have been looking at multi-line-mode(regexp) but that seems to imply all consecutive lines until the next regex match are assumed to be part of the same message. > > I hope I can do something where all matching ICIDs are treated as part of one line, that can be parsed separately. > > Not sure if this is possible with multi-line-mode *or* with some patterndb wizardry. > > Has anyone addressed this? > > Thanks for any working-examples/guidance/sympathy (in roughly that order :-) > > Jim > > > > > ---- jrhendri@roadrunner.com <mailto:jrhendri@roadrunner.com> wrote: >> Hi, >> >> I am trying to parse data elements out of a variable number of log lines that all are associated by a single unique key. >> >> Specifically - they are Cisco IronPort email logs that have various "ID" fields (MID - message ID is the most common) >> >> >> Essentially I want to pull the MID out of the line marked marked: >> >> "Start MID (\d+) <other stuff>" >> >> and then process every line that matches that specific MID value as part of the message. >> >> Note: they all have this string included somewhere: >> >> "MID (\d+) " >> >> Up to a reasonable timeout - or ended by: >> >> "Message finished mid (\d+) done" with the matching ID. >> >> Is this possible with syslog-ng? (OSE or PE?) >> >> I thought I had seen something using patterndb but I cannot seem to find the reference >> >> Clearly there will be interleaved lines with *different* MIDs that need to be processed independently. >> >> Thanks in advance! >> Jim > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I fixed the issue with udp being dropped at the system level by changing the linux core files but this time restarting the system, now I know they are missing somewhere between getting to the system and syslog-ng :( I'm lost once again... From: chebannedmeagain@hotmail.com To: syslog-ng@lists.balabit.hu Subject: Re: Ubuntu Precise -ng filling out buffer, dropping messages Date: Tue, 29 Apr 2014 09:43:06 -0700 Thanks for responding! The exact version is of syslog-ng is 3.3.4. Here's something you may find of interest. 1K Messages root@badbox:/proc/net# nc -lku 514 > /tmp/testing123.txt root@goodbox:/proc/sys/net# loggen --inet --dgram --size 500 --rate 1000 --interval 30 badbox.cbf 514 average rate = 996.43 msg/sec, count=29893 <--------------- sent root@badbox:/proc/net# cat /tmp/testing123.txt | wc -l 24081 <------------ received/processed Lower the rate to 150 messages per second: root@goodbox:/proc/sys/net# loggen --inet --dgram --size 500 --rate 150 --interval 30 badbox.cbf 514 average rate = 149.03 msg/sec, count=4471 < -------------------- sent root@badbox:/proc/net# cat /tmp/testing123.txt | wc -l 4471 < -------------------------- received/processed At this point, given the test above, I don't know if this is a system issue or a syslog-ng issue. It seems to be system, but I really can't tell what.
From: syslog-ng-request@lists.balabit.hu Subject: syslog-ng Digest, Vol 108, Issue 24 To: syslog-ng@lists.balabit.hu Date: Tue, 29 Apr 2014 13:25:01 +0200
Send syslog-ng mailing list submissions to syslog-ng@lists.balabit.hu
To subscribe or unsubscribe via the World Wide Web, visit https://lists.balabit.hu/mailman/listinfo/syslog-ng or, via email, send a message with subject or body 'help' to syslog-ng-request@lists.balabit.hu
You can reach the person managing the list at syslog-ng-owner@lists.balabit.hu
When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..."
Today's Topics:
1. Re: Ubuntu Precise -ng filling out buffer, dropping messages (Gergely Nagy) 2. [Bug 279] Syslog-ng central loging server seg fault gentoo (bugzilla@bugzilla.balabit.com) 3. [Bug 279] Syslog-ng central loging server seg fault gentoo (bugzilla@bugzilla.balabit.com) 4. Re: Pattern DB Parser "Default Values" (Gergely Nagy) 5. Re: syslog-ng does not start if destination host not found (Gergely Nagy) 6. [Bug 275] lib/filter/filter-in-list.c does not compile under Solaris 10 (bugzilla@bugzilla.balabit.com) 7. [Bug 279] Syslog-ng central loging server seg fault gentoo (bugzilla@bugzilla.balabit.com) 8. Re: Basic (?) multi line question (Jim Hendrick)
----------------------------------------------------------------------
Message: 1 Date: Tue, 29 Apr 2014 12:24:09 +0200 From: Gergely Nagy <algernon@balabit.hu> Subject: Re: [syslog-ng] Ubuntu Precise -ng filling out buffer, dropping messages To: syslog-ng@lists.balabit.hu Message-ID: <87mwf4xwl2.fsf@balabit.hu> Content-Type: text/plain
Hi!
Chaman Chakalaka <chebannedmeagain@hotmail.com> writes:
I'm trying to process ~800 UDP messages second, which I don't think is much. The current setup worked fine in Ubuntu 10.04 (Lucid) and syslog-ng 2.6 (I believe). I'm running into what I believe is receive buffer problems on Ubuntu Server 12.04 (Precise) w/ ng 3.XX
First of all, what's the exact version of your syslog-ng? Precise has a fairly old version, one that's... not exactly the best release. I'd suggest you give a try to the packages at: http://asylum.madhouse-project.org/projects/debian/
I'd suggest the syslog-ng 3.5 branch from there, and see if the problem persists with an upgraded syslog-ng. If it persists, let us know, and we'll help debug the issue further.
-- |8]
------------------------------
Message: 2 Date: Tue, 29 Apr 2014 12:25:38 +0200 (CEST) From: bugzilla@bugzilla.balabit.com Subject: [syslog-ng] [Bug 279] Syslog-ng central loging server seg fault gentoo To: syslog-ng@lists.balabit.hu Message-ID: <20140429102538.E2BAC39DC88@lists.balabit.hu> Content-Type: text/plain; charset="UTF-8"
https://bugzilla.balabit.com/show_bug.cgi?id=279
Gergely Nagy <algernon@balabit.hu> changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |algernon@balabit.hu AssignedTo|bazsi@balabit.hu |algernon@balabit.hu
-- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
------------------------------
Message: 3 Date: Tue, 29 Apr 2014 12:29:08 +0200 (CEST) From: bugzilla@bugzilla.balabit.com Subject: [syslog-ng] [Bug 279] Syslog-ng central loging server seg fault gentoo To: syslog-ng@lists.balabit.hu Message-ID: <20140429102908.849C539DC78@lists.balabit.hu> Content-Type: text/plain; charset="UTF-8"
https://bugzilla.balabit.com/show_bug.cgi?id=279
--- Comment #1 from Gergely Nagy <algernon@balabit.hu> 2014-04-29 12:29:08 --- Without debug symbols, the backtrace is fairly useless for debugging purposes, unfortunately. It would help tremendously, if you could reproduce the problem with a non-stripped binary, so we see the functions in the backtrace.
Meanwhile, can I ask what config you use on the host where the segmentation fault happened? Maybe we can figure something out from that...
Thanks!
-- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
------------------------------
Message: 4 Date: Tue, 29 Apr 2014 12:41:44 +0200 From: Gergely Nagy <algernon@balabit.hu> Subject: Re: [syslog-ng] Pattern DB Parser "Default Values" To: syslog-ng@lists.balabit.hu Cc: Balazs Scheidler <bazsi@balabit.com> Message-ID: <87iopsxvrr.fsf@balabit.hu> Content-Type: text/plain
David Hauck <davidh@netacquire.com> writes:
I was wondering if there was a way to specify default values for pattern DB parsers that include a value, but where the parsed value is <null>[/empty]?
In particular if I have something like the following:
<pattern>test message; field1=@ESTRING:field1: @field2=@ESTRING:field2: @field3=@ESTRING:: @field4=@ESTRING:field4: @</pattern>
I'd like to be able to do something like either, 1:
<pattern>test message; field1=@ESTRING:field1<foo>: @field2=@ESTRING:field2<bar>: @field3=@ESTRING:: @field4=@ESTRING:field4<beef>: @</pattern>
Or 2:
<pattern>test message; field1=@ESTRING:field1: @field2=@ESTRING:field2: @field3=@ESTRING:: @field4=@ESTRING:field4: @</pattern> <values> <value name="field1.default">foo</value> <value name="field2.default">bar</value> <value name="field4.default">beef</value>
Just curious...
You can use ${field1:-foo} in templates, to set a default if none is set. It doesn't work for empty fields, though, but that can be worked around with an $(if $(length $field1) eq 0 "default" $field1) template, possibly in a rewrite rule.
Though, maybe ${field1:-foo} should work for empty values too, not just unset ones (to mimic shell better, which does just that). I can make it do so, if that'd be desired, would make it unnecessary to use the $(if) hack.
@Bazsi: What do you think?
-- |8]
------------------------------
Message: 5 Date: Tue, 29 Apr 2014 12:59:56 +0200 From: Gergely Nagy <algernon@balabit.hu> Subject: Re: [syslog-ng] syslog-ng does not start if destination host not found To: syslog-ng@lists.balabit.hu Message-ID: <87eh0gxuxf.fsf@balabit.hu> Content-Type: text/plain
"Bendler, Ehren" <ebendler@ciena.com> writes:
[...]
If this is the intended behavior, that's fine too. We can deploy our own patch to the afsocket module if it isn't going to be changed in a release.
No, this is definitely not the intended behaviour. Some change between 3.3.5 and 3.5.7 broke the fix, I'll go ahead and restore the intended behaviour. Thanks for reporting the issue!
Unfortunately, I can't help with the other issue at the moment, but I'll try to revisit it later.
-- |8]
------------------------------
Message: 6 Date: Tue, 29 Apr 2014 13:09:22 +0200 (CEST) From: bugzilla@bugzilla.balabit.com Subject: [syslog-ng] [Bug 275] lib/filter/filter-in-list.c does not compile under Solaris 10 To: syslog-ng@lists.balabit.hu Message-ID: <20140429110922.7AD9939DC99@lists.balabit.hu> Content-Type: text/plain; charset="UTF-8"
https://bugzilla.balabit.com/show_bug.cgi?id=275
--- Comment #2 from Gergely Nagy <algernon@balabit.hu> 2014-04-29 13:09:22 --- I think we can change the code to use find_cr_or_lf(), instead of using getline(), or reimplement something like getline() in terms of find_cr_or_lf() + fgets (or mmap or something). That would solve the problem without having to add much to misc.c. I'll see what I can do.
-- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
------------------------------
Message: 7 Date: Tue, 29 Apr 2014 13:21:01 +0200 (CEST) From: bugzilla@bugzilla.balabit.com Subject: [syslog-ng] [Bug 279] Syslog-ng central loging server seg fault gentoo To: syslog-ng@lists.balabit.hu Message-ID: <20140429112101.BD6F339DCA1@lists.balabit.hu> Content-Type: text/plain; charset="UTF-8"
https://bugzilla.balabit.com/show_bug.cgi?id=279
--- Comment #2 from Martin <hlavacek@gmx.com> 2014-04-29 13:21:02 --- I thought that I have recompiled syslog with debug symbols because I have added --enable-debug to my ebuild:
syslog1 ~ # syslog-ng -V syslog-ng 3.4.7 Installer-Version: 3.4.7 Revision: ssh+git://algernon@git.balabit/var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.4#detached_from_v3.4.3#999a7a6102d40da44b75a2acf78e54244164771f Compile-Date: Apr 29 2014 13:06:40 Available-Modules: affile,afprog,afsocket-notls,afsocket-tls,afuser,basicfuncs,confgen,csvparser,dbparser,syslogformat,cryptofuncs,system-source,afamqp,afsocket Enable-Debug: on Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: off Enable-Pcre: on
You can see that opt "Enable-Debug:" is ON. It is not enought? If not can you please give me any advice how should I recompile this binary in proper way in gentoo?
Size of binary is: syslog1 ~ # ls -lah /usr/sbin/syslog-ng 16K -rwxr-xr-x 1 root root 15K Apr 29 13:18 /usr/sbin/syslog-ng*
Configure options by emerge: ./configure --prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --libdir=/usr/lib64 --disable-silent-rules --disable-dependency-tracking --with-ivykis=internal --with-libmongo-client=internal --sysconfdir=/etc/syslog-ng --localstatedir=/var/lib/syslog-ng --with-pidfile-dir=/var/run --with-module-dir=/usr/lib64/syslog-ng --enable-debug --with-systemdsystemunitdir=/usr/lib/systemd/system --disable-systemd --disable-linux-caps --disable-geoip --enable-ipv6 --disable-json --disable-mongodb --enable-pcre --disable-smtp --disable-spoof-source --disable-sql --enable-ssl --enable-tcp-wrapper
-- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
------------------------------
Message: 8 Date: Tue, 29 Apr 2014 07:24:58 -0400 From: Jim Hendrick <jrhendri@roadrunner.com> Subject: Re: [syslog-ng] Basic (?) multi line question To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Message-ID: <535F8C0A.5060104@roadrunner.com> Content-Type: text/plain; charset="iso-8859-1"
Thanks all for the thoughts -
I will try to write up some of the patterns and correlations, starting with the most simple.
This would (I think) be a valuable addition to track different logs that have some dynamic id as a key.
(ultimately I am hoping to parse specific data out of these multi-line beasties and be able to populate a database directly from syslog-ng)
I will work on writing this up this week.
Thanks again! Jim
On 04/29/2014 04:53 AM, Tusa Viktor wrote:
Hi!
If you know the format of all the messages which possibly contains a MID, you can write patterns for them and then you can use correlation to extract information from these messages. But it only works with special conditions, I think it wouldn't work in your case. But it wouldn't be so hard to create such functionality in syslog-ng, so if you open a github issue in http://github.com/balabit/syslog-ng, some of us will try to make it work.
Best Regards, Viktor
On Tue, Apr 29, 2014 at 8:14 AM, C. L. Martinez <carlopmart@gmail.com <mailto:carlopmart@gmail.com>> wrote:
Hi Jim,
Some time ago, I have tried the same: correlate logs for Ironport devices. And my conclusion was: impossible. I loose a lot info and some correlated logs are wrong ...
The only approach that maybe should work with opensource tools, IMO, is rsyslog+sec.pl <http://sec.pl>. But, as a Orangepeel says, logstash can be an option.
Bye.
On Mon, Apr 28, 2014 at 2:44 PM, <jrhendri@roadrunner.com <mailto:jrhendri@roadrunner.com>> wrote: > Hmmm - crickets :-) > > I have some examples like this: > <date> <host> <program>: Info: New SMTP ICID [0-9]{9} <rest of message> > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} <rest of message> > <date> <host> <program>: Info: New SMTP DCID [0-9]{9} <rest of message> > <date> <host> <program>: Info: Message done DCID [0-9]{9} MID [0-9]{9} <rest of message> > <date> <host> <program>: Info: ICID [0-9]{9} close > > this is only an example to illustrate the different message elements that contain different kinds of IDs. > > The issue is there will be interleaving with *different* ICID (inbound connections from different SMTP servers) each sending multiple MIDs (message IDs) and also different DCID (destination connections *to* different mail relays). > > I have been looking at multi-line-mode(regexp) but that seems to imply all consecutive lines until the next regex match are assumed to be part of the same message. > > I hope I can do something where all matching ICIDs are treated as part of one line, that can be parsed separately. > > Not sure if this is possible with multi-line-mode *or* with some patterndb wizardry. > > Has anyone addressed this? > > Thanks for any working-examples/guidance/sympathy (in roughly that order :-) > > Jim > > > > > ---- jrhendri@roadrunner.com <mailto:jrhendri@roadrunner.com> wrote: >> Hi, >> >> I am trying to parse data elements out of a variable number of log lines that all are associated by a single unique key. >> >> Specifically - they are Cisco IronPort email logs that have various "ID" fields (MID - message ID is the most common) >> >> >> Essentially I want to pull the MID out of the line marked marked: >> >> "Start MID (\d+) <other stuff>" >> >> and then process every line that matches that specific MID value as part of the message. >> >> Note: they all have this string included somewhere: >> >> "MID (\d+) " >> >> Up to a reasonable timeout - or ended by: >> >> "Message finished mid (\d+) done" with the matching ID. >> >> Is this possible with syslog-ng? (OSE or PE?) >> >> I thought I had seen something using patterndb but I cannot seem to find the reference >> >> Clearly there will be interleaved lines with *different* MIDs that need to be processed independently. >> >> Thanks in advance! >> Jim > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On 04/29/2014 12:20 PM, Chaman Chakalaka wrote:
I fixed the issue with udp being dropped at the system level by changing the linux core files but this time restarting the system, now I know they are missing somewhere between getting to the system and syslog-ng :(
I'm lost once again...
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- trysource s_network_udp { udp(so_rcvbuf(33554432) log_fetch_limit(20000) log_iw_size(1000000) ); };
This assumes that you have a large net.core.rmem_max net.core.rmem_max = 52428800 See how that goes.
participants (2)
-
Chaman Chakalaka
-
Evan Rempel