- syslog-ng - lists.balabit.hu

ZMQ transport, source and destination
by Laci Mészáros 04 May '14

04 May '14

Hy all, I am László Mészáros a student at Szechenyi Istvan University learning software engineering. I've made a proposal for Google Summer of Code to implement the zeromq transport, source and destination. The organization accepted it, so during this summer I will work on it. You can find the proposal here: https://github.com/balabit/syslog-ng/wiki/GSoC2014-Proposal:-ZMQ-transport,… I made a blog where you can follow the status, it can be found here: http://lmesz.tumblr.com/post/84705072064/introduction-aims I think one part it will be fun and on another it will be difficult, so I would like to ask all of you, if you have some use cases on your shelf about how you would/will use zmq for logging via network, please share it with us. You can send it to the list or to me personally it would be very useful during the implementation. Br, Laci

1 0

MongoDB destination TLS support
by György Demarcsek Ifj. 01 May '14

01 May '14

Hi All! My name is György Demarcsek and I'm a Computer Science BSc student at Budapest University of Technology and Economics. I will graduate in 2015, but planning to go for the MSc as well. I participate in the Google Summer of Code program this year, and my proposal "TLS Support for the MongoDB destination" has been accepted by the syslog-ng project. You may verify it right here: https://algernon.blogs.balabit.com/2014/04/gsoc2014-syslog-ng-accepted-proj… So this summer, I'll be coding TLS support for the MongoDB client ( https://github.com/algernon/libmongo-client) library used by syslog-ng ose. I'm really enthusiastic, I really want to do a great job. I will regularly report about the development process on my blog - created mostly for this - at http://demarcsek92.blogspot.hu/search/label/gsoc2014 so you can all follow my process and comment on anything. Suggestions, tips, critics are all welcome from anyone, especially developers and fellow GSoC participants contributing to syslog-ng ose. Thanks folks, happy hacking to all of you! :) Cheers, Gyuri

1 0

Pattern DB Parser "Default Values"
by David Hauck 01 May '14

01 May '14

Hi, I was wondering if there was a way to specify default values for pattern DB parsers that include a value, but where the parsed value is <null>[/empty]? In particular if I have something like the following: <pattern>test message; field1=@ESTRING:field1: @field2=@ESTRING:field2: @field3=@ESTRING:: @field4=@ESTRING:field4: @</pattern> I'd like to be able to do something like either, 1: <pattern>test message; field1=@ESTRING:field1<foo>: @field2=@ESTRING:field2<bar>: @field3=@ESTRING:: @field4=@ESTRING:field4<beef>: @</pattern> Or 2: <pattern>test message; field1=@ESTRING:field1: @field2=@ESTRING:field2: @field3=@ESTRING:: @field4=@ESTRING:field4: @</pattern> <values> <value name="field1.default">foo</value> <value name="field2.default">bar</value> <value name="field4.default">beef</value> Just curious... Thanks, -David

4 5

Re: [syslog-ng] Ubuntu Precise -ng filling out buffer, dropping messages
by Chaman Chakalaka 29 Apr '14

29 Apr '14

Thanks for responding! The exact version is of syslog-ng is 3.3.4. Here's something you may find of interest. 1K Messages root@badbox:/proc/net# nc -lku 514 > /tmp/testing123.txt root@goodbox:/proc/sys/net# loggen --inet --dgram --size 500 --rate 1000 --interval 30 badbox.cbf 514 average rate = 996.43 msg/sec, count=29893 <--------------- sent root@badbox:/proc/net# cat /tmp/testing123.txt | wc -l 24081 <------------ received/processed Lower the rate to 150 messages per second: root@goodbox:/proc/sys/net# loggen --inet --dgram --size 500 --rate 150 --interval 30 badbox.cbf 514 average rate = 149.03 msg/sec, count=4471 < -------------------- sent root@badbox:/proc/net# cat /tmp/testing123.txt | wc -l 4471 < -------------------------- received/processed At this point, given the test above, I don't know if this is a system issue or a syslog-ng issue. It seems to be system, but I really can't tell what. > From: syslog-ng-request(a)lists.balabit.hu > Subject: syslog-ng Digest, Vol 108, Issue 24 > To: syslog-ng(a)lists.balabit.hu > Date: Tue, 29 Apr 2014 13:25:01 +0200 > > Send syslog-ng mailing list submissions to > syslog-ng(a)lists.balabit.hu > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.balabit.hu/mailman/listinfo/syslog-ng > or, via email, send a message with subject or body 'help' to > syslog-ng-request(a)lists.balabit.hu > > You can reach the person managing the list at > syslog-ng-owner(a)lists.balabit.hu > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of syslog-ng digest..." > > > Today's Topics: > > 1. Re: Ubuntu Precise -ng filling out buffer, dropping messages > (Gergely Nagy) > 2. [Bug 279] Syslog-ng central loging server seg fault gentoo > (bugzilla(a)bugzilla.balabit.com) > 3. [Bug 279] Syslog-ng central loging server seg fault gentoo > (bugzilla(a)bugzilla.balabit.com) > 4. Re: Pattern DB Parser "Default Values" (Gergely Nagy) > 5. Re: syslog-ng does not start if destination host not found > (Gergely Nagy) > 6. [Bug 275] lib/filter/filter-in-list.c does not compile under > Solaris 10 (bugzilla(a)bugzilla.balabit.com) > 7. [Bug 279] Syslog-ng central loging server seg fault gentoo > (bugzilla(a)bugzilla.balabit.com) > 8. Re: Basic (?) multi line question (Jim Hendrick) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 29 Apr 2014 12:24:09 +0200 > From: Gergely Nagy <algernon(a)balabit.hu> > Subject: Re: [syslog-ng] Ubuntu Precise -ng filling out buffer, > dropping messages > To: syslog-ng(a)lists.balabit.hu > Message-ID: <87mwf4xwl2.fsf(a)balabit.hu> > Content-Type: text/plain > > Hi! > > Chaman Chakalaka <chebannedmeagain(a)hotmail.com> writes: > > > I'm trying to process ~800 UDP messages second, which I don't think is > > much. The current setup worked fine in Ubuntu 10.04 (Lucid) and > > syslog-ng 2.6 (I believe). I'm running into what I believe is receive > > buffer problems on Ubuntu Server 12.04 (Precise) w/ ng 3.XX > > First of all, what's the exact version of your syslog-ng? Precise has a > fairly old version, one that's... not exactly the best release. I'd > suggest you give a try to the packages at: > http://asylum.madhouse-project.org/projects/debian/ > > I'd suggest the syslog-ng 3.5 branch from there, and see if the problem > persists with an upgraded syslog-ng. If it persists, let us know, and > we'll help debug the issue further. > > -- > |8] > > > > ------------------------------ > > Message: 2 > Date: Tue, 29 Apr 2014 12:25:38 +0200 (CEST) > From: bugzilla(a)bugzilla.balabit.com > Subject: [syslog-ng] [Bug 279] Syslog-ng central loging server seg > fault gentoo > To: syslog-ng(a)lists.balabit.hu > Message-ID: <20140429102538.E2BAC39DC88(a)lists.balabit.hu> > Content-Type: text/plain; charset="UTF-8" > > https://bugzilla.balabit.com/show_bug.cgi?id=279 > > > Gergely Nagy <algernon(a)balabit.hu> changed: > > What |Removed |Added > ---------------------------------------------------------------------------- > CC| |algernon(a)balabit.hu > AssignedTo|bazsi(a)balabit.hu |algernon(a)balabit.hu > > > > > -- > Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email > ------- You are receiving this mail because: ------- > You are watching all bug changes. > > > ------------------------------ > > Message: 3 > Date: Tue, 29 Apr 2014 12:29:08 +0200 (CEST) > From: bugzilla(a)bugzilla.balabit.com > Subject: [syslog-ng] [Bug 279] Syslog-ng central loging server seg > fault gentoo > To: syslog-ng(a)lists.balabit.hu > Message-ID: <20140429102908.849C539DC78(a)lists.balabit.hu> > Content-Type: text/plain; charset="UTF-8" > > https://bugzilla.balabit.com/show_bug.cgi?id=279 > > > > > > --- Comment #1 from Gergely Nagy <algernon(a)balabit.hu> 2014-04-29 12:29:08 --- > Without debug symbols, the backtrace is fairly useless for debugging purposes, unfortunately. It would help tremendously, if you could reproduce the problem > with a non-stripped binary, so we see the functions in the backtrace. > > Meanwhile, can I ask what config you use on the host where the segmentation fault happened? Maybe we can figure something out from that... > > Thanks! > > > -- > Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email > ------- You are receiving this mail because: ------- > You are watching all bug changes. > > > ------------------------------ > > Message: 4 > Date: Tue, 29 Apr 2014 12:41:44 +0200 > From: Gergely Nagy <algernon(a)balabit.hu> > Subject: Re: [syslog-ng] Pattern DB Parser "Default Values" > To: syslog-ng(a)lists.balabit.hu > Cc: Balazs Scheidler <bazsi(a)balabit.com> > Message-ID: <87iopsxvrr.fsf(a)balabit.hu> > Content-Type: text/plain > > David Hauck <davidh(a)netacquire.com> writes: > > > I was wondering if there was a way to specify default values for > > pattern DB parsers that include a value, but where the parsed value is > > <null>[/empty]? > > > > In particular if I have something like the following: > > > > <pattern>test message; field1=@ESTRING:field1: @field2=@ESTRING:field2: @field3=@ESTRING:: @field4=@ESTRING:field4: @</pattern> > > > > I'd like to be able to do something like either, 1: > > > > <pattern>test message; field1=@ESTRING:field1<foo>: @field2=@ESTRING:field2<bar>: @field3=@ESTRING:: @field4=@ESTRING:field4<beef>: @</pattern> > > > > Or 2: > > > > <pattern>test message; field1=@ESTRING:field1: @field2=@ESTRING:field2: @field3=@ESTRING:: @field4=@ESTRING:field4: @</pattern> > > <values> > > <value name="field1.default">foo</value> > > <value name="field2.default">bar</value> > > <value name="field4.default">beef</value> > > > > Just curious... > > You can use ${field1:-foo} in templates, to set a default if none is > set. It doesn't work for empty fields, though, but that can be worked > around with an $(if $(length $field1) eq 0 "default" $field1) template, > possibly in a rewrite rule. > > Though, maybe ${field1:-foo} should work for empty values too, not just > unset ones (to mimic shell better, which does just that). I can make it > do so, if that'd be desired, would make it unnecessary to use the $(if) > hack. > > @Bazsi: What do you think? > > -- > |8] > > > > ------------------------------ > > Message: 5 > Date: Tue, 29 Apr 2014 12:59:56 +0200 > From: Gergely Nagy <algernon(a)balabit.hu> > Subject: Re: [syslog-ng] syslog-ng does not start if destination host > not found > To: syslog-ng(a)lists.balabit.hu > Message-ID: <87eh0gxuxf.fsf(a)balabit.hu> > Content-Type: text/plain > > "Bendler, Ehren" <ebendler(a)ciena.com> writes: > > [...] > > If this is the intended behavior, that's fine too. We can deploy our > > own patch to the afsocket module if it isn't going to be changed in a > > release. > > No, this is definitely not the intended behaviour. Some change between > 3.3.5 and 3.5.7 broke the fix, I'll go ahead and restore the intended > behaviour. Thanks for reporting the issue! > > Unfortunately, I can't help with the other issue at the moment, but I'll > try to revisit it later. > > -- > |8] > > > > ------------------------------ > > Message: 6 > Date: Tue, 29 Apr 2014 13:09:22 +0200 (CEST) > From: bugzilla(a)bugzilla.balabit.com > Subject: [syslog-ng] [Bug 275] lib/filter/filter-in-list.c does not > compile under Solaris 10 > To: syslog-ng(a)lists.balabit.hu > Message-ID: <20140429110922.7AD9939DC99(a)lists.balabit.hu> > Content-Type: text/plain; charset="UTF-8" > > https://bugzilla.balabit.com/show_bug.cgi?id=275 > > > > > > --- Comment #2 from Gergely Nagy <algernon(a)balabit.hu> 2014-04-29 13:09:22 --- > I think we can change the code to use find_cr_or_lf(), instead of using getline(), or reimplement something like getline() in terms of find_cr_or_lf() + fgets > (or mmap or something). That would solve the problem without having to add much to misc.c. I'll see what I can do. > > > -- > Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email > ------- You are receiving this mail because: ------- > You are watching all bug changes. > > > ------------------------------ > > Message: 7 > Date: Tue, 29 Apr 2014 13:21:01 +0200 (CEST) > From: bugzilla(a)bugzilla.balabit.com > Subject: [syslog-ng] [Bug 279] Syslog-ng central loging server seg > fault gentoo > To: syslog-ng(a)lists.balabit.hu > Message-ID: <20140429112101.BD6F339DCA1(a)lists.balabit.hu> > Content-Type: text/plain; charset="UTF-8" > > https://bugzilla.balabit.com/show_bug.cgi?id=279 > > > > > > --- Comment #2 from Martin <hlavacek(a)gmx.com> 2014-04-29 13:21:02 --- > I thought that I have recompiled syslog with debug symbols because I have added --enable-debug to my ebuild: > > syslog1 ~ # syslog-ng -V > syslog-ng 3.4.7 > Installer-Version: 3.4.7 > Revision: ssh+git://algernon@git.balabit/var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.4#detached_from_v3.4.3#999a7a6102d40da44b75a2acf78e54244164771f > Compile-Date: Apr 29 2014 13:06:40 > Available-Modules: > affile,afprog,afsocket-notls,afsocket-tls,afuser,basicfuncs,confgen,csvparser,dbparser,syslogformat,cryptofuncs,system-source,afamqp,afsocket > Enable-Debug: on > Enable-GProf: off > Enable-Memtrace: off > Enable-IPv6: on > Enable-Spoof-Source: off > Enable-TCP-Wrapper: on > Enable-Linux-Caps: off > Enable-Pcre: on > > You can see that opt "Enable-Debug:" is ON. It is not enought? If not can you please give me any advice how should I recompile this binary in proper way in > gentoo? > > Size of binary is: > syslog1 ~ # ls -lah /usr/sbin/syslog-ng > 16K -rwxr-xr-x 1 root root 15K Apr 29 13:18 /usr/sbin/syslog-ng* > > Configure options by emerge: > ./configure --prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share > --sysconfdir=/etc --localstatedir=/var/lib --libdir=/usr/lib64 --disable-silent-rules --disable-dependency-tracking --with-ivykis=internal > --with-libmongo-client=internal --sysconfdir=/etc/syslog-ng --localstatedir=/var/lib/syslog-ng --with-pidfile-dir=/var/run > --with-module-dir=/usr/lib64/syslog-ng --enable-debug --with-systemdsystemunitdir=/usr/lib/systemd/system --disable-systemd --disable-linux-caps > --disable-geoip --enable-ipv6 --disable-json --disable-mongodb --enable-pcre --disable-smtp --disable-spoof-source --disable-sql --enable-ssl > --enable-tcp-wrapper > > > -- > Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email > ------- You are receiving this mail because: ------- > You are watching all bug changes. > > > ------------------------------ > > Message: 8 > Date: Tue, 29 Apr 2014 07:24:58 -0400 > From: Jim Hendrick <jrhendri(a)roadrunner.com> > Subject: Re: [syslog-ng] Basic (?) multi line question > To: Syslog-ng users' and developers' mailing list > <syslog-ng(a)lists.balabit.hu> > Message-ID: <535F8C0A.5060104(a)roadrunner.com> > Content-Type: text/plain; charset="iso-8859-1" > > Thanks all for the thoughts - > > I will try to write up some of the patterns and correlations, starting > with the most simple. > > This would (I think) be a valuable addition to track different logs that > have some dynamic id as a key. > > (ultimately I am hoping to parse specific data out of these multi-line > beasties and be able to populate a database directly from syslog-ng) > > I will work on writing this up this week. > > Thanks again! > Jim > > > On 04/29/2014 04:53 AM, Tusa Viktor wrote: > > Hi! > > > > If you know the format of all the messages which possibly contains a > > MID, you can write patterns for them and then you can use correlation > > to extract information from these messages. But it only works with > > special conditions, I think it wouldn't work in your case. But it > > wouldn't be so hard to create such functionality in syslog-ng, so if > > you open a github issue in http://github.com/balabit/syslog-ng, some > > of us will try to make it work. > > > > Best Regards, > > Viktor > > > > > > On Tue, Apr 29, 2014 at 8:14 AM, C. L. Martinez <carlopmart(a)gmail.com > > <mailto:carlopmart@gmail.com>> wrote: > > > > Hi Jim, > > > > Some time ago, I have tried the same: correlate logs for Ironport > > devices. And my conclusion was: impossible. I loose a lot info and > > some correlated logs are wrong ... > > > > The only approach that maybe should work with opensource tools, IMO, > > is rsyslog+sec.pl <http://sec.pl>. But, as a Orangepeel says, > > logstash can be an > > option. > > > > Bye. > > > > On Mon, Apr 28, 2014 at 2:44 PM, <jrhendri(a)roadrunner.com > > <mailto:jrhendri@roadrunner.com>> wrote: > > > Hmmm - crickets :-) > > > > > > I have some examples like this: > > > <date> <host> <program>: Info: New SMTP ICID [0-9]{9} <rest of > > message> > > > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} > > <rest of message> > > > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} > > <rest of message> > > > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} > > <rest of message> > > > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} > > <rest of message> > > > <date> <host> <program>: Info: New SMTP DCID [0-9]{9} <rest of > > message> > > > <date> <host> <program>: Info: Message done DCID [0-9]{9} MID > > [0-9]{9} <rest of message> > > > <date> <host> <program>: Info: ICID [0-9]{9} close > > > > > > this is only an example to illustrate the different message > > elements that contain different kinds of IDs. > > > > > > The issue is there will be interleaving with *different* ICID > > (inbound connections from different SMTP servers) each sending > > multiple MIDs (message IDs) and also different DCID (destination > > connections *to* different mail relays). > > > > > > I have been looking at multi-line-mode(regexp) but that seems to > > imply all consecutive lines until the next regex match are assumed > > to be part of the same message. > > > > > > I hope I can do something where all matching ICIDs are treated > > as part of one line, that can be parsed separately. > > > > > > Not sure if this is possible with multi-line-mode *or* with some > > patterndb wizardry. > > > > > > Has anyone addressed this? > > > > > > Thanks for any working-examples/guidance/sympathy (in roughly > > that order :-) > > > > > > Jim > > > > > > > > > > > > > > > ---- jrhendri(a)roadrunner.com <mailto:jrhendri@roadrunner.com> wrote: > > >> Hi, > > >> > > >> I am trying to parse data elements out of a variable number > > of log lines that all are associated by a single unique key. > > >> > > >> Specifically - they are Cisco IronPort email logs that have > > various "ID" fields (MID - message ID is the most common) > > >> > > >> > > >> Essentially I want to pull the MID out of the line marked marked: > > >> > > >> "Start MID (\d+) <other stuff>" > > >> > > >> and then process every line that matches that specific MID > > value as part of the message. > > >> > > >> Note: they all have this string included somewhere: > > >> > > >> "MID (\d+) " > > >> > > >> Up to a reasonable timeout - or ended by: > > >> > > >> "Message finished mid (\d+) done" with the matching ID. > > >> > > >> Is this possible with syslog-ng? (OSE or PE?) > > >> > > >> I thought I had seen something using patterndb but I cannot > > seem to find the reference > > >> > > >> Clearly there will be interleaved lines with *different* MIDs > > that need to be processed independently. > > >> > > >> Thanks in advance! > > >> Jim > > > > > > > > ______________________________________________________________________________ > > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > > Documentation: > > http://www.balabit.com/support/documentation/?product=syslog-ng > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > > > ______________________________________________________________________________ > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > Documentation: > > http://www.balabit.com/support/documentation/?product=syslog-ng > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > > > > > > > > ______________________________________________________________________________ > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > >

2 2

Correct Usage of Multiple 'pattern' Databases
by David Hauck 29 Apr '14

29 Apr '14

Hello, I've only recently dug into some more intricate 'syslog-ng' configurations and had a question regarding 'log' construct blocks where multiple 'parser' references exist. I've been trying to do something like the following (testing with the supplied example pattern databases): log { filter(f_auth); parser("login"); parser("sshd"); parser("su"); parser("sudo"); log { filter(f_class_system); ... }; }; The problem I'm having is that extracted values from matched rules appear to be lost when the matched rule exists in a pattern db *other than the last referenced parser() db*. Specifically, if a rule is matched in the 'sshd' db above the following 'f_class_system' filter (which attempts to match '.classifier.class') *does not* match; however, if a rule is matched in the 'sudo' db above the 'f_class_system' filter *does* match. I'm sure this is perfectly explainable, but I can't find any documentation/Google references specifically outlining this behaviour. Given the above and in order to work around this I assume I would have to, either: 1) combine all of the rules into a single db file, or 2) break out each 'parser' reference into a separate embedded 'log' construct (not ideal since the filtering et mechanics in each would be identical and for maintenance reasons I'd like to consolidate these into a single 'log' construct). Both options are less than ideal. Is there a better way? Really appreciate any help you might be able to offer. Thanks, -David

9 17

syslog-ng does not start if destination host not found
by Bendler, Ehren 29 Apr '14

29 Apr '14

In syslog-ng 3.5.4.1, this seems to be a recurrence of an issue that has appeared several times before over the years, most recently said to be fixed/changed in v3.3.5 here (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=660897) We had previously been using syslog-ng 3.0.4, which would fail this test gracefully and retry the connection based on the time_reopen() parameter. We don't use DNS, but we generate a hard-coded /etc/hosts file after startup that associates internal names with a customer chosen subnet address so that we can collect logs without editing the syslog-ng.conf file while we are running. As in, we know that the primary controller will always be present with the name "CTXP", but we can't promise that it will have a particular address because the customer can change it, so we have to write it out to /etc/hosts at a point in the startup process *after* syslog-ng is started. * Starting syslog-ng Error resolving hostname; host='CTXP' Error initializing message pipeline; ..[FAIL(2)] If this is the intended behavior, that's fine too. We can deploy our own patch to the afsocket module if it isn't going to be changed in a release. Another thing I need help with (when I use a hard-coded IP in syslog-ng.conf for testing) is how to get this message to go away: * Starting syslog-ng WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='10', new_log_iw_size='100', min_log_fifo_size='12000' [OK] Here are the settings we have for these parameters: source remote_lms { syslog(ip(0.0.0.0) port(5149) max-connections(120) log_iw_size(1200) so_keepalive(yes)); }; With log_fifo_size(12000) set in the global options. The documentation says I did the math correctly here, with log_fifo_size = 10 * log_iw_size(), and log_iw_size() = 10 * max-connections()? This is the only remote source in our configuration. Thanks, Ehren Bendler

2 1

Ubuntu Precise -ng filling out buffer, dropping messages
by Chaman Chakalaka 29 Apr '14

29 Apr '14

Hi there, I'm trying to process ~800 UDP messages second, which I don't think is much. The current setup worked fine in Ubuntu 10.04 (Lucid) and syslog-ng 2.6 (I believe). I'm running into what I believe is receive buffer problems on Ubuntu Server 12.04 (Precise) w/ ng 3.XX Via tcpdump, I can attest all messages are getting to the server. I'm listening to port 514 UDP. From syslog-ng stats, I don't see any "dropped" statistics (always 0), but I know for a fact that when the message load is more than 150, I start seeing message loss. All messages received are sent to files on local disk after matching some filters. I have tested this with loggen and no filters -- more messages are sent than written to file. netstat -su shows increasing number of receive errors when going beyond 150 msgs per second. It also shows the receive queue hover in the 20000's. I have tried setting the so_rcvbuf to a relatively large value as described in the admin guide (2 097 152 bytes). The default net.core.rmem_max on ubuntu server 12.04 is 8388608. I have played with this number, up and down with no luck -- messages are still lost. I have spent hours reading documentation, doing packet captures, playing with values in -ng and the host, etc, but I'm at a loss. Any hints as to what I should look into? And more direct questions, Will so_rcvbuf declared on the src modify the system's rmem_max or do I have to do it on both? Why is it recommended in the admin guide to not modify the rmem_default? Thanks any advice would be helpful.

2 1

[Ask for help] How to resolve these two memory leak problem? thanks!
by Xufeng Zhang 23 Apr '14

23 Apr '14

Hello all, I still met below two memory leaks problem when using remote udp connection: ==11004== 20 bytes in 2 blocks are definitely lost in loss record 18 of 596 ==11004== at 0x4A05F58: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==11004== by 0x305927C6B1: strdup (strdup.c:43) ==11004== by 0x305E6529F2: _cfg_lexer_lex (cfg-lex.l:199) ==11004== by 0x305E6284C1: cfg_lexer_lex (cfg-lexer.c:759) ==11004== by 0x305E65BF76: rewrite_expr_parse (rewrite-expr-grammar.c:2959) ==11004== by 0x305E654732: T.99 (cfg-parser.h:83) ==11004== by 0x305E655294: main_parse (cfg-grammar.y:584) ==11004== by 0x305E625EE2: cfg_run_parser (cfg-parser.h:83) ==11004== by 0x305E625FF5: cfg_read_config (cfg.c:384) ==11004== by 0x305E642A6E: main_loop_init (mainloop.c:680) ==11004== by 0x401774: main (main.c:246) _cfg_lexer_lex (cfg-lex.l:199) is: yylval->cptr = strdup(yyextra->string_buffer->str); ==11004== 40,960 bytes in 40 blocks are definitely lost in loss record 596 of 596 ==11004== at 0x4A05F58: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==11004== by 0x305DA478F4: g_malloc (in /lib64/libglib-2.0.so.0.2600.0) ==11004== by 0x305E641778: log_writer_flush (logwriter.c:950) ==11004== by 0x305E64185C: log_writer_work_perform (logwriter.c:130) ==11004== by 0x305E6418C7: log_writer_io_flush_output (logwriter.c:210) ==11004== by 0x305E6605E6: iv_fd_poll_and_run (iv_fd.c:167) ==11004== by 0x305E660D93: iv_main (iv_main_posix.c:117) ==11004== by 0x305E641D86: main_loop_run (mainloop.c:737) ==11004== by 0x4017A0: main (main.c:267) log_writer_flush (logwriter.c:950) is: self->line_buffer->str = g_malloc(self->line_buffer->allocated_len); These were found in syslog-ng-3.4.2, and I don't think syslog-ng upstream has already fixed these problem, Does anybody know how to fix it? Thanks in advance! Xufeng

1 1

Millisecond Resolution Timestamps
by David Hauck 22 Apr '14

22 Apr '14

Hello, I'm using the following global options in order to format messages with millisecond resolution: ts_format(iso); frac_digits(3); Although the initial (syslog-ng starting) message appears to include sub-second resolution subsequent messages do not: 20140415 10:19:23.590 notice syslog(syslog-ng):syslog-ng starting up; version='3.5.4.1' ... 20140415 10:19:33.000 notice authpriv(su):FAILED su for test by root ... 20140415 10:20:11.000 notice user(root):test This includes messages originating from any number of sources (including all processes that log via syslog()), *except* messages originating from the kernel (these always seem to have sub-second resolution). Does anyone have any ideas what might be going on here? Thanks, -David PS: the timestamp formatting above is done via a simple template. Regardless of this (re-)formatting nominal iso messages also exhibit this limitation. For e.g.,: 2014-04-14T15:23:48.000-07:00 host99738728 nasysconfd: exit code 0 for /netacquire/bin/sysconf/osinfo read

3 4

CSV-Parser Conditions
by Justin B 20 Apr '14

20 Apr '14

Need information on this.I am using the CSV-PARSER and I want to put a condition that if a column is empty do not apply the CSV-PARSER but write the line. Primarily i get the application logs and i dont want to apply the CSV-Parser on those log lines where the whole error stack is there. Can someone help on this?? -- Kaladhar

2 5