- syslog-ng - lists.balabit.hu

MongoDB destination TLS support
by György Demarcsek Ifj. 01 May '14

01 May '14

Hi All! My name is György Demarcsek and I'm a Computer Science BSc student at Budapest University of Technology and Economics. I will graduate in 2015, but planning to go for the MSc as well. I participate in the Google Summer of Code program this year, and my proposal "TLS Support for the MongoDB destination" has been accepted by the syslog-ng project. You may verify it right here: https://algernon.blogs.balabit.com/2014/04/gsoc2014-syslog-ng-accepted-proj… So this summer, I'll be coding TLS support for the MongoDB client ( https://github.com/algernon/libmongo-client) library used by syslog-ng ose. I'm really enthusiastic, I really want to do a great job. I will regularly report about the development process on my blog - created mostly for this - at http://demarcsek92.blogspot.hu/search/label/gsoc2014 so you can all follow my process and comment on anything. Suggestions, tips, critics are all welcome from anyone, especially developers and fellow GSoC participants contributing to syslog-ng ose. Thanks folks, happy hacking to all of you! :) Cheers, Gyuri

1 0

Pattern DB Parser "Default Values"
by David Hauck 01 May '14

01 May '14

Hi, I was wondering if there was a way to specify default values for pattern DB parsers that include a value, but where the parsed value is <null>[/empty]? In particular if I have something like the following: <pattern>test message; field1=@ESTRING:field1: @field2=@ESTRING:field2: @field3=@ESTRING:: @field4=@ESTRING:field4: @</pattern> I'd like to be able to do something like either, 1: <pattern>test message; field1=@ESTRING:field1<foo>: @field2=@ESTRING:field2<bar>: @field3=@ESTRING:: @field4=@ESTRING:field4<beef>: @</pattern> Or 2: <pattern>test message; field1=@ESTRING:field1: @field2=@ESTRING:field2: @field3=@ESTRING:: @field4=@ESTRING:field4: @</pattern> <values> <value name="field1.default">foo</value> <value name="field2.default">bar</value> <value name="field4.default">beef</value> Just curious... Thanks, -David

4 5

Re: [syslog-ng] Ubuntu Precise -ng filling out buffer, dropping messages
by Chaman Chakalaka 29 Apr '14

29 Apr '14

Thanks for responding! The exact version is of syslog-ng is 3.3.4. Here's something you may find of interest. 1K Messages root@badbox:/proc/net# nc -lku 514 > /tmp/testing123.txt root@goodbox:/proc/sys/net# loggen --inet --dgram --size 500 --rate 1000 --interval 30 badbox.cbf 514 average rate = 996.43 msg/sec, count=29893 <--------------- sent root@badbox:/proc/net# cat /tmp/testing123.txt | wc -l 24081 <------------ received/processed Lower the rate to 150 messages per second: root@goodbox:/proc/sys/net# loggen --inet --dgram --size 500 --rate 150 --interval 30 badbox.cbf 514 average rate = 149.03 msg/sec, count=4471 < -------------------- sent root@badbox:/proc/net# cat /tmp/testing123.txt | wc -l 4471 < -------------------------- received/processed At this point, given the test above, I don't know if this is a system issue or a syslog-ng issue. It seems to be system, but I really can't tell what. > From: syslog-ng-request(a)lists.balabit.hu > Subject: syslog-ng Digest, Vol 108, Issue 24 > To: syslog-ng(a)lists.balabit.hu > Date: Tue, 29 Apr 2014 13:25:01 +0200 > > Send syslog-ng mailing list submissions to > syslog-ng(a)lists.balabit.hu > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.balabit.hu/mailman/listinfo/syslog-ng > or, via email, send a message with subject or body 'help' to > syslog-ng-request(a)lists.balabit.hu > > You can reach the person managing the list at > syslog-ng-owner(a)lists.balabit.hu > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of syslog-ng digest..." > > > Today's Topics: > > 1. Re: Ubuntu Precise -ng filling out buffer, dropping messages > (Gergely Nagy) > 2. [Bug 279] Syslog-ng central loging server seg fault gentoo > (bugzilla(a)bugzilla.balabit.com) > 3. [Bug 279] Syslog-ng central loging server seg fault gentoo > (bugzilla(a)bugzilla.balabit.com) > 4. Re: Pattern DB Parser "Default Values" (Gergely Nagy) > 5. Re: syslog-ng does not start if destination host not found > (Gergely Nagy) > 6. [Bug 275] lib/filter/filter-in-list.c does not compile under > Solaris 10 (bugzilla(a)bugzilla.balabit.com) > 7. [Bug 279] Syslog-ng central loging server seg fault gentoo > (bugzilla(a)bugzilla.balabit.com) > 8. Re: Basic (?) multi line question (Jim Hendrick) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 29 Apr 2014 12:24:09 +0200 > From: Gergely Nagy <algernon(a)balabit.hu> > Subject: Re: [syslog-ng] Ubuntu Precise -ng filling out buffer, > dropping messages > To: syslog-ng(a)lists.balabit.hu > Message-ID: <87mwf4xwl2.fsf(a)balabit.hu> > Content-Type: text/plain > > Hi! > > Chaman Chakalaka <chebannedmeagain(a)hotmail.com> writes: > > > I'm trying to process ~800 UDP messages second, which I don't think is > > much. The current setup worked fine in Ubuntu 10.04 (Lucid) and > > syslog-ng 2.6 (I believe). I'm running into what I believe is receive > > buffer problems on Ubuntu Server 12.04 (Precise) w/ ng 3.XX > > First of all, what's the exact version of your syslog-ng? Precise has a > fairly old version, one that's... not exactly the best release. I'd > suggest you give a try to the packages at: > http://asylum.madhouse-project.org/projects/debian/ > > I'd suggest the syslog-ng 3.5 branch from there, and see if the problem > persists with an upgraded syslog-ng. If it persists, let us know, and > we'll help debug the issue further. > > -- > |8] > > > > ------------------------------ > > Message: 2 > Date: Tue, 29 Apr 2014 12:25:38 +0200 (CEST) > From: bugzilla(a)bugzilla.balabit.com > Subject: [syslog-ng] [Bug 279] Syslog-ng central loging server seg > fault gentoo > To: syslog-ng(a)lists.balabit.hu > Message-ID: <20140429102538.E2BAC39DC88(a)lists.balabit.hu> > Content-Type: text/plain; charset="UTF-8" > > https://bugzilla.balabit.com/show_bug.cgi?id=279 > > > Gergely Nagy <algernon(a)balabit.hu> changed: > > What |Removed |Added > ---------------------------------------------------------------------------- > CC| |algernon(a)balabit.hu > AssignedTo|bazsi(a)balabit.hu |algernon(a)balabit.hu > > > > > -- > Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email > ------- You are receiving this mail because: ------- > You are watching all bug changes. > > > ------------------------------ > > Message: 3 > Date: Tue, 29 Apr 2014 12:29:08 +0200 (CEST) > From: bugzilla(a)bugzilla.balabit.com > Subject: [syslog-ng] [Bug 279] Syslog-ng central loging server seg > fault gentoo > To: syslog-ng(a)lists.balabit.hu > Message-ID: <20140429102908.849C539DC78(a)lists.balabit.hu> > Content-Type: text/plain; charset="UTF-8" > > https://bugzilla.balabit.com/show_bug.cgi?id=279 > > > > > > --- Comment #1 from Gergely Nagy <algernon(a)balabit.hu> 2014-04-29 12:29:08 --- > Without debug symbols, the backtrace is fairly useless for debugging purposes, unfortunately. It would help tremendously, if you could reproduce the problem > with a non-stripped binary, so we see the functions in the backtrace. > > Meanwhile, can I ask what config you use on the host where the segmentation fault happened? Maybe we can figure something out from that... > > Thanks! > > > -- > Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email > ------- You are receiving this mail because: ------- > You are watching all bug changes. > > > ------------------------------ > > Message: 4 > Date: Tue, 29 Apr 2014 12:41:44 +0200 > From: Gergely Nagy <algernon(a)balabit.hu> > Subject: Re: [syslog-ng] Pattern DB Parser "Default Values" > To: syslog-ng(a)lists.balabit.hu > Cc: Balazs Scheidler <bazsi(a)balabit.com> > Message-ID: <87iopsxvrr.fsf(a)balabit.hu> > Content-Type: text/plain > > David Hauck <davidh(a)netacquire.com> writes: > > > I was wondering if there was a way to specify default values for > > pattern DB parsers that include a value, but where the parsed value is > > <null>[/empty]? > > > > In particular if I have something like the following: > > > > <pattern>test message; field1=@ESTRING:field1: @field2=@ESTRING:field2: @field3=@ESTRING:: @field4=@ESTRING:field4: @</pattern> > > > > I'd like to be able to do something like either, 1: > > > > <pattern>test message; field1=@ESTRING:field1<foo>: @field2=@ESTRING:field2<bar>: @field3=@ESTRING:: @field4=@ESTRING:field4<beef>: @</pattern> > > > > Or 2: > > > > <pattern>test message; field1=@ESTRING:field1: @field2=@ESTRING:field2: @field3=@ESTRING:: @field4=@ESTRING:field4: @</pattern> > > <values> > > <value name="field1.default">foo</value> > > <value name="field2.default">bar</value> > > <value name="field4.default">beef</value> > > > > Just curious... > > You can use ${field1:-foo} in templates, to set a default if none is > set. It doesn't work for empty fields, though, but that can be worked > around with an $(if $(length $field1) eq 0 "default" $field1) template, > possibly in a rewrite rule. > > Though, maybe ${field1:-foo} should work for empty values too, not just > unset ones (to mimic shell better, which does just that). I can make it > do so, if that'd be desired, would make it unnecessary to use the $(if) > hack. > > @Bazsi: What do you think? > > -- > |8] > > > > ------------------------------ > > Message: 5 > Date: Tue, 29 Apr 2014 12:59:56 +0200 > From: Gergely Nagy <algernon(a)balabit.hu> > Subject: Re: [syslog-ng] syslog-ng does not start if destination host > not found > To: syslog-ng(a)lists.balabit.hu > Message-ID: <87eh0gxuxf.fsf(a)balabit.hu> > Content-Type: text/plain > > "Bendler, Ehren" <ebendler(a)ciena.com> writes: > > [...] > > If this is the intended behavior, that's fine too. We can deploy our > > own patch to the afsocket module if it isn't going to be changed in a > > release. > > No, this is definitely not the intended behaviour. Some change between > 3.3.5 and 3.5.7 broke the fix, I'll go ahead and restore the intended > behaviour. Thanks for reporting the issue! > > Unfortunately, I can't help with the other issue at the moment, but I'll > try to revisit it later. > > -- > |8] > > > > ------------------------------ > > Message: 6 > Date: Tue, 29 Apr 2014 13:09:22 +0200 (CEST) > From: bugzilla(a)bugzilla.balabit.com > Subject: [syslog-ng] [Bug 275] lib/filter/filter-in-list.c does not > compile under Solaris 10 > To: syslog-ng(a)lists.balabit.hu > Message-ID: <20140429110922.7AD9939DC99(a)lists.balabit.hu> > Content-Type: text/plain; charset="UTF-8" > > https://bugzilla.balabit.com/show_bug.cgi?id=275 > > > > > > --- Comment #2 from Gergely Nagy <algernon(a)balabit.hu> 2014-04-29 13:09:22 --- > I think we can change the code to use find_cr_or_lf(), instead of using getline(), or reimplement something like getline() in terms of find_cr_or_lf() + fgets > (or mmap or something). That would solve the problem without having to add much to misc.c. I'll see what I can do. > > > -- > Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email > ------- You are receiving this mail because: ------- > You are watching all bug changes. > > > ------------------------------ > > Message: 7 > Date: Tue, 29 Apr 2014 13:21:01 +0200 (CEST) > From: bugzilla(a)bugzilla.balabit.com > Subject: [syslog-ng] [Bug 279] Syslog-ng central loging server seg > fault gentoo > To: syslog-ng(a)lists.balabit.hu > Message-ID: <20140429112101.BD6F339DCA1(a)lists.balabit.hu> > Content-Type: text/plain; charset="UTF-8" > > https://bugzilla.balabit.com/show_bug.cgi?id=279 > > > > > > --- Comment #2 from Martin <hlavacek(a)gmx.com> 2014-04-29 13:21:02 --- > I thought that I have recompiled syslog with debug symbols because I have added --enable-debug to my ebuild: > > syslog1 ~ # syslog-ng -V > syslog-ng 3.4.7 > Installer-Version: 3.4.7 > Revision: ssh+git://algernon@git.balabit/var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.4#detached_from_v3.4.3#999a7a6102d40da44b75a2acf78e54244164771f > Compile-Date: Apr 29 2014 13:06:40 > Available-Modules: > affile,afprog,afsocket-notls,afsocket-tls,afuser,basicfuncs,confgen,csvparser,dbparser,syslogformat,cryptofuncs,system-source,afamqp,afsocket > Enable-Debug: on > Enable-GProf: off > Enable-Memtrace: off > Enable-IPv6: on > Enable-Spoof-Source: off > Enable-TCP-Wrapper: on > Enable-Linux-Caps: off > Enable-Pcre: on > > You can see that opt "Enable-Debug:" is ON. It is not enought? If not can you please give me any advice how should I recompile this binary in proper way in > gentoo? > > Size of binary is: > syslog1 ~ # ls -lah /usr/sbin/syslog-ng > 16K -rwxr-xr-x 1 root root 15K Apr 29 13:18 /usr/sbin/syslog-ng* > > Configure options by emerge: > ./configure --prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share > --sysconfdir=/etc --localstatedir=/var/lib --libdir=/usr/lib64 --disable-silent-rules --disable-dependency-tracking --with-ivykis=internal > --with-libmongo-client=internal --sysconfdir=/etc/syslog-ng --localstatedir=/var/lib/syslog-ng --with-pidfile-dir=/var/run > --with-module-dir=/usr/lib64/syslog-ng --enable-debug --with-systemdsystemunitdir=/usr/lib/systemd/system --disable-systemd --disable-linux-caps > --disable-geoip --enable-ipv6 --disable-json --disable-mongodb --enable-pcre --disable-smtp --disable-spoof-source --disable-sql --enable-ssl > --enable-tcp-wrapper > > > -- > Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email > ------- You are receiving this mail because: ------- > You are watching all bug changes. > > > ------------------------------ > > Message: 8 > Date: Tue, 29 Apr 2014 07:24:58 -0400 > From: Jim Hendrick <jrhendri(a)roadrunner.com> > Subject: Re: [syslog-ng] Basic (?) multi line question > To: Syslog-ng users' and developers' mailing list > <syslog-ng(a)lists.balabit.hu> > Message-ID: <535F8C0A.5060104(a)roadrunner.com> > Content-Type: text/plain; charset="iso-8859-1" > > Thanks all for the thoughts - > > I will try to write up some of the patterns and correlations, starting > with the most simple. > > This would (I think) be a valuable addition to track different logs that > have some dynamic id as a key. > > (ultimately I am hoping to parse specific data out of these multi-line > beasties and be able to populate a database directly from syslog-ng) > > I will work on writing this up this week. > > Thanks again! > Jim > > > On 04/29/2014 04:53 AM, Tusa Viktor wrote: > > Hi! > > > > If you know the format of all the messages which possibly contains a > > MID, you can write patterns for them and then you can use correlation > > to extract information from these messages. But it only works with > > special conditions, I think it wouldn't work in your case. But it > > wouldn't be so hard to create such functionality in syslog-ng, so if > > you open a github issue in http://github.com/balabit/syslog-ng, some > > of us will try to make it work. > > > > Best Regards, > > Viktor > > > > > > On Tue, Apr 29, 2014 at 8:14 AM, C. L. Martinez <carlopmart(a)gmail.com > > <mailto:carlopmart@gmail.com>> wrote: > > > > Hi Jim, > > > > Some time ago, I have tried the same: correlate logs for Ironport > > devices. And my conclusion was: impossible. I loose a lot info and > > some correlated logs are wrong ... > > > > The only approach that maybe should work with opensource tools, IMO, > > is rsyslog+sec.pl <http://sec.pl>. But, as a Orangepeel says, > > logstash can be an > > option. > > > > Bye. > > > > On Mon, Apr 28, 2014 at 2:44 PM, <jrhendri(a)roadrunner.com > > <mailto:jrhendri@roadrunner.com>> wrote: > > > Hmmm - crickets :-) > > > > > > I have some examples like this: > > > <date> <host> <program>: Info: New SMTP ICID [0-9]{9} <rest of > > message> > > > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} > > <rest of message> > > > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} > > <rest of message> > > > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} > > <rest of message> > > > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} > > <rest of message> > > > <date> <host> <program>: Info: New SMTP DCID [0-9]{9} <rest of > > message> > > > <date> <host> <program>: Info: Message done DCID [0-9]{9} MID > > [0-9]{9} <rest of message> > > > <date> <host> <program>: Info: ICID [0-9]{9} close > > > > > > this is only an example to illustrate the different message > > elements that contain different kinds of IDs. > > > > > > The issue is there will be interleaving with *different* ICID > > (inbound connections from different SMTP servers) each sending > > multiple MIDs (message IDs) and also different DCID (destination > > connections *to* different mail relays). > > > > > > I have been looking at multi-line-mode(regexp) but that seems to > > imply all consecutive lines until the next regex match are assumed > > to be part of the same message. > > > > > > I hope I can do something where all matching ICIDs are treated > > as part of one line, that can be parsed separately. > > > > > > Not sure if this is possible with multi-line-mode *or* with some > > patterndb wizardry. > > > > > > Has anyone addressed this? > > > > > > Thanks for any working-examples/guidance/sympathy (in roughly > > that order :-) > > > > > > Jim > > > > > > > > > > > > > > > ---- jrhendri(a)roadrunner.com <mailto:jrhendri@roadrunner.com> wrote: > > >> Hi, > > >> > > >> I am trying to parse data elements out of a variable number > > of log lines that all are associated by a single unique key. > > >> > > >> Specifically - they are Cisco IronPort email logs that have > > various "ID" fields (MID - message ID is the most common) > > >> > > >> > > >> Essentially I want to pull the MID out of the line marked marked: > > >> > > >> "Start MID (\d+) <other stuff>" > > >> > > >> and then process every line that matches that specific MID > > value as part of the message. > > >> > > >> Note: they all have this string included somewhere: > > >> > > >> "MID (\d+) " > > >> > > >> Up to a reasonable timeout - or ended by: > > >> > > >> "Message finished mid (\d+) done" with the matching ID. > > >> > > >> Is this possible with syslog-ng? (OSE or PE?) > > >> > > >> I thought I had seen something using patterndb but I cannot > > seem to find the reference > > >> > > >> Clearly there will be interleaved lines with *different* MIDs > > that need to be processed independently. > > >> > > >> Thanks in advance! > > >> Jim > > > > > > > > ______________________________________________________________________________ > > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > > Documentation: > > http://www.balabit.com/support/documentation/?product=syslog-ng > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > > > ______________________________________________________________________________ > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > Documentation: > > http://www.balabit.com/support/documentation/?product=syslog-ng > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > > > > > > > > ______________________________________________________________________________ > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > >

2 2

Correct Usage of Multiple 'pattern' Databases
by David Hauck 29 Apr '14

29 Apr '14

Hello, I've only recently dug into some more intricate 'syslog-ng' configurations and had a question regarding 'log' construct blocks where multiple 'parser' references exist. I've been trying to do something like the following (testing with the supplied example pattern databases): log { filter(f_auth); parser("login"); parser("sshd"); parser("su"); parser("sudo"); log { filter(f_class_system); ... }; }; The problem I'm having is that extracted values from matched rules appear to be lost when the matched rule exists in a pattern db *other than the last referenced parser() db*. Specifically, if a rule is matched in the 'sshd' db above the following 'f_class_system' filter (which attempts to match '.classifier.class') *does not* match; however, if a rule is matched in the 'sudo' db above the 'f_class_system' filter *does* match. I'm sure this is perfectly explainable, but I can't find any documentation/Google references specifically outlining this behaviour. Given the above and in order to work around this I assume I would have to, either: 1) combine all of the rules into a single db file, or 2) break out each 'parser' reference into a separate embedded 'log' construct (not ideal since the filtering et mechanics in each would be identical and for maintenance reasons I'd like to consolidate these into a single 'log' construct). Both options are less than ideal. Is there a better way? Really appreciate any help you might be able to offer. Thanks, -David

9 17

syslog-ng does not start if destination host not found
by Bendler, Ehren 29 Apr '14

29 Apr '14

In syslog-ng 3.5.4.1, this seems to be a recurrence of an issue that has appeared several times before over the years, most recently said to be fixed/changed in v3.3.5 here (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=660897) We had previously been using syslog-ng 3.0.4, which would fail this test gracefully and retry the connection based on the time_reopen() parameter. We don't use DNS, but we generate a hard-coded /etc/hosts file after startup that associates internal names with a customer chosen subnet address so that we can collect logs without editing the syslog-ng.conf file while we are running. As in, we know that the primary controller will always be present with the name "CTXP", but we can't promise that it will have a particular address because the customer can change it, so we have to write it out to /etc/hosts at a point in the startup process *after* syslog-ng is started. * Starting syslog-ng Error resolving hostname; host='CTXP' Error initializing message pipeline; ..[FAIL(2)] If this is the intended behavior, that's fine too. We can deploy our own patch to the afsocket module if it isn't going to be changed in a release. Another thing I need help with (when I use a hard-coded IP in syslog-ng.conf for testing) is how to get this message to go away: * Starting syslog-ng WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='10', new_log_iw_size='100', min_log_fifo_size='12000' [OK] Here are the settings we have for these parameters: source remote_lms { syslog(ip(0.0.0.0) port(5149) max-connections(120) log_iw_size(1200) so_keepalive(yes)); }; With log_fifo_size(12000) set in the global options. The documentation says I did the math correctly here, with log_fifo_size = 10 * log_iw_size(), and log_iw_size() = 10 * max-connections()? This is the only remote source in our configuration. Thanks, Ehren Bendler

2 1

Ubuntu Precise -ng filling out buffer, dropping messages
by Chaman Chakalaka 29 Apr '14

29 Apr '14

Hi there, I'm trying to process ~800 UDP messages second, which I don't think is much. The current setup worked fine in Ubuntu 10.04 (Lucid) and syslog-ng 2.6 (I believe). I'm running into what I believe is receive buffer problems on Ubuntu Server 12.04 (Precise) w/ ng 3.XX Via tcpdump, I can attest all messages are getting to the server. I'm listening to port 514 UDP. From syslog-ng stats, I don't see any "dropped" statistics (always 0), but I know for a fact that when the message load is more than 150, I start seeing message loss. All messages received are sent to files on local disk after matching some filters. I have tested this with loggen and no filters -- more messages are sent than written to file. netstat -su shows increasing number of receive errors when going beyond 150 msgs per second. It also shows the receive queue hover in the 20000's. I have tried setting the so_rcvbuf to a relatively large value as described in the admin guide (2 097 152 bytes). The default net.core.rmem_max on ubuntu server 12.04 is 8388608. I have played with this number, up and down with no luck -- messages are still lost. I have spent hours reading documentation, doing packet captures, playing with values in -ng and the host, etc, but I'm at a loss. Any hints as to what I should look into? And more direct questions, Will so_rcvbuf declared on the src modify the system's rmem_max or do I have to do it on both? Why is it recommended in the admin guide to not modify the rmem_default? Thanks any advice would be helpful.

2 1

[Ask for help] How to resolve these two memory leak problem? thanks!
by Xufeng Zhang 23 Apr '14

23 Apr '14

Hello all, I still met below two memory leaks problem when using remote udp connection: ==11004== 20 bytes in 2 blocks are definitely lost in loss record 18 of 596 ==11004== at 0x4A05F58: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==11004== by 0x305927C6B1: strdup (strdup.c:43) ==11004== by 0x305E6529F2: _cfg_lexer_lex (cfg-lex.l:199) ==11004== by 0x305E6284C1: cfg_lexer_lex (cfg-lexer.c:759) ==11004== by 0x305E65BF76: rewrite_expr_parse (rewrite-expr-grammar.c:2959) ==11004== by 0x305E654732: T.99 (cfg-parser.h:83) ==11004== by 0x305E655294: main_parse (cfg-grammar.y:584) ==11004== by 0x305E625EE2: cfg_run_parser (cfg-parser.h:83) ==11004== by 0x305E625FF5: cfg_read_config (cfg.c:384) ==11004== by 0x305E642A6E: main_loop_init (mainloop.c:680) ==11004== by 0x401774: main (main.c:246) _cfg_lexer_lex (cfg-lex.l:199) is: yylval->cptr = strdup(yyextra->string_buffer->str); ==11004== 40,960 bytes in 40 blocks are definitely lost in loss record 596 of 596 ==11004== at 0x4A05F58: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==11004== by 0x305DA478F4: g_malloc (in /lib64/libglib-2.0.so.0.2600.0) ==11004== by 0x305E641778: log_writer_flush (logwriter.c:950) ==11004== by 0x305E64185C: log_writer_work_perform (logwriter.c:130) ==11004== by 0x305E6418C7: log_writer_io_flush_output (logwriter.c:210) ==11004== by 0x305E6605E6: iv_fd_poll_and_run (iv_fd.c:167) ==11004== by 0x305E660D93: iv_main (iv_main_posix.c:117) ==11004== by 0x305E641D86: main_loop_run (mainloop.c:737) ==11004== by 0x4017A0: main (main.c:267) log_writer_flush (logwriter.c:950) is: self->line_buffer->str = g_malloc(self->line_buffer->allocated_len); These were found in syslog-ng-3.4.2, and I don't think syslog-ng upstream has already fixed these problem, Does anybody know how to fix it? Thanks in advance! Xufeng

1 1

Millisecond Resolution Timestamps
by David Hauck 22 Apr '14

22 Apr '14

Hello, I'm using the following global options in order to format messages with millisecond resolution: ts_format(iso); frac_digits(3); Although the initial (syslog-ng starting) message appears to include sub-second resolution subsequent messages do not: 20140415 10:19:23.590 notice syslog(syslog-ng):syslog-ng starting up; version='3.5.4.1' ... 20140415 10:19:33.000 notice authpriv(su):FAILED su for test by root ... 20140415 10:20:11.000 notice user(root):test This includes messages originating from any number of sources (including all processes that log via syslog()), *except* messages originating from the kernel (these always seem to have sub-second resolution). Does anyone have any ideas what might be going on here? Thanks, -David PS: the timestamp formatting above is done via a simple template. Regardless of this (re-)formatting nominal iso messages also exhibit this limitation. For e.g.,: 2014-04-14T15:23:48.000-07:00 host99738728 nasysconfd: exit code 0 for /netacquire/bin/sysconf/osinfo read

3 4

CSV-Parser Conditions
by Justin B 20 Apr '14

20 Apr '14

Need information on this.I am using the CSV-PARSER and I want to put a condition that if a column is empty do not apply the CSV-PARSER but write the line. Primarily i get the application logs and i dont want to apply the CSV-Parser on those log lines where the whole error stack is there. Can someone help on this?? -- Kaladhar

2 5

pdbtool 'patternize'
by David Hauck 18 Apr '14

18 Apr '14

Hello, Does anyone have an explanation for why a "pdbtool patternize" generated pattern db indicates it is version '3'? I'm running the latest version of syslog-ng (3.5.4.1) so I was expecting that this would produce a version '4' pattern db. Easy enough to change in the generated XML, just wondering why the latest generator wouldn't create the latest version. Also, what is the nominal format for the log messages that the 'patternize' command is able to process (i.e., would this be logs that contain the nominally formatted syslog-ng output - e.g., via the default template: template("$ISODATE $HOST $MSGHDR$MSG\n");). I've seen some output that appears to suggest there's some nominal decoding of the input log messages. Thanks, -David

5 13

syslog-ng uses 100% cpu
by Yakup Kaya 17 Apr '14

17 Apr '14

Hello everyone, My problem is syslog-ng is using 100% CPU. When I trace the process for about 1 minute I get output as follows. As you can see epoll_wait and clock_gettime system calls are causing the problem. My syslog-ng version is 3.3.4. In the older versions I did not have such problems (e.g. 2.0.9). I would like to ask what exactly these system calls are used for and is there a configuration option or parameter to disable these calls or tune them? *top command output;* top - 15:28:46 up 4 days, 16:44, 7 users, load average: 1.33, 1.35, 1.33 Tasks: 245 total, 3 running, 240 sleeping, 2 stopped, 0 zombie Cpu(s): 9.2%us, 16.5%sy, 0.0%ni, 72.4%id, 1.9%wa, 0.1%hi, 0.0%si, 0.0%st Mem: 4029172k total, 3958124k used, 71048k free, 322556k buffers Swap: 4192928k total, 479076k used, 3713852k free, 1613548k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 10192 root 20 0 8304 3908 2452 R 99.4 0.1 850:33.93 syslog-ng *strace command output:* strace -f -c -p `pgrep -f '/sbin/syslog-ng'` Process 10192 attached - interrupt to quit % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 54.37 0.073938 0 5740561 epoll_wait 45.54 0.061930 0 5740562 clock_gettime 0.03 0.000035 0 1884 fcntl64 0.01 0.000020 0 1488 370 read 0.01 0.000018 0 408 writev 0.01 0.000015 0 2166 epoll_ctl 0.01 0.000008 0 172 78 accept 0.01 0.000007 0 797 write 0.01 0.000007 0 848 182 setsockopt 0.00 0.000006 0 489 gettimeofday 0.00 0.000006 0 408 _llseek 0.00 0.000000 0 92 close 0.00 0.000000 0 190 alarm 0.00 0.000000 0 15 stat64 ------ ----------- ----------- --------- --------- ---------------- 100.00 0.135990 11490080 630 total -- Yakup KAYA (B.Sc., CCNA) K?demli Sistem Destek Uzman?/Senior System Support Specialist Labris Teknoloji A.S,. Silikon Blok 1 NK 24 ODTÜ-Teknokent / Ankara, TURKEY Tel: +90 312 210 11 13 Fax: +90 312 210 14 92 yakup.kaya(a)labrisnetworks.com yakup.kaya(a)labris.eu www.labrisnetworks.com

2 2

[Bug 261] New: DOCBUG: --rekey
by bugzilla＠bugzilla.balabit.com 15 Apr '14

15 Apr '14

https://bugzilla.balabit.com/show_bug.cgi?id=261 Summary: DOCBUG: --rekey Product: syslog-ng Version: 3.5.x Platform: PC OS/Version: Linux Status: NEW Severity: minor Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.balabit(a)faxm0dem.org Type of the Report: --- Estimated Hours: 0.0 In Example 2.3. (Using the rekey() option) from §2.10.1. (Specifying data types in value-pairs) the documentation claims: "The rekey() option can be used with the format-json template-function as well, using the following syntax: $(format-json --key .cee.* --rekey --shift 4 --add-prefix events.)" However, when using this syntax, the config parser fails with the following error: "Error parsing config, Error compiling template (Error parsing value-pairs: expected an equal sign in key=value pair)" The correct syntax (which works, as suggested by algeron on IRC) would be the following: $(format-json --rekey .cee.* --add-prefix events. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

Fix several memory leaks caused by configuration reloading
by Xufeng Zhang 14 Apr '14

14 Apr '14

Hello all, Patch 1 is try to resolve below valgrind memory leak: ==25354== 26,112 bytes in 32 blocks are definitely lost in loss record 619 of 619 ==25354== at 0x4A05F58: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==25354== by 0x3A05A478F4: g_malloc (in /lib64/libglib-2.0.so.0.2600.0) ==25354== by 0x3A036416D2: log_writer_flush (logwriter.c:1020) ==25354== by 0x3A03641812: log_writer_deinit (logwriter.c:1138) ==25354== by 0x523AF4A: affile_dw_deinit (logpipe.h:268) ==25354== by 0x523B7EB: affile_dd_deinit (logpipe.h:268) ==25354== by 0x3A0362961E: cfg_tree_stop (logpipe.h:268) ==25354== by 0x3A036420DF: main_loop_reload_config_apply (mainloop.c:498) ==25354== by 0x3A036613FA: iv_signal_event (iv_signal.c:170) ==25354== by 0x3A0365FE48: iv_event_raw_got_event (iv_event_raw_posix.c:89) ==25354== by 0x3A03660511: iv_fd_poll_and_run (iv_fd.c:163) ==25354== by 0x3A03660C93: iv_main (iv_main_posix.c:117) Patch 3 is try to resolve below valgrind memory leak: ==25354== 1,107 (176 direct, 931 indirect) bytes in 1 blocks are definitely lost in loss record 594 of 619 ==25354== at 0x4A05F58: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==25354== by 0x3A05A478F4: g_malloc (in /lib64/libglib-2.0.so.0.2600.0) ==25354== by 0x3A0362FFEF: g_process_set_argv_space (gprocess.c:502) ==25354== by 0x40164C: main (main.c:196) The memory leak problem which resolved by patch 2 is very clear. The only concern from me is patch 1, would it cause any side effect? Thanks!

1 3

Syslog-ng build issue (Autoconf?)
by Bendler, Ehren 11 Apr '14

11 Apr '14

Our build environment: Autoconf 2.68 Automake 1.11.3 m4 1.4.16 libtool 2.4.2 gcc 4.6.3 (cross compiling to PPC) We get this error when building syslog-ng 3.5.4.1: ./configure: line 12794: syntax error near unexpected token `-Wno-pointer-sign,' ./configure: line 12794: `AX_CFLAGS_GCC_OPTION(-Wno-pointer-sign, CFLAGS_NOWARN_POINTER_SIGN)' When I comment out that line in configure.ac (since we don't use that flag), I get an error from libmongo complaining about not having automake 1.14+ and that it only tries to build it because I changed the AC file. I tried to get around that by disabling mongodb support in the configure flags, but it still tries to build that library. I'd prefer to just get rid of the initial error. At the moment I am thinking it is related to our GNU Autotools versions, but I need confirmation of that before I can go ask the powers that be for updates. Or if that isn't the problem, alternate suggestions are appreciated. -Ehren Bendler

1 1

insider 2014-04: incubator packaged; tailing logs; PCI-DSS; eCSI;
by Czanik Péter 10 Apr '14

10 Apr '14

Dear syslog-ng users, This is the 33th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. FEATURED NEWS syslog-ng incubator packaged and documented ------------------------------------------- Last month we had a nice list of new syslog-ng incubator features. By that time it was not much documented and was packaged only for a few platforms. Luckily both of these changed, as new features are now well documented in blogs and additional platforms are now supported. https://czanik.blogs.balabit.com/2014/03/syslog-ng-incubator-packaged/ Tailing SSB logs remotely ------------------------- Perhaps the most important new feature introduced in syslog-ng Store Box 3F2 is the REST-based API to access stored logs and SSB’s indexing engine. It can be used for “tail -f logfile.txt | grep foo” functionality by creating a simple shell script. https://gyp.blogs.balabit.com/2014/03/stuff-to-do-with-the-rest-api-tailing… Google Summer of Code (GsoC) ---------------------------- Student application deadline is over and we are happy to announce, that there are many promising applications. Right now we don't know yet how many of these we can accept, the number of slots will only be known by the end of the month. Stay tuned! PCI-DSS compliance and log management ------------------------------------- After the new version of PCI DSS was published in November, we decided to revisit our very popular Regulatory Compliance and System Logging white paper. Since PCI DSS is such an important standard, we decided to dedicate a white paper exclusively to this topic. In addition to highlighting the changes in version 3.0, we have changed the format of the paper to include information on how log management can help you meet each requirement – not just number 10 – as well as detailed information on how syslog-ng Premium Edition and the syslog-ng Store Box can help you fulfill PCI-DSS requirements. https://pages.balabit.com/pci-dss-compliance-and-log-management.html Become an eCSI Officer - free of charge --------------------------------------- Unknown hackers, industrial espionage, or malicious codes spread by zombie networks are not the greatest risks today: your own users do most of the real damage. To help you overcome these challenges, we have created an information-packed education series. The eCSI video series consists of separate modules that you can watch one-by-one, covering various aspects of logging, log management, and privileged activity management from the ground up. https://www.balabit.com/lp/ecsi-training-camp SHORT NEWS * LOADays 2014 experiences: https://czanik.blogs.balabit.com/2014/04/loadays-2014/ * Why somebody chose syslog-ng over rival logging solutions: http://sharknet.us/2014/04/04/a-review-of-syslog-ng/ SYSLOG-NG AT CONFERENCES The next confirmed events are: * Infosecurity Europe, 29 April - 1 May, London, United Kingdom: http://www.infosec.co.uk/, Giving a talk titled "Finding method in the madness: the challenges of the automatic classification of log messages". The talk will be given by Balazs Scheidler (Bazsi) who you will also be able to find at the BalaBit booth at the event. * LinuxTag, 8-10 May, Berlin, Germany: http://linuxtag.org, Giving a talk titled "Finding method in the madness: the challenges of the automatic classification of log messages". BalaBit will also be sponsoring the event so you'll be able to find us and get T-shirts and talk with our engineers. The talk will be a revised version of the talk given a week before at Infosecurity Europe. NEW RELEASES * Check git if you are impatient :) Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

Stats on destinations with macro's ?
by Scot 09 Apr '14

09 Apr '14

Is there a trick to get stats on destinations with macros ? I get stats on my FIFO, local, net work destinations but not on the destinations with macros. Also I don't see any difference in the stats output using any level level 1,2,3. Syslog-ng restarted after each conf change Sent from my iPad

6 11

[PATCH] patterndb: Fix test_patterndb crash when testing outside-of-rule db lookups.
by Nick Alcock 08 Apr '14

08 Apr '14

First test_patterndb_tags_outside_of_rule() nulls out the messages pointer, then it calls clean_pattern_db(), which unconditionally dereferences it. This will never work. Guard the use of messages in clean_pattern_db() with a nullity check. Signed-off-by: Nick Alcock <nix(a)esperi.org.uk> --- modules/dbparser/tests/test_patterndb.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/modules/dbparser/tests/test_patterndb.c b/modules/dbparser/tests/test_patterndb.c index 7ad7f09..ea7c576 100644 --- a/modules/dbparser/tests/test_patterndb.c +++ b/modules/dbparser/tests/test_patterndb.c @@ -69,8 +69,11 @@ create_pattern_db(gchar *pdb) void clean_pattern_db(void) { - g_ptr_array_foreach(messages, (GFunc) log_msg_unref, NULL); - g_ptr_array_free(messages, TRUE); + if (messages) + { + g_ptr_array_foreach(messages, (GFunc) log_msg_unref, NULL); + g_ptr_array_free(messages, TRUE); + } pattern_db_free(patterndb); patterndb = NULL; -- 1.9.1.171.gc28c581

3 2

[Bug 276] New: bundled rabbitmq-c does not compile on Solaris
by bugzilla＠bugzilla.balabit.com 07 Apr '14

07 Apr '14

https://bugzilla.balabit.com/show_bug.cgi?id=276 Summary: bundled rabbitmq-c does not compile on Solaris Product: syslog-ng Version: 3.5.x Platform: PC OS/Version: Solaris Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: michael.hocke(a)nyu.edu Type of the Report: --- Estimated Hours: 0.0 Created an attachment (id=94) --> (https://bugzilla.balabit.com/attachment.cgi?id=94) patch for Makefile.in and amqp_private.h The bundled rabbitmq-c library needs htonll and ntohll that do not exist in Solaris 10. I added a couple defines to modules/afamqp/rabbitmq-c/librabbitmq/amqp_private.h that implement those two functions. I also had to force the linker to add -lsocket and -lnsl for a successful compile. This is probably more of a problem with rabbitmq-c but since it is bundled with syslog-ng I thought I should report it here. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

AUTO: James B Horwath is out of the office (returning 04/14/2014)
by James B Horwath 07 Apr '14

07 Apr '14

I am out of the office until 04/14/2014. I will be out of the office until 04/14 with limited access to e-mail. I will do my best to return any urgent communications. If you have an urgent issue please contact Sunil Gupta at (610)-807-6710. Regards Jim Horwath IT Security Services & Governance, Corporate BTS and IT Security/Risk Tel 610-807-8795 Mobile 610-533-6972 3900 Burgess Place, Bethlehem, Pa. 18017 Jim_Horwath(a)glic.com Note: This is an automated response to your message "syslog-ng Digest, Vol 108, Issue 5" sent on 4/7/2014 6:00:02. This is the only notification you will receive while this person is away. ----------------------------------------- This message, and any attachments to it, may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are notified that any use, dissemination, distribution, copying, or communication of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately by return e-mail and delete the message and any attachments. Thank you.

1 0

[Bug 277] New: test_basicfuncs crashes, apparently due to $(env ...) of a nonexistent variable
by bugzilla＠bugzilla.balabit.com 07 Apr '14

07 Apr '14

https://bugzilla.balabit.com/show_bug.cgi?id=277 Summary: test_basicfuncs crashes, apparently due to $(env ...) of a nonexistent variable Product: syslog-ng Version: 3.5.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: nix(a)esperi.org.uk Type of the Report: bug Estimated Hours: 0.0 This is with syslog-ng 3.5.4.1, fresh out of git, glib 2.38.2, and gcc 4.8.2. Backtrace: #0 0x00007f4209096d1a in strlen () from /lib/libc.so.6 #1 0x00007f4209631550 in g_string_insert_len (len=-1, val=0x0, pos=-1, string=0x219d380) at gstring.c:440 #2 g_string_insert_len (string=0x219d380, pos=-1, val=0x0, len=-1) at gstring.c:428 #3 0x00007f4209b82563 in tf_env (msg=<optimized out>, argc=1, argv=<optimized out>, result=0x219d380) at modules/basicfuncs/misc-funcs.c:54 #4 0x00007f4209942788 in log_template_append_format_with_context (self=self@entry=0x2196d40, messages=messages@entry=0x7fff01a9f1e8, num_messages=num_messages@entry=1, opts=0x2196828, opts@entry=0x0, tz=tz@entry=0, seq_num=999, seq_num@entry=32578, context_id=0x40c143 "test-context-id", context_id@entry=0x0, result=result@entry=0x219d380) at lib/template/templates.c:1385 #5 0x00007f4209942b9f in log_template_append_format (self=self@entry=0x2196d40, lm=lm@entry=0x219c3c0, opts=opts@entry=0x0, tz=tz@entry=0, seq_num=seq_num@entry=32578, context_id=context_id@entry=0x0, result=0x219d380) at lib/template/templates.c:1412 #6 0x00007f4209942c0b in log_template_format (self=self@entry=0x2196d40, lm=lm@entry=0x219c3c0, opts=opts@entry=0x0, tz=tz@entry=0, seq_num=32578, seq_num@entry=999, context_id=0x0, context_id@entry=0x40c143 "test-context-id", result=result@entry=0x219d380) at lib/template/templates.c:1419 #7 0x0000000000409457 in assert_template_format_with_escaping (template=template@entry=0x40b408 "$(env OHHELLO)", escaping=escaping@entry=0, expected=expected@entry=0x40c55e "") at libtest/template_lib.c:108 #8 0x00000000004094ba in assert_template_format (template=template@entry=0x40b408 "$(env OHHELLO)", expected=expected@entry=0x40c55e "") at libtest/template_lib.c:93 #9 0x0000000000409161 in test_misc_funcs () at modules/basicfuncs/tests/test_basicfuncs.c:101 #10 0x0000000000408c4a in main (argc=<optimized out>, argv=<optimized out>) at modules/basicfuncs/tests/test_basicfuncs.c:115 Somehow, between assert_template_format_with_escaping() and tf_env(), we end up with a 1-entry argc and an argv containing an empty string; this then turns into a request to insert a null pointer at position -1. I'd say glib is right to coredump when faced with that :) -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

stale submodules for ivykis and libmongo-client in 3.5.x
by Nix 04 Apr '14

04 Apr '14

spindle 1469 /usr/src/syslog-ng/syslog-ng% git submodule update fatal: reference is not a tree: ec9066dcc66b33e3767b4729f86844549f004657 fatal: reference is not a tree: cf44084c01a4228244f0dc079c43f94f67462537 Unable to checkout 'ec9066dcc66b33e3767b4729f86844549f004657' in submodule path 'lib/ivykis' Unable to checkout 'cf44084c01a4228244f0dc079c43f94f67462537' in submodule path 'modules/afmongodb/libmongo-client' That this has been true for a month makes me wonder how many people are actually building syslog-ng from git... -- NULL && (void)

2 2

syslog-ng: lua, monitoring
by Peter Czanik 04 Apr '14

04 Apr '14

Hi, As not everyone is subscribed to the BalaBit blog rss feed, here are a couple of interesting blogs how to use syslog-ng in even more situations: Extending syslog-ng with lua scripts as destination: https://talien.blogs.balabit.com/2014/03/lua-destination-examples/ Monitoring with syslog-ng: https://talien.blogs.balabit.com/2014/03/monitoring-with-syslog-ng/ Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

2 1

SQL destination - multiple destinations (tables) within one database ?
by Tomáš Novosad 03 Apr '14

03 Apr '14

Hello, i'd like to ask, if it si possible, somehow, to share one database connection across mutliple SQL destinations? Consider this setup (shortened): destination table_a { sql( type(psql) host("localhost") username("my_login") password("secret") database("my_db") table("table_a") columns(...) values (...) ); }; destination table_b { sql( type(psql) host("localhost") username("my_login") password("secret") database("my_db") table("table_b") columns(...) values (...) ); }; destination table_c { sql( type(psql) host("localhost") username("my_login") password("secret") database("my_db") table("table_c") columns(...) values (...) ); }; when i'm using this setup, syslog open 3 separate connections to database, 1 connection for each destination. Is it possible to configure syslog-ng to use only 1 connection, as database and login are the same for all 3 destinations ? If there would be, let's say 50 different tables in single DB, there would be 50 separate connections, which is a waste of resources, as Postgres assign memory for each connection. Also the config in syslog would be much simpler and easier to maintain, if the connection would be described in some section, and destination would define table name and point to some previously defined DB connection. Thanks for helping -- Tomáš Novosad LinuxBox.cz, s.r.o. 28. října 168, 709 00 Ostrava tel.: +420 591 166 221 mobil: +420 737 238 655 email: tomas.novosad(a)linuxbox.cz jabber: novosad(a)linuxbox.cz www.linuxbox.cz mobil servis: +420 737 238 656 email servis: servis(a)linuxbox.cz

2 3

syslog-ng not reconnecting when using tcp
by prasad padiyar 02 Apr '14

02 Apr '14

Hi, we use syslog-ng on one of our nodes as client and rsyslogd as server. We have configured tcp to connect to server. What we see is when the reboot the complete cluster (has node where the server runs and other node where the client runs,) only the server node is rebooted. The client node where syslog-ng runs is not rebooted. The issue is after we reboot the syslog-ng connection is not re-established and hence we are not getting any logs on the server. After going through some of the posts related to the problem and the admin guide, it mentions about time_reopen() value to solve this issue. In my case, i have configured time_reopen(40). So it means at 40s interval the tcp connection is tired to be re-established. But i see that this is not happening. Can you please help me resolve this issue. Also can you let me know how many times and at what interval this connection re-establishment is tried. Below are some of the logs that could be helpful. syslog-ng.conf: cat /etc/syslog-ng/syslog-ng.conf # syslog-ng configuration file. # # This should behave pretty much like the original syslog on RedHat. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # # 20000925 gb(a)sysfive.com # # Updated by Frank Crawford (<Frank.Crawford(a)ac3.com.au>) - 10 Aug 2002 # - for Red Hat 7.3 # - totally do away with klogd # - add message "kernel:" as is done with klogd. # # Updated by Frank Crawford (<Frank.Crawford(a)ac3.com.au>) - 22 Aug 2002 # - use the log_prefix option as per Balazs Scheidler's email # # Updated by Jose Pedro Oliveira (<jpo at di.uminho.pt>) - 05 Apr 2003 # - corrected filters 'f_filter2' and 'f_filter6' # these filters were only allowing messages of one specific # priority level; they should be allowing messages from that # priority and upper levels. # # Updated by Jose Pedro Oliveira (<jpo at di.uminho.pt>) - 25 Jan 2005 # - Don't sync the d_mail destination # # Updated by Jose Pedro Oliveira (<jpo at di.uminho.pt>) - 01 Feb 2005 # - /proc/kmsg is a file not a pipe. # ( https://lists.balabit.hu/pipermail/syslog-ng/2005-February/006963.html) # options { sync (0); time_reopen (40); log_fifo_size (1000); stats(86400); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source s_sys { file ("/proc/kmsg" log_prefix("kernel: ")); unix-stream ("/dev/log"); internal(); # udp(ip(0.0.0.0) port(514)); }; destination d_cons { file("/dev/console"); }; #destination d_mesg { file("/var/log/messages"); }; #destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" sync(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_mlal { usertty("*"); }; destination tcp-to-master { tcp("169.254.1.82" localip("169.254.1.66") localport(601) port(601) ); }; #filter f_filter1 { facility(kern); }; filter f_filter2 { level(info..emerg) and not facility(mail,authpriv,cron) and not match("Connection broken to AF_INET") and not match("Error connecting to remote host AF_INET"); }; filter f_filter3 { facility(authpriv); }; filter f_filter4 { facility(mail); }; filter f_filter5 { level(emerg); }; filter f_filter6 { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_filter7 { facility(local7); }; filter f_filter8 { facility(cron); }; log { source(s_sys); filter(f_filter2); destination(tcp-to-master); }; #log { source(s_sys); filter(f_filter1); destination(d_cons); }; # Redirecting the logs to /var/log/ of CLA, to avoid log files filling up AHUB3-A file system #log { source(s_sys); filter(f_filter2); destination(d_mesg); }; #log { source(s_sys); filter(f_filter3); destination(d_auth); }; log { source(s_sys); filter(f_filter4); destination(d_mail); }; log { source(s_sys); filter(f_filter5); destination(d_mlal); }; log { source(s_sys); filter(f_filter6); destination(d_spol); }; log { source(s_sys); filter(f_filter7); destination(d_boot); }; log { source(s_sys); filter(f_filter8); destination(d_cron); }; netstat output when the cluster was rebooted: Transition from ESTABLISHED to CLOSED_WAIT tcp 0 0 169.254.1.67:601 169.254.1.82:601 ESTABLISHED 3771/syslog-ng 10.34.37.082224000 tcp 1 0 169.254.1.67:601 169.254.1.82:601 CLOSE_WAIT 3771/syslog-ng CLOSED_WAIT to LAST_ACK tcp 1 0 169.254.1.67:601 169.254.1.82:601 CLOSE_WAIT 3771/syslog-ng 10.34.43.555017000 tcp 1 1 169.254.1.67:601 169.254.1.82:601 LAST_ACK - 10.34.43.644520000 LAST_ACK to CLOSED: tcp 1 1 169.254.1.67:601 169.254.1.82:601 LAST_ACK - 10.36.29.754533000 tcp 1 1 169.254.1.67:601 169.254.1.82:601 LAST_ACK - 10.36.29.801733000 tcp 1 1 169.254.1.67:601 169.254.1.82:601 LAST_ACK - 10.36.29.846228000 10.36.29.892427000 10.36.29.931181000 SYN_SENT: 10.36.43.192808000 tcp 0 1 169.254.1.67:601 169.254.1.82:601 SYN_SENT 3771/syslog-ng 10.36.43.238811000 tcp 0 1 169.254.1.67:601 169.254.1.82:601 SYN_SENT 3771/syslog-ng 10.36.43.279561000 tcp 0 1 169.254.1.67:601 169.254.1.82:601 SYN_SENT 3771/syslog-ng continues... 10.36.46.180840000 tcp 0 1 169.254.1.67:601 169.254.1.82:601 SYN_SENT 3771/syslog-ng 10.36.46.225891000 10.36.46.275090000 10.36.46.318485000 10.36.46.359169000 The server node comes up and it start rsyslogd after some more time. But when this starts, there is no syn packets received by rsyslogd and hence the connection is not established. Can you please let me know where exactly 40s(configured time_reopen value) coming into picture here in the netstat output? syslog-ng version: syslog-ng 1.6.12 ----------------------------------------------------------------------------------------------------------------------------------------------------- I got a reply from one of the balabit guys that i have to use version 3.4. I am currently trying to use it. But i still have some questions below. 1. How many number of times the connection is retried. 2. when exactly the connection retry happens (when will it get to know when it has to retry) 3. time_reopen(40), what exactly is 40seconds here. -- Regards, Prasad

2 2

SQL destination - inserting data into ARRAY column
by Tomáš Novosad 02 Apr '14

02 Apr '14

Hello everyone, i'm using syslog-ng to parse/store some logs into PostgreSQL database. I've got full control of the data being logged, because it is produced in my perl code. Data are logged into syslog and written unchanged into file, and in separate log statement, data are parsed into macros and inserted into database. There is a column which hold a list of email addresses. The column in database is defined as array of text (mail_to TEXT[]). This works fine, as the value is being parsed into macro, and this is enclosed in brackets. In detail log statement looks like: mail_to="addr1,addr2" which is parsed with pattern db into a macro ${mflog.to} then the macro is surrounded by brackets - this is done with set statement: set("{${mfstat.to}}", value("mfstat.to")); Syslog then insert this value into postgres:'{addr1,addr2}' Postgres successfuly parses the string and store it as two-element array [addr1,addr2]. This works just fine, but now to my problem. I also want to store another ARRAY colum, which is not of TEXT type, but a custom data type. The definition in database looks like: (parts part_data[]) where part_data is my custom type defined in postgres. Problems comes when i try to insert such data via syslog. My perl program produces this line which is sent to syslog: parts="(part1,20,bla),(part2,234,bli)" this is again parsed into macro and macro is again surronded by brackets. So syslog tries to insert this '{(part1,20,bla),(part2,234,bli)}' which fails because target column if of type part_data[], but postgres parses the string as text[]. Therfore i need to explicitly instruct postgres to do a data conversion. This can be easily achieved by adding the conversion to input string '{(part1,20,bla),(part2,234,bli)}'::part_data[] however i do no know how to achieve this, as the literal text '::part_data[]' is just outside the single qutes. I just do not know how to instruct syslog-ng to add extra text ::part_data[] just after the last single quote. All the inserted values are simple strings and i do not know how to add some extramodifications to that strings. Is there any way to do such thing? Similar thing would be for example running some function on inserted value. Let's say i would like the insert to look like this "INSERT into table (col1) VALUES (some_postgres_func(val1))" How do i set the sql destination in syslog-ng to achieve this? Many thanks for any ideas. -- Tomáš Novosad

2 2