- syslog-ng - lists.balabit.hu

Re: [syslog-ng] syslog-ng memory usage grows
by Jim Hendrick 24 Dec '14

24 Dec '14

I will certainly try. Send me a link, and Thanks! Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Tibor Benke <ihrwein(a)gmail.com> Date:12/17/2014 5:42 PM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Cc: wernli(a)in2p3.fr, jrhendri(a)roadrunner.com Subject: Re: [syslog-ng] syslog-ng memory usage grows Hi All, I checked your valgrind log and the redis module and I think the leak is in redis.c:137: redisCommand(self->c, "ping"); This returns a reply object, but it's ignored and never freed. If we make a patch on GitHub, could you test it? Regards, Tibor 2014-12-17 18:36 GMT+01:00 <jrhendri(a)roadrunner.com>: Hi Fabien, Hope the season is treating you well. I ran syslog-ng for 10 minutes (recall, ~7000 EPS incoming) under valgrind and here is the output (attached). It looks like there is some sort of leak as the process from yesterday had grown to over 16GB before I stopped it for the valgrind test. Thanks for the help (as always). And please let me know if I can provide any other data. Jim ---- Fabien Wernli <wernli(a)in2p3.fr> wrote: > Hi Jim, > > On Tue, Dec 16, 2014 at 01:24:28PM -0500, Jim Hendrick wrote: > > I'm not sure what I am asking, other than general advice on: > > - performance using patterndb > > performance is awesome, I wouldn't worry about it > although it would help to have some counters on the parsers in the stats > interface. That bieng said, it could help you to increase the stats level. > > > - performance using redis destination > > I can't comment on that one, sorry > > > - advice on debugging where this memory growth is happening > > Could you run syslog-ng inside `valgrind --tool=memcheck > --trace-children=yes --leak-check=yes`? > > The problem is this will really slow down your instance, but let it run a > few minutes nevertheless and pastebin the result somewhere. > > > As a rough measure - I have a syslog-ng process that has been running for > > less than 3 hours and right now is using 1.52GB of resident memory (shown by > > "top") > > I'm suspecting `format-json` to be the source of the leak, or the flow > control cache. These have been the leak source of our own observations many > times, although many patches have solved the issues. > > > (I would rather parse things in syslog-ng, but I *could* do this all using > > logstash/grok if this proves too much for patterndb at this load) > > if you're worried about performance, I wouldn't even consider to begin thinking about comparing > LS/grok to patterndb > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

3 3

insider 2014-12: incubator 0.4.1 released; anonymization; monitoring;
by Czanik, Péter 18 Dec '14

18 Dec '14

Dear syslog-ng users, This is the 39th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. FEATURED NEWS syslog-ng incubator 0.4.1 released ------------------------ We are proud to announce that version 0.4.1 of syslog-ng incubator is released. It is compatible with syslog-ng version 3.6. Some features previously included in the incubator such as Riemann and Graphite support have been ported to syslog-ng. It also adds many interesting new features: kafka and java destination support, ZMQ source and destination and a grok parser. Ports and packages are available for Fedora, FreeBSD and openSUSE, but with some of the features missing, as dependencies are not yet available packaged. For details, read https://czanik.blogs.balabit.com/2014/12/syslog-ng-incubator-0-4-1-released/ Data Privacy, Anonymization and Log Data ---------------------------------------- Data privacy is becoming a hot topic, especially in the European Union. New features in syslog-ng can be used to become compliant with regulations. Read about encrypting, anonymizing logs and pseudonymization at https://jluby.blogs.balabit.com/2014/12/08/data-privacy-anonymization-and-l… The trinity of monitoring: syslog-ng, Riemann and Elasticsearch -------------------------------------------------------------- BalaBit organizes OpenAcademy in Budapest, Hungary, as a way of continuous learning. Last time we had a guest, Fabien Wernli (on IRC better known as faxmodem), who talked about how syslog-ng can be integrated with Riemann and Elasticsearch. The video starts in Hungarian, but the presentation is in English: https://www.youtube.com/watch?v=Y4uzJm5VLh4 NEW RELEASES syslog-ng 3.6.2: https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.6.2 syslog-ng incubator 0.4.0 & 0.4.1: https://github.com/balabit/syslog-ng-incubator/releases Happy Holidays! Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

syslog-ng memory usage grows
by Jim Hendrick 17 Dec '14

17 Dec '14

Hi, I am continuing to test syslog-ng as the parser and shipper into an elasticsearch cluster. Right now I have syslog-ng 3.6.1 receiving logs at about ~7000 EPS and running them through a patterndb parser that splits out the (bluecoat proxy) fields into key-value pairs, writing them to a redis destination using format-json. syslog-ng 3.6.1 Installer-Version: 3.6.1 Revision: Compile-Date: Dec 9 2014 19:42:20 Available-Modules: dbparser,json-plugin,afuser,affile,afmongodb,tfgeoip,afprog,redis,afstomp,afsql,afsocket,system-source,afamqp,pseudofile,confgen,afsocket-tls,csvparser,basicfuncs,graphite,syslogformat,afsocket-notls,linux-kmsg-format,cryptofuncs Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off I then use logstash to pull from redis and feed elasticsearch (the thought being that this would provide a buffer for the messages) Over the weekend I had syslog-ng crash (unfortunately no core) but now that I am watching it more closely, I see what appears to be continuous growth in memory usage (which leads me to suppose this was the cause of the crash). I'm not sure what I am asking, other than general advice on: - performance using patterndb - performance using redis destination - advice on debugging where this memory growth is happening As a rough measure - I have a syslog-ng process that has been running for less than 3 hours and right now is using 1.52GB of resident memory (shown by "top") I am including two config files. One has a single source, no filtering, single destination. The other splits logs from three upstream syslog-ng servers thinking this would provide "threading" with multiple patterndb parsers (I have read that patterndb is single threaded). Any thoughts on how to figure out where the memory is going? Or recommendations on whether or not using patterndb in this way is a particularly dumb idea :-) (I would rather parse things in syslog-ng, but I *could* do this all using logstash/grok if this proves too much for patterndb at this load) Thanks! Jim

4 3

Email alerts en syslog-ng 3.3.5
by Roberto Carna 17 Dec '14

17 Dec '14

Dear, I have this syslog platform: - Debian 7 - Syslog-ng 3.3.5.-4 (installed via apt-get) - Php-syslog-ng (graphical interface) - Postfix I need to configure syslog-ng in order to send email alerts/notifications about some specific events. Is it possible to implement with this syslog-ng version and what is the way??? Thanks a lot!!! Roberto

2 4

RFC: debugger
by Scheidler, Balázs 17 Dec '14

17 Dec '14

Hi, I was involved in a number of projects where syslog-ng was used in a relatively complex setup: we wanted to structure incoming log messages and transform them into nice & clean JSON messages. In a number of cases, this required several parsers to get right (csv-parser, db-parser and sometimes regexps) and the resulting configuration was pretty complicated. I've figured, that if we have a domain specific language in the form of the syslog-ng configuration file, we should also have a debugger that makes it easy to find out what went wrong. And this is what I have did, some proof-of-concept code is available on the f/debugger branch on github: https://github.com/balabit/syslog-ng/tree/f/debugger Here's how it works: # start up syslog-ng with the -i option to indicate we # want the debugger enabled $ sbin/syslog-ng -iFe Once syslog-ng starts up, you'll see something like this: Waiting for breakpoint... For now, all processing elements within syslog-ng (called LogPipe) breaks into the debugger. Later, I want to implement real breakpoints, where you can selectively attach breakpoints to elements in the configuration file. # on another console send a log message to syslog-ng: $ logger almafa This is what happens on the syslog-ng side: Breakpoint hit etc/syslog-ng.conf:11:2 11 unix-stream("log"); Dec 15 16:14:05 bazsi: almafa (syslog-ng) At a breakpoint, syslog-ng displays where the break has occurred, quotes the source line and displays the message that is being processed. Now it becomes interactive, you can inspect the message with various commands: (syslog-ng) help syslog-ng interactive console, the following commands are available help, h, or ? Display this help continue or c Continue until the next breakpoint print, p Print the current log message drop, d Drop the current message quit, q Tell syslog-ng to exit (syslog-ng) p $MSG almafa (syslog-ng) p MESSAGE=almafa PROGRAM=bazsi LEGACY_MSGHDR=bazsi: .unix.pid=18703 .unix.uid=1000 .unix.gid=1000 TAGS= (syslog-ng) p "$(format-json --key *)" {"_unix":{"uid":"1000","pid":"18703","gid":"1000"}, ... } With the "continue" command, you can request syslog-ng to proceed to the next LogPipe, where it stops again. "drop" drops the current message, "quit" tells syslog-ng to exit. I would really like to extend this to further ways, like: - real breakpoints, instead of stopping everywhere - inject messages - change messages to make configuration testable - single-step (to the next primitive LogPipe) and step-over (to the next 'real' LogPipe that is found in the configuration file) - interactive REPL to template functions to make it easier to interact with them What do you think? -- Bazsi

2 1

Syslog-ng Drops Logs
by guamar＠IMIST.MA 15 Dec '14

15 Dec '14

Hello, i'm IT technical at CNRSt Morocco. I work on a project to centralize logs with syslog-ng , but so some logs are dropped . I have 10 clients and on each client over 12 website is hosted. thank you to find attached the config file. thanks. Cordialement. ---------------------------------------------------------------------------------------------------------------------------------------------- Thami Guamar Service informatique Institut Marocain de l'Information Scientifique et Technique (IMIST)- Centre National pour la Recherche Scientifique et Technique (CNRST) Angle avenue Allal El Fassi, avenue des FAR,Quartier Hay Ryad B.P. 8027 Rabat-Nations Unies 10102 Rabat, Maroc E-mail :guamar@imist.ma Tél : (212) 5 37 56 99 00/36 Fax : 212 5 37 56 99 01

4 3

Create a Log file for Each server.
by Jason Long 10 Dec '14

10 Dec '14

Hello all. How are you? I have a windows sever with Syslog agent installed on it and it forward all logs to my Linux box. My Syslog-NG collected it very well but I want to Forward another Windows Events into my Syslog server but How can I modify my current config file for create another file for this new server? I use Logstash and Kibana on my Linux Box, Can it detect my new Log file automatically? # syslog-ng configuration file. # # This should behave pretty much like the original syslog on RedHat. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source s_netsyslog { udp(ip(0.0.0.0) port(514) flags(no-hostname)); tcp(ip(0.0.0.0) port(514) flags(no-hostname)); }; destination d_netsyslog { file("/var/log/network.log" owner("root") group("root") perm(0644)); }; log { source(s_netsyslog); destination(d_netsyslog); }; as you see it collect Syslog from any IP address. Cheers.

2 3

Re: [syslog-ng] [syslog-ng 3.6.1] booting with openrc, but syslog-ng thinks i boot with sytemd
by Balazs Scheidler 08 Dec '14

08 Dec '14

Hi Lone_Wolf, Could you pls give the patch a try? Should fix your issue. Thanks. On Thu, Dec 4, 2014 at 12:43 PM, Juhász, Viktor <viktor.juhasz(a)balabit.com> wrote: > Hi, > > There is a pull request on to 3.6/master: > > https://github.com/balabit/syslog-ng/pull/327 > > I created it on morning, but it is not listed in the pull requests list > > Viktor > > > On Thu, Dec 4, 2014 at 12:27 PM, Balazs Scheidler <bazsi77(a)gmail.com> > wrote: > >> Hi, >> >> Viktor could you pls share the fix for this? I know I promised to port >> it, but got distracted. >> >> Thanks. >> On Dec 4, 2014 11:36 AM, "LoneVVolf" <lonewolf(a)xs4all.nl> wrote: >> >>> On 04-12-14 09:58, Tibor Benke wrote: >>> >>> I think you've found a bug in service-management.c: >>> >>> >>> https://github.com/balabit/syslog-ng/blob/master/lib/service-management.c#L… >>> >>> If you have syslog-ng compiled with --enable-systemd, it will always >>> be the active service-management unit. I thought we addressed this in a >>> patch by jviktor. >>> >>> Tibor, >>> >>> Is that code run during compiling/building or at run-time ? >>> Archlinux developers build packages on machines with an archlinux base >>> installation with systemd installed & active . >>> If service-management.c is executed at build-time, that would explain >>> why syslog-ng tries to use systemd on my system. >>> >>> Given the multi-platform support of syslog-ng i would expect such checks >>> to be executed at run-time . >>> >>> >>> >>> >>> 2014-12-04 0:04 GMT+01:00 Balazs Scheidler <bazsi77(a)gmail.com>: >>> >>>> Also, you might want to use the system() source, which tries to >>>> determine the optimal setup to fetch local messages. So no need to use I >>>> unix-dgram or internal directly. >>>> >>> Balazs, thanks for the tip. After the problem is fixed, i'll look >>> into using that. >>> >>> >>> ______________________________________________________________________________ >>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >>> Documentation: >>> http://www.balabit.com/support/documentation/?product=syslog-ng >>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>> >>> >>> > -- Bazsi

3 2

Sorry, it is just a test
by Juhász, Viktor 05 Dec '14

05 Dec '14

Test

1 0

[syslog-ng 3.6.1] booting with openrc, but syslog-ng thinks i boot with sytemd
by LoneVVolf 04 Dec '14

04 Dec '14

Hi, First i'd like to say i have been using syslog-ng for years and am very pleased with it. Thank you for a reliable and stable product. I'm running archlinux x86_64 and boot with openrc 0.13.5 . I do have systemd installed so i can use udev functionality. Today I updated from syslog-ng 3.5.6 to 3.6.1 and found this in rc.log : [2014-12-03T13:27:21.835298] Using /dev/log Unix socket with systemd is not possible. Changing to systemd-syslog source, which supports socket activation.; * Starting syslog-ng ... [2014-12-03T13:27:21.890729] Using /dev/log Unix socket with systemd is not possible. Changing to systemd-syslog source, which supports socket activation.; [2014-12-03T13:27:21.902289] Failed to acquire /run/systemd/journal/syslog socket, disabling systemd-syslog source; [ ok ] This is the source part of my /etc/syslog-ng/syslog-ng.conf source src { unix-dgram("/dev/log"); internal(); file("/proc/kmsg"); }; I use the official archlinux syslog-ng package, https://www.archlinux.org/packages/extra/x86_64/syslog-ng/ . It uses these build flags : | ./configure--prefix=/usr--sysconfdir=/etc/syslog-ng--libexecdir=/usr/lib \ --sbindir=/usr/bin--localstatedir=/var/lib/syslog-ng--datadir=/usr/share/syslog-ng \ --with-pidfile-dir=/run--disable-spoof-source--enable-ipv6--enable-sql \ --enable-systemd--with-systemdsystemunitdir=/usr/lib/systemd/system| syslog-ng does appear to work normally, but i'd like to get rid of the incorrect error message. Lone_Wolf (full rc.log & syslog-ng.conf attached)

3 6