December 2014 - syslog-ng - lists.balabit.hu

RFC: debugger
by Scheidler, Balázs 17 Dec '14

17 Dec '14

Hi, I was involved in a number of projects where syslog-ng was used in a relatively complex setup: we wanted to structure incoming log messages and transform them into nice & clean JSON messages. In a number of cases, this required several parsers to get right (csv-parser, db-parser and sometimes regexps) and the resulting configuration was pretty complicated. I've figured, that if we have a domain specific language in the form of the syslog-ng configuration file, we should also have a debugger that makes it easy to find out what went wrong. And this is what I have did, some proof-of-concept code is available on the f/debugger branch on github: https://github.com/balabit/syslog-ng/tree/f/debugger Here's how it works: # start up syslog-ng with the -i option to indicate we # want the debugger enabled $ sbin/syslog-ng -iFe Once syslog-ng starts up, you'll see something like this: Waiting for breakpoint... For now, all processing elements within syslog-ng (called LogPipe) breaks into the debugger. Later, I want to implement real breakpoints, where you can selectively attach breakpoints to elements in the configuration file. # on another console send a log message to syslog-ng: $ logger almafa This is what happens on the syslog-ng side: Breakpoint hit etc/syslog-ng.conf:11:2 11 unix-stream("log"); Dec 15 16:14:05 bazsi: almafa (syslog-ng) At a breakpoint, syslog-ng displays where the break has occurred, quotes the source line and displays the message that is being processed. Now it becomes interactive, you can inspect the message with various commands: (syslog-ng) help syslog-ng interactive console, the following commands are available help, h, or ? Display this help continue or c Continue until the next breakpoint print, p Print the current log message drop, d Drop the current message quit, q Tell syslog-ng to exit (syslog-ng) p $MSG almafa (syslog-ng) p MESSAGE=almafa PROGRAM=bazsi LEGACY_MSGHDR=bazsi: .unix.pid=18703 .unix.uid=1000 .unix.gid=1000 TAGS= (syslog-ng) p "$(format-json --key *)" {"_unix":{"uid":"1000","pid":"18703","gid":"1000"}, ... } With the "continue" command, you can request syslog-ng to proceed to the next LogPipe, where it stops again. "drop" drops the current message, "quit" tells syslog-ng to exit. I would really like to extend this to further ways, like: - real breakpoints, instead of stopping everywhere - inject messages - change messages to make configuration testable - single-step (to the next primitive LogPipe) and step-over (to the next 'real' LogPipe that is found in the configuration file) - interactive REPL to template functions to make it easier to interact with them What do you think? -- Bazsi

2 1

Syslog-ng Drops Logs
by guamar＠IMIST.MA 15 Dec '14

15 Dec '14

Hello, i'm IT technical at CNRSt Morocco. I work on a project to centralize logs with syslog-ng , but so some logs are dropped . I have 10 clients and on each client over 12 website is hosted. thank you to find attached the config file. thanks. Cordialement. ---------------------------------------------------------------------------------------------------------------------------------------------- Thami Guamar Service informatique Institut Marocain de l'Information Scientifique et Technique (IMIST)- Centre National pour la Recherche Scientifique et Technique (CNRST) Angle avenue Allal El Fassi, avenue des FAR,Quartier Hay Ryad B.P. 8027 Rabat-Nations Unies 10102 Rabat, Maroc E-mail :guamar@imist.ma Tél : (212) 5 37 56 99 00/36 Fax : 212 5 37 56 99 01

4 3

Create a Log file for Each server.
by Jason Long 10 Dec '14

10 Dec '14

Hello all. How are you? I have a windows sever with Syslog agent installed on it and it forward all logs to my Linux box. My Syslog-NG collected it very well but I want to Forward another Windows Events into my Syslog server but How can I modify my current config file for create another file for this new server? I use Logstash and Kibana on my Linux Box, Can it detect my new Log file automatically? # syslog-ng configuration file. # # This should behave pretty much like the original syslog on RedHat. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source s_netsyslog { udp(ip(0.0.0.0) port(514) flags(no-hostname)); tcp(ip(0.0.0.0) port(514) flags(no-hostname)); }; destination d_netsyslog { file("/var/log/network.log" owner("root") group("root") perm(0644)); }; log { source(s_netsyslog); destination(d_netsyslog); }; as you see it collect Syslog from any IP address. Cheers.

2 3

Re: [syslog-ng] [syslog-ng 3.6.1] booting with openrc, but syslog-ng thinks i boot with sytemd
by Balazs Scheidler 08 Dec '14

08 Dec '14

Hi Lone_Wolf, Could you pls give the patch a try? Should fix your issue. Thanks. On Thu, Dec 4, 2014 at 12:43 PM, Juhász, Viktor <viktor.juhasz(a)balabit.com> wrote: > Hi, > > There is a pull request on to 3.6/master: > > https://github.com/balabit/syslog-ng/pull/327 > > I created it on morning, but it is not listed in the pull requests list > > Viktor > > > On Thu, Dec 4, 2014 at 12:27 PM, Balazs Scheidler <bazsi77(a)gmail.com> > wrote: > >> Hi, >> >> Viktor could you pls share the fix for this? I know I promised to port >> it, but got distracted. >> >> Thanks. >> On Dec 4, 2014 11:36 AM, "LoneVVolf" <lonewolf(a)xs4all.nl> wrote: >> >>> On 04-12-14 09:58, Tibor Benke wrote: >>> >>> I think you've found a bug in service-management.c: >>> >>> >>> https://github.com/balabit/syslog-ng/blob/master/lib/service-management.c#L… >>> >>> If you have syslog-ng compiled with --enable-systemd, it will always >>> be the active service-management unit. I thought we addressed this in a >>> patch by jviktor. >>> >>> Tibor, >>> >>> Is that code run during compiling/building or at run-time ? >>> Archlinux developers build packages on machines with an archlinux base >>> installation with systemd installed & active . >>> If service-management.c is executed at build-time, that would explain >>> why syslog-ng tries to use systemd on my system. >>> >>> Given the multi-platform support of syslog-ng i would expect such checks >>> to be executed at run-time . >>> >>> >>> >>> >>> 2014-12-04 0:04 GMT+01:00 Balazs Scheidler <bazsi77(a)gmail.com>: >>> >>>> Also, you might want to use the system() source, which tries to >>>> determine the optimal setup to fetch local messages. So no need to use I >>>> unix-dgram or internal directly. >>>> >>> Balazs, thanks for the tip. After the problem is fixed, i'll look >>> into using that. >>> >>> >>> ______________________________________________________________________________ >>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >>> Documentation: >>> http://www.balabit.com/support/documentation/?product=syslog-ng >>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>> >>> >>> > -- Bazsi

3 2

Sorry, it is just a test
by Juhász, Viktor 05 Dec '14

05 Dec '14

Test

1 0

[syslog-ng 3.6.1] booting with openrc, but syslog-ng thinks i boot with sytemd
by LoneVVolf 04 Dec '14

04 Dec '14

Hi, First i'd like to say i have been using syslog-ng for years and am very pleased with it. Thank you for a reliable and stable product. I'm running archlinux x86_64 and boot with openrc 0.13.5 . I do have systemd installed so i can use udev functionality. Today I updated from syslog-ng 3.5.6 to 3.6.1 and found this in rc.log : [2014-12-03T13:27:21.835298] Using /dev/log Unix socket with systemd is not possible. Changing to systemd-syslog source, which supports socket activation.; * Starting syslog-ng ... [2014-12-03T13:27:21.890729] Using /dev/log Unix socket with systemd is not possible. Changing to systemd-syslog source, which supports socket activation.; [2014-12-03T13:27:21.902289] Failed to acquire /run/systemd/journal/syslog socket, disabling systemd-syslog source; [ ok ] This is the source part of my /etc/syslog-ng/syslog-ng.conf source src { unix-dgram("/dev/log"); internal(); file("/proc/kmsg"); }; I use the official archlinux syslog-ng package, https://www.archlinux.org/packages/extra/x86_64/syslog-ng/ . It uses these build flags : | ./configure--prefix=/usr--sysconfdir=/etc/syslog-ng--libexecdir=/usr/lib \ --sbindir=/usr/bin--localstatedir=/var/lib/syslog-ng--datadir=/usr/share/syslog-ng \ --with-pidfile-dir=/run--disable-spoof-source--enable-ipv6--enable-sql \ --enable-systemd--with-systemdsystemunitdir=/usr/lib/systemd/system| syslog-ng does appear to work normally, but i'd like to get rid of the incorrect error message. Lone_Wolf (full rc.log & syslog-ng.conf attached)

3 6