- syslog-ng - lists.balabit.hu

Create a Log file for Each server.
by Jason Long 10 Dec '14

10 Dec '14

Hello all. How are you? I have a windows sever with Syslog agent installed on it and it forward all logs to my Linux box. My Syslog-NG collected it very well but I want to Forward another Windows Events into my Syslog server but How can I modify my current config file for create another file for this new server? I use Logstash and Kibana on my Linux Box, Can it detect my new Log file automatically? # syslog-ng configuration file. # # This should behave pretty much like the original syslog on RedHat. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source s_netsyslog { udp(ip(0.0.0.0) port(514) flags(no-hostname)); tcp(ip(0.0.0.0) port(514) flags(no-hostname)); }; destination d_netsyslog { file("/var/log/network.log" owner("root") group("root") perm(0644)); }; log { source(s_netsyslog); destination(d_netsyslog); }; as you see it collect Syslog from any IP address. Cheers.

2 3

Re: [syslog-ng] [syslog-ng 3.6.1] booting with openrc, but syslog-ng thinks i boot with sytemd
by Balazs Scheidler 08 Dec '14

08 Dec '14

Hi Lone_Wolf, Could you pls give the patch a try? Should fix your issue. Thanks. On Thu, Dec 4, 2014 at 12:43 PM, Juhász, Viktor <viktor.juhasz(a)balabit.com> wrote: > Hi, > > There is a pull request on to 3.6/master: > > https://github.com/balabit/syslog-ng/pull/327 > > I created it on morning, but it is not listed in the pull requests list > > Viktor > > > On Thu, Dec 4, 2014 at 12:27 PM, Balazs Scheidler <bazsi77(a)gmail.com> > wrote: > >> Hi, >> >> Viktor could you pls share the fix for this? I know I promised to port >> it, but got distracted. >> >> Thanks. >> On Dec 4, 2014 11:36 AM, "LoneVVolf" <lonewolf(a)xs4all.nl> wrote: >> >>> On 04-12-14 09:58, Tibor Benke wrote: >>> >>> I think you've found a bug in service-management.c: >>> >>> >>> https://github.com/balabit/syslog-ng/blob/master/lib/service-management.c#L… >>> >>> If you have syslog-ng compiled with --enable-systemd, it will always >>> be the active service-management unit. I thought we addressed this in a >>> patch by jviktor. >>> >>> Tibor, >>> >>> Is that code run during compiling/building or at run-time ? >>> Archlinux developers build packages on machines with an archlinux base >>> installation with systemd installed & active . >>> If service-management.c is executed at build-time, that would explain >>> why syslog-ng tries to use systemd on my system. >>> >>> Given the multi-platform support of syslog-ng i would expect such checks >>> to be executed at run-time . >>> >>> >>> >>> >>> 2014-12-04 0:04 GMT+01:00 Balazs Scheidler <bazsi77(a)gmail.com>: >>> >>>> Also, you might want to use the system() source, which tries to >>>> determine the optimal setup to fetch local messages. So no need to use I >>>> unix-dgram or internal directly. >>>> >>> Balazs, thanks for the tip. After the problem is fixed, i'll look >>> into using that. >>> >>> >>> ______________________________________________________________________________ >>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >>> Documentation: >>> http://www.balabit.com/support/documentation/?product=syslog-ng >>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>> >>> >>> > -- Bazsi

3 2

Sorry, it is just a test
by Juhász, Viktor 05 Dec '14

05 Dec '14

Test

1 0

[syslog-ng 3.6.1] booting with openrc, but syslog-ng thinks i boot with sytemd
by LoneVVolf 04 Dec '14

04 Dec '14

Hi, First i'd like to say i have been using syslog-ng for years and am very pleased with it. Thank you for a reliable and stable product. I'm running archlinux x86_64 and boot with openrc 0.13.5 . I do have systemd installed so i can use udev functionality. Today I updated from syslog-ng 3.5.6 to 3.6.1 and found this in rc.log : [2014-12-03T13:27:21.835298] Using /dev/log Unix socket with systemd is not possible. Changing to systemd-syslog source, which supports socket activation.; * Starting syslog-ng ... [2014-12-03T13:27:21.890729] Using /dev/log Unix socket with systemd is not possible. Changing to systemd-syslog source, which supports socket activation.; [2014-12-03T13:27:21.902289] Failed to acquire /run/systemd/journal/syslog socket, disabling systemd-syslog source; [ ok ] This is the source part of my /etc/syslog-ng/syslog-ng.conf source src { unix-dgram("/dev/log"); internal(); file("/proc/kmsg"); }; I use the official archlinux syslog-ng package, https://www.archlinux.org/packages/extra/x86_64/syslog-ng/ . It uses these build flags : | ./configure--prefix=/usr--sysconfdir=/etc/syslog-ng--libexecdir=/usr/lib \ --sbindir=/usr/bin--localstatedir=/var/lib/syslog-ng--datadir=/usr/share/syslog-ng \ --with-pidfile-dir=/run--disable-spoof-source--enable-ipv6--enable-sql \ --enable-systemd--with-systemdsystemunitdir=/usr/lib/systemd/system| syslog-ng does appear to work normally, but i'd like to get rid of the incorrect error message. Lone_Wolf (full rc.log & syslog-ng.conf attached)

3 6

How can I have Persian and Arabic names in Syslog?
by Jason Long 26 Nov '14

26 Nov '14

Hello Folks.How are you?I installed a Syslog agent on Windows server and some of directories on Windows server are Persian. In Linux box I installed Logstash,Syslog-ng and Kibana but when my Linux receive Logs from Windows server the name shown as "????". How can I solve it? Cheers.

1 0

Re: [syslog-ng] Remote tags
by Nikolay P 25 Nov '14

25 Nov '14

Thank you, Sascha for sharing your example of config. That is exactly how I set it up after Fabian's answer. I used .SDATA.meta(a)11111.tags to store my tags and on the server they were seen as a string of comma separated values, so to match particular ones I used filters like this filter f_example { match( "example" value( ".SDATA.meta(a)11111.tags" ) ); } Later I realized that I am trying to invent my own wheel because what I was trying to achieve is to separate logs generated by system programs from the logs generated by userland programs. Now I am filtering them based on facility. But I am still using .SDATA.meta. On every computer I set this rewrite r_is_computer { set( "true" value( ".SDATA.meta(a)11111.is_computer" ) ); }; In this way on server end I can easily distinguish logs generated by computers from logs generated by different network equipment and I do not have to use IP based filtering. Nikolay. ----- Original Message ----- From: "Sascha Lucas" <Sascha.Lucas(a)gisa.de> To: syslog-ng(a)lists.balabit.hu Sent: Tuesday, November 25, 2014 2:34:00 AM GMT -05:00 US/Canada Eastern Subject: Re: [syslog-ng] Remote tags Hi Nikolay, > Could anyone here advice me if it is possible to set a tags() on a log > entry on one machine, send this log message to a remote syslog-ng and > use this tags() in a filter on the remote machine? As Fabien pointed out, it is possible. I'm doing something similar using rfc5424 protocol: The first thing I do is rewriting the log to append local scoped macro data into the sdata structure (here I'm using $SOURCEIP, where you want $tags). When I read rfc5424 I remember, that there are custom data-structures where you can store your tags (I decided to abuse .SDATA.origin.ip for my purpose): rewrite r_sdata { set("$SOURCEIP" value(".SDATA.origin.ip")); }; The second thing is to use the syslog-driver (capable of sending and receiving rfc5424): destination d_logserver { syslog("X.X.X.X" transport("udp")); }; And finely the log line log { source(s_network); source(src); rewrite(r_sdata); destination(d_logserver); }; On the server I have a source capable of rfc5424: source s_network { syslog( transport("udp") flags(validate-utf8) so-rcvbuf(2097152)); }; The transferred Information is directly available on the server in the macro ${.SDATA.origin.ip}. Your tags may be a bit special, because multiple tags would be transferred as a comma separated string. Matching on a single tag would probably mean to rewrite the log again. This time with something like set("${.SDATA.your.structure}. " value("tags"));. HTH, Sascha. Aufsichtsratsvorsitzender: Herbert Vogel Geschäftsführung: Michael Krüger Sitz der Gesellschaft: Halle/Saale Registergericht: Amtsgericht Stendal | Handelsregister-Nr. HRB 208414 UST-ID-Nr. DE 158253683 Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Empfänger sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail oder des Inhalts dieser Mail sind nicht gestattet. Diese Kommunikation per E-Mail ist nicht gegen den Zugriff durch Dritte geschützt. Die GISA GmbH haftet ausdrücklich nicht für den Inhalt und die Vollständigkeit von E-Mails und den gegebenenfalls daraus entstehenden Schaden. Sollte trotz der bestehenden Viren-Schutzprogramme durch diese E-Mail ein Virus in Ihr System gelangen, so haftet die GISA GmbH - soweit gesetzlich zulässig - nicht für die hieraus entstehenden Schäden. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

Remote tags
by Nikolay P 25 Nov '14

25 Nov '14

Hi, list. Could anyone here advice me if it is possible to set a tags() on a log entry on one machine, send this log message to a remote syslog-ng and use this tags() in a filter on the remote machine? So that on machine 1: source src { internal( tags("computer", "test") ) }; destination rem { udp("example.org" port(514)); }; log { source(src); destination(rem) }; And on machine 2 (example.org) source rem { udp(ip("0.0.0.0") port(514)); }; filter f_tag_test{ tags("test") }; destination test_dest { file("/var/log/test.log"); }; log { source(rem); filter(f_tag_test); destination(test_dest) }; So, should I be able to read tags ("computer" and "test") set by machine 1 on machine 2?

3 2

SQL escaping
by Nikolay P 24 Nov '14

24 Nov '14

Hello. My syslog-ng (version 3.4.8) logs some messages into the PostgreSQL database. I am going to setup a web page which will display those logs. I submitted log entry "alert('123')" surrounded by "script" html tags and displayed content of the database table on a web page. Javascript alerted "123" which means it is a XSS vulnerability. I am not in control of incoming log messages and they can containt anything, so I have no trust to them. Is there anything I can do from the syslog-ng side of things to close this XSS vulnerability or I have to deal with it in my Web application?

2 1

unsubscribe
by Simão Mata 20 Nov '14

20 Nov '14

unsubscribe On 20/11/14 12:00, syslog-ng-request(a)lists.balabit.hu wrote: > Send syslog-ng mailing list submissions to > syslog-ng(a)lists.balabit.hu > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.balabit.hu/mailman/listinfo/syslog-ng > or, via email, send a message with subject or body 'help' to > syslog-ng-request(a)lists.balabit.hu > > You can reach the person managing the list at > syslog-ng-owner(a)lists.balabit.hu > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of syslog-ng digest..." > > > Today's Topics: > > 1. syslog-ng 3.6.1 packages (Czanik) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 19 Nov 2014 15:23:03 +0100 > From: Czanik, P?ter <peter.czanik(a)balabit.com> > Subject: [syslog-ng] syslog-ng 3.6.1 packages > To: "Syslog-ng users' and developers' mailing list" > <syslog-ng(a)lists.balabit.hu> > Message-ID: > <CANcUavu8PeuFqC+u81vTc1Agf2=_Cee1+1hYoSjWv0eOR+AJtw(a)mail.gmail.com> > Content-Type: text/plain; charset=UTF-8 > > Hi, > Recently many people asked me in public and private about syslog-ng > 3.6.1 packages. To make my life easier, I put together a blog about > the current situation. It's available at > https://czanik.blogs.balabit.com/2014/11/syslog-ng-ose-3-6-1-packages/ > Bye, > Peter Czanik (CzP) <peter.czanik(a)balabit.com> > BalaBit IT Security / syslog-ng upstream > http://czanik.blogs.balabit.com/ > https://twitter.com/PCzanik > > > ------------------------------ > > _______________________________________________ > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > https://lists.balabit.hu/mailman/listinfo/syslog-ng > > > End of syslog-ng Digest, Vol 115, Issue 11 > ****************************************** > -- <http://www.wimdu.com>*Wimdu GmbH - * Voltastraße 5, 13355 Berlin, Germany *Managing Directors -* Arne Bleckwenn, Hinrich Dreiling *Commercial Register Number -* 129773 B Berlin <http://www.facebook.com/wimdu> <http://www.twitter.com/wimdu> <http://www.google.com/+wimdu> <http://www.youtube.com/wimtubechannel> <http://www.linkedin.com/company/wimdu-gmbh>

1 0

syslog-ng 3.6.1 packages
by Czanik, Péter 19 Nov '14

19 Nov '14

Hi, Recently many people asked me in public and private about syslog-ng 3.6.1 packages. To make my life easier, I put together a blog about the current situation. It's available at https://czanik.blogs.balabit.com/2014/11/syslog-ng-ose-3-6-1-packages/ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

Re: [syslog-ng] How can I add multi server in my Syslog-ng Configuration?
by Jim Hendrick 18 Nov '14

18 Nov '14

Sorry to ask such a basic question, but are you sure the access point is *sending* logs? Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Jason Long <hack3rcon(a)yahoo.com> Date:11/18/2014 6:27 AM (GMT-05:00) To: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] How can I add multi server in my Syslog-ng Configuration? Hello all. How are you? I have a windows sever with Syslog agent installed on it and it forward all logs to my Linux box. My Syslog-NG collected it very well but I want to forward my Access point log to syslog-ng too. My Syslog-NG configuration is : # syslog-ng configuration file. # # This should behave pretty much like the original syslog on RedHat. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source s_netsyslog { udp(ip(0.0.0.0) port(514) flags(no-hostname)); tcp(ip(0.0.0.0) port(514) flags(no-hostname)); }; destination d_netsyslog { file("/var/log/network.log" owner("root") group("root") perm(0644)); }; log { source(s_netsyslog); destination(d_netsyslog); }; as you see it collect Syslog from any IP address but why my Syslog-ng can't receive Access point log? Cheers.

2 1

How can I add multi server in my Syslog-ng Configuration?
by Jason Long 18 Nov '14

18 Nov '14

Hello all.How are you?I have a windows sever with Syslog agent installed on it and it forward all logs to my Linux box. My Syslog-NG collected it very well but I want to forward my Access point log to syslog-ng too. My Syslog-NG configuration is : # syslog-ng configuration file.## This should behave pretty much like the original syslog on RedHat. But# it could be configured a lot smarter.## See syslog-ng(8) and syslog-ng.conf(5) for more information.# options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes);}; source s_netsyslog { udp(ip(0.0.0.0) port(514) flags(no-hostname)); tcp(ip(0.0.0.0) port(514) flags(no-hostname));}; destination d_netsyslog { file("/var/log/network.log" owner("root") group("root") perm(0644)); }; log { source(s_netsyslog); destination(d_netsyslog); }; as you see it collect Syslog from any IP address but why my Syslog-ng can't receive Access point log? Cheers.

1 0

syslog-ng 3.7.0alpha1 is released
by Czanik, Péter 17 Nov '14

17 Nov '14

Hi, We are proud to announce that syslog-ng 3.7.0alpha1 is released. This is the first alpha release of the syslog-ng OSE 3.7 branch. Changes compared to the latest stable release (3.6.1): Features It is possible to create templates without braces. User defined template-function support added. User can define template functions in her/his configuration the same way she/he would define a template. $(format-cim) template function added into an SCL module. A new choice for inherit-properties implemented that will merge all name-value pairs into the new synthetic message, with the most recent being beferred over older values. Developer notes Added implementation for user-defined template functions. A new API added, user_template_function_register() that allows registering a LogTemplate instance as a template function, dynamically. Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Balazs Scheidler, Fabien Wernli, Gergely Nagy, Laszlo Budai, Peter Czanik, Viktor Juhasz, Viktor Tusa Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

3 2

What is the main reason for forward Windows Logs to Linux Box?
by Jason Long 14 Nov '14

14 Nov '14

Hello Folks.How are you?I have a question and Please accept my apology if it is silly. I forward Windows Log via Snare into my Linux box, But Can I ask why a network admin do it? Why some people don't use Windows Log program? I received all Windows Logs in Linux with Windows Audit and I don't know how can I analysis it easily!!! Cheers.

2 1

Syslog-ng Drops Logs
by Yalin Aksoy 07 Nov '14

07 Nov '14

We have lots of traffic going through syslog-ng in our system (5000 logs per second), so some logs are dropped because of 'log_fifo_size()'. I've look around the web and found flush-lines and flush-timeout methods, but also it failed in every configuration. Related parts of my syslog-ng conf looks like that at the moment. " log_fifo_size(4096); flush_lines(100); flush_timeout(1000); " If I increase fifo size to ~ 16000 ,syslog-ng consumes too much memory for my system to operate. Is there any other way to stop that leak and what is the best practice to use? Thanks in advance. -Yalin

3 3

insider 2014-11: 3.6.1 released; JSON; SSB 4LTS; TLS mutual auth; PE 5F2;
by Czanik, Péter 06 Nov '14

06 Nov '14

Dear syslog-ng users, This is the 38th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. FEATURED NEWS syslog-ng 3.6.1 released ------------------------ We are proud to announce that culminating almost a year’s worth of development, syslog-ng 3.6.1 has been released! This is the first stable release from the 3.6 branch, the successor to 3.5, which was originally released in November, 2013. Read more about the highlights of this release at https://czanik.blogs.balabit.com/2014/10/syslog-ng-3-6-1-is-released/ or all the technical details at https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.6.1 Sending JSON formatted messages with syslog-ng ---------------------------------------------- The use of name-value pairs greatly enhances the usability of collected logs. There are many ways in syslog-ng to turn log messages into name-value pairs: parsers. From this post you can learn how to parse log messages using syslog-ng and send the resulting name-value pairs in JSON format to Loggly. Read our guest blog at https://www.loggly.com/blog/sending-json-format-logs-syslog-ng/ This article focuses on JSON and Loggly, but once your logs are parsed, name-value pairs can also be sent to Graphite, Riemann, Redis, AMQP, MongoDB, and so on. SSB 4LTS is released -------------------- One of the products building on the foundations of the open source syslog-ng is SSB (syslog-ng Store Box). It provides a complete log management solution with a web interface based on syslog-ng. The new long term support release comes with many new features and performance improvements. Read more at https://jluby.blogs.balabit.com/2014/10/30/syslog-ng-store-box-4lts-release… TLS encryption and mutual authentication ---------------------------------------- Configuring TLS encryption and mutual authentication can be a challenging job as it needs both strong OpenSSL and syslog-ng skills. This tutorial provides step by step instructions about the necessary steps: creating certificates using OpenSSL and configuring syslog-ng to use them. The tutorial was originally published on linux.com: http://www.linux.com/community/blogs/129-servers/790912-tls-encryption-and-… and now also available on balabit.com at http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guid… syslog-ng PE 5F2 released ------------------------- The second feature release of syslog-ng PE is released. The primary focus is data privacy: anonymization and pseudonymization of incoming log messages. Other features include journal support, SMTP destination. Read more about it at https://jluby.blogs.balabit.com/2014/10/28/introducing-syslog-ng-premium-ed… Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

Launching command with a certain value after extracting it from patterndb
by C. L. Martinez 05 Nov '14

05 Nov '14

Hi all, Is it possible to trigger a command after extracting a field using patterndb? For example I have the following log: Nov 4 15:18:10 myserver01 info ftps[876]: Rule Allow <ALLOW>: - MAP user:mytest IP:1.1.1.1 With patterndb, I can extract field user with for example, a value of $user. Can I trigger a command like "cat $user >> /tmp/users.log" without calling a script?? Thanks.

3 3

How can I disable SSLv3 in syslog-ng 3.3.2 client config to sovle CVE-2014-3566(SSLv3 Fallback Vulnerabilit)?
by bluebenben 05 Nov '14

05 Nov '14

Hi guys In my project I am using syslog-ng as syslog client and send log via TLS. We all know that recently there is one new security flaw which is Poodle(CVE-2014-3566 - SSLv3 Fallback Vulnerability) This requires disabling SSLv3 I have checked admin guide of syslog-ng 3.3.2 but I am able to find the option Could you please let me know the way? Alternatively I think I may achieve the object by disable SSLv3 ciphers used by syslog-ng client original ciphers used by us is ALL:!SSLv2:!MEDIUM:!LOW:!EXP:!ADH:!ECDH:!PSK:!MD5:@STRENGTH I may change it to ALL:!SSLv3:!SSLv2:!MEDIUM:!LOW:!EXP:!ADH:!ECDH:!PSK:!MD5:@STRENGTH Bug this will make syslog-ng only supports TLS1.2 and cause negative impact to interoperability Thanks Jason

2 1

Elasticsearch destination
by Russell Fulton 31 Oct '14

31 Oct '14

Hi We are already using the open source version of syslog-ng and I am about to set up some elastic search instances and would much prefer to feed data direct from syslog-ng rather than go through logstash (I already have a heap of patterndb parsers and performance should be way better!) I have spent an hour or so with Google and have found various references to elastic search destination being available but I can find no mention of it in the release notes for 3.6.1. I have also downloaded the the tarball and unpacked it but could not find any evidence of the module , nore is there any mention of it in the manual. As of now what is the recommended way of getting parsed data from OS syslog-ng into ES? Thanks, Russell

6 17

Re: [syslog-ng] Elasticsearch destination
by Jim Hendrick 31 Oct '14

31 Oct '14

If we use syslog-ng, hadoop, redis, elasticsearch and kibana we could call it the SHREK stack ... Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Fabien Wernli <wernli(a)in2p3.fr> Date:10/31/2014 7:43 AM (GMT-05:00) To: syslog-ng(a)lists.balabit.hu Subject: Re: [syslog-ng] Elasticsearch destination Hi, On Fri, Oct 31, 2014 at 07:05:47AM -0400, Jim Hendrick wrote: > area. That is one of the reasons I am trying to have syslog-ng do as > much as possible so I can remove "L" and only use "E" and "K" :-) yeah, let's call it the SNEK stack, or SNERK when using Riemann too :o) ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

5 4

Re: [syslog-ng] Elasticsearch destination
by Jim Hendrick 31 Oct '14

31 Oct '14

Or reman or "R" .. Hey .. this might actually be a thing! Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Fabien Wernli <wernli(a)in2p3.fr> Date:10/31/2014 7:43 AM (GMT-05:00) To: syslog-ng(a)lists.balabit.hu Subject: Re: [syslog-ng] Elasticsearch destination Hi, On Fri, Oct 31, 2014 at 07:05:47AM -0400, Jim Hendrick wrote: > area. That is one of the reasons I am trying to have syslog-ng do as > much as possible so I can remove "L" and only use "E" and "K" :-) yeah, let's call it the SNEK stack, or SNERK when using Riemann too :o) ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

Re: [syslog-ng] syslog-ng as "shipper" into ELK stack
by Jim Hendrick 28 Oct '14

28 Oct '14

Thanks!I am on vacation this week but I look forward to trying out these suggestions. Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Alexandre Biancalana <biancalana(a)gmail.com> Date:10/06/2014 11:42 AM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng as "shipper" into ELK stack On Mon, Oct 6, 2014 at 9:36 AM, Radu Gheorghe <radu.gheorghe(a)sematext.com> wrote: Hi Jim, With rabbitmq you have the advantage that you can install the RabbitMQ river and have Elasticsearch pull logs from Rabbit instead of having another [moving] piece pull logs from Rabbit and push them to ES. So you'd have a simpler setup that also makes sure ES isn't overwhelmed (because ES is pulling). Another point that can be an advantage is that with Redis you are limited to the RAM memory available on the machine (all your data need to fit in memory) , with RabbitMQ you have disk persistence that can help in cases where you need to stop ES consumption by any reason. There are some problems with this approach: - the river only runs on one node at a time, which may become a bottleneck - rivers are deprecated (or will be) so the ES side isn't actively maintained. I've seen failover issues (node running the river goes down, another node should start the river but doesn't) which needed river delete + recreate to kick the process in again I didn't know that, I will check it out. Logstash started by recommending RabbitMQ as the queue between two Logstash instances, but now moved to Redis. Apparently the reason is that Redis plays nicely with Logstash, and Rabbit didn't, here's a quote from the guide: "Previous versions of this guide used AMQP via RabbitMQ. Due to the complexity of AMQP as well as performance issues related to the Bunny driver we use, we're now recommending Redis instead." Best regards, Radu -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Sat, Oct 4, 2014 at 5:09 AM, Jim Hendrick <jrhendri(a)roadrunner.com> wrote: Thanks. Why rabbitmq instead of redis? Is it faster, or does it offer some additional functions? Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Alexandre Biancalana <biancalana(a)gmail.com> Date:10/03/2014 7:01 PM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng as "shipper" into ELK stack On Thu, Oct 2, 2014 at 9:33 PM, Jim Hendrick <jrhendri(a)roadrunner.com> wrote: Hi, I am working on configuring Elasticsearch, Logstash & Kibana (ELK) to test it as a backend search tool for large volumes of logs. I decided to put Redis in front of Logstash as a "broker" for the incoming logs, and syslog-ng as the "shipper" so it looks like this: syslog-ng ==> redis ==> logstash ==> elasticsearch ==> apache ==> kibana I've been using the following: syslog-ng => rabbitmq => elasticsearch syslog-ng + patterndb to parse logs and write then in json format on rabbitmq, after that is just use elasticsearch amqp river to consume the queue. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

4 4

improving SQLite writing performances (WAS: Re: inserts into a sqlite3 database are not delayed)
by Davide Alberani 27 Oct '14

27 Oct '14

On Wed, Oct 15, 2014 at 10:56 AM, Balazs Scheidler <bazsi77(a)gmail.com> wrote: > > It immediately runs the query, flush_lines() controls the transaction size, > which can be enabled using > flags(explicit-commits). Ok, I see and I can confirm noticeable improvements on a system under heavy I/O and CPU load using explicit-commits (or better: I see it gets a lot worse disabling it, since I already had it on ;)) Plus, I'm seeing minor improvements with session_statements set to "PRAGMA synchronous=OFF; PRAGMA count_changes=OFF;" Beside this, has anyone found other ways to improve writing performances using syslog-ng with sqlite? On (physical) systems with high load, I've noticed that the system is much more responsive if the elevator is set to noop or deadline instead of cfq. But I guess this is highly dependent on the specific circumstances of the system. Thanks! -- Davide Alberani <davide.alberani(a)gmail.com> [PGP KeyID: 0x465BFD47] http://www.mimante.net/

2 1

errors running configure for incubator on Ubuntu 12.04
by Russell Fulton 25 Oct '14

25 Oct '14

rful011@secmontst01:~/src/syslog-ng-incubator$ ./configure checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane… yes [ snip ] checking whether g++ accepts -g... yes checking dependency style of g++... gcc3 checking for inline... inline ./configure: line 4534: syntax error near unexpected token `shared' ./configure: line 4534: `LT_INIT(shared dlopen)’ I have looked at configure and config.log to try and figure out what it is actually checking for. ‘inline’ is to generic to google for :( any ideas what is wrong? Russell

2 1

syslog-ng 3.6.1 is released
by Czanik, Péter 22 Oct '14

22 Oct '14

Hi, We are proud to announce that culminating almost a year’s worth of development, syslog-ng 3.6.1 has been released! This is the first stable release from the 3.6 branch, the successor to 3.5, which was originally released in November, 2013. Read more about the highlights of this release at https://czanik.blogs.balabit.com/2014/10/syslog-ng-3-6-1-is-released/ or all the technical details at https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.6.1 Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0