- syslog-ng - lists.balabit.hu

Re: [syslog-ng] Feedback for GSoC project - RIak Destination for Syslog-ng
by Gergely Nagy 06 May '15

06 May '15

Hi! I'm the mentor for the Riak destination for syslog-ng project, please allow me to answer the questions below: >>>>> "Fred" == Fred Dushin <fdushin(a)basho.com> writes: Fred> As far as I understand, you're talking about a mapping from keys to Fred> sets, but I'm unclear on a few things. The idea is to map a set of log messages to a Riak Set. Where both the key used for the set, and its contents are configurable by the user. There are no plans for a default at this time. There are many ways to configure a syslog-ng=>Riak setup with a destination like the one planned. One is to turn each log message (after parsing) to a Riak Map, and push those maps into a Riak Set. Another way is to format the parsed log messages (with all the extracted fiels, if any) into JSON, and push those into a set. So, for example, given the following syslog line: May 6 14:42:18 eowyn avahi-daemon[27812]: Invalid response packet from host fe80::5d0f:d53a:7b6:3680. We'd end up with a JSON like this: {"timestamp": "2015-05-06T14:42:18+02:00", "host": "eowyn", "program": "avahi-daemon", "pid": 27812, "message": "Invalid response packet from host fe80::5d0f:d53a:7b6:3680.", "avahi-daemon": { "type": "warning", "message": "Invalid response packet", "host": "fe80::5d0f:d53a:7b6:3680" } } We could either add that to a Riak set as-is, or turn it into a Riak map first. Fred> What are the keys you are thinking about? Time stamps? If Fred> timestamps, these are presumably the timestamps of the syslog Fred> event? Whatever the user configures. They may be time stamps (rounded, for predictable keys), or a combination of program name + current date (day granuality). Fred> Just a word of warning, if so. You might find a lot of Fred> variation in timestamp formats and granularity. Perhaps you Fred> can get something reliable out of syslog-ng, We get something sensible out of syslog-ng. But in the end, it is up to the user to configure the template used for keys. There may - and probably will - be examples, but no default. Fred> but that won't help you in the case where syslog-ng is Fred> functioning as a syslog relay, and you want to preserve the Fred> timestamp of the originator, which you should, if you want to Fred> preserve integrity of the logs (e.g, for compliance). In case of syslog-ng, we actually have access to a few kinds of timestamps: the timestamp from the log message (if any), the timestamp of receipt, and the current time. The granularity of timestamps is configurable to some extent. Fred> Or are you talking about a key being a (course grained) Fred> timestamp, say, an integral value in UTC seconds, for example? Fred> And the value(s) being all logs in that interval? Is that your Fred> motivation for sets? That's one way, yes. One could also use something like $PROGRAM/$YEAR-$MONTH-$DAY as key, if the program doesn't produce more than a megabyte of logs a day. So with the example above, our key in case of that log would be avahi-daemon/2015-05-06, and the message would be an element of the set underneath the key. Fred> How much of the syslog payload are you planning to parse? The destination itself is not going to do any parsing. Other parts of syslog-ng do that, and it is up to the user to set up a pipeline that feeds the destination. The source may be syslog, HTTP logs, the Journal, or any of the other sources syslog-ng supports. How much parsing is done, and what gets extracted, is no concern to the destination plugin. Fred> Another interesting problem is that the STRUCTURED-DATA element of Fred> 5424 uses OIDs to discriminate different data types that are encoded Fred> in the header. And while there is a kind of loosely coupled authority Fred> for OIDs, there is no infrastructure for determining a parsing Fred> strategy for these fields. They could really be anything, in the worst Fred> case. As far as I remember, syslog-ng treats all STRUCTURED-DATA elements as strings. But there are tools within syslog-ng to allow converting to other data types, but that must be done explicitly. Fred> But regardless of the deeply structured data, you could get some very Fred> interesting traction by just taking standard headers and indexing them Fred> through Yokozuna. Certainly, indexing the body of a syslog message is Fred> a great idea, as these messages are generally unstructured and fodder Fred> for lucene. This is something that Logstash/ElasticSearch can do Fred> pretty effectively today, and it would be cool to see the same in Riak Fred> + some syslog provider. Yep! When I proposed the idea, using Yokozuna is something I had in mind. Combine the parsing abilities of syslog-ng, Riak for archival purposes, and Yokozuna for searching. That sounds like a match made in heaven. Fred> Finally, it would be really nice if you could structure your plugin in Fred> such a way that they could eventually be ported to rsyslog [2]. The Fred> rsyslogd daemon is deployed by default on certain Linux favors and Fred> enjoys fairly widespread distribution. You might be able to get it Fred> supported in that community, as well. Part of the project is writing a small library to send data to Riak, From C. Just enough for syslog-ng's needs. That library could be used by rsyslog, too (like the MongoDB library originally written for syslog-ng's purposes is used by rsyslog too). But sharing more code than that is not practical, the two daemons work in widely different ways. -- |8]

1 0

Re: [syslog-ng] Feedback for GSoC project - RIak Destination for Syslog-ng
by Gergely Nagy 05 May '15

05 May '15

>>>>> "Christopher" == Christopher Meiklejohn <cmeiklejohn(a)basho.com> writes: >> The intention is to store log messages in each element of the set: >> either as a string (syslog or json, or whatever else the user sees fit), >> or as a map of key-value pairs (where values themselves can be maps >> too). >> >> On average, the log messages are a few kilobytes in size. There may be >> exceptions, but >1mb ones are fairly rare. How much data the set would >> hold... now that's a question that can't really be answered. It is >> really up to the syslog-ng user to configure that. Christopher> I’m referring to the size of the entire set, not the objects that will be members of Christopher> the set. Therefore, the performance penalty seen when using large objects would Christopher> be observed as soon as the size of the entire set (or map) has reached ~1 MB. Christopher> Given that restriction, I’d imagine you would only be able to store a few messages Christopher> in each set. That granularity seems like you are no longer getting the benefits Christopher> of the set. The granularity is configurable by the user: if they have small (say, a few hundred byte long) messages, then we can store a reasonable amount of them in a single Set. For example, assuming an average length of 384 bytes / message (the longest line in today's logs on my laptop), a Set would be able to store about 2k messages. That's not too bad. Christopher> Additionally, the primary benefit of the data types in Riak is that they converge Christopher> deterministically when dealing with concurrent Christopher> operations. Not only that: using sets makes the keys predictable. If I want to retrieve the logs, with sets, I can retrieve the 2015-05-01T15:12:10-T15:12:15 key for example, and have all the logs from those 5 seconds. If I used one message per Riak object, it would be much harder to read the data back. There may be multiple threads adding to the same set, so the deterministic convergence is useful still. Christopher> I’m curious if the set is the right choice here; could Christopher> you just use a custom set format inside of a normal Christopher> Riak object That's an option worth considering, yes. Thanks! Christopher> (or store one message per Riak object, given the write will be an Christopher> immutable log entry?) That's the first goal of the project, because that's the easiest and most straightforward to implement. The downside of one message per Riak object is that it's hard to retrieve the data, because making the keys predictable is not going to be easy. Boxing a few hundred (or few thousand) together into a Set has the advantage of making the keys predictable at the cost of transferring more data to the client when looking for a subset of the logs within the set. -- |8]

1 0

Re: [syslog-ng] Feedback for GSoC project - RIak Destination for Syslog-ng
by Gergely Nagy 05 May '15

05 May '15

>>>>> "Christopher" == Christopher Meiklejohn <cmeiklejohn(a)basho.com> writes: Christopher> I’m a bit concerned with your use of the set embedded in the Christopher> map. The original idea was to use a Set directly. The Set-in-Map thing was just a thought experiment (Map-in-Set would make more sense). Christopher> Large objects have traditionally been a big problem in Riak due Christopher> to the use of distributed Erlang and head of line blocking. I’m Christopher> curious if you could elaborate on what type of data you will be Christopher> storing in the set: how big you expect each item to be, how big you Christopher> expect the map to be, and the overall layout of data inside of the Christopher> data structure. The intention is to store log messages in each element of the set: either as a string (syslog or json, or whatever else the user sees fit), or as a map of key-value pairs (where values themselves can be maps too). On average, the log messages are a few kilobytes in size. There may be exceptions, but >1mb ones are fairly rare. How much data the set would hold... now that's a question that can't really be answered. It is really up to the syslog-ng user to configure that. Hope that made it a little more clear. -- |8], mentor for the riak destination for syslog-ng GSoC project

1 0

Feedback for GSoC project - RIak Destination for Syslog-ng
by Parth Oberoi 02 May '15

02 May '15

Hello Everyone, My proposal[0] for adding Riak as a destination to Syslog-ng was accepted for GSoC2015, so before my coding period begins i.e. 25th May, I wanted to do some ground work and take some feedback from the potential users. The proposal[0] contains all the initial implementation details which might require improvement . Therefore if some of you could take a look and give me some feedback on it, would help me greatly. Below are a few links to important snippets of the proposal. [0] - https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-Riak-destinati… *Goals for the project :-* https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-Riak-destinati… *Development Tools :-* https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-Riak-destinati… *Deliverables :-* https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-Riak-destinati… *My scheduled Plan:-* https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-Riak-destinati… Thanks in advance. Regards, Parth Oberoi (@htrap)

1 0

insider 2015-04: 3.7 beta1; RedHat Summit; Hadoop; GSoC; RFC5424;
by Czanik, Péter 30 Apr '15

30 Apr '15

Dear syslog-ng users, This is the 41th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng 3.7 beta1 released ---------------------------- In the first beta of syslog-ng OSE 3.7, the Java and Python destinations were migrated from the syslog-ng incubator to the syslog-ng core. Also, syslog-ng now includes an interactive syslog-ng.conf debugger. Read about known problems and how to get started at https://czanik.blogs.balabit.com/2015/04/first-beta-of-syslog-ng-3-7-releas… syslog-ng at RedHat Summit -------------------------- BalaBit is a general sponsor of the Red Hat Summit. The event will be in Boston, June 23-26. http://www.redhat.com/summit/ Visit the BalaBit booth to talk about syslog-ng and log management! syslog-ng Hadoop support ------------------------ With the release of syslog-ng PE 5F3, support for Hadoop arrived. It will also be part of the upcoming syslog-ng OSE 3.7 release. Read more about how syslog-ng can send logs to a HDFS destination: https://jluby.blogs.balabit.com/2015/04/08/sending-logs-to-hadoop-using-sys… With this solution BalaBit became a certified MapR technology partner: https://www.mapr.com/partners/partner/balabit-it-security syslog-ng in Google Summer of Code 2015 --------------------------------------- This year we participate in GSoC again. The syslog-ng team received many good proposals, it will be a tough choice to decide on the finalists. We plan to have an IRC meeting with the participation of the winning GSoC students and members of the syslog-ng community. A separate e-mail will announce the exact time. The project and idea list is available at https://github.com/balabit/syslog-ng/wiki/GSoC2015-idea-&-project-list Forwarding name value pairs using RFC5424 ----------------------------------------- Recently many users asked how to forward the name of a file source over the network. It can easily be done using the RFC5424 syslog protocol. The same method can also be used to create and forward name value pairs between servers in general. Read more about it at https://czanik.blogs.balabit.com/2015/03/using-rfc5424-syslog-to-forward-fi… syslog-ng @ LOADays 2015 ------------------------ This year syslog-ng was presented again at LOADays, a small but very exciting Linux admin conference in Antwerp. The basics of syslog-ng, the importance of name value pairs and the latest developments in syslog-ng were presented. For details, check: https://czanik.blogs.balabit.com/2015/04/loadays-2015-logging-elasticsearch… NEW RELEASES syslog-ng 3.7.0beta1: https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.7.0beta1 Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

syslog-ng memory leak ? valgrind attached
by Jim Hendrick 29 Apr '15

29 Apr '15

From 3.6.2 OSE built with json & redis Seeing steady memory growth (> 10 GB after ~ a day at ~7000 EPS load) This was from a few minutes of runtime. (~5) Is this a known issue or did I trip over something? Thanks! Jim

5 4

Re: [syslog-ng] syslog-ng memory leak ? valgrind attached
by Jim Hendrick 29 Apr '15

29 Apr '15

I'll try this tomorrow. Took today off to get some work done on the house. Thanks,Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: "Czanik, Péter" <peter.czanik(a)balabit.com> Date: 04/29/2015 9:57 AM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng memory leak ? valgrind attached For the redis destination it's a know problem, was quickly fixed after 3.6.2 release: https://github.com/balabit/syslog-ng/commit/53606941b1b6b6726c4945f3560984a… If you use my unofficial syslog-ng packages, let me know, and I can add this patch. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik On Wed, Apr 29, 2015 at 3:49 PM, Jim Hendrick <jrhendri(a)roadrunner.com> wrote: From 3.6.2 OSE built with json & redis Seeing steady memory growth (> 10 GB after ~ a day at ~7000 EPS load) This was from a few minutes of runtime. (~5) Is this a known issue or did I trip over something? Thanks! Jim ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

Re: [syslog-ng] syslog-ng memory leak ? valgrind attached
by Jim Hendrick 29 Apr '15

29 Apr '15

Queues are often full. (separate issue ). Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Fabien Wernli <wernli(a)in2p3.fr> Date: 04/29/2015 9:55 AM (GMT-05:00) To: syslog-ng(a)lists.balabit.hu Subject: Re: [syslog-ng] syslog-ng memory leak ? valgrind attached Hi Jim, On Wed, Apr 29, 2015 at 09:49:05AM -0400, Jim Hendrick wrote: > From 3.6.2 OSE built with json & redis > > Seeing steady memory growth (> 10 GB after ~ a day at ~7000 EPS load) > > This was from a few minutes of runtime. (~5) > > Is this a known issue or did I trip over something? We're having similar issues when the workload tickles your figures. One possible explanation seems to be that the memory from queues is never released [1]. Are your queues always empty or do they do regular work? Oh, then there was this one too [2] [1] https://github.com/balabit/syslog-ng/issues/287 [2] https://github.com/balabit/syslog-ng/issues/446 ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

syslog-ng.conf configuration for log files every 15 minutes
by Akshay Surve 28 Apr '15

28 Apr '15

Hi, Currently, I have used the syslog-ng.conf to setup hourly files using the following config - destination d_clicks {file("/var/log/local-pid$PID-click-$YEAR-$MONTH-$DAY-$HOUR.log");}; Is there a way I could get files every 15 minutes? Best, Akshay

2 1

my fedora/epel packages update to use riemann-c-client 1.6.1
by Czanik, Péter 27 Apr '15

27 Apr '15

Hi, I just finished updating my fedora/epel packages on copr to riemann-c-client 1.6.1. This version brings gnutls support to riemann-c-client. As personally I don't have access to a riemann server, please test it and let me know if it works for you (I'm also interested in success stories, not just failures :-) ). https://copr.fedoraproject.org/coprs/czanik/syslog-ng36epel6/ https://copr.fedoraproject.org/coprs/czanik/syslog-ng36/ https://copr.fedoraproject.org/coprs/czanik/syslog-ng37/ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0