- syslog-ng - lists.balabit.hu

Re: [syslog-ng] Feedback for GSoC project - RIak Destination for Syslog-ng
by Gergely Nagy 05 May '15

05 May '15

>>>>> "Christopher" == Christopher Meiklejohn <cmeiklejohn(a)basho.com> writes: >> The intention is to store log messages in each element of the set: >> either as a string (syslog or json, or whatever else the user sees fit), >> or as a map of key-value pairs (where values themselves can be maps >> too). >> >> On average, the log messages are a few kilobytes in size. There may be >> exceptions, but >1mb ones are fairly rare. How much data the set would >> hold... now that's a question that can't really be answered. It is >> really up to the syslog-ng user to configure that. Christopher> I’m referring to the size of the entire set, not the objects that will be members of Christopher> the set. Therefore, the performance penalty seen when using large objects would Christopher> be observed as soon as the size of the entire set (or map) has reached ~1 MB. Christopher> Given that restriction, I’d imagine you would only be able to store a few messages Christopher> in each set. That granularity seems like you are no longer getting the benefits Christopher> of the set. The granularity is configurable by the user: if they have small (say, a few hundred byte long) messages, then we can store a reasonable amount of them in a single Set. For example, assuming an average length of 384 bytes / message (the longest line in today's logs on my laptop), a Set would be able to store about 2k messages. That's not too bad. Christopher> Additionally, the primary benefit of the data types in Riak is that they converge Christopher> deterministically when dealing with concurrent Christopher> operations. Not only that: using sets makes the keys predictable. If I want to retrieve the logs, with sets, I can retrieve the 2015-05-01T15:12:10-T15:12:15 key for example, and have all the logs from those 5 seconds. If I used one message per Riak object, it would be much harder to read the data back. There may be multiple threads adding to the same set, so the deterministic convergence is useful still. Christopher> I’m curious if the set is the right choice here; could Christopher> you just use a custom set format inside of a normal Christopher> Riak object That's an option worth considering, yes. Thanks! Christopher> (or store one message per Riak object, given the write will be an Christopher> immutable log entry?) That's the first goal of the project, because that's the easiest and most straightforward to implement. The downside of one message per Riak object is that it's hard to retrieve the data, because making the keys predictable is not going to be easy. Boxing a few hundred (or few thousand) together into a Set has the advantage of making the keys predictable at the cost of transferring more data to the client when looking for a subset of the logs within the set. -- |8]

1 0

Re: [syslog-ng] Feedback for GSoC project - RIak Destination for Syslog-ng
by Gergely Nagy 05 May '15

05 May '15

>>>>> "Christopher" == Christopher Meiklejohn <cmeiklejohn(a)basho.com> writes: Christopher> I’m a bit concerned with your use of the set embedded in the Christopher> map. The original idea was to use a Set directly. The Set-in-Map thing was just a thought experiment (Map-in-Set would make more sense). Christopher> Large objects have traditionally been a big problem in Riak due Christopher> to the use of distributed Erlang and head of line blocking. I’m Christopher> curious if you could elaborate on what type of data you will be Christopher> storing in the set: how big you expect each item to be, how big you Christopher> expect the map to be, and the overall layout of data inside of the Christopher> data structure. The intention is to store log messages in each element of the set: either as a string (syslog or json, or whatever else the user sees fit), or as a map of key-value pairs (where values themselves can be maps too). On average, the log messages are a few kilobytes in size. There may be exceptions, but >1mb ones are fairly rare. How much data the set would hold... now that's a question that can't really be answered. It is really up to the syslog-ng user to configure that. Hope that made it a little more clear. -- |8], mentor for the riak destination for syslog-ng GSoC project

1 0

Feedback for GSoC project - RIak Destination for Syslog-ng
by Parth Oberoi 02 May '15

02 May '15

Hello Everyone, My proposal[0] for adding Riak as a destination to Syslog-ng was accepted for GSoC2015, so before my coding period begins i.e. 25th May, I wanted to do some ground work and take some feedback from the potential users. The proposal[0] contains all the initial implementation details which might require improvement . Therefore if some of you could take a look and give me some feedback on it, would help me greatly. Below are a few links to important snippets of the proposal. [0] - https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-Riak-destinati… *Goals for the project :-* https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-Riak-destinati… *Development Tools :-* https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-Riak-destinati… *Deliverables :-* https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-Riak-destinati… *My scheduled Plan:-* https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-Riak-destinati… Thanks in advance. Regards, Parth Oberoi (@htrap)

1 0

insider 2015-04: 3.7 beta1; RedHat Summit; Hadoop; GSoC; RFC5424;
by Czanik, Péter 30 Apr '15

30 Apr '15

Dear syslog-ng users, This is the 41th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng 3.7 beta1 released ---------------------------- In the first beta of syslog-ng OSE 3.7, the Java and Python destinations were migrated from the syslog-ng incubator to the syslog-ng core. Also, syslog-ng now includes an interactive syslog-ng.conf debugger. Read about known problems and how to get started at https://czanik.blogs.balabit.com/2015/04/first-beta-of-syslog-ng-3-7-releas… syslog-ng at RedHat Summit -------------------------- BalaBit is a general sponsor of the Red Hat Summit. The event will be in Boston, June 23-26. http://www.redhat.com/summit/ Visit the BalaBit booth to talk about syslog-ng and log management! syslog-ng Hadoop support ------------------------ With the release of syslog-ng PE 5F3, support for Hadoop arrived. It will also be part of the upcoming syslog-ng OSE 3.7 release. Read more about how syslog-ng can send logs to a HDFS destination: https://jluby.blogs.balabit.com/2015/04/08/sending-logs-to-hadoop-using-sys… With this solution BalaBit became a certified MapR technology partner: https://www.mapr.com/partners/partner/balabit-it-security syslog-ng in Google Summer of Code 2015 --------------------------------------- This year we participate in GSoC again. The syslog-ng team received many good proposals, it will be a tough choice to decide on the finalists. We plan to have an IRC meeting with the participation of the winning GSoC students and members of the syslog-ng community. A separate e-mail will announce the exact time. The project and idea list is available at https://github.com/balabit/syslog-ng/wiki/GSoC2015-idea-&-project-list Forwarding name value pairs using RFC5424 ----------------------------------------- Recently many users asked how to forward the name of a file source over the network. It can easily be done using the RFC5424 syslog protocol. The same method can also be used to create and forward name value pairs between servers in general. Read more about it at https://czanik.blogs.balabit.com/2015/03/using-rfc5424-syslog-to-forward-fi… syslog-ng @ LOADays 2015 ------------------------ This year syslog-ng was presented again at LOADays, a small but very exciting Linux admin conference in Antwerp. The basics of syslog-ng, the importance of name value pairs and the latest developments in syslog-ng were presented. For details, check: https://czanik.blogs.balabit.com/2015/04/loadays-2015-logging-elasticsearch… NEW RELEASES syslog-ng 3.7.0beta1: https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.7.0beta1 Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

syslog-ng memory leak ? valgrind attached
by Jim Hendrick 29 Apr '15

29 Apr '15

From 3.6.2 OSE built with json & redis Seeing steady memory growth (> 10 GB after ~ a day at ~7000 EPS load) This was from a few minutes of runtime. (~5) Is this a known issue or did I trip over something? Thanks! Jim

5 4

Re: [syslog-ng] syslog-ng memory leak ? valgrind attached
by Jim Hendrick 29 Apr '15

29 Apr '15

I'll try this tomorrow. Took today off to get some work done on the house. Thanks,Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: "Czanik, Péter" <peter.czanik(a)balabit.com> Date: 04/29/2015 9:57 AM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng memory leak ? valgrind attached For the redis destination it's a know problem, was quickly fixed after 3.6.2 release: https://github.com/balabit/syslog-ng/commit/53606941b1b6b6726c4945f3560984a… If you use my unofficial syslog-ng packages, let me know, and I can add this patch. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik On Wed, Apr 29, 2015 at 3:49 PM, Jim Hendrick <jrhendri(a)roadrunner.com> wrote: From 3.6.2 OSE built with json & redis Seeing steady memory growth (> 10 GB after ~ a day at ~7000 EPS load) This was from a few minutes of runtime. (~5) Is this a known issue or did I trip over something? Thanks! Jim ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

Re: [syslog-ng] syslog-ng memory leak ? valgrind attached
by Jim Hendrick 29 Apr '15

29 Apr '15

Queues are often full. (separate issue ). Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Fabien Wernli <wernli(a)in2p3.fr> Date: 04/29/2015 9:55 AM (GMT-05:00) To: syslog-ng(a)lists.balabit.hu Subject: Re: [syslog-ng] syslog-ng memory leak ? valgrind attached Hi Jim, On Wed, Apr 29, 2015 at 09:49:05AM -0400, Jim Hendrick wrote: > From 3.6.2 OSE built with json & redis > > Seeing steady memory growth (> 10 GB after ~ a day at ~7000 EPS load) > > This was from a few minutes of runtime. (~5) > > Is this a known issue or did I trip over something? We're having similar issues when the workload tickles your figures. One possible explanation seems to be that the memory from queues is never released [1]. Are your queues always empty or do they do regular work? Oh, then there was this one too [2] [1] https://github.com/balabit/syslog-ng/issues/287 [2] https://github.com/balabit/syslog-ng/issues/446 ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

syslog-ng.conf configuration for log files every 15 minutes
by Akshay Surve 28 Apr '15

28 Apr '15

Hi, Currently, I have used the syslog-ng.conf to setup hourly files using the following config - destination d_clicks {file("/var/log/local-pid$PID-click-$YEAR-$MONTH-$DAY-$HOUR.log");}; Is there a way I could get files every 15 minutes? Best, Akshay

2 1

my fedora/epel packages update to use riemann-c-client 1.6.1
by Czanik, Péter 27 Apr '15

27 Apr '15

Hi, I just finished updating my fedora/epel packages on copr to riemann-c-client 1.6.1. This version brings gnutls support to riemann-c-client. As personally I don't have access to a riemann server, please test it and let me know if it works for you (I'm also interested in success stories, not just failures :-) ). https://copr.fedoraproject.org/coprs/czanik/syslog-ng36epel6/ https://copr.fedoraproject.org/coprs/czanik/syslog-ng36/ https://copr.fedoraproject.org/coprs/czanik/syslog-ng37/ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

getting started with 3.7 beta1
by Czanik, Péter 23 Apr '15

23 Apr '15

Hi, I put together a quick getting started for syslog-ng 3.7 beta1: https://czanik.blogs.balabit.com/2015/04/first-beta-of-syslog-ng-3-7-releas… If you run into trouble, don't hesitate writing about it here or posting a bugreport to https://github.com/balabit/syslog-ng/issues/ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0