- syslog-ng - lists.balabit.hu

Re: [syslog-ng] syslog-ng memory leak ? valgrind attached
by Jim Hendrick 29 Apr '15

29 Apr '15

I'll try this tomorrow. Took today off to get some work done on the house. Thanks,Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: "Czanik, Péter" <peter.czanik(a)balabit.com> Date: 04/29/2015 9:57 AM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng memory leak ? valgrind attached For the redis destination it's a know problem, was quickly fixed after 3.6.2 release: https://github.com/balabit/syslog-ng/commit/53606941b1b6b6726c4945f3560984a… If you use my unofficial syslog-ng packages, let me know, and I can add this patch. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik On Wed, Apr 29, 2015 at 3:49 PM, Jim Hendrick <jrhendri(a)roadrunner.com> wrote: From 3.6.2 OSE built with json & redis Seeing steady memory growth (> 10 GB after ~ a day at ~7000 EPS load) This was from a few minutes of runtime. (~5) Is this a known issue or did I trip over something? Thanks! Jim ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

Re: [syslog-ng] syslog-ng memory leak ? valgrind attached
by Jim Hendrick 29 Apr '15

29 Apr '15

Queues are often full. (separate issue ). Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Fabien Wernli <wernli(a)in2p3.fr> Date: 04/29/2015 9:55 AM (GMT-05:00) To: syslog-ng(a)lists.balabit.hu Subject: Re: [syslog-ng] syslog-ng memory leak ? valgrind attached Hi Jim, On Wed, Apr 29, 2015 at 09:49:05AM -0400, Jim Hendrick wrote: > From 3.6.2 OSE built with json & redis > > Seeing steady memory growth (> 10 GB after ~ a day at ~7000 EPS load) > > This was from a few minutes of runtime. (~5) > > Is this a known issue or did I trip over something? We're having similar issues when the workload tickles your figures. One possible explanation seems to be that the memory from queues is never released [1]. Are your queues always empty or do they do regular work? Oh, then there was this one too [2] [1] https://github.com/balabit/syslog-ng/issues/287 [2] https://github.com/balabit/syslog-ng/issues/446 ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

syslog-ng.conf configuration for log files every 15 minutes
by Akshay Surve 28 Apr '15

28 Apr '15

Hi, Currently, I have used the syslog-ng.conf to setup hourly files using the following config - destination d_clicks {file("/var/log/local-pid$PID-click-$YEAR-$MONTH-$DAY-$HOUR.log");}; Is there a way I could get files every 15 minutes? Best, Akshay

2 1

my fedora/epel packages update to use riemann-c-client 1.6.1
by Czanik, Péter 27 Apr '15

27 Apr '15

Hi, I just finished updating my fedora/epel packages on copr to riemann-c-client 1.6.1. This version brings gnutls support to riemann-c-client. As personally I don't have access to a riemann server, please test it and let me know if it works for you (I'm also interested in success stories, not just failures :-) ). https://copr.fedoraproject.org/coprs/czanik/syslog-ng36epel6/ https://copr.fedoraproject.org/coprs/czanik/syslog-ng36/ https://copr.fedoraproject.org/coprs/czanik/syslog-ng37/ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

getting started with 3.7 beta1
by Czanik, Péter 23 Apr '15

23 Apr '15

Hi, I put together a quick getting started for syslog-ng 3.7 beta1: https://czanik.blogs.balabit.com/2015/04/first-beta-of-syslog-ng-3-7-releas… If you run into trouble, don't hesitate writing about it here or posting a bugreport to https://github.com/balabit/syslog-ng/issues/ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

syslog-ng 3.7 beta1 packages for Fedora and EPEL7
by Czanik, Péter 22 Apr '15

22 Apr '15

Hi, If you have some time, please check our latest beta on Fedora or EPEL7! Packages are available at: https://copr.fedoraproject.org/coprs/czanik/syslog-ng37/ * Tue Apr 21 2015 Peter Czanik <peter.czanik(a)balabit.com> - 3.6.0beta1-3 - update to 3.7.0beta1 - temp: disable man pages (missing from release) - temp: disable smp build - add python support - add java support - move json support to the core package due to system() - update configuration version to 3.7 EPEL6 is not supported rigth now due to: https://github.com/balabit/syslog-ng/issues/317 Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

Issues running syslog-ng-ctl program locally to determine logging issues
by Andrew Bell 20 Apr '15

20 Apr '15

Thank you very much! Adding that flag/setting worked and I was able to see a control socket file appear in my local syslog-ng instance (in a location where I did have write permission) . Using syslog-ng-ctl with this control file, I was able to finally see some helpful output on logs that were processed and/or dropped by syslog-ng. Still working on figuring out the root cause behind why logs aren't being processed by syslog-ng, but getting syslog-ng-ctl to work correctly has assisted greatly with troubleshooting. Thanks again! -Andrew

1 0

Re: [syslog-ng] Issues running syslog-ng-ctl program locally to determine logging issues
by Andrew Bell 17 Apr '15

17 Apr '15

Also would like to throw out that I am open to using strace or some other debug hook program to figure out the cause here if that would help better, just would like a pointer as to how I would go about using it to identify dropped logs. I don't believe running syslog-ng with the debug setting on its own indicates if a logged event has been dropped or not (but I suppose it could indicate that the log never made its way over). Thanks, Andrew From: Andrew Bell Sent: Wednesday, April 15, 2015 11:55 AM To: syslog-ng(a)lists.balabit.hu Subject: Issues running syslog-ng-ctl program locally to determine logging issues Hello, Wondering if someone can help me with an issue that I am currently having. I am running syslog-ng as a local non-root user (I have it installed to a local instance under my home directory) and am trying to use the syslog-ng-ctl program in order to diagnose why certain log files are not passing through syslog-ng to their log destination (trying to rule out if it is a problem with the location where the logs originate OR if there is an issue with the syslog-ng setup and it is dropping or otherwise failing to forward those incoming logs). However, when I try to run syslog-ng-ctl under my local account I get the following error, where it appears to reference a control socket that is located in a directory that is owned and operated by the root account (I should note that I am unable to sudo as root with my setup on the system). /home/abell/syslog-ng-install/opt/syslog-ng/sbin$ ./syslog-ng-ctl verbose Error connecting control socket, socket='/opt/syslog-ng/var/run/syslog-ng.ctl', error='No such file or directory' This type of error above is also appearing when I run syslog-ng program itself with the debug ('-d -v') options set Error opening control socket, bind() failed; socket='/opt/syslog-ng/var/run/syslog-ng.ctl', error='No such file or directory (2)' I did some searching online and did find some manuals on syslog-ng-ctl that said I could specify a different directory where the control socket resides to one that isn't in the root directory using the "-c" flag. This mailing list post also mentioned that running a local instance of syslog-ng-ctl should automatically create a local instance of the control socket file located in the /var directory where the local instance resides (so in my case, I imagine it would be /home/abell/syslog-ng-install/var ???). Here's the reference link -> https://lists.balabit.hu/pipermail/syslog-ng/2009-June/013027.html - there weren't any next messages to this thread so it is difficult to tell if this solution worked for the original author here or not... So I created that "/var" directory as he suggested but I still don't see the .ctl file created when I go to stop and restart syslog-ng ... and so I don't really have anything to specify for the "-c" option as an alternative. I tried touching the control socket file in that directory as a zero byte file, but still nothing /home/abell/syslog-ng-install/opt/syslog-ng/var$ touch syslog-ng.ctl /home/abell/syslog-ng-install/opt/syslog-ng/var$ cd .. /home/abell/syslog-ng-install/opt/syslog-ng$ cd sbin /home/abell/syslog-ng-install/opt/syslog-ng/sbin$ ./syslog-ng-ctl verbose --set=on -c /home/abell/syslog-ng-install/opt/syslog-ng/var/syslog-ng.ctl Error connecting control socket, socket='/home/abell/syslog-ng-install/opt/syslog-ng/var/syslog-ng.ctl', error='Connection refused' Any ideas as to how I can get this control socket file created on my local syslog-ng - or if I can tell syslog-ng itself to specify spinning up the control socket file itself in a different location than the root one? I'd really like to be able to use this tool to figure out what is going on with my syslog-ng setup. Let me know if there's any other details or information that I can provide. Thanks, Andrew

4 4

Issues running syslog-ng-ctl program locally to determine logging issues
by Andrew Bell 15 Apr '15

15 Apr '15

Hello, Wondering if someone can help me with an issue that I am currently having. I am running syslog-ng as a local non-root user (I have it installed to a local instance under my home directory) and am trying to use the syslog-ng-ctl program in order to diagnose why certain log files are not passing through syslog-ng to their log destination (trying to rule out if it is a problem with the location where the logs originate OR if there is an issue with the syslog-ng setup and it is dropping or otherwise failing to forward those incoming logs). However, when I try to run syslog-ng-ctl under my local account I get the following error, where it appears to reference a control socket that is located in a directory that is owned and operated by the root account (I should note that I am unable to sudo as root with my setup on the system). /home/abell/syslog-ng-install/opt/syslog-ng/sbin$ ./syslog-ng-ctl verbose Error connecting control socket, socket='/opt/syslog-ng/var/run/syslog-ng.ctl', error='No such file or directory' This type of error above is also appearing when I run syslog-ng program itself with the debug ('-d -v') options set Error opening control socket, bind() failed; socket='/opt/syslog-ng/var/run/syslog-ng.ctl', error='No such file or directory (2)' I did some searching online and did find some manuals on syslog-ng-ctl that said I could specify a different directory where the control socket resides to one that isn't in the root directory using the "-c" flag. This mailing list post also mentioned that running a local instance of syslog-ng-ctl should automatically create a local instance of the control socket file located in the /var directory where the local instance resides (so in my case, I imagine it would be /home/abell/syslog-ng-install/var ???). Here's the reference link -> https://lists.balabit.hu/pipermail/syslog-ng/2009-June/013027.html - there weren't any next messages to this thread so it is difficult to tell if this solution worked for the original author here or not... So I created that "/var" directory as he suggested but I still don't see the .ctl file created when I go to stop and restart syslog-ng ... and so I don't really have anything to specify for the "-c" option as an alternative. I tried touching the control socket file in that directory as a zero byte file, but still nothing /home/abell/syslog-ng-install/opt/syslog-ng/var$ touch syslog-ng.ctl /home/abell/syslog-ng-install/opt/syslog-ng/var$ cd .. /home/abell/syslog-ng-install/opt/syslog-ng$ cd sbin /home/abell/syslog-ng-install/opt/syslog-ng/sbin$ ./syslog-ng-ctl verbose --set=on -c /home/abell/syslog-ng-install/opt/syslog-ng/var/syslog-ng.ctl Error connecting control socket, socket='/home/abell/syslog-ng-install/opt/syslog-ng/var/syslog-ng.ctl', error='Connection refused' Any ideas as to how I can get this control socket file created on my local syslog-ng - or if I can tell syslog-ng itself to specify spinning up the control socket file itself in a different location than the root one? I'd really like to be able to use this tool to figure out what is going on with my syslog-ng setup. Let me know if there's any other details or information that I can provide. Thanks, Andrew

2 1

syslog-ng-3.7beta1 released
by Czanik, Péter 14 Apr '15

14 Apr '15

3.7.0beta1 This is the first beta release of the upcoming syslog-ng OSE 3.7 branch. Further releases will focus on fixes and small Getting started ... documentations. Changes compared to the previous alpha release: Features - Added batched event sending support for riemann destination driver which makes the riemann destination respect flush-lines(), and send event in batches of configurable amount (defaults to 1). In case of an error, all messages within the batch will be dropped. Dropped messages, and messages that result in formatting errors do not count towards the batch size. There is no timeout, but messages will be flushed upon deinit. - Added IPv6 netmask filter for selecting only messages sent by a host whose IP address belongs to the specified IPv6 subnet. - Added syslog-ng debug bundle generator script for collecting debug related information. - Added a new macro, called HOSTID which is a 32-bit number generated by a cryptographically secure PRNG. Its purpose is to identify the syslog-ng host, thus it is the same for every message generated on the same host. - Added a new macro, called UNIQID which is a practically unique ID generated from the HOSTID and the RCPTID in the format of HOSTID@RCPTID. Uniqid is a derived value: it is built up from the always available hostid and the optional rcptid. In other words: uniqid is an extension over rcptid. For that reason use-rcptid has been deprecated and use-uniqid could be use instead. - Added a reset option to syslog-ng-ctl stats. With this option the non-stored stats counters can be zeroed. - Java-destination driver ported from syslog-ng-incubator. Purpose of having Java destination driver is to provide the right way to support all player in the "Java related logging ecosystem" (Kafka, HDFS, ElasticSearch, ...). Java dest driver is a special driver, a bridge between the C and the Java world from syslog-ng point of view. - Python language support is ported from syslog-ng incubator and has been completely reworked. Now, it is possible to implement template functions in Python language and also destination drivers. Main purpose of supporting Python language is to implement a nice interactive syslog-ng config debugger for syslog-ng. - New builtin interactive syslog-ng.conf debugger implemented for syslog-ng. The debugger has a Python frontend which contains a full Completer (just press TABs and works like bash) Enhancements - Extended the set of supported characters to every printable ASCII's except ., [ and ] in extract-prefix for json-parser(). - OpenSSL set as a hard dependency for syslog-ng because the newly added hostid and uniqid features requires a CPRNG provided by OpenSSL. After OpenSSL is a hard dependency - non-embedded crypto lib is not a real option, so the support of having such a crypto lib discontinued - all SSL-dependent features enabled by default - Added string-delimiters option to csvparser to support multi character delimiters in CSV parsing. - Upgrade RabbitMQ submodule to the upstream. - Extended rcpt-id to 64 bits (formerly it was 48 bits). Fixes - Fixed the encoding of characters below 32 if escaping is enabled in templates. Templated outputs never contained references to characters below 32, essentially they were dropped from the output for two reasons: - the prefixing backslash was removed from the code - the format_uint32_padded() function produced no outputs in base 8 - Fixed afstomp destination port issue. It always tried to connect to the port 0. - Fixed compilation where the monolitic libsystemd was not available. - Fixed memleak in db-parser which could happen at every reload. - Fixed a class of rule conflicts in db-parser: Because an error in the pdb load algorithms, some rules would conflict which shouldn't have done that. The problem was that several programs would use the same RADIX tree to store their patterns. Merging independent programs meant that if they the same pattern listed, it would clash, even though their $PROGRAM is different. There were multiple issues: - we looked up pattern string directly, even they might have contained @parser <https://github.com/parser>@ references. It was simply not designed that way and only worked as long as we didn't have the possibility to use parsers in program names - we could merge programs with the same prefix, e.g. su, supervise/syslog-ng and supervise/logindexd would clash, on "su", which is a common prefix for all three. The solution involved in using a separate hash table for loading, which at the end is turned into the radix tree. - Fixed a crash around affile at the first message delivery when templates were used. - Excluded "tags" from riemann destination driver as an attribute which conflicts with reserved keyword - Fixed a docbook related compilation error: there was a hardcoded path that caused build to fail if docbook is not on that path. Debian based platforms did not affected by this problem. Now a new option was created for ./configure that is --enable-manpages that enables the generation of manpages using docbook from online source. '--with-docbook=PATH' gives you the opportunity to specify the path for your own installed docbook. Developer notes - filter: fix external filter plugin lookup The filter_plugin rule expected an LL_IDENTIFIER and filter_comparison expected a string which in turn is an LL_IDENTIFIER or LL_STRING. It caused a conflict in the grammar which prevented to load external filter plugins. Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Balazs Scheidler, Botond Borsits, Fabien Wernli, Gergely Nagy, Gergo Nagy, Gyorgy Pasztor, Istvan Adam Mozes, Laszlo Budai, Manikandan-Selvaganesh, Peter Czanik, Robert Fekete, Tibor Benke, Viktor Juhasz, Vincent Bernat, Zoltan Fried, Zoltan Pallagi. — View it on GitHub <https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.7.0beta1>.

1 1

Re: [syslog-ng] db-parser reuse for multiple logs?
by Jim Hendrick 14 Apr '15

14 Apr '15

I think it would depend on the filter. If it uses something that is readily available then maybe filter first. If is a match within the message, then possibly not. The patterndb parser is very fast. I have a single system pulling 24 fields out of a 7000 event per second load and it shows no sign of stress. Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Mikkel Leth Carlsen <mlca(a)tdc.dk> Date: 04/10/2015 2:11 AM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] db-parser reuse for multiple logs? Hi Evan Thanks! I'm not too familiar with how syslog-ng actually implements the configuration and processes syslog messages, but I would think that parsing before filtering could have an impact on performance? I.e. I would want to exclude unwanted data before applying the parser - which I assume is a more costly operation even if it does not match? Perhaps by inserting: filter(filter_host1_or_host2); before the parser below? /Mikkel > -----Oprindelig meddelelse----- > Fra: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng- > bounces(a)lists.balabit.hu] På vegne af Evan Rempel > Sendt: 8. april 2015 17:38 > Til: syslog-ng(a)lists.balabit.hu > Emne: Re: [syslog-ng] db-parser reuse for multiple logs? > > I don't have an answer to your actual question, however, you can work > around it by > > log { > source(src_udp); > parser(myparser); > log { > filter(filter_host1); > destination(dst_host1); > flags(final); > }; > log { > filter(filter_host2); > destination(dst_host2); > flags(final); > }; > }; > > Hope that helps. > > Evan. > > On 04/08/2015 04:31 AM, Mikkel Leth Carlsen wrote: > > Hi > > > > Are db-parsers defined in syslog-ng configurations not reusable for > multiple logs? A simplified example (syslog 3.6.2): > > > > parser myparser { > > db_parser( > > file("/usr/local/etc/patterndb.d/myparser.xml") > > ); > > }; > > > > template mytemplate { > > template("${A};${B};${C}\n"); > > } > > > > filter filter_host1 { > > netmask(10.0.0.1/255.255.255.255); > > }; > > > > filter filter_host2 { > > netmask(10.0.0.2/255.255.255.255); > > }; > > > > destination dst_host1 { > > file("host1.log" perm(0644) template(mytemplate)); > > }; > > > > > > destination dst_host2 { > > file("host2.log" perm(0644) template(mytemplate)); > > }; > > > > log { > > source(src_udp); > > filter(filter_host1); > > parser(myparser); > > destination(dst_host1); > > flags(final); > > }; > > > > log { > > source(src_udp); > > filter(filter_host2); > > parser(myparser); > > destination(dst_host2); > > flags(final); > > }; > > > > This seems to work as expected and 'syslog-ng -s' does not report any > problems, but I see the following in the syslog-ng internal log: > > > > Internal error, duplicate configuration elements refer to the same > persistent config; name='db- > parser(/usr/local/etc/patterndb.d/myparser.xml)' > > Internal error, duplicate configuration elements refer to the same > persistent config; name='db- > parser(/usr/local/etc/patterndb.d/myparser.xml)' > > > > /Mikkel > > > _______________________________________________________________________ > _______ > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > _______________________________________________________________________ > _______ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

5 7

Re: [syslog-ng] Understanding Sources & Destinations
by Jim Hendrick 10 Apr '15

10 Apr '15

This could be used for example to process historical log data as part of an investigation. Say you have a search tool (like ELK or something that needs preprocessing to json, sql, etc). Being able to send those logs through a command line version that does the exact same processing as the real time one would be extremely valuable. Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: "Scheidler, Balázs" <balazs.scheidler(a)balabit.com> Date: 04/10/2015 12:39 AM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] Understanding Sources & Destinations Hi, The command line tool should work primarily between stdin/stdout just like for instance awk. We may make it possible to fire up tcp listeners but the primary use-case is batch processing already received data. Another use case is to debug configuration snippets in a faster way than edit/reload/test cycle that nromal syslog-ng offers. On Apr 10, 2015 12:55 AM, "Asadullah Hussain" <asadxflow(a)gmail.com> wrote: Hi Guys, I am working to understand the task of developing syslog-ng as a command line tool (GSoC2015) and I am trying to figure what "possible outputs" the command line tool will take. As per my understanding the input & outputs to syslog are defined as Sources & Destinations (which are places where applications output their logs from respectively). I have read the documentation about sources & destinations [Table 6.2 ](http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guides/en/syslog-ng-ose-guide-admin/html-single/index.html#chapter-sources) and my understanding is that syslog has following 3 types of sources: files, streams on tcp/ip sockets and stdout from a program. and 3 types of destinations: files or remote hosts (TCP/IP socket) and applications (mentioned in section 7). So are these all possible types of inputs/outputs that syslog-ng can have? which have to be supported by the command line tool. Or am I missing some points? -- Cheers, Asadullah Hussain ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

2 1

db-parser reuse for multiple logs?
by Mikkel Leth Carlsen 10 Apr '15

10 Apr '15

Hi Are db-parsers defined in syslog-ng configurations not reusable for multiple logs? A simplified example (syslog 3.6.2): parser myparser { db_parser( file("/usr/local/etc/patterndb.d/myparser.xml") ); }; template mytemplate { template("${A};${B};${C}\n"); } filter filter_host1 { netmask(10.0.0.1/255.255.255.255); }; filter filter_host2 { netmask(10.0.0.2/255.255.255.255); }; destination dst_host1 { file("host1.log" perm(0644) template(mytemplate)); }; destination dst_host2 { file("host2.log" perm(0644) template(mytemplate)); }; log { source(src_udp); filter(filter_host1); parser(myparser); destination(dst_host1); flags(final); }; log { source(src_udp); filter(filter_host2); parser(myparser); destination(dst_host2); flags(final); }; This seems to work as expected and 'syslog-ng -s' does not report any problems, but I see the following in the syslog-ng internal log: Internal error, duplicate configuration elements refer to the same persistent config; name='db-parser(/usr/local/etc/patterndb.d/myparser.xml)' Internal error, duplicate configuration elements refer to the same persistent config; name='db-parser(/usr/local/etc/patterndb.d/myparser.xml)' /Mikkel

4 4

Understanding Sources & Destinations
by Asadullah Hussain 10 Apr '15

10 Apr '15

Hi Guys, I am working to understand the task of developing syslog-ng as a command line tool (GSoC2015) and I am trying to figure what "possible outputs" the command line tool will take. As per my understanding the input & outputs to syslog are defined as Sources & Destinations (which are places where applications output their logs from respectively). I have read the documentation about sources & destinations [Table 6.2 ]( http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guid…) and my understanding is that syslog has following 3 types of sources: files, streams on tcp/ip sockets and stdout from a program. and 3 types of destinations: files or remote hosts (TCP/IP socket) and applications (mentioned in section 7). So are these all possible types of inputs/outputs that syslog-ng can have? which have to be supported by the command line tool. Or am I missing some points? -- Cheers, Asadullah Hussain

2 1

UDP errors and lost UDP messages
by Matt Zagrabelny 07 Apr '15

07 Apr '15

Greetings list! Using syslog-ng 3.1 with Debian Squeeze, 2.6.32-5-amd64. The system has 8GB of RAM. I'm losing some UDP logs. I know to not use UDP - we use TLS for our Debian systems, but our Cisco gear leaves us with few options. According to netstat, the rate is anywhere from 600 to 3000 UDP errors per second. Using a tcpdump query of "dst port 514" show about the same rate of UDP traffic coming to the system. I've bumped the buffer size according to various docs: $ head -n -0 /proc/sys/net/core/[rw]mem_* ==> /proc/sys/net/core/rmem_default <== 16777216 ==> /proc/sys/net/core/rmem_max <== 16777216 ==> /proc/sys/net/core/wmem_default <== 16777216 ==> /proc/sys/net/core/wmem_max <== 16777216 And the udp specific memory limits: $ head -n -0 /proc/sys/net/ipv4/*udp* ==> /proc/sys/net/ipv4/udp_mem <== 768384 1024512 1536768 ==> /proc/sys/net/ipv4/udp_rmem_min <== 16777216 ==> /proc/sys/net/ipv4/udp_wmem_min <== 16777216 My UDP source for syslog-ng is also using a larger buffer: $ grep -A4 -B1 'udp(' /etc/syslog-ng/syslog-ng.conf source s_udp { udp( keep_hostname(yes) so_rcvbuf(16777216) ); }; According to syslog-ng-ctl stats the system is processing ~270 UDP messages per second. This hasn't really changed since I've made the kernel variable tweaks, nor after changing the so_rcvbuf size either. Any ideas of what to look for next? Thanks! -m

3 4

so_rcvbuf default value of 0
by Matt Zagrabelny 07 Apr '15

07 Apr '15

Greetings, Just looking for confirmation about the so_rcvbuf parameter for the udp() source driver. The default is 0. Does that mean that syslog-ng uses the default value from the kernel? ie. cat /proc/sys/net/core/rmem_default 124928 If that is true, then perhaps the syslog-ng admin docs could mention that. I've scanned "man 7 socket" and don't seen anything that suggests that a value of 0 for so_rcvbuf means that the socket will inherit the value. Thanks for the help! -m

2 2

Compiling Github Source (not Balabit source)
by Asadullah Hussain 03 Apr '15

03 Apr '15

Hello, I have recently started exploring the syslog source with plan of contributing in future. I am able to run syslog using the source from Balabit site. But is there a way to compile the Github source? I have tried using Autoconf/Autoscan to generate the configure files but without success. I just have future pull requests in mind as Github source and Balabit source differ in a few files). -- Cheers, Asad

3 4

Re: [syslog-ng] Compiling Github Source (not Balabit source)
by Laszlo Budai 03 Apr '15

03 Apr '15

Hi Asad, I've created a docker image which can helps you: https://registry.hub.docker.com/u/lbudai/docker-syslog-ng-build-env/ L. Sent from my HTC ----- Reply message ----- From: "Asadullah Hussain" <asadxflow(a)gmail.com> To: <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] Compiling Github Source (not Balabit source) Date: Thu, Apr 2, 2015 11:33 Hello, I have recently started exploring the syslog source with plan of contributing in future. I am able to run syslog using the source from Balabit site. But is there a way to compile the Github source? I have tried using Autoconf/Autoscan to generate the configure files but without success. I just have future pull requests in mind as Github source and Balabit source differ in a few files). -- Cheers, Asad ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

Re: [syslog-ng] Compiling Github Source (not Balabit source) (Asadullah Hussain)
by Anshul Kushwaha 02 Apr '15

02 Apr '15

Hi Asad, You can try running autoconf ./autogen.sh ./configure make make install for compiling from the github source. This worked well for me. If there are errors while running autoconf, try posting them here or on irc. Regards, Anshul Kushwaha

1 0

Stats missing from remote sources in syslog-ng 3.5?
by Pradeep Sanders 31 Mar '15

31 Mar '15

Hello, I'm seeing a similar problem to what was reported here: http://communities.balabit.com/balabit/topics/stats-from-syslog-remote-sour… <http://communities.balabit.com/balabit/topics/stats-from-syslog-remote-sour…> I'm trying to upgrade from the RHEL 6 default 3.2 version to 3.5 or 3.6, in order to enable threaded support. I've tried a 3.6 build from the following location: https://copr-be.cloud.fedoraproject.org/results/czanik/syslog-ng36epel6/epe… <https://copr-be.cloud.fedoraproject.org/results/czanik/syslog-ng36epel6/epe…> However, this build consumes all my host memory (48GB) in a matter of hours before starting on swap space. I've tried a 3.5.4 and a 3.5.6 package but both of them exhibit the same issue, with remote sources not reported in "syslog-ng-ctl stats". Does anyone have any advice on how I might fix the stats for remote sources in 3.5? Thanks, -Pradeep

2 1

interrogate daemon for parameter values?
by Matt Zagrabelny 30 Mar '15

30 Mar '15

Greetings, Is there a way to interrogate the running daemon to find out what the *effective* value is for things? For instance, I'd like to know what the effective value of so_rcvbuf is for a udp source. cat /proc/sys/net/core/rmem_default 1024 syslog-ng.conf: udp(); # uses default value of 0 service syslog-ng start echo 512 > /proc/sys/net/core/rmem_default Here is where I'd like to interrogate the running daemon and find that the udp driver is using a value of 1024 for so_rcvbuf. Any chance of doing that? Thanks! -m

2 1

[filter] unable to squelch annoying spam
by Robin Blanchard 28 Mar '15

28 Mar '15

Hi List, I've got some Solaris machines emitting some particularly annoying spam that I cannot seem to squelch. I've tried filtering on just about every MACRO that I can think might catch it, all to no avail. Here's the snippet from running syslog-ng in debug/foreground: # syslog-ng -Fdve 2>&1 |grep alloc_extra_sgl_frame Incoming log entry; line='<4>Mar 27 19:00:55 alloc_extra_sgl_frame failed' And here's the relevant filter bit (the other strings are doing their job). filter solaris_alloc { not ( match('alloc_extra_sgl_frame' value("MESSAGE")) or match('alloc_extra_sgl_frame' value("MSGHDR")) or match('alloc_extra_sgl_frame' value("FACILITY")) or match('alloc_extra_sgl_frame' value("PRIORITY")) or match('alloc_extra_sgl_frame' value("MSGID")) or match('ext-arq alloc fail.' value("MESSAGE")) or match('ext-arq alloc fail.' value("MSGHDR")) or match('/pci@0,0/pci8086,3c06@2,2/pci1000,3080@0' value("MESSAGE")) or match('/pci@0,0/pci8086,3c06@2,2/pci1000,3080@0' value("MSGHDR")) ); }; # syslog-ng --version syslog-ng 3.5.6 Installer-Version: 3.5.6 Revision: Compile-Date: Aug 13 2014 13:54:36 Available-Modules: affile,afprog,afsocket-notls,afsocket-tls,afsocket,afstomp,afuser,basicfuncs,confgen,cryptofuncs,csvparser,dbparser,linux-kmsg-format,syslogformat,system-source Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on What else should I try? -- Robin P. Blanchard Nephila Advisors Infrastructure Administrator +1 615.823.8516 ext 4516 ---------------------------------------------------------------------------------- The information in this e-mail, and any attachment therein, is confidential and for use by the addressee only. Any use, disclosure, reproduction, modification or distribution of the contents of this e-mail, or any part thereof, other than by the intended recipient, is strictly prohibited. If you are not the intended recipient, please return the e-mail to the sender and delete it from your computer. This email is for information purposes only, nothing contained herein constitutes an offer to sell or buy securities, as such an offer may only be made from a properly authorized offering document. Although Nephila attempts to sweep e-mail and attachments for viruses, it does not guarantee that either are virus-free and accepts no liability for any damage sustained as a result of viruses. -----------------------------------------------------------------------------------

2 1

GSoC proposal: Python connector (maxvt)
by Max Timchenko 27 Mar '15

27 Mar '15

Dear all, I’ve posted a proposal for the Python connector at: https://github.com/balabit/syslog-ng/wiki/GSoC-2015-Proposal:-syslog-ng-for… Any comments are welcome. There are a few questions at the bottom; repeating them here for convenience: 1. Python 3, Python 2, or both? Supporting only 3 allows using nifty idioms and language features, but there are still users who prefer to speak 2. Constrained platforms may not have enough resources to run both. 2. Are there any priorities among the different components in the project? A total of 5 components are specified. 3. Viability of one-component-at-a-time approach — whether you would prefer to see each part as it is done to completion, or proceed in parallel for all components (eg., wrapper APIs for all, then Python APIs for all, then completed documentation and examples for all). Yours, — Max

2 1

GSoC 2015: Java Language Binding
by Garmine 42 25 Mar '15

25 Mar '15

Hi! I started coding coding around 6 years ago, first using Java and later extending my knowledge to many other languages including C, Python, FORTH, Active Oberon and Bash just to name a few of many. Right now I'm a student at Budapest University of Technology and looking for a challenge for the summer. Recently a friend of mine drew my attention to syslog-ng and so I started exploring the project. Not only because it's an important Unix application but also because I developed some logging utilities of my own and as a OSS user I would love to start paying back to the community using the knowledge I learnt from them. I am working on issue 257 at the moment and I study the code and infrastructure above all to truly understand what I'm doing rather than just copying functionality. (And sometimes figuring out configuration issues as Arch Linux doesn't seem to be the most popular development platform for syslog-ng :)) As I have significant experience in C and Java* I believe I could do the most if I worked to enable Java-Syslog interoperability. * I wrote OpenGL&CL experiments, an overly complicated BrainF#@k interpreter and contributed to a game called ScrumbleShip in C. I also developed a primitive web browser resembling the Line Mode Browser from scratch using an own HTML parser, a JSON parser, contributed to a Minecraft server's code and wrote many other smaller and larger programs in Java. Due to my interest in OpenGL I also gathered knowledge about JNI. (And I like to talk a lot!) Kind regards, Miklos Kozak

1 0

GSoC Syslog-ng WebSocket question
by Mohammadi, Mahmoud 25 Mar '15

25 Mar '15

Dear WebSocket Mentors, This is Mahmoud Mohammadi, a graduate student in UNC Charlotte, USA. Currently I am researching on GSoC syslog-ng ideas. In the WebSocket protocol, as you know there are different tasks and issues to consider such as initial handshaking, data exchange, data formats and ping/pong communication commands. So is it required to write a library for implementing the WebSocket protocol in C or C++ or just use an existing library( recommended by you or the programmer) to integrating it with syslog-ng for the required functionalities? And also can you please is there a rough estimation( if possible) about the variety of different data formats or commands exchanging between the server and client or they would be determined during the project? Sincerely, Mahmoud

1 0