- syslog-ng - lists.balabit.hu

redis/tcp destination throughput and dropped messages
by Nicholas Radonicich 21 Jul '14

21 Jul '14

Hello, hoping someone might be able to help... I am attempting to build a syslog server with outputs to Logstash/Elasticsearch via redis but am having issues with dropped messages in such that i cannot get over ~4000 messages per second out to a destination (TCP or redis) without the rest being dropped. I have tried various tuning on the tcp output with flush_lines() and threaded() to no avail, as well as some things on the inputs. Both redis and logstash (used for tcp inputs) were on the localhost. redis-benchmark has no issues doing >70000 LPUSH/second and the file based destinations of syslog-ng are not dropping messages (we are sending almost 80k/s) so I don’t know where I am going wrong. loggen was accepting > 110000mps its just on the outputs it seems. Thanks, Nick The information in this message, including in all attachments, is confidential or privileged. In the event you have received this message in error and are not the intended recipient, you are hereby advised that any use, copying or reproduction of this document is strictly forbidden. Please notify immediately the sender of this error and destroy this message, including its attachments, as the case may be. L'information apparaissant dans ce message électronique et dans les documents qui y sont joints est de nature confidentielle ou privilégiée. Si ce message vous est parvenu par erreur et que vous n'en êtes pas le destinataire visé, vous êtes par les présentes avisé que toute utilisation, copie ou distribution de ce message est strictement interdite. Vous êtes donc prié d’en informer immédiatement l’expéditeur et de détruire ce message, ainsi que les documents qui y sont joints, le cas échéant. Ce courriel provient de Nicholas.Radonicich(a)cogeco.com . Pour assurer la livraison de futurs envois, veuillez inclure la présente adresse courriel à votre carnet d’adresses ou votre liste d’expéditeurs autorisés. Si vous ne souhaitez plus recevoir de messages promotionnels de la part de Cogeco, veuillez envoyer un courriel à desabonnement(a)cogeco.com<mailto:desabonnement@cogeco.com?Subject=Unsubscribe%20to%20mail%20from%20Nicholas.Radonicich@cogeco.com>. Politique en matière de protection des renseignements personnels de Cogeco et Engagement en matière d’anti-spam<http://www.cogeco.ca/cable/entreprise/cca/vie_privee.html> – Contactez-nous<http://www.cogeco.ca/web/qc/fr/residentiel/soutien/contact_telephone.php> Cogeco Câble Canada, 5 Place Ville-Marie, Bureau 1700, Montréal, Québec, H3B 0B3 -- This email is from Nicholas.Radonicich(a)cogeco.com . To ensure the delivery of future emails, please add the current email address to your address book or safe senders list. If you no longer wish to receive promotional emails from Cogeco, please send an email to unsubscribe(a)cogeco.com<mailto:unsubscribe@cogeco.com?Subject=Unsubscribe%20to%20mail%20from%20Nicholas.Radonicich@cogeco.com>. Privacy Policy and Anti-spam Commitment<http://www.cogeco.ca/cable/corporate/cgo/privacy.html> - Contact us<http://www.cogeco.ca/web/on/en/residential/support/contact_phone.php> Cogeco Cable Canada, 5 Place Ville-Marie, Suite 1700, Montreal, Quebec, H3B 0B3

2 2

syslog-ng 3.5.5 has been released
by devel＠balabit.hu 21 Jul '14

21 Jul '14

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.5.5 SUMMARY : new stable release DATE : Jul 21, 2014 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.5.5) has been released. For latest fixes in the 3.5.x feature branch you are recommended to upgrade to this version. CHANGES: 3.5.5 Mon, 21 Jul 2014 17:00:00 +0200 This is the fifth bug-fix release for the 3.5.x series. Starting with this release, the 3.5 branch replaces 3.4 as the current "stable" version. Upgrading from earlier versions - and especially earlier branches - is highly recommended. New features ============ * The facility() filter now supports ranges, allowing one to capture a range of facilities: facility(local0..local6). * The $(format-json) template function now supports SEQNUM. * The upstart script in contrib has been improved to allow the user to override certain settings (such as the configuration file, and the pidfile), and allow extra options to be specified. Bugfixes ======== * Major memory leaks have been fixed in various parts of syslog-ng, including but not limited to network and file destinations and sources, value-pairs (thus affecting mongodb, $(format-json), and others), template handling and rewrite rules. Some of these leaks only happened during reload, some leaked on every message. They have now been plugged. * The value-pairs framework has seen a lot of updates, too: - It now honors time zone settings, so everything that uses value-pairs, will correctly take time zone settings into account. - Using globs in the key() option of value pairs will now also search macros for matches (thus key("D*") will match DATE too). - The syslog-proto (RFC5424) scope now correctly includes SDATA. - A bug that resulted in hierarchical elements being misparsed and made to appear in random places have been fixed. * The body() and subject() options of the SMTP destinations are now mandatory, and a double-free was also fixed in the driver. * Minor fixes were made to the $(env) template function, so that when looking up an empty environment variable, we will no longer trigger a glib warning. A misleading error message was also corrected. * The pdbtool patternize tool will now output version 4 PDB XML. * The configure script will not try to configure the AMQP module if no suitable version of python is installed. This avoids a difficult to diagnose error message during compilation. Developer news ============== * When compiled with internal ivykis, we will now install its headers too, along with our own. This is to make it easier for the Incubator to be compiled against a syslog-ng with embedded ivykis. * Plugin symbols are now loaded differently, in a way that lets plugins load further plugins, and use already loaded symbols. This is required for the various scripting destinations in the Incubator. * Starting with this release, we are installing the testing helpers as a convenience library (available under the syslog-ng-test pkg-config module). This allows third party projects to use our helpers. Credits ======= syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Balazs Scheidler <bazsi(a)balabit.hu> Daniel Gados Fabien Wernli Gergely Nagy <algernon(a)balabit.hu> Laszlo Budai <lbudai(a)balabit.hu> Laszlo Meszaros <lmesz(a)balabit.hu> Nick Alcock <nix(a)esperi.org.uk> Peter Czanik <czanik(a)balabit.hu> Peter Gyongyosi <gyp(a)balabit.hu> Sebastien Badia <seb(a)sebian.fr> Tibor Benke <tibor.benke(a)balabit.com> Viktor Juhasz <viktor.juhasz(a)balabit.com> Viktor Tusa <tusa(a)balabit.hu> Xufeng Zhang <xufeng.zhang(a)windriver.com> Zoltan Pallagi <pzolee(a)balabit.hu> DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

(no subject)
by Radu Gheorghe 21 Jul '14

21 Jul '14

Hi, This is my first post here, so I have to start by thanking all the contributors for an awesome product :) My question is about adding an array to a JSON document. What I'm trying to do is to send a message like this: @cee: {"message": "test message", "tags":["test", "message"]} My template looks a like this: template("@cee: $(format-json --pair message=\"$MSG\" --pair tags="test")\n") This works fine for a single tag, but how can I add multiple ones? The broader use-case is that I want to add tags to logs matching a specific filter. For example: ---------------------- filter user_tests { facility(user) and message(test) }; destination logsene_tests { syslog("logsene-receiver-syslog.sematext.com" transport("tcp") port(514) template("@cee: $(format-json --pair message=\"$MSG\" --pair tags=\"test\")\n") ); }; log { source(all_syslog); filter(user_tests); destination(logsene_tests); flags(final); }; ---------------------- If there's a better way to add multiple tags to a log, please tell me - I'm good with making big changes if it leads to a cleaner/better config. Best regards, Radu -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/

3 7

syslog-ng 2.1.4 - file sources are read only when reload or restart is performed?
by Rafał Radecki 18 Jul '14

18 Jul '14

Hi All ;) I am using a virtual machine with: Red Hat Enterprise Linux Server release 5.7 (Tikanga) Linux logserver01 2.6.18-274.el5 #1 SMP Fri Jul 8 17:36:59 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux syslog-ng 2.1.4 I have following configuration: ... 9 options { 10 create_dirs (yes); 11 dir_group (root); 12 dir_owner (root); 13 dir_perm (0700); 14 group (root); 15 owner (root); 16 perm (0600); 17 flush_lines(1); 18 flush_timeout (1000); 19 keep_hostname (yes); 20 log_fifo_size (1); 21 use_dns (no); 22 use_fqdn (no); 23 }; ... 39 source s_stdout { 40 # file ("/logs/stdout.log" flags(no-parse) follow_freq(1)); 41 file ("/logs/stdout.log" flags(no-parse)); 42 }; ... 61 destination d_stdout { file("/var/log/$YEAR$MONTH$DAY/stdout"); }; ... 80 log { source(s_stdout); destination(d_stdout); }; The problem is that changes in /logs/stdout.log are only visible in /var/log/20140717/stdout when I perform /etc/init.d/syslog-ng reload or restart. I tried several settings of flush_*, log_fifo_size and follow_freq but with no luck :D Is it a problem withe the version that I use (quite old :D ) or is there a mistake in my configuration maybe? BR, Rafal.

3 3

Logs sent to wrong destination
by Renato Bezerra 18 Jul '14

18 Jul '14

Hi, I'm using syslog-ng in a long time, but recently i noted that, in some cases, the log has sent to a wrong destination. I have many devices sending logs to my host, the problem appears when the server receive webservers logs, they are delivered to a different destination and I don't known how. here is the configuration: destination apache { file("/var/log/webserver/$R_YEAR-$R_MONTH-$R_DAY-$R_HOUR" owner(ll) group(ll) perm(0644) dir_perm(0755) create_dirs(yes)); }; filter f_apache { ( host("xxx.xxx.xxx.82") or host("xxx.xxx.xxx.137") ); }; log { source(aaa); filter(f_apache); destination(apache); }; The ip address xxx.xxx.xxx.137 send a duplicate log event to another directory, without any other configuration. Have you seen this?

3 4

Apache Kafka output module
by Pierre-Yves Ritschard 11 Jul '14

11 Jul '14

Hi list, I submitted a PR for an Apache Kafka [1] output module. This is my first syslog-ng module, so it might need a bit of guidance. This provides the ability to create a partitioned stream of logs based on a message field or static key. A typical destination configuration will look like: destination d_kafka { kafka( properties(metadata.broker.list("localhost:9092")) topic("syslog") payload("$(format-json --scope all-nv-pairs --scope core)") partition("static-key") # partition(random) # partition(field("$PROGRAM")) ); }; For reference, this allows us to work on logs locally with syslog, extract fields, then publish on a queue whose consumers will index logs in our data stores. [1]. http://kafka.apache.org

2 1

patterndb and context - access fields from initial message
by Atom2 11 Jul '14

11 Jul '14

Hi guys, I did have a conversation today on IRC with algernon (again many thanks to him) about the problem I am faced with and we agreed that I should send my question to the list. My aim is to consolidate a bunch of related syslog messages into a single message once the final message has come in (or a timeout has occured for whatever reason). To do this I use patterndb together with a defined context linking related messages to the context. Unfortunately the length of the context (i.e. the number of messages within the context) is unknown from the outset. Basically it (mainly) depends on how long the context is active for - which can be any time and is completely unforseeable. Now my problem is as follows: From the initial message I parse (using patterndb) certain fields into a number of named MACROS. This is all solved and works very well. When the final message comes in (or in case a timeout occurs) I would like to trigger an action (triggered by match or timeout respectively) to generate and subsequently inject a message about the full context using certain fields, mainly from the final message, but also including a number of fields from the inital message of the context. Generally a field is available using the syntax ${MACRO}@# where @# denotes a reference to a previous message within the same context - so ${MACRO}@5 would use the value of MACRO from a message with a distance of 5 from the curren message (where the distance of the current message to itself is 1 - so @5 references the 4th previous message). That also works very well but obviously requires that one knows which message to refer to using the distance value. The problem comes when the context-length is unknown and thus the distance of the initial message to the final message is not a fixed, pre-determined number but rather varies. Although there seems to be a variable/macro indicating/storing the context-length which (according to the manual) can be accessed by $(context-length) [NOTE: the paranthesis instead of the curly braces] the combination of the two does not seem to work out: I was expecting that ${MACRO}@$(context-length) would provide me with the required information - that is: the value of the parsed field from the initial message - but my construct unfortunately fails. Is there anything I am doing wrong here (i.e. is there any quoting/esacping required for certain characters), is my thought process flawed or does this not work/is it a bug. In case it does not work, how would I be able to achieve my goal. Many thanks and best regards, Atom2 P.S. The version of syslog-ng I am using is 3.4.7, compiled from sources, using the latest stable gentoo ebuild.

4 6

Syslog-ng completely unusable after upgrade
by James Lay 10 Jul '14

10 Jul '14

Well shoot...here's what I got...I'm using the stock syslog-ng.conf installed with 3.5.4.1 just for testing: ############################################################################# # Default syslog-ng.conf file which collects all local logs into a # single file called /var/log/messages. # @version: 3.3 @include "scl.conf" source s_local { system(); internal(); }; source s_network { udp(); }; destination d_local { file("/var/log/messages"); }; log { source(s_local); # uncomment this line to open port 514 to receive messages #source(s_network); destination(d_local); }; Running strace here's the last bit: /usr/sbin/syslog-ng -t -d -v -F -f /usr/etc/syslog-ng.conf open("/usr/lib/libsyslog-ng-3.3.11.so", O_RDONLY) = 10 read(10, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0@0\1\000"..., 512) = 512 fstat64(10, {st_mode=S_IFREG|0755, st_size=1427313, ...}) = 0 mmap2(NULL, 447784, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 10, 0) = 0xb79d7000 mmap2(0xb7a40000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 10, 0x68) = 0xb7a40000 mmap2(0xb7a44000, 1320, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7a44000 close(10) = 0 write(2, "iv_tls_user_register: called aft"..., 43iv_tls_user_register: called after iv_init ) = 43 rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0 tgkill(5172, 5172, SIGABRT) = 0 --- SIGABRT (Aborted) @ 0 (0) --- +++ killed by SIGABRT +++ Process 5172 detached And that's it....I'm glad I'm doing this on a virtual machine. I've not been able to anything on this online...thoughts or help are appreciated. James

2 2

Configure option --disable-json doesn't seem to function
by James Lay 10 Jul '14

10 Jul '14

Topic says it: syslog-ng-3.5.4.1 ./configure --prefix=/usr --enable-dynamic-linking --disable-ipv6 --enable-spoof-source --disable-json checking python module: json... no configure: error: failed to find required module json configure: error: ./configure.gnu failed for modules/afamqp/rabbitmq-c It appears to still be looking. Thank you. James

2 9

syslog problem
by Riyas Ahamed 09 Jul '14

09 Jul '14

Hi, I have configured syslog-ng but I cannot able to see logs of network devices in syslog-ng server. Please help me to sort-out the problem. Regards, N.B.Riaz Ahmed http://www.csscorp.com/common/email-disclaimer.php

4 4