- syslog-ng - lists.balabit.hu

Insider 2018-02: New website; One Identity; Docker; Error messages; Elastic 6; DevConf;
by Czanik, Péter 08 Feb '18

08 Feb '18

Dear syslog-ng users, This is the 64th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS New syslog-ng website --------------------- Our new website – http://syslog-ng.com/ – is now live. It is a central hub of information for all syslog-ng related content, including commercial products built around syslog-ng. All documentation and blogs were migrated here. The old syslog-ng.org site will go away once all open source related content found its place on the new site. Feedback is much appreciated! Balabit acquired by One Identity -------------------------------- You might have already heard that Balabit, the company behind syslog-ng, has been acquired by One Identity (http://www.oneidentity.com) and you may be wondering what would happen to syslog-ng in the future. Learn how Balázs Scheidler, original author of syslog-ng, sees the future: https://lists.balabit.hu/pipermail/syslog-ng/2018-January/024254.html Tl;dr: nothing to worry about! Logging in Docker using syslog-ng --------------------------------- Collecting logs can be especially important in a containerized environment, where you start and stop containers around the clock, and the container might not even exist anymore by the time you investigate an event. Read this white paper to find out how to: Run your central log server in Docker Collect Docker infrastructure logs using syslog-ng Collect logs from containers using Docker volumes https://syslog-ng.com/white-papers Common syslog-ng error messages and their solutions --------------------------------------------------- In this post, we explain a few common syslog-ng error and warning messages, what they mean, and how to solve them. https://syslog-ng.com/blog/common-syslog-ng-error-messages-solutions/ Getting started with Elasticsearch 6 on RHEL/CentOS --------------------------------------------------- Version 6 of the Elastic Stack has now been available for some time, packed with new features and improved performance. Compatibility of syslog-ng was checked already during the alpha phase of development, as syslog-ng is becoming popular among Elasticsearch users: it can greatly simplify logging to Elasticsearch. There are no major changes from a syslog-ng point of view but – to improve your copy & paste experience – I updated my getting started guide from Elastic Stack 5 to 6. https://syslog-ng.com/blog/syslog-ng-and-elasticsearch-6-getting-started-on… DevConf 2018 ------------ DevConf is a yearly conference for developers, administrators, and users of Linux and related technologies. It is organized by Red Hat in Brno, home to one of their major development centers. Since the majority of syslog-ng users are running their logging server on Red Hat Enterprise Linux or CentOS, it is important to know their needs and the directions where RHEL is heading. It is also a good possibility to discuss logging-related questions with Red Hat engineers in person. https://syslog-ng.com/blog/devconf-long-live-containerization/ UPCOMING EVENTS SCALE: Logging Docker using syslog-ng: https://www.socallinuxexpo.org/scale/16x/presentations/logging-docker-using… Pasadena Convention Center March 8 - 11, 2018 Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://syslog-ng.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

problem elasticsearch2 creating index per HOST
by Abe Lebo 07 Feb '18

07 Feb '18

LS, i want to create an Elasticsearch index per host i receive logs from (also syslog-ng) , somehow that fails. in have a setup that creates an index per day (see below), that works. I can build a file output per host per day by adding the $HOST to the filename, that works too.(see below) but if add $HOST to the indexname it no longer creates indices. Anybody know what's wrong? Thanks in advance, Abel ...... destination d_testfile { file( "/tmp/testlogabel-$HOST$YEAR$MONTH" flags(syslog-protocol) template("$(format-json @timestamp=$ISODATE --scope nv-pairs)\n") ); }; ################################################################################ # Elasticsearch destination ################################################################################ destination d_es_per_day { elasticsearch2( client-lib-dir("/usr/share/elasticsearch/lib/:/usr/lib/syslog-ng/3.13/java-modules/") index("test-abel_$YEAR.$MONTH.$DAY") type("test") cluster-url("http://node01:9400 http://node02:9400 http://node03:9400") client-mode("http") template("$(format-json @timestamp=$ISODATE --scope nv-pairs)\n") ); }; ###### # sending logs to ES destination log { source(s_net); source(s_src); filter(f_abel); parser(p_abel); destination(d_es_per_day); flags(flow-control); };

2 2

OSE, Shared Storage, & Load Balanced syslog Messages
by Pierce, Stevan - PHX 07 Feb '18

07 Feb '18

I am working on setting up OSE as a relay and would like to have storage set aside in case a destination upstream from the relay goes down. How does load balancing syslog messages work when clustering at the OS level? If I am sharing storage across multiple relays, how would I go about setting it up so that messages from one server does not overwrite another server? swp

2 1

Rsyslog relay or syslog-ng ?
by Scot 06 Feb '18

06 Feb '18

Hi, Hoping someone has seen an easy fix for this. Sorry is it's specifically referenced somewhere I'm not seeing. Dealing with a vendor who is not able to leverage the RFC headers or TCP input. We have rsyslog relays in remote sites sending TCP/514 to syslog-ng and others locally sending directly to syslog-ng TCP/UDP 514. The devices sending directly to syslog-ng are reporting to the IDS correctly. Hosts relaying through rsyslog are showing a source address of the relay. */etc/rsyslog.d/forward.conf* *$ActionQueueFileName fwdRule1 # unique name prefix for spool files* *$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)* *$ActionQueueSaveOnShutdown on # save messages to disk on shutdown* *$ActionQueueType LinkedList # run asynchronously* *$ActionResumeRetryCount -1 # infinite retries if host is down* *# remote host is: name/ip:port, e.g. 192.168.0.1:514 <http://192.168.0.1:514>, port optional* **.* @@syslog-ngIP:514* */etc/syslog-ng/conf.d* *source s_net_tcp {tcp(ip(0.0.0.0) port(514) max-connections(300) keep_hostname(yes) so_rcvbuf(262142) log_iw_size(25000)); };* *source s_net_udp {syslog(ip(0.0.0.0) keep_hostname(yes) port(514) transport("udp") flags(no-hostname) so_rcvbuf(262142));};* *destination d_ids {network("IDSHOSTNAME" spoof_source(yes) transport(udp) port(514) flags(syslog-protocol)); };* *log { source(s_net_udp);* * channel {filter(f_ids); destination (d_ids);};* * channel {parser(pattern_db); destination (d_es);};* *};* *log { source(s_net_tcp);* * channel {filter(f_ids); destination (d_ids);};* * channel {parser(pattern_db); destination (d_es);};* *};*

3 4

RFC5424 sending - dest=unix-stream. Framing?
by Declan White 05 Feb '18

05 Feb '18

I have syslog-ng pumping data into a unix-stream destination socketfilething. Data going into this domain socket has lots of adventures, and then emerges as a source unix-stream on another host. Both have flags(syslog-protocol), but the data gets framing errors at the receiver. I think it's become a bytestream (instead of a message stream) enroute, and the recv() is relying on only getting one message at a time, because it can't know the message length any other way, as unix-stream gets messages without framing(?). Is there a way to salvage this? (I noticed TLS might add framing, but since the tunnel is already TLS ...) -- Declan White

2 5

Re: [syslog-ng] Help with ES destination error
by Marco Mignone 01 Feb '18

01 Feb '18

Hi Fabien, I have tried the following: - emptying all index/docs in ES - create the test/test index with the CURL in my email - pointing syslog ES destination to the test/test index This resulted in the same error again. I have tried to change the template to just output all nv-pairs and use a complete new index - same error. Grabbing some packet capture now to see if I can spot anything wrong. Marco > On 28 Jan 2018, at 14:19, Fabien Wernli <wernli(a)in2p3.fr> wrote: > > Hi, > > The reason I asked you to configure syslogng to index to "test" was to make > sure you are in the same conditions as your curl command. > You might for instance have a mapping template matching fw-* but not test. > > Please either configure syslogng to index to test, or use the same fw- index > on the curl cmdline. > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq >

2 5

Syslog-ng as basic realtime reliable logfile replicator. Possible?
by Declan White 31 Jan '18

31 Jan '18

Hullo. I'm trying to fit syslog-ng around a basic problem and looking for tips. I have log files growing on one machine that I want to follow and reliably replicate to a central machine, so it's effectively a basic 'tail -f' job. It seems simple, but as I try and close out the possible error conditions it's getting hairier and hairier. e.g. - by default, there is nothing you can make with syslog-ng alone that will not lose data during a network or endpoint outage. - transporting metadata can tell you which file the data is from, but not where in the file it's from, so you can't really tell if you have duplicate data, or missed data. (The inode number might be handy too) - behaviour around input file truncation is fuzzy. That a truncation has occured might be useful metadata to send (if you're looking for people fiddling logs). - It doesn't seem to be able to encode binary/NULs in the logs, so it cannot relay data from 'untrusted' application logs? - Not sure what it does with very long lines. Loses data? I'm not necessarily looking to get syslog-ng to recreate the file exactly, just to send enough information to allow something else to work out the full order of events. Googling around to see how others solve this problem, I see people doing infinite rsync loops, or installing large Java beasties, or paying someone else to make it all go away. I tried using rsyslog, but it melted down into a screaming puddle of nondeterministic threading. Is what I'm attempting really as hard as it seems? - D

5 5

Re: [syslog-ng] Help with ES destination error
by Fabien Wernli 28 Jan '18

28 Jan '18

Hi, The reason I asked you to configure syslogng to index to "test" was to make sure you are in the same conditions as your curl command. You might for instance have a mapping template matching fw-* but not test. Please either configure syslogng to index to test, or use the same fw- index on the curl cmdline.

1 0

Help with ES destination error
by Marco Mignone 26 Jan '18

26 Jan '18

Hi all, Hope you can help me solve this scenario. I am receiving messages from a firewall, extracting KV pairs with a custom parser_db, sending them to an elasticsearch destination applying a JSON template. All works fine until I apply the JSON template to the Elasticsearch destination - when I do this in the config and a message is received I get the following from Syslog: syslog-ng | [2018-01-25T17:59:00.873601] Outgoing message; message='{"timestamp":"2018-01-25 17:59:00","src":{"port":"62118","ip":"192.168.xx.xx","if":"X1","hostname":"hostname.domain.co.uk"},"sn":"xxxx","sid":"5165","priority":"1","nipspri":"3","msg":"IPS Detection Alert","ipscat":"WEB-TLS SSLv2.0 Client Hello 2","fw":{"ip":"x.xxx.xxx.xx","action":"NA"},"dst":{"port":"443","ip":"40.xxx.xxx.xx","if":"X5"},"_classifier":{"rule_id":"71593655-6fa7-4fca-9617-480e79703215","class":"IPSDetection"},"PROGRAM":"id=Firewall","LEGACY_MSGHDR":"id=Firewall ","HOST_FROM":"xx.xx.xx.xx","HOST":"xx.xx.xx.xx"}\x0a’ syslog-ng | 17:59:00.897 [?3?] ERROR - {"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse"}],"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"not_x_content_exception","reason":"Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"}} My setup is all on Ubuntu/Docker and is using Syslog-ng 3.13.2 and ES 6.1. Configuration is quite simple: source s_net { udp( ip(0.0.0.0),port(514) #,flags(no-parse) ); }; #this parse all the field from the previous message parser sonicwall { db-parser(file("/etc/syslog-ng/patterndb.d/sonicwall-pattern.xml")); }; #some more parsing due to firewall sending dynamic lenght messages for sources and destinations parser split-sonicwall-srcdata { csv-parser( columns("src.ip", "src.port", "src.if","src.hostname") delimiters(chars(":"), strings(" ")) template("${src.data}") ); }; parser split-sonicwall-dstdata { csv-parser( columns("dst.ip", "dst.port", "dst.if","dst.hostname") delimiters(chars(":"), strings(" ")) template("${dst.data}") ); }; #finally the template applied template t_sonicwall { template("$(format-json --scope all-nv-pairs --exclude MESSAGE,SOURCE,src.data,dst.data,mfield,cfield,nfield)\n"); }; destination d_elasticsearch { elasticsearch2( client-lib-dir("/jarfiles/*.jar:/usr/lib/syslog-ng/3.13/java-modules/elastic-jest-client/*.jar:/usr/lib/syslog-ng/3.13/java-modules/") index("fw-${YEAR}.${MONTH}.$(lowercase '${.classifier.class}')") type("syslog") client_mode("http") cluster("docker-cluster") cluster_url("http://elasticsearch:9200") template(t_sonicwall) flush-limit("1") ); }; log { source(s_net); parser(sonicwall); parser(split-sonicwall-srcdata); parser(split-sonicwall-dstdata); destination(d_elasticsearch); destination(d_file); }; Although if I try to manually put the content of the message with a curl POST request it works (not some info is masked with xx): curl -X POST -H "Content-Type: application/json" -H "Cache-Control: no-cache" -d '{ "timestamp": "2018-01-25 17:59:00", "src.port": "62118", "src.ip": "xx.xx.xx.xx", "src.if": "X1", "src.hostname": "hostname.domain.co.uk", "sn": “xxxx", "sid": "5165", "priority": "1", "nipspri": "3", "msg": "IPS Detection Alert", "ipscat": "WEB-TLS SSLv2.0 Client Hello 2", "fw.ip": "xx.xx.xx.xx", "fw.action": "NA", "dst.port": "443", "dst.ip": "xx.xx.xx.xx", "dst.if": "X5", "_classifier.rule_id": "71593655-6fa7-4fca-9617-480e79703215", "_classifier.class": "IPSDetection", "PROGRAM": “id=Firewall", "LEGACY_MSGHDR": "id=Firewall ", "HOST_FROM": "xx.xx.xx.xx", "HOST": "xx.xx.xx.xx" }' "http://es:9200/test/test/“ {"_index":"test","_type":"test","_id":"ZzaCLmEB5A1B2qoXkU_p","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":1,"_primary_term":1} Any suggestions would be much appreciated. Thanks, Marco Mignone

2 2

Excessive memory usage
by Patrick Hemmer 26 Jan '18

26 Jan '18

We're seeing an issue on our systems where syslog-ng is consuming an excessive amount of memory, approximately 380mb (RSS) right now on one of our hosts. I'm not sure what's causing this as our config is pretty minimal, and there are no destinations unavailable which would cause it to buffer. We do have `log-msg-size(65536)` and `log-fifo-size(10000)` which might have caused it to buffer a large amount some time in the past, but would it not release these buffers? Version 3.13.2 on CentOS/7 (RPM from https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng313/) Config: @version:3.13 @include "scl.conf" # This configuration is meant to utilize systemd journald options { flush_lines(0); time_reopen(10); log-msg-size(65536); log_fifo_size(10000); chain_hostnames(off); use_dns(no); use_fqdn(yes); keep_hostname(yes); create_dirs(no); stats_freq(3600); }; source s_system { system(); }; source s_internal { internal(); }; destination d_journald { unix-dgram("/dev/log" flush-lines(1) template("${MSG}\n")); }; log { source(s_internal); destination(d_journald); }; destination d_loghost { syslog("syslog.ecom.chewy.com" frac-digits(3) log-fifo-size(10000) ); }; log { source(s_system); destination(d_loghost); }; Stats: # syslog-ng-ctl stats SourceName;SourceId;SourceInstance;State;Type;Number dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;dropped;163025 dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;processed;489030242 dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;queued;1 dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;written;488867216 destination;d_journald;;a;processed;1997 src.internal;s_internal#0;;a;processed;1997 src.internal;s_internal#0;;a;stamp;1515709115 center;;received;a;processed;1997 destination;d_loghost;;a;processed;489030242 source;s_internal;;a;processed;1997 center;;queued;a;processed;489032239 global;scratch_buffers_count;;a;queued;25769803782 dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;dropped;0 dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;processed;845 dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;queued;0 dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;written;845 global;payload_reallocs;;a;processed;2932894697 src.journald;;journal;a;processed;489030242 src.journald;;journal;a;stamp;1515710081 global;sdata_updates;;a;processed;0 global;scratch_buffers_bytes;;a;queued;0 global;msg_clones;;a;processed;135 global;internal_queue_length;;a;processed;0 Thanks -Patrick

2 3