- syslog-ng - lists.balabit.hu

Re: [syslog-ng] Help with ES destination error
by Marco Mignone 01 Feb '18

01 Feb '18

Hi Fabien, I have tried the following: - emptying all index/docs in ES - create the test/test index with the CURL in my email - pointing syslog ES destination to the test/test index This resulted in the same error again. I have tried to change the template to just output all nv-pairs and use a complete new index - same error. Grabbing some packet capture now to see if I can spot anything wrong. Marco > On 28 Jan 2018, at 14:19, Fabien Wernli <wernli(a)in2p3.fr> wrote: > > Hi, > > The reason I asked you to configure syslogng to index to "test" was to make > sure you are in the same conditions as your curl command. > You might for instance have a mapping template matching fw-* but not test. > > Please either configure syslogng to index to test, or use the same fw- index > on the curl cmdline. > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq >

2 5

Syslog-ng as basic realtime reliable logfile replicator. Possible?
by Declan White 31 Jan '18

31 Jan '18

Hullo. I'm trying to fit syslog-ng around a basic problem and looking for tips. I have log files growing on one machine that I want to follow and reliably replicate to a central machine, so it's effectively a basic 'tail -f' job. It seems simple, but as I try and close out the possible error conditions it's getting hairier and hairier. e.g. - by default, there is nothing you can make with syslog-ng alone that will not lose data during a network or endpoint outage. - transporting metadata can tell you which file the data is from, but not where in the file it's from, so you can't really tell if you have duplicate data, or missed data. (The inode number might be handy too) - behaviour around input file truncation is fuzzy. That a truncation has occured might be useful metadata to send (if you're looking for people fiddling logs). - It doesn't seem to be able to encode binary/NULs in the logs, so it cannot relay data from 'untrusted' application logs? - Not sure what it does with very long lines. Loses data? I'm not necessarily looking to get syslog-ng to recreate the file exactly, just to send enough information to allow something else to work out the full order of events. Googling around to see how others solve this problem, I see people doing infinite rsync loops, or installing large Java beasties, or paying someone else to make it all go away. I tried using rsyslog, but it melted down into a screaming puddle of nondeterministic threading. Is what I'm attempting really as hard as it seems? - D

5 5

Re: [syslog-ng] Help with ES destination error
by Fabien Wernli 28 Jan '18

28 Jan '18

Hi, The reason I asked you to configure syslogng to index to "test" was to make sure you are in the same conditions as your curl command. You might for instance have a mapping template matching fw-* but not test. Please either configure syslogng to index to test, or use the same fw- index on the curl cmdline.

1 0

Help with ES destination error
by Marco Mignone 26 Jan '18

26 Jan '18

Hi all, Hope you can help me solve this scenario. I am receiving messages from a firewall, extracting KV pairs with a custom parser_db, sending them to an elasticsearch destination applying a JSON template. All works fine until I apply the JSON template to the Elasticsearch destination - when I do this in the config and a message is received I get the following from Syslog: syslog-ng | [2018-01-25T17:59:00.873601] Outgoing message; message='{"timestamp":"2018-01-25 17:59:00","src":{"port":"62118","ip":"192.168.xx.xx","if":"X1","hostname":"hostname.domain.co.uk"},"sn":"xxxx","sid":"5165","priority":"1","nipspri":"3","msg":"IPS Detection Alert","ipscat":"WEB-TLS SSLv2.0 Client Hello 2","fw":{"ip":"x.xxx.xxx.xx","action":"NA"},"dst":{"port":"443","ip":"40.xxx.xxx.xx","if":"X5"},"_classifier":{"rule_id":"71593655-6fa7-4fca-9617-480e79703215","class":"IPSDetection"},"PROGRAM":"id=Firewall","LEGACY_MSGHDR":"id=Firewall ","HOST_FROM":"xx.xx.xx.xx","HOST":"xx.xx.xx.xx"}\x0a’ syslog-ng | 17:59:00.897 [?3?] ERROR - {"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse"}],"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"not_x_content_exception","reason":"Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"}} My setup is all on Ubuntu/Docker and is using Syslog-ng 3.13.2 and ES 6.1. Configuration is quite simple: source s_net { udp( ip(0.0.0.0),port(514) #,flags(no-parse) ); }; #this parse all the field from the previous message parser sonicwall { db-parser(file("/etc/syslog-ng/patterndb.d/sonicwall-pattern.xml")); }; #some more parsing due to firewall sending dynamic lenght messages for sources and destinations parser split-sonicwall-srcdata { csv-parser( columns("src.ip", "src.port", "src.if","src.hostname") delimiters(chars(":"), strings(" ")) template("${src.data}") ); }; parser split-sonicwall-dstdata { csv-parser( columns("dst.ip", "dst.port", "dst.if","dst.hostname") delimiters(chars(":"), strings(" ")) template("${dst.data}") ); }; #finally the template applied template t_sonicwall { template("$(format-json --scope all-nv-pairs --exclude MESSAGE,SOURCE,src.data,dst.data,mfield,cfield,nfield)\n"); }; destination d_elasticsearch { elasticsearch2( client-lib-dir("/jarfiles/*.jar:/usr/lib/syslog-ng/3.13/java-modules/elastic-jest-client/*.jar:/usr/lib/syslog-ng/3.13/java-modules/") index("fw-${YEAR}.${MONTH}.$(lowercase '${.classifier.class}')") type("syslog") client_mode("http") cluster("docker-cluster") cluster_url("http://elasticsearch:9200") template(t_sonicwall) flush-limit("1") ); }; log { source(s_net); parser(sonicwall); parser(split-sonicwall-srcdata); parser(split-sonicwall-dstdata); destination(d_elasticsearch); destination(d_file); }; Although if I try to manually put the content of the message with a curl POST request it works (not some info is masked with xx): curl -X POST -H "Content-Type: application/json" -H "Cache-Control: no-cache" -d '{ "timestamp": "2018-01-25 17:59:00", "src.port": "62118", "src.ip": "xx.xx.xx.xx", "src.if": "X1", "src.hostname": "hostname.domain.co.uk", "sn": “xxxx", "sid": "5165", "priority": "1", "nipspri": "3", "msg": "IPS Detection Alert", "ipscat": "WEB-TLS SSLv2.0 Client Hello 2", "fw.ip": "xx.xx.xx.xx", "fw.action": "NA", "dst.port": "443", "dst.ip": "xx.xx.xx.xx", "dst.if": "X5", "_classifier.rule_id": "71593655-6fa7-4fca-9617-480e79703215", "_classifier.class": "IPSDetection", "PROGRAM": “id=Firewall", "LEGACY_MSGHDR": "id=Firewall ", "HOST_FROM": "xx.xx.xx.xx", "HOST": "xx.xx.xx.xx" }' "http://es:9200/test/test/“ {"_index":"test","_type":"test","_id":"ZzaCLmEB5A1B2qoXkU_p","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":1,"_primary_term":1} Any suggestions would be much appreciated. Thanks, Marco Mignone

2 2

Excessive memory usage
by Patrick Hemmer 26 Jan '18

26 Jan '18

We're seeing an issue on our systems where syslog-ng is consuming an excessive amount of memory, approximately 380mb (RSS) right now on one of our hosts. I'm not sure what's causing this as our config is pretty minimal, and there are no destinations unavailable which would cause it to buffer. We do have `log-msg-size(65536)` and `log-fifo-size(10000)` which might have caused it to buffer a large amount some time in the past, but would it not release these buffers? Version 3.13.2 on CentOS/7 (RPM from https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng313/) Config: @version:3.13 @include "scl.conf" # This configuration is meant to utilize systemd journald options { flush_lines(0); time_reopen(10); log-msg-size(65536); log_fifo_size(10000); chain_hostnames(off); use_dns(no); use_fqdn(yes); keep_hostname(yes); create_dirs(no); stats_freq(3600); }; source s_system { system(); }; source s_internal { internal(); }; destination d_journald { unix-dgram("/dev/log" flush-lines(1) template("${MSG}\n")); }; log { source(s_internal); destination(d_journald); }; destination d_loghost { syslog("syslog.ecom.chewy.com" frac-digits(3) log-fifo-size(10000) ); }; log { source(s_system); destination(d_loghost); }; Stats: # syslog-ng-ctl stats SourceName;SourceId;SourceInstance;State;Type;Number dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;dropped;163025 dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;processed;489030242 dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;queued;1 dst.syslog;d_loghost#0;tcp,syslog.ecom.chewy.com:601;a;written;488867216 destination;d_journald;;a;processed;1997 src.internal;s_internal#0;;a;processed;1997 src.internal;s_internal#0;;a;stamp;1515709115 center;;received;a;processed;1997 destination;d_loghost;;a;processed;489030242 source;s_internal;;a;processed;1997 center;;queued;a;processed;489032239 global;scratch_buffers_count;;a;queued;25769803782 dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;dropped;0 dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;processed;845 dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;queued;0 dst.unix-dgram;d_journald#0;unix-dgram,localhost.afunix:/dev/log;a;written;845 global;payload_reallocs;;a;processed;2932894697 src.journald;;journal;a;processed;489030242 src.journald;;journal;a;stamp;1515710081 global;sdata_updates;;a;processed;0 global;scratch_buffers_bytes;;a;queued;0 global;msg_clones;;a;processed;135 global;internal_queue_length;;a;processed;0 Thanks -Patrick

2 3

SUMO Logic Integration
by Scot 25 Jan '18

25 Jan '18

Hi all, Thought I had asked already but I don't see it, so sorry if it's a repost. I now have the need to switch all my syslog-ng -> SPLUNK feeds over to SUMO. Including json/beats going through syslog-ng :( Sumo supports a syslog tcp/udp collection, I setup an "Installed Collector" local to the environment. Seems to work fine until you you need to match categories in sumo. The sumo config seems to be pretty limited per instance so looking at spinning up multiple instances of sumo just to support Category mapping. Not happy with the product. Anyone working with syslog-ng and SUMO who would like to collaborate? Scot

1 0

pulling in "sar" data
by Jim Hendrick 24 Jan '18

24 Jan '18

Hi list, Has anyone added sar data to syslog-ng? I know of a couple "brute-force" ways like running a cron to run the sar command and dump it to a file. But before I do that I wanted to check and see if there was a more elegant solution. The overall problem would be to run this across 10,000+ servers with minimal need to change configurations. (for example - if there was a need to change the parameters to sar, or change the frequency of the pull). I guess I was hoping for some add-on that either directly pulled the performance data from the kernel or could include a config that would specify what data elements would be included. FYI - we do use syslog-ng PE but this seems fairly generic so I thought I'd ask the list. Given the scale of the problem, pulling "all" data even relatively frequently (say every 1-5 minutes) would result in a huge volume increase in our logging solution (where we pay by the ingested GB...) Thoughts? Advice? Thanks in advance! Jim

2 1

new build of syslog-ng 3.13.2 has new dependencies
by Evan Rempel 23 Jan '18

23 Jan '18

The copr repo at https://copr-be.cloud.fedoraproject.org/results/czanik/syslog-ng313/epel-7-… has a new build of syslog-ng. The actual version of syslog-ng has not changed, just the build. The package has gone from version 3.13.2-1 to 3.13.2-2 and new dependencies have been added. Specifically the 4 below --> Processing Dependency: libmongoc-1.0.so.0(LIBMONGOC_1.0)(64bit) for package: syslog-ng-3.13.2-2.el7.centos.x86_64 --> Processing Dependency: libbson-1.0.so.0(LIBBSON_1.0)(64bit) for package: syslog-ng-3.13.2-2.el7.centos.x86_64 --> Processing Dependency: libmongoc-1.0.so.0()(64bit) for package: syslog-ng-3.13.2-2.el7.centos.x86_64 --> Processing Dependency: libbson-1.0.so.0()(64bit) for package: syslog-ng-3.13.2-2.el7.centos.x86_64 Since the version of syslog-ng has not changed, how come the dependencies have changed? Is this a mistake and these dependencies are not actually needed? I ask because I am running the package 3.13.2-1 and not having any difficulties (I am not using the mondo module). If the dependencies are not required, can they be removed and a new build created? -- Evan Rempel

2 3

format-json incorrectly parsing some events
by Alicia Smith 19 Jan '18

19 Jan '18

I'm using syslog-ng rpm version 3.12.1-2 on CentOS 7 When we receive events remotely from another CentOS 7 host it uses the RFC5424 format and parses the messages correctly. However we have some hosts that are older and still using rsyslog which is using the RFC3164 format - those events do not parse correctly. My question is what is the best way to get syslog-ng to parse them? This is how they come out: {"TAGS":".source.test","SOURCEIP":"127.0.0.1","SOURCE":"test","SEQNUM":"26","PROGRAM":"info","PRIORITY":"notice","MESSAGE":" mig-agent 10430 - - - [info] refreshing agent environment","LEGACY_MSGHDR":"info ","HOST_FROM":" syslog-dev1.private.mdc1.mozilla.com","HOST":"sanvmadm1.ops.mdc1.mozilla.com","FILE_NAME":"/var/log/test.log","FACILITY":"user","DATE":"Jan 17 23:57:52","CATEGORY":"syslog"] Notice the Program says "info" and the mig-agent and pid are in the message key's value. This is a correctly parsed event that has those fields parsed properly: {"TAGS":".source.moz_net","SOURCEIP":"127.0.0.1","SOURCE":"moz_net","SEQNUM":"35","PROGRAM":"mig-agent","PRIORITY":"info","PID":"2698","MESSAGE":"- - - [info] Public IP retrieval failed through proxy http://proxy.dmz.scl3.mozilla.com:3128 - Get https://api.mig.mozilla.org/api/v1//ip: proxyconnect tcp: dial tcp 10.22.74.78:3128: i/o timeout","LEGACY_MSGHDR":"mig-agent[2698]: ","HOST_FROM":"localhost6.localdomain","HOST":" syslog-dev1.private.mdc1.mozilla.com","FACILITY":"daemon","DATE":"Jan 18 00:02:25","CATEGORY":"syslog"} destination d_amqp { amqp( vhost("/") host("localhost") port(5672) exchange("eventtask") exchange-type("direct") routing-key("eventtask") body("$(format-json --scope selected_macros --scope nv_pairs)") persistent(no) username("rabbituser") password("*****") ); }; -- Alicia Smith @phrozyn Information Security Engineer asmith(a)mozilla.com

2 1

Balabit acquired by One Identity
by Scheidler, Balázs 18 Jan '18

18 Jan '18

Dear syslog-ng users, You might have already heard that Balabit, the company behind syslog-ng has been acquired by One Identity (http://www.oneidentity.com) and you may be wondering what would happen to syslog-ng in the future. I'd like to share a bit of personal details: I've founded the syslog-ng project back in 1998, during my university years, and a bit later on, I was also one of the founders of Balabit. I've worked on syslog-ng as a personal hobby until about 2007, at which point Balabit has officially started sponsoring the project, building products on top and also providing professional services. Although I was CEO and later CTO of Balabit, I always remained an active member of the syslog-ng community. Whenever I've found some leisure time (e.g. on a plane :), I was working on syslog-ng, and I've kept doing that parallel to Balabit establishing a professional dev team that worked on syslog-ng and related products. All of this work went straight to the open source project: https://github.com/balabit/syslog-ng/graphs/contributors With that story, I’d like to assure you that this acquisition will not negatively affect syslog-ng. One Identity considers the log management products of Balabit an important part of the acquisition and will continue to develop both the commercial and the open source branches further. Of course, I also have a role in the combined entity, and will continue to keep working on syslog-ng personally. I seem to be expected to be sitting on planes much more often :) One Identity as a business will also keep working on syslog-ng, and are committed to keeping it open source, adding new features, and so on, just like we did in the past. As a first step of this process, the syslog-ng product line will receive its own homepage in the coming weeks. Stay tuned, and keep using syslog-ng -- Bazsi

1 0