June 2016 - syslog-ng - lists.balabit.hu

Custom plugin/module
by Dmitry Shmulevich 16 Jun '16

16 Jun '16

Hello, I'm looking for an advice. I'm sending the logs from my machine to a third-party server. The protocol between the client (syslog-ng service) and the third-party server is quite simple: for each received log entry the server replies with a confirmation message on the same tcp socket. Currently I'm using a simple syslog-ng config to send logs via network driver to a specified ip/port. However, since syslog-ng service doesn't receive/read the messages from the server, the connection eventually dies. Is there any way to modify syslog-ng config to allow reads after writes? I didn't find anything similar in the documentation. An alternative way seems to be implementing a syslog-ng module or a plugin. Would that be the right approach? It seems that I can customize _LogProtoClient structure, and create a plugin. But I have no clue on how the overall API looks like, and what are the steps to build the module. Could you guys point me to the right resources? Is there any sample plugin, or a guidance for developing custom plugins? Thank you in advance, Dmitry

3 3

Embedded syslog-ng: BMW i3 all-electric car
by Czanik, Péter 13 Jun '16

13 Jun '16

Hi, A few years ago I wrote about syslog-ng running on all Kindle e-readers. Recently I found, that BMW also uses syslog-ng in the i3 all-electric car: https://czanik.blogs.balabit.com/2016/06/embedded-syslog-ng-bmw-i3-all-elec… If you have another interesting use case, either embedded or the more traditional central server, I'd be glad to add it to our "Powered by syslog-ng" page at https://syslog-ng.org/powered-by-syslog-ng/ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

Re: [syslog-ng] Is there a standard for naming tag/value pairs when parsing
by jrhendri 12 Jun '16

12 Jun '16

There is something to be said for making it easy to use syslog-ng with the major players and right now that is splunk. I would hasten to add keeping json actively used and the native elasticsearch destination current also makes it easy to position syslog-ng as a standard collection and distribution layer. Within 2 to 3 years something will leapfrog splunk and if syslog-ng is ready, it will make that transition much easier. Jim Sent from my Verizon, Samsung Galaxy smartphone -------- Original message --------From: "Scheidler, Balázs" <balazs.scheidler(a)balabit.com> Date: 6/12/16 5:44 AM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] Is there a standard for naming tag/value pairs when parsing Well, CEE is pretty much dead, and I didn't see too much activity wrt lumberjack either. I would rather see consolidation instead of further fragmentation in this area. Cheers Bazsi You are the last person I thought would point me toward the splunk CIM. Given the support that Balabit has put behind CEE and then lumberjack and even the experimental patternDB schema (https://github.com/balabit/syslog-ng-patterndb/blob/master/SCHEMAS.txt) I was sure you would steer me toward lumberjack. At first glance the splunk CIM appears to be structured around and partially dependant on some of the data flows of the splunk product. I'll continue to review it but at this point I am still open to alternate suggestions. Evan. On 06/11/2016 11:45 AM, Scheidler, Balázs wrote: There's common information model at splunk or the field dictionary of CEF, of arcsight fame. I would probably use the splunk one, except if you plan to use arcsight at the end. On Jun 11, 2016 18:32, "Evan Rempel" <erempel(a)uvic.ca> wrote: There was a project by Mitre (https://www.mitre.org/) called the Common Event Expression (https://cee.mitre.org/) that was going to be the official standard for metadata names for events, but that project has been stopped. Other than the two references that the CEE project has for logging standardization efforts, does anyone know of any major efforts by any group to define a standard for metadata naming? Evan. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

UTF-8 handling
by Robin Blanchard 10 Jun '16

10 Jun '16

Hi, I've got syslog-ng-3.7.3-3.el7.centos.x86_64 feeding logstash-2.2.4-1.noarch. I've had flags(sanitize-utf8) enabled on the relevant syslog-ng listener, yet LS continues to complain with ':expected_charset=>"UTF-8"', to the point of LS ultimately choking. The underlying messages are indeed garbage, eg: $ pcregrep -n "[\x80-\xFF]" garbage.txt |tail -1 299:\\xc2\u0003\u0003\\xe0\u0001|\\x9d\\x9bX\\xa3wt}\\xe4\u001e\\xd6g\u0013j\\x9c!j\\xd9#*\\x95#\\xf8�� Should I instead be employing flags(validate-utf8) ? Ultimately, I'd like for syslog-ng to either re-encode messages into proper UTF8, or -- if not possible -- just squelch/drop them…. Thanks in advance -------------------------------------------------------------------------------------------------------------------------- This email has been sent to you on behalf of Nephila Advisors LLC (“Advisors”). Advisors provides consultancy services to Nephila Capital Ltd. (“Capital”), an investment advisor managed and carrying on business in Bermuda. Advisors and its employees do not act as agents for Capital or the funds it advises and do not have the authority to bind Capital or such funds to any transaction or agreement. The information in this e-mail, and any attachment therein, is confidential and for use by the addressee only. Any use, disclosure, reproduction, modification or distribution of the contents of this e-mail, or any part thereof, other than by the intended recipient, is strictly prohibited. If you are not the intended recipient, please return the e-mail to the sender and delete it from your computer. This email is for information purposes only, nothing contained herein constitutes an offer to sell or buy securities, as such an offer may only be made from a properly authorized offering document. Although Nephila attempts to sweep e-mail and attachments for viruses, it does not guarantee that either are virus-free and accepts no liability for any damage sustained as a result of viruses. --------------------------------------------------------------------------------------------------------------------------

1 0

Invalid parsing of syslog messages having timezone
by Nutan Shinde 10 Jun '16

10 Jun '16

Hi, Following is the syslog message received from Cisco router : *Mar 1 09:30:25.249 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel2, changed state to down As, you can see UTC is included in the above timestamp. That is why value of $PROGRAM is UTC and $MSGONLY is %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel2, changed state to down. What should I include in the syslog-ng.conf so that time zone is ignored?

3 2

grouping-by parser questions
by Fekete, Róbert 07 Jun '16

07 Jun '16

Hi Bazsi, I've started to document the grouping-by parser, and have a few questions/comments about it: * It seems that some of the grouping-by options are the same (or very similar) to the correlation-related attributes of the pattern database, but have different names. Could we name them consistently where they are the same? (I haven't checked the correlation module from Rust, but maybe we could align that as well.) For example: grouping-by | patterndb scope | context-scope timeout | context-timeout aggregate | message or action * In the original commit message, you mention three possible values for the 'scope' option, whereas the context-scope in the patterndb has four (program). Are these deliberately different, or they use the same code? * grouping-by doesn't look to me as an actual parser. From the existing objects, it resembles a filter more (IMHO), but I'd rather categorize it as something else that transforms/processes the incoming data, and should be therefore in a separate configuration object (along with the geoip parser). Robert

2 2

Problems to receive multiple/single log lines
by Jorge Pereira 04 Jun '16

04 Jun '16

Hi everyone, Question: Is it possible to send multiple/single log lines to the same destination? In this case, exist some way to differentiate the receiving of multiple/single lines? Below my client and server setup. # Client Side <MY CLIENT CONF> options { threaded(yes); flush_lines(0); use-dns(no); normalize-hostnames(yes); keep-hostname(yes); }; destination d_collector_waf { tcp("syslog-server.internal.net" port(514) keep-alive(on) flags(no-parse) ); }; source s_modsec_log { # This file is multiple-line, below the content. # # <SAMPLE CONTENT> #--bfd16c01-H-- #Message: Access denied with code 401 (phase 2). Pattern match "(?i:(?:[\\;\\|\\`]\\W*?\\bcc|\\b(wget|curl))\\b|\\/cc(?:[\\'\"\\|\\;\\`\\-\\s]|$))" at ARGS:a. [file "/usr/local/openresty/nginx/conf/waf/www.happystage.tk.conf"] [line "177"] [id "950907"] [rev "2"] [msg "System Command Injection"] [data "Matched Data: wget found within ARGS:a: wget"] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "8"] [tag "OWASP_CRS/WEB_ATTACK/COMMAND_INJECTION"] [tag "WASCTC/WASC-31"] [tag "OWASP_TOP_10/A1"] [tag "PCI/6.5.2"] #Action: Intercepted (phase 2) #Apache-Handler: IIS #Stopwatch: 1465000225000663 664069 (- - -) #Stopwatch2: 1465000225000663 664069; combined=239, p1=138, p2=80, p3=0, p4=0, p5=20, sr=10, sw=1, l=0, gc=0 #Producer: ModSecurity for nginx (STABLE)/2.8.0 (http://www.modsecurity.org/) 200911012341. #Server: ModSecurity Standalone #Engine-Mode: "ENABLED" # # </SAMPLE CONTENT> # The below regex in "multi-line-prefix()" works fine, transform the inputstream in a single line. # below debug output: # Incoming log entry; line='--3215e80c-Z--\x0a\x0a--4b80ac01-A--\x0a[04/Jun/2016:00:31:57 +0000] AiA7Ac8cA0AWAcAcAcA1@cYc 206.128.156.45 0 127.0.0.1 80\x0a--4b80ac01-B--\x0aGET /WAF-Testing/?a=wget HTTP/1.1\x0ahost: www.happystage.tk\x0arequest-id: 2016-06-04T00:31:57Z|51d35e61f4|206.128.156.45|EcWqiPItjA\x0aaccept: */*\x0auser-agent: curl/7.47.0\x0a\x0a--4b80ac01-H--\x0aMessage: Access denied with code 401 (phase 2). Pattern match "(?i:(?:[\\;\\|\\`]\\W*?\\bcc|\\b(wget|curl))\\b|\\/cc(?:[\\\'\"\\|\\;\\`\\-\\s]|$))" at ARGS:a. [file "/usr/local/openresty/nginx/conf/waf/www.happystage.tk.conf"] [line "177"] [id "950907"] [rev "2"] [msg "System Command Injection"] [data "Matched Data: wget found within ARGS:a: wget"] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "8"] [tag "OWASP_CRS/WEB_ATTACK/COMMAND_INJECTION"] [tag "WASCTC/WASC-31"] [tag "OWASP_TOP_10/A1"] [tag "PCI/6.5.2"]\x0aAction: Intercepted (phase 2)\x0aApache-Handler: IIS\x0aStopwatch: 1465000317000372 373530 (- - -)\x0aStopwatch2: 1465000317000372 373530; combined=221, p1=137, p2=63, p3=0, p4=0, p5=2' file("/usr/local/openresty/nginx/logs/waf/www.mydomain.com" follow_freq(1) flags(no-parse) multi-line-mode(regexp) override_program("ng_waf:www.mydomain.com") multi-line-prefix("--[a-fA-F0-9]{8}-Z--") ); }; source s_access_log { # The content of this file is a single-line delimited by a "\n" file("/usr/local/openresty/nginx/logs/access.log" program_override("ng_access") follow_freq(1) flags(no-parse) ); }; log { source(s_modsec_log); source(s_access_log); destination(d_collector); }; </MY CLIENT CONF> My setver setup: <SERVER CONF> options { flush-lines(100); log-fifo-size(1000); threaded(yes); use-dns(no); normalize-hostnames(yes); keep-hostname(yes); }; source s_collector { tcp(ip(0.0.0.0) port(514) keep-alive(on) flags(no-parse) ); }; filter f_nginx_waf { match("ng_modsec" value("PROGRAM")); }; rewrite r_nginx_waf { subst("ng_modsec:", "", value("PROGRAM")); }; destination d_nginx_waf { file("/var/log/syslog-ng/nginx/waf/${PROGRAM}_log" create_dirs(yes) owner("root") group("root") perm(0640) dir_perm(0750) flags(no-parse) template("${MSG}") ); }; log { source(s_collector); filter(f_nginx_waf); rewrite(r_nginx_waf); destination(d_nginx_waf); flags(flow-control); }; filter f_nginx_access { match("ng_access" value("PROGRAM")); }; destination d_nginx_access { file("/var/log/syslog-ng/nginx/${PROGRAM}_log" create_dirs(yes) owner("root") group("root") perm(0640) dir_perm(0750) flags(no-parse) template("${MSG}") ); }; log { source(s_collector); filter(f_nginx_access); destination(d_nginx_access); flags(flow-control); }; </SERVER CONF>

1 0

How to roll journald cursor in persist file
by Matwey V. Kornilov 02 Jun '16

02 Jun '16

Hello, I am running syslog-ng 3.7.1 and journald v210. The goal I want to reach is the following. I would like to replicate local system logs to remote endpoint using standard UDP protocol. So, I've just disabled all destinations except the following one: destination logserver { udp("10.0.0.47" port(514)); }; log { source(src); destination(logserver); }; The issue is the following. It seems that syslog-ng 3.7.1 uses journald natively for system() source. And I like it. The issue is that when I start syslog-ng it tries to sync all existing local journald data (about 5 GB of compressed journald data, 1.5 years of logs) from the beginning. I found in modules/systemd-journal/journal-reader.c that syslog-ng stores journald cursor in the persists file. I would like to point the cursor to the end of journald database in order to avoid unneeded log sync. The question is how could I do that in opensource syslog-ng version?

3 3

Elasticsearch with Syslog-ng
by Ivan Adji - Krstev 01 Jun '16

01 Jun '16

Hi all, I have the following error when i trying to run syslog-ng. [2016-05-31T11:17:31.298897] Seeking the journal to the last cursor position; cursor='s=6b605b6b981a471ea3d1abe52861ae82;i=21a8;b=42ab72c238874f7d8afc3c2cf0fea52d;m=38dbdc6d9;t=533be9608d083;x=140acfae276a2e55' [2016-05-31T11:17:31.299921] Module loaded and initialized successfully; module='syslogformat' [2016-05-31T11:17:31.299958] Failed to acquire /run/systemd/journal/syslog socket, disabling systemd-syslog source; [2016-05-31T11:17:31.796854] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-31T11:17:31.797915] Add path to classpath: /usr/lib64/syslog-ng/java-modules/commons-codec-1.9.jar; [2016-05-31T11:17:31.798191] Add path to classpath: /usr/lib64/syslog-ng/java-modules/commons-lang3-3.4.jar; [2016-05-31T11:17:31.798466] Add path to classpath: /usr/lib64/syslog-ng/java-modules/commons-logging-1.2.jar; [2016-05-31T11:17:31.798821] Add path to classpath: /usr/lib64/syslog-ng/java-modules/dummy.jar; [2016-05-31T11:17:31.799058] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-v2.jar; [2016-05-31T11:17:31.799296] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic.jar; [2016-05-31T11:17:31.799503] Add path to classpath: /usr/lib64/syslog-ng/java-modules/gson-2.6.2.jar; [2016-05-31T11:17:31.799778] Add path to classpath: /usr/lib64/syslog-ng/java-modules/guava-19.0.jar; [2016-05-31T11:17:31.799988] Add path to classpath: /usr/lib64/syslog-ng/java-modules/hdfs.jar; [2016-05-31T11:17:31.800249] Add path to classpath: /usr/lib64/syslog-ng/java-modules/http.jar; [2016-05-31T11:17:31.800477] Add path to classpath: /usr/lib64/syslog-ng/java-modules/httpasyncclient-4.1.1.jar; [2016-05-31T11:17:31.800684] Add path to classpath: /usr/lib64/syslog-ng/java-modules/httpclient-4.5.2.jar; [2016-05-31T11:17:31.800940] Add path to classpath: /usr/lib64/syslog-ng/java-modules/httpcore-4.4.4.jar; [2016-05-31T11:17:31.801215] Add path to classpath: /usr/lib64/syslog-ng/java-modules/httpcore-nio-4.4.4.jar; [2016-05-31T11:17:31.801449] Add path to classpath: /usr/lib64/syslog-ng/java-modules/jest-2.0.2.jar; [2016-05-31T11:17:31.801667] Add path to classpath: /usr/lib64/syslog-ng/java-modules/jest-common-2.0.2.jar; [2016-05-31T11:17:31.801932] Add path to classpath: /usr/lib64/syslog-ng/java-modules/kafka.jar; [2016-05-31T11:17:31.802119] Add path to classpath: /usr/lib64/syslog-ng/java-modules/log4j-1.2.16.jar; [2016-05-31T11:17:31.802353] Add path to classpath: /usr/lib64/syslog-ng/java-modules/slf4j-api-1.7.13.jar; [2016-05-31T11:17:31.802569] Add path to classpath: /usr/lib64/syslog-ng/java-modules/syslog-ng-common.jar; [2016-05-31T11:17:31.802849] Add path to classpath: /usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-31T11:17:31.803043] Add path to classpath: /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar; [2016-05-31T11:17:31.803264] Add path to classpath: /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar; [2016-05-31T11:17:31.803497] Add path to classpath: /usr/share/elasticsearch/lib/commons-cli-1.3.1.jar; [2016-05-31T11:17:31.803746] Add path to classpath: /usr/share/elasticsearch/lib/compiler-0.8.13.jar; [2016-05-31T11:17:31.803992] Add path to classpath: /usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar; [2016-05-31T11:17:31.804249] Add path to classpath: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar; [2016-05-31T11:17:31.804436] Add path to classpath: /usr/share/elasticsearch/lib/guava-18.0.jar; [2016-05-31T11:17:31.804651] Add path to classpath: /usr/share/elasticsearch/lib/hppc-0.7.1.jar; [2016-05-31T11:17:31.804877] Add path to classpath: /usr/share/elasticsearch/lib/jackson-core-2.6.6.jar; [2016-05-31T11:17:31.805109] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.6.jar; [2016-05-31T11:17:31.805383] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.6.jar; [2016-05-31T11:17:31.805663] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.6.jar; [2016-05-31T11:17:31.805908] Add path to classpath: /usr/share/elasticsearch/lib/jna-4.1.0.jar; [2016-05-31T11:17:31.806160] Add path to classpath: /usr/share/elasticsearch/lib/joda-convert-1.2.jar; [2016-05-31T11:17:31.806402] Add path to classpath: /usr/share/elasticsearch/lib/joda-time-2.8.2.jar; [2016-05-31T11:17:31.806654] Add path to classpath: /usr/share/elasticsearch/lib/jsr166e-1.1.0.jar; [2016-05-31T11:17:31.806889] Add path to classpath: /usr/share/elasticsearch/lib/jts-1.13.jar; [2016-05-31T11:17:31.807087] Add path to classpath: /usr/share/elasticsearch/lib/log4j-1.2.17.jar; [2016-05-31T11:17:31.807260] Add path to classpath: /usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar; [2016-05-31T11:17:31.807476] Add path to classpath: /usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar; [2016-05-31T11:17:31.807759] Add path to classpath: /usr/share/elasticsearch/lib/lucene-core-5.5.0.jar; [2016-05-31T11:17:31.808003] Add path to classpath: /usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar; [2016-05-31T11:17:31.808261] Add path to classpath: /usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar; [2016-05-31T11:17:31.808476] Add path to classpath: /usr/share/elasticsearch/lib/lucene-join-5.5.0.jar; [2016-05-31T11:17:31.808653] Add path to classpath: /usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar; [2016-05-31T11:17:31.808929] Add path to classpath: /usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar; [2016-05-31T11:17:31.809140] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar; [2016-05-31T11:17:31.809362] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar; [2016-05-31T11:17:31.809595] Add path to classpath: /usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar; [2016-05-31T11:17:31.809823] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar; [2016-05-31T11:17:31.810023] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar; [2016-05-31T11:17:31.810229] Add path to classpath: /usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar; [2016-05-31T11:17:31.810427] Add path to classpath: /usr/share/elasticsearch/lib/netty-3.10.5.Final.jar; [2016-05-31T11:17:31.810628] Add path to classpath: /usr/share/elasticsearch/lib/securesm-1.0.jar; [2016-05-31T11:17:31.810885] Add path to classpath: /usr/share/elasticsearch/lib/snakeyaml-1.15.jar; [2016-05-31T11:17:31.811065] Add path to classpath: /usr/share/elasticsearch/lib/spatial4j-0.5.jar; [2016-05-31T11:17:31.811279] Add path to classpath: /usr/share/elasticsearch/lib/t-digest-3.0.jar; [2016-05-31T11:17:33.037026] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-31T11:17:33.073362] Error initializing message pipeline; Im running with Elasticsearch with the following configuration: destination d_es { java( class-path("/usr/lib64/syslog-ng/java-modules/*.jar:/usr/share/elasticsearch/lib/*.jar") class-name("org.syslog_ng.elasticsearch.ElasticSearchDestination") option("index", "syslog-ng_${YEAR}.${MONTH}.${DAY}") option("type", "test") option("cluster", "elasticsearch") # option("flush_limit", "100") option( "message_template", "$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") ); }; Any hints on this ? Ivan

4 13