- syslog-ng - lists.balabit.hu

UTF-8 handling
by Robin Blanchard 10 Jun '16

10 Jun '16

Hi, I've got syslog-ng-3.7.3-3.el7.centos.x86_64 feeding logstash-2.2.4-1.noarch. I've had flags(sanitize-utf8) enabled on the relevant syslog-ng listener, yet LS continues to complain with ':expected_charset=>"UTF-8"', to the point of LS ultimately choking. The underlying messages are indeed garbage, eg: $ pcregrep -n "[\x80-\xFF]" garbage.txt |tail -1 299:\\xc2\u0003\u0003\\xe0\u0001|\\x9d\\x9bX\\xa3wt}\\xe4\u001e\\xd6g\u0013j\\x9c!j\\xd9#*\\x95#\\xf8�� Should I instead be employing flags(validate-utf8) ? Ultimately, I'd like for syslog-ng to either re-encode messages into proper UTF8, or -- if not possible -- just squelch/drop them…. Thanks in advance -------------------------------------------------------------------------------------------------------------------------- This email has been sent to you on behalf of Nephila Advisors LLC (“Advisors”). Advisors provides consultancy services to Nephila Capital Ltd. (“Capital”), an investment advisor managed and carrying on business in Bermuda. Advisors and its employees do not act as agents for Capital or the funds it advises and do not have the authority to bind Capital or such funds to any transaction or agreement. The information in this e-mail, and any attachment therein, is confidential and for use by the addressee only. Any use, disclosure, reproduction, modification or distribution of the contents of this e-mail, or any part thereof, other than by the intended recipient, is strictly prohibited. If you are not the intended recipient, please return the e-mail to the sender and delete it from your computer. This email is for information purposes only, nothing contained herein constitutes an offer to sell or buy securities, as such an offer may only be made from a properly authorized offering document. Although Nephila attempts to sweep e-mail and attachments for viruses, it does not guarantee that either are virus-free and accepts no liability for any damage sustained as a result of viruses. --------------------------------------------------------------------------------------------------------------------------

1 0

Invalid parsing of syslog messages having timezone
by Nutan Shinde 10 Jun '16

10 Jun '16

Hi, Following is the syslog message received from Cisco router : *Mar 1 09:30:25.249 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel2, changed state to down As, you can see UTC is included in the above timestamp. That is why value of $PROGRAM is UTC and $MSGONLY is %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel2, changed state to down. What should I include in the syslog-ng.conf so that time zone is ignored?

3 2

grouping-by parser questions
by Fekete, Róbert 07 Jun '16

07 Jun '16

Hi Bazsi, I've started to document the grouping-by parser, and have a few questions/comments about it: * It seems that some of the grouping-by options are the same (or very similar) to the correlation-related attributes of the pattern database, but have different names. Could we name them consistently where they are the same? (I haven't checked the correlation module from Rust, but maybe we could align that as well.) For example: grouping-by | patterndb scope | context-scope timeout | context-timeout aggregate | message or action * In the original commit message, you mention three possible values for the 'scope' option, whereas the context-scope in the patterndb has four (program). Are these deliberately different, or they use the same code? * grouping-by doesn't look to me as an actual parser. From the existing objects, it resembles a filter more (IMHO), but I'd rather categorize it as something else that transforms/processes the incoming data, and should be therefore in a separate configuration object (along with the geoip parser). Robert

2 2

Problems to receive multiple/single log lines
by Jorge Pereira 04 Jun '16

04 Jun '16

Hi everyone, Question: Is it possible to send multiple/single log lines to the same destination? In this case, exist some way to differentiate the receiving of multiple/single lines? Below my client and server setup. # Client Side <MY CLIENT CONF> options { threaded(yes); flush_lines(0); use-dns(no); normalize-hostnames(yes); keep-hostname(yes); }; destination d_collector_waf { tcp("syslog-server.internal.net" port(514) keep-alive(on) flags(no-parse) ); }; source s_modsec_log { # This file is multiple-line, below the content. # # <SAMPLE CONTENT> #--bfd16c01-H-- #Message: Access denied with code 401 (phase 2). Pattern match "(?i:(?:[\\;\\|\\`]\\W*?\\bcc|\\b(wget|curl))\\b|\\/cc(?:[\\'\"\\|\\;\\`\\-\\s]|$))" at ARGS:a. [file "/usr/local/openresty/nginx/conf/waf/www.happystage.tk.conf"] [line "177"] [id "950907"] [rev "2"] [msg "System Command Injection"] [data "Matched Data: wget found within ARGS:a: wget"] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "8"] [tag "OWASP_CRS/WEB_ATTACK/COMMAND_INJECTION"] [tag "WASCTC/WASC-31"] [tag "OWASP_TOP_10/A1"] [tag "PCI/6.5.2"] #Action: Intercepted (phase 2) #Apache-Handler: IIS #Stopwatch: 1465000225000663 664069 (- - -) #Stopwatch2: 1465000225000663 664069; combined=239, p1=138, p2=80, p3=0, p4=0, p5=20, sr=10, sw=1, l=0, gc=0 #Producer: ModSecurity for nginx (STABLE)/2.8.0 (http://www.modsecurity.org/) 200911012341. #Server: ModSecurity Standalone #Engine-Mode: "ENABLED" # # </SAMPLE CONTENT> # The below regex in "multi-line-prefix()" works fine, transform the inputstream in a single line. # below debug output: # Incoming log entry; line='--3215e80c-Z--\x0a\x0a--4b80ac01-A--\x0a[04/Jun/2016:00:31:57 +0000] AiA7Ac8cA0AWAcAcAcA1@cYc 206.128.156.45 0 127.0.0.1 80\x0a--4b80ac01-B--\x0aGET /WAF-Testing/?a=wget HTTP/1.1\x0ahost: www.happystage.tk\x0arequest-id: 2016-06-04T00:31:57Z|51d35e61f4|206.128.156.45|EcWqiPItjA\x0aaccept: */*\x0auser-agent: curl/7.47.0\x0a\x0a--4b80ac01-H--\x0aMessage: Access denied with code 401 (phase 2). Pattern match "(?i:(?:[\\;\\|\\`]\\W*?\\bcc|\\b(wget|curl))\\b|\\/cc(?:[\\\'\"\\|\\;\\`\\-\\s]|$))" at ARGS:a. [file "/usr/local/openresty/nginx/conf/waf/www.happystage.tk.conf"] [line "177"] [id "950907"] [rev "2"] [msg "System Command Injection"] [data "Matched Data: wget found within ARGS:a: wget"] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "8"] [tag "OWASP_CRS/WEB_ATTACK/COMMAND_INJECTION"] [tag "WASCTC/WASC-31"] [tag "OWASP_TOP_10/A1"] [tag "PCI/6.5.2"]\x0aAction: Intercepted (phase 2)\x0aApache-Handler: IIS\x0aStopwatch: 1465000317000372 373530 (- - -)\x0aStopwatch2: 1465000317000372 373530; combined=221, p1=137, p2=63, p3=0, p4=0, p5=2' file("/usr/local/openresty/nginx/logs/waf/www.mydomain.com" follow_freq(1) flags(no-parse) multi-line-mode(regexp) override_program("ng_waf:www.mydomain.com") multi-line-prefix("--[a-fA-F0-9]{8}-Z--") ); }; source s_access_log { # The content of this file is a single-line delimited by a "\n" file("/usr/local/openresty/nginx/logs/access.log" program_override("ng_access") follow_freq(1) flags(no-parse) ); }; log { source(s_modsec_log); source(s_access_log); destination(d_collector); }; </MY CLIENT CONF> My setver setup: <SERVER CONF> options { flush-lines(100); log-fifo-size(1000); threaded(yes); use-dns(no); normalize-hostnames(yes); keep-hostname(yes); }; source s_collector { tcp(ip(0.0.0.0) port(514) keep-alive(on) flags(no-parse) ); }; filter f_nginx_waf { match("ng_modsec" value("PROGRAM")); }; rewrite r_nginx_waf { subst("ng_modsec:", "", value("PROGRAM")); }; destination d_nginx_waf { file("/var/log/syslog-ng/nginx/waf/${PROGRAM}_log" create_dirs(yes) owner("root") group("root") perm(0640) dir_perm(0750) flags(no-parse) template("${MSG}") ); }; log { source(s_collector); filter(f_nginx_waf); rewrite(r_nginx_waf); destination(d_nginx_waf); flags(flow-control); }; filter f_nginx_access { match("ng_access" value("PROGRAM")); }; destination d_nginx_access { file("/var/log/syslog-ng/nginx/${PROGRAM}_log" create_dirs(yes) owner("root") group("root") perm(0640) dir_perm(0750) flags(no-parse) template("${MSG}") ); }; log { source(s_collector); filter(f_nginx_access); destination(d_nginx_access); flags(flow-control); }; </SERVER CONF>

1 0

How to roll journald cursor in persist file
by Matwey V. Kornilov 02 Jun '16

02 Jun '16

Hello, I am running syslog-ng 3.7.1 and journald v210. The goal I want to reach is the following. I would like to replicate local system logs to remote endpoint using standard UDP protocol. So, I've just disabled all destinations except the following one: destination logserver { udp("10.0.0.47" port(514)); }; log { source(src); destination(logserver); }; The issue is the following. It seems that syslog-ng 3.7.1 uses journald natively for system() source. And I like it. The issue is that when I start syslog-ng it tries to sync all existing local journald data (about 5 GB of compressed journald data, 1.5 years of logs) from the beginning. I found in modules/systemd-journal/journal-reader.c that syslog-ng stores journald cursor in the persists file. I would like to point the cursor to the end of journald database in order to avoid unneeded log sync. The question is how could I do that in opensource syslog-ng version?

3 3

Elasticsearch with Syslog-ng
by Ivan Adji - Krstev 01 Jun '16

01 Jun '16

Hi all, I have the following error when i trying to run syslog-ng. [2016-05-31T11:17:31.298897] Seeking the journal to the last cursor position; cursor='s=6b605b6b981a471ea3d1abe52861ae82;i=21a8;b=42ab72c238874f7d8afc3c2cf0fea52d;m=38dbdc6d9;t=533be9608d083;x=140acfae276a2e55' [2016-05-31T11:17:31.299921] Module loaded and initialized successfully; module='syslogformat' [2016-05-31T11:17:31.299958] Failed to acquire /run/systemd/journal/syslog socket, disabling systemd-syslog source; [2016-05-31T11:17:31.796854] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-31T11:17:31.797915] Add path to classpath: /usr/lib64/syslog-ng/java-modules/commons-codec-1.9.jar; [2016-05-31T11:17:31.798191] Add path to classpath: /usr/lib64/syslog-ng/java-modules/commons-lang3-3.4.jar; [2016-05-31T11:17:31.798466] Add path to classpath: /usr/lib64/syslog-ng/java-modules/commons-logging-1.2.jar; [2016-05-31T11:17:31.798821] Add path to classpath: /usr/lib64/syslog-ng/java-modules/dummy.jar; [2016-05-31T11:17:31.799058] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-v2.jar; [2016-05-31T11:17:31.799296] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic.jar; [2016-05-31T11:17:31.799503] Add path to classpath: /usr/lib64/syslog-ng/java-modules/gson-2.6.2.jar; [2016-05-31T11:17:31.799778] Add path to classpath: /usr/lib64/syslog-ng/java-modules/guava-19.0.jar; [2016-05-31T11:17:31.799988] Add path to classpath: /usr/lib64/syslog-ng/java-modules/hdfs.jar; [2016-05-31T11:17:31.800249] Add path to classpath: /usr/lib64/syslog-ng/java-modules/http.jar; [2016-05-31T11:17:31.800477] Add path to classpath: /usr/lib64/syslog-ng/java-modules/httpasyncclient-4.1.1.jar; [2016-05-31T11:17:31.800684] Add path to classpath: /usr/lib64/syslog-ng/java-modules/httpclient-4.5.2.jar; [2016-05-31T11:17:31.800940] Add path to classpath: /usr/lib64/syslog-ng/java-modules/httpcore-4.4.4.jar; [2016-05-31T11:17:31.801215] Add path to classpath: /usr/lib64/syslog-ng/java-modules/httpcore-nio-4.4.4.jar; [2016-05-31T11:17:31.801449] Add path to classpath: /usr/lib64/syslog-ng/java-modules/jest-2.0.2.jar; [2016-05-31T11:17:31.801667] Add path to classpath: /usr/lib64/syslog-ng/java-modules/jest-common-2.0.2.jar; [2016-05-31T11:17:31.801932] Add path to classpath: /usr/lib64/syslog-ng/java-modules/kafka.jar; [2016-05-31T11:17:31.802119] Add path to classpath: /usr/lib64/syslog-ng/java-modules/log4j-1.2.16.jar; [2016-05-31T11:17:31.802353] Add path to classpath: /usr/lib64/syslog-ng/java-modules/slf4j-api-1.7.13.jar; [2016-05-31T11:17:31.802569] Add path to classpath: /usr/lib64/syslog-ng/java-modules/syslog-ng-common.jar; [2016-05-31T11:17:31.802849] Add path to classpath: /usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-31T11:17:31.803043] Add path to classpath: /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar; [2016-05-31T11:17:31.803264] Add path to classpath: /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar; [2016-05-31T11:17:31.803497] Add path to classpath: /usr/share/elasticsearch/lib/commons-cli-1.3.1.jar; [2016-05-31T11:17:31.803746] Add path to classpath: /usr/share/elasticsearch/lib/compiler-0.8.13.jar; [2016-05-31T11:17:31.803992] Add path to classpath: /usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar; [2016-05-31T11:17:31.804249] Add path to classpath: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar; [2016-05-31T11:17:31.804436] Add path to classpath: /usr/share/elasticsearch/lib/guava-18.0.jar; [2016-05-31T11:17:31.804651] Add path to classpath: /usr/share/elasticsearch/lib/hppc-0.7.1.jar; [2016-05-31T11:17:31.804877] Add path to classpath: /usr/share/elasticsearch/lib/jackson-core-2.6.6.jar; [2016-05-31T11:17:31.805109] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.6.jar; [2016-05-31T11:17:31.805383] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.6.jar; [2016-05-31T11:17:31.805663] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.6.jar; [2016-05-31T11:17:31.805908] Add path to classpath: /usr/share/elasticsearch/lib/jna-4.1.0.jar; [2016-05-31T11:17:31.806160] Add path to classpath: /usr/share/elasticsearch/lib/joda-convert-1.2.jar; [2016-05-31T11:17:31.806402] Add path to classpath: /usr/share/elasticsearch/lib/joda-time-2.8.2.jar; [2016-05-31T11:17:31.806654] Add path to classpath: /usr/share/elasticsearch/lib/jsr166e-1.1.0.jar; [2016-05-31T11:17:31.806889] Add path to classpath: /usr/share/elasticsearch/lib/jts-1.13.jar; [2016-05-31T11:17:31.807087] Add path to classpath: /usr/share/elasticsearch/lib/log4j-1.2.17.jar; [2016-05-31T11:17:31.807260] Add path to classpath: /usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar; [2016-05-31T11:17:31.807476] Add path to classpath: /usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar; [2016-05-31T11:17:31.807759] Add path to classpath: /usr/share/elasticsearch/lib/lucene-core-5.5.0.jar; [2016-05-31T11:17:31.808003] Add path to classpath: /usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar; [2016-05-31T11:17:31.808261] Add path to classpath: /usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar; [2016-05-31T11:17:31.808476] Add path to classpath: /usr/share/elasticsearch/lib/lucene-join-5.5.0.jar; [2016-05-31T11:17:31.808653] Add path to classpath: /usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar; [2016-05-31T11:17:31.808929] Add path to classpath: /usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar; [2016-05-31T11:17:31.809140] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar; [2016-05-31T11:17:31.809362] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar; [2016-05-31T11:17:31.809595] Add path to classpath: /usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar; [2016-05-31T11:17:31.809823] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar; [2016-05-31T11:17:31.810023] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar; [2016-05-31T11:17:31.810229] Add path to classpath: /usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar; [2016-05-31T11:17:31.810427] Add path to classpath: /usr/share/elasticsearch/lib/netty-3.10.5.Final.jar; [2016-05-31T11:17:31.810628] Add path to classpath: /usr/share/elasticsearch/lib/securesm-1.0.jar; [2016-05-31T11:17:31.810885] Add path to classpath: /usr/share/elasticsearch/lib/snakeyaml-1.15.jar; [2016-05-31T11:17:31.811065] Add path to classpath: /usr/share/elasticsearch/lib/spatial4j-0.5.jar; [2016-05-31T11:17:31.811279] Add path to classpath: /usr/share/elasticsearch/lib/t-digest-3.0.jar; [2016-05-31T11:17:33.037026] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-31T11:17:33.073362] Error initializing message pipeline; Im running with Elasticsearch with the following configuration: destination d_es { java( class-path("/usr/lib64/syslog-ng/java-modules/*.jar:/usr/share/elasticsearch/lib/*.jar") class-name("org.syslog_ng.elasticsearch.ElasticSearchDestination") option("index", "syslog-ng_${YEAR}.${MONTH}.${DAY}") option("type", "test") option("cluster", "elasticsearch") # option("flush_limit", "100") option( "message_template", "$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") ); }; Any hints on this ? Ivan

4 13

Syslog-NG with Elasticsearch
by Ivan Adji - Krstev 30 May '16

30 May '16

Does syslog-ng 3.8 supports the latest Elasticsearch version 2 ? Kind regards Ivan

2 3

Web Visualize logs
by Ivan Adji - Krstev 26 May '16

26 May '16

Hi all, Me again :) ! Is there any other tool, free open source that can show me the log in some kind of graphs and can make some triggers etc. Like LogAnalyzer ? I want to change the LogAnalyzer and try other different web interfaces etc. Any suggestions ? I see Elasticsearch or Kibana but, as i see it now, they use some different protocols and styles (i still can't understand how is working) and im not sure if i can integrate with syslog-ng, but also i need a free tool :( . As im using Syslog-NG with MongoDB all i need is something that can present all this log and have options for sending e-mails if we have disaster or something and can query fast all this logs. LogAnalyzers is ok but i want to see different styles. Thanks in advanced ! Ivan

3 13

Transform file path
by Alexey Vlasov 25 May '16

25 May '16

Hi, I intend to manage the distribution of the Apache log-files for each virtualhost using syslog-ng. I write the following in Apache vhost configue: <VirtualHost *> SetEnv V3WUSER w_test-l26-apache-_b8649b LogFormat "%{V3WUSER}e %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" xcombined CustomLog /var/log/apache_aux2_worker2/access_pipe.log xcombined </IfModule> as a result I get this kind of format of the log file: w_test-l26-apache-_b8649b 10.0.2.24 - - [24/May/2016:12:41:33 +0300] "GET / HTTP/1.1" 200 - "http://example.com/" "ELinks (0.11.7; Linux 3.14.46-1gb-csm x86_64; 158x45)" where the first field is the unique identifier of the virtualhost. Next step I write the following in syslog-ng: source src_apache_piped_logs { pipe("/var/log/apache_aux2_worker2/access_pipe.log" keep_timestamp(no)); }; destination dst_apache_piped_logs { file("/var/log/virtwww/${V3WUSER}/access.log" template("$MSGONLY\n") template-escape(no)); }; log { source(src_apache_piped_logs); destination(dst_apache_piped_logs); }; Сonsequently I would like to have each log file of the virtualhost placed in its own catalog and the path should contain the first field of the message (${V3WUSER} = w_test-l26-apache-_b8649b) Is it actually possible? Thanks in advance for the help.

4 3

WANTED: feature owner of syslog-ng command line tool :)
by Noémi Ványi 24 May '16

24 May '16

Hello! A new feature is going to be developed, as a part of GSOC, so syslog-ng can be started as a command line tool. What could the CLI do for you? - pipelines of complex filtering, rewriting could be tested more easily, thus configuration could become simpler than before - existing logs could be transformed The format of the new command line options is not decided yet. So your help and insight is needed to design the interface and the usage of the tool that you and other people around the world would use. :) If you are intereted in contributing to the CLI, thus contributing to the syslog-ng community, please send an email to this thread. :) I am available at gitter for further info. :) Regards kvch

4 5