- syslog-ng - lists.balabit.hu

Invalid parsing of syslog messages having timezone
by Nutan Shinde 10 Jun '16

10 Jun '16

Hi, Following is the syslog message received from Cisco router : *Mar 1 09:30:25.249 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel2, changed state to down As, you can see UTC is included in the above timestamp. That is why value of $PROGRAM is UTC and $MSGONLY is %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel2, changed state to down. What should I include in the syslog-ng.conf so that time zone is ignored?

3 2

grouping-by parser questions
by Fekete, Róbert 07 Jun '16

07 Jun '16

Hi Bazsi, I've started to document the grouping-by parser, and have a few questions/comments about it: * It seems that some of the grouping-by options are the same (or very similar) to the correlation-related attributes of the pattern database, but have different names. Could we name them consistently where they are the same? (I haven't checked the correlation module from Rust, but maybe we could align that as well.) For example: grouping-by | patterndb scope | context-scope timeout | context-timeout aggregate | message or action * In the original commit message, you mention three possible values for the 'scope' option, whereas the context-scope in the patterndb has four (program). Are these deliberately different, or they use the same code? * grouping-by doesn't look to me as an actual parser. From the existing objects, it resembles a filter more (IMHO), but I'd rather categorize it as something else that transforms/processes the incoming data, and should be therefore in a separate configuration object (along with the geoip parser). Robert

2 2

Problems to receive multiple/single log lines
by Jorge Pereira 04 Jun '16

04 Jun '16

Hi everyone, Question: Is it possible to send multiple/single log lines to the same destination? In this case, exist some way to differentiate the receiving of multiple/single lines? Below my client and server setup. # Client Side <MY CLIENT CONF> options { threaded(yes); flush_lines(0); use-dns(no); normalize-hostnames(yes); keep-hostname(yes); }; destination d_collector_waf { tcp("syslog-server.internal.net" port(514) keep-alive(on) flags(no-parse) ); }; source s_modsec_log { # This file is multiple-line, below the content. # # <SAMPLE CONTENT> #--bfd16c01-H-- #Message: Access denied with code 401 (phase 2). Pattern match "(?i:(?:[\\;\\|\\`]\\W*?\\bcc|\\b(wget|curl))\\b|\\/cc(?:[\\'\"\\|\\;\\`\\-\\s]|$))" at ARGS:a. [file "/usr/local/openresty/nginx/conf/waf/www.happystage.tk.conf"] [line "177"] [id "950907"] [rev "2"] [msg "System Command Injection"] [data "Matched Data: wget found within ARGS:a: wget"] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "8"] [tag "OWASP_CRS/WEB_ATTACK/COMMAND_INJECTION"] [tag "WASCTC/WASC-31"] [tag "OWASP_TOP_10/A1"] [tag "PCI/6.5.2"] #Action: Intercepted (phase 2) #Apache-Handler: IIS #Stopwatch: 1465000225000663 664069 (- - -) #Stopwatch2: 1465000225000663 664069; combined=239, p1=138, p2=80, p3=0, p4=0, p5=20, sr=10, sw=1, l=0, gc=0 #Producer: ModSecurity for nginx (STABLE)/2.8.0 (http://www.modsecurity.org/) 200911012341. #Server: ModSecurity Standalone #Engine-Mode: "ENABLED" # # </SAMPLE CONTENT> # The below regex in "multi-line-prefix()" works fine, transform the inputstream in a single line. # below debug output: # Incoming log entry; line='--3215e80c-Z--\x0a\x0a--4b80ac01-A--\x0a[04/Jun/2016:00:31:57 +0000] AiA7Ac8cA0AWAcAcAcA1@cYc 206.128.156.45 0 127.0.0.1 80\x0a--4b80ac01-B--\x0aGET /WAF-Testing/?a=wget HTTP/1.1\x0ahost: www.happystage.tk\x0arequest-id: 2016-06-04T00:31:57Z|51d35e61f4|206.128.156.45|EcWqiPItjA\x0aaccept: */*\x0auser-agent: curl/7.47.0\x0a\x0a--4b80ac01-H--\x0aMessage: Access denied with code 401 (phase 2). Pattern match "(?i:(?:[\\;\\|\\`]\\W*?\\bcc|\\b(wget|curl))\\b|\\/cc(?:[\\\'\"\\|\\;\\`\\-\\s]|$))" at ARGS:a. [file "/usr/local/openresty/nginx/conf/waf/www.happystage.tk.conf"] [line "177"] [id "950907"] [rev "2"] [msg "System Command Injection"] [data "Matched Data: wget found within ARGS:a: wget"] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "8"] [tag "OWASP_CRS/WEB_ATTACK/COMMAND_INJECTION"] [tag "WASCTC/WASC-31"] [tag "OWASP_TOP_10/A1"] [tag "PCI/6.5.2"]\x0aAction: Intercepted (phase 2)\x0aApache-Handler: IIS\x0aStopwatch: 1465000317000372 373530 (- - -)\x0aStopwatch2: 1465000317000372 373530; combined=221, p1=137, p2=63, p3=0, p4=0, p5=2' file("/usr/local/openresty/nginx/logs/waf/www.mydomain.com" follow_freq(1) flags(no-parse) multi-line-mode(regexp) override_program("ng_waf:www.mydomain.com") multi-line-prefix("--[a-fA-F0-9]{8}-Z--") ); }; source s_access_log { # The content of this file is a single-line delimited by a "\n" file("/usr/local/openresty/nginx/logs/access.log" program_override("ng_access") follow_freq(1) flags(no-parse) ); }; log { source(s_modsec_log); source(s_access_log); destination(d_collector); }; </MY CLIENT CONF> My setver setup: <SERVER CONF> options { flush-lines(100); log-fifo-size(1000); threaded(yes); use-dns(no); normalize-hostnames(yes); keep-hostname(yes); }; source s_collector { tcp(ip(0.0.0.0) port(514) keep-alive(on) flags(no-parse) ); }; filter f_nginx_waf { match("ng_modsec" value("PROGRAM")); }; rewrite r_nginx_waf { subst("ng_modsec:", "", value("PROGRAM")); }; destination d_nginx_waf { file("/var/log/syslog-ng/nginx/waf/${PROGRAM}_log" create_dirs(yes) owner("root") group("root") perm(0640) dir_perm(0750) flags(no-parse) template("${MSG}") ); }; log { source(s_collector); filter(f_nginx_waf); rewrite(r_nginx_waf); destination(d_nginx_waf); flags(flow-control); }; filter f_nginx_access { match("ng_access" value("PROGRAM")); }; destination d_nginx_access { file("/var/log/syslog-ng/nginx/${PROGRAM}_log" create_dirs(yes) owner("root") group("root") perm(0640) dir_perm(0750) flags(no-parse) template("${MSG}") ); }; log { source(s_collector); filter(f_nginx_access); destination(d_nginx_access); flags(flow-control); }; </SERVER CONF>

1 0

How to roll journald cursor in persist file
by Matwey V. Kornilov 02 Jun '16

02 Jun '16

Hello, I am running syslog-ng 3.7.1 and journald v210. The goal I want to reach is the following. I would like to replicate local system logs to remote endpoint using standard UDP protocol. So, I've just disabled all destinations except the following one: destination logserver { udp("10.0.0.47" port(514)); }; log { source(src); destination(logserver); }; The issue is the following. It seems that syslog-ng 3.7.1 uses journald natively for system() source. And I like it. The issue is that when I start syslog-ng it tries to sync all existing local journald data (about 5 GB of compressed journald data, 1.5 years of logs) from the beginning. I found in modules/systemd-journal/journal-reader.c that syslog-ng stores journald cursor in the persists file. I would like to point the cursor to the end of journald database in order to avoid unneeded log sync. The question is how could I do that in opensource syslog-ng version?

3 3

Elasticsearch with Syslog-ng
by Ivan Adji - Krstev 02 Jun '16

02 Jun '16

Hi all, I have the following error when i trying to run syslog-ng. [2016-05-31T11:17:31.298897] Seeking the journal to the last cursor position; cursor='s=6b605b6b981a471ea3d1abe52861ae82;i=21a8;b=42ab72c238874f7d8afc3c2cf0fea52d;m=38dbdc6d9;t=533be9608d083;x=140acfae276a2e55' [2016-05-31T11:17:31.299921] Module loaded and initialized successfully; module='syslogformat' [2016-05-31T11:17:31.299958] Failed to acquire /run/systemd/journal/syslog socket, disabling systemd-syslog source; [2016-05-31T11:17:31.796854] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-31T11:17:31.797915] Add path to classpath: /usr/lib64/syslog-ng/java-modules/commons-codec-1.9.jar; [2016-05-31T11:17:31.798191] Add path to classpath: /usr/lib64/syslog-ng/java-modules/commons-lang3-3.4.jar; [2016-05-31T11:17:31.798466] Add path to classpath: /usr/lib64/syslog-ng/java-modules/commons-logging-1.2.jar; [2016-05-31T11:17:31.798821] Add path to classpath: /usr/lib64/syslog-ng/java-modules/dummy.jar; [2016-05-31T11:17:31.799058] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-v2.jar; [2016-05-31T11:17:31.799296] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic.jar; [2016-05-31T11:17:31.799503] Add path to classpath: /usr/lib64/syslog-ng/java-modules/gson-2.6.2.jar; [2016-05-31T11:17:31.799778] Add path to classpath: /usr/lib64/syslog-ng/java-modules/guava-19.0.jar; [2016-05-31T11:17:31.799988] Add path to classpath: /usr/lib64/syslog-ng/java-modules/hdfs.jar; [2016-05-31T11:17:31.800249] Add path to classpath: /usr/lib64/syslog-ng/java-modules/http.jar; [2016-05-31T11:17:31.800477] Add path to classpath: /usr/lib64/syslog-ng/java-modules/httpasyncclient-4.1.1.jar; [2016-05-31T11:17:31.800684] Add path to classpath: /usr/lib64/syslog-ng/java-modules/httpclient-4.5.2.jar; [2016-05-31T11:17:31.800940] Add path to classpath: /usr/lib64/syslog-ng/java-modules/httpcore-4.4.4.jar; [2016-05-31T11:17:31.801215] Add path to classpath: /usr/lib64/syslog-ng/java-modules/httpcore-nio-4.4.4.jar; [2016-05-31T11:17:31.801449] Add path to classpath: /usr/lib64/syslog-ng/java-modules/jest-2.0.2.jar; [2016-05-31T11:17:31.801667] Add path to classpath: /usr/lib64/syslog-ng/java-modules/jest-common-2.0.2.jar; [2016-05-31T11:17:31.801932] Add path to classpath: /usr/lib64/syslog-ng/java-modules/kafka.jar; [2016-05-31T11:17:31.802119] Add path to classpath: /usr/lib64/syslog-ng/java-modules/log4j-1.2.16.jar; [2016-05-31T11:17:31.802353] Add path to classpath: /usr/lib64/syslog-ng/java-modules/slf4j-api-1.7.13.jar; [2016-05-31T11:17:31.802569] Add path to classpath: /usr/lib64/syslog-ng/java-modules/syslog-ng-common.jar; [2016-05-31T11:17:31.802849] Add path to classpath: /usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-31T11:17:31.803043] Add path to classpath: /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar; [2016-05-31T11:17:31.803264] Add path to classpath: /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar; [2016-05-31T11:17:31.803497] Add path to classpath: /usr/share/elasticsearch/lib/commons-cli-1.3.1.jar; [2016-05-31T11:17:31.803746] Add path to classpath: /usr/share/elasticsearch/lib/compiler-0.8.13.jar; [2016-05-31T11:17:31.803992] Add path to classpath: /usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar; [2016-05-31T11:17:31.804249] Add path to classpath: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar; [2016-05-31T11:17:31.804436] Add path to classpath: /usr/share/elasticsearch/lib/guava-18.0.jar; [2016-05-31T11:17:31.804651] Add path to classpath: /usr/share/elasticsearch/lib/hppc-0.7.1.jar; [2016-05-31T11:17:31.804877] Add path to classpath: /usr/share/elasticsearch/lib/jackson-core-2.6.6.jar; [2016-05-31T11:17:31.805109] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.6.jar; [2016-05-31T11:17:31.805383] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.6.jar; [2016-05-31T11:17:31.805663] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.6.jar; [2016-05-31T11:17:31.805908] Add path to classpath: /usr/share/elasticsearch/lib/jna-4.1.0.jar; [2016-05-31T11:17:31.806160] Add path to classpath: /usr/share/elasticsearch/lib/joda-convert-1.2.jar; [2016-05-31T11:17:31.806402] Add path to classpath: /usr/share/elasticsearch/lib/joda-time-2.8.2.jar; [2016-05-31T11:17:31.806654] Add path to classpath: /usr/share/elasticsearch/lib/jsr166e-1.1.0.jar; [2016-05-31T11:17:31.806889] Add path to classpath: /usr/share/elasticsearch/lib/jts-1.13.jar; [2016-05-31T11:17:31.807087] Add path to classpath: /usr/share/elasticsearch/lib/log4j-1.2.17.jar; [2016-05-31T11:17:31.807260] Add path to classpath: /usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar; [2016-05-31T11:17:31.807476] Add path to classpath: /usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar; [2016-05-31T11:17:31.807759] Add path to classpath: /usr/share/elasticsearch/lib/lucene-core-5.5.0.jar; [2016-05-31T11:17:31.808003] Add path to classpath: /usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar; [2016-05-31T11:17:31.808261] Add path to classpath: /usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar; [2016-05-31T11:17:31.808476] Add path to classpath: /usr/share/elasticsearch/lib/lucene-join-5.5.0.jar; [2016-05-31T11:17:31.808653] Add path to classpath: /usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar; [2016-05-31T11:17:31.808929] Add path to classpath: /usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar; [2016-05-31T11:17:31.809140] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar; [2016-05-31T11:17:31.809362] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar; [2016-05-31T11:17:31.809595] Add path to classpath: /usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar; [2016-05-31T11:17:31.809823] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar; [2016-05-31T11:17:31.810023] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar; [2016-05-31T11:17:31.810229] Add path to classpath: /usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar; [2016-05-31T11:17:31.810427] Add path to classpath: /usr/share/elasticsearch/lib/netty-3.10.5.Final.jar; [2016-05-31T11:17:31.810628] Add path to classpath: /usr/share/elasticsearch/lib/securesm-1.0.jar; [2016-05-31T11:17:31.810885] Add path to classpath: /usr/share/elasticsearch/lib/snakeyaml-1.15.jar; [2016-05-31T11:17:31.811065] Add path to classpath: /usr/share/elasticsearch/lib/spatial4j-0.5.jar; [2016-05-31T11:17:31.811279] Add path to classpath: /usr/share/elasticsearch/lib/t-digest-3.0.jar; [2016-05-31T11:17:33.037026] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-31T11:17:33.073362] Error initializing message pipeline; Im running with Elasticsearch with the following configuration: destination d_es { java( class-path("/usr/lib64/syslog-ng/java-modules/*.jar:/usr/share/elasticsearch/lib/*.jar") class-name("org.syslog_ng.elasticsearch.ElasticSearchDestination") option("index", "syslog-ng_${YEAR}.${MONTH}.${DAY}") option("type", "test") option("cluster", "elasticsearch") # option("flush_limit", "100") option( "message_template", "$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") ); }; Any hints on this ? Ivan

4 13

Syslog-NG with Elasticsearch
by Ivan Adji - Krstev 30 May '16

30 May '16

Does syslog-ng 3.8 supports the latest Elasticsearch version 2 ? Kind regards Ivan

2 3

Web Visualize logs
by Ivan Adji - Krstev 26 May '16

26 May '16

Hi all, Me again :) ! Is there any other tool, free open source that can show me the log in some kind of graphs and can make some triggers etc. Like LogAnalyzer ? I want to change the LogAnalyzer and try other different web interfaces etc. Any suggestions ? I see Elasticsearch or Kibana but, as i see it now, they use some different protocols and styles (i still can't understand how is working) and im not sure if i can integrate with syslog-ng, but also i need a free tool :( . As im using Syslog-NG with MongoDB all i need is something that can present all this log and have options for sending e-mails if we have disaster or something and can query fast all this logs. LogAnalyzers is ok but i want to see different styles. Thanks in advanced ! Ivan

3 13

Transform file path
by Alexey Vlasov 25 May '16

25 May '16

Hi, I intend to manage the distribution of the Apache log-files for each virtualhost using syslog-ng. I write the following in Apache vhost configue: <VirtualHost *> SetEnv V3WUSER w_test-l26-apache-_b8649b LogFormat "%{V3WUSER}e %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" xcombined CustomLog /var/log/apache_aux2_worker2/access_pipe.log xcombined </IfModule> as a result I get this kind of format of the log file: w_test-l26-apache-_b8649b 10.0.2.24 - - [24/May/2016:12:41:33 +0300] "GET / HTTP/1.1" 200 - "http://example.com/" "ELinks (0.11.7; Linux 3.14.46-1gb-csm x86_64; 158x45)" where the first field is the unique identifier of the virtualhost. Next step I write the following in syslog-ng: source src_apache_piped_logs { pipe("/var/log/apache_aux2_worker2/access_pipe.log" keep_timestamp(no)); }; destination dst_apache_piped_logs { file("/var/log/virtwww/${V3WUSER}/access.log" template("$MSGONLY\n") template-escape(no)); }; log { source(src_apache_piped_logs); destination(dst_apache_piped_logs); }; Сonsequently I would like to have each log file of the virtualhost placed in its own catalog and the path should contain the first field of the message (${V3WUSER} = w_test-l26-apache-_b8649b) Is it actually possible? Thanks in advance for the help.

4 3

WANTED: feature owner of syslog-ng command line tool :)
by Noémi Ványi 24 May '16

24 May '16

Hello! A new feature is going to be developed, as a part of GSOC, so syslog-ng can be started as a command line tool. What could the CLI do for you? - pipelines of complex filtering, rewriting could be tested more easily, thus configuration could become simpler than before - existing logs could be transformed The format of the new command line options is not decided yet. So your help and insight is needed to design the interface and the usage of the tool that you and other people around the world would use. :) If you are intereted in contributing to the CLI, thus contributing to the syslog-ng community, please send an email to this thread. :) I am available at gitter for further info. :) Regards kvch

4 5

Syslog-NG with MongoDB
by Ivan Adji - Krstev 24 May '16

24 May '16

Hi all, What is the best practice for storing all those logs in one central environment. I have one Linux Box running Syslog-NG with LogAnalyzer and MongoDB ( for now ), and is the best way to configure and use it with MongoDB or with MariaDB ( MySQL ) ? I have once install MySQL but it was getting very slow as the logs getting bigger and bigger ( for one week ). Now i have done with MongoDB ( still testing ) but i have problem as LogAnalyzer does not show me the real pictures, i have no Date info, no Facility, no serverity, Hosts, syslogtag, i just have ProcessID. Any hints on this ? I have the following configuration on the syslog-ng.cfg: destination d_mongodb { mongodb( servers("localhost:27017") database("logs") # uri('mongodb://localhost/syslog-ng') collection("syslog") value-pairs( scope("selected-macros" "nv-pairs" "sdata") ) ); }; Kind regards Ivan

5 10