May 2016 - syslog-ng - lists.balabit.hu

Re: [syslog-ng] Web Visualize logs (Ivan Adji - Krstev)
by Szabó, István 07 Sep '16

07 Sep '16

Hi, For using syslog-ng together with elasticsearch and Kibana this is a pretty good description, also giving you a good understanding of what it enables you to do: https://czanik.blogs.balabit.com/2015/10/how-to-parse-data-with-syslog-ng-s… https://czanik.blogs.balabit.com/2015/12/elasticsearch-and-syslog-ng-fast-a… /Istvan

9 24

powered by syslog-ng
by Czanik, Péter 19 Jul '16

19 Jul '16

Hi, We have now a "Powered by syslog-ng" page on our website. It's available at https://syslog-ng.org/powered-by-syslog-ng/ If your project / company / product / etc. uses syslog-ng, we are happy to include you on the list too, just e-mail me directly! Some of the more interesting names from the list are Facebook, Kindle, FreeNAS and of course all the major Linux distributions. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

2 1

How to roll journald cursor in persist file
by Matwey V. Kornilov 02 Jun '16

02 Jun '16

Hello, I am running syslog-ng 3.7.1 and journald v210. The goal I want to reach is the following. I would like to replicate local system logs to remote endpoint using standard UDP protocol. So, I've just disabled all destinations except the following one: destination logserver { udp("10.0.0.47" port(514)); }; log { source(src); destination(logserver); }; The issue is the following. It seems that syslog-ng 3.7.1 uses journald natively for system() source. And I like it. The issue is that when I start syslog-ng it tries to sync all existing local journald data (about 5 GB of compressed journald data, 1.5 years of logs) from the beginning. I found in modules/systemd-journal/journal-reader.c that syslog-ng stores journald cursor in the persists file. I would like to point the cursor to the end of journald database in order to avoid unneeded log sync. The question is how could I do that in opensource syslog-ng version?

3 3

Elasticsearch with Syslog-ng
by Ivan Adji - Krstev 01 Jun '16

01 Jun '16

Hi all, I have the following error when i trying to run syslog-ng. [2016-05-31T11:17:31.298897] Seeking the journal to the last cursor position; cursor='s=6b605b6b981a471ea3d1abe52861ae82;i=21a8;b=42ab72c238874f7d8afc3c2cf0fea52d;m=38dbdc6d9;t=533be9608d083;x=140acfae276a2e55' [2016-05-31T11:17:31.299921] Module loaded and initialized successfully; module='syslogformat' [2016-05-31T11:17:31.299958] Failed to acquire /run/systemd/journal/syslog socket, disabling systemd-syslog source; [2016-05-31T11:17:31.796854] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-31T11:17:31.797915] Add path to classpath: /usr/lib64/syslog-ng/java-modules/commons-codec-1.9.jar; [2016-05-31T11:17:31.798191] Add path to classpath: /usr/lib64/syslog-ng/java-modules/commons-lang3-3.4.jar; [2016-05-31T11:17:31.798466] Add path to classpath: /usr/lib64/syslog-ng/java-modules/commons-logging-1.2.jar; [2016-05-31T11:17:31.798821] Add path to classpath: /usr/lib64/syslog-ng/java-modules/dummy.jar; [2016-05-31T11:17:31.799058] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic-v2.jar; [2016-05-31T11:17:31.799296] Add path to classpath: /usr/lib64/syslog-ng/java-modules/elastic.jar; [2016-05-31T11:17:31.799503] Add path to classpath: /usr/lib64/syslog-ng/java-modules/gson-2.6.2.jar; [2016-05-31T11:17:31.799778] Add path to classpath: /usr/lib64/syslog-ng/java-modules/guava-19.0.jar; [2016-05-31T11:17:31.799988] Add path to classpath: /usr/lib64/syslog-ng/java-modules/hdfs.jar; [2016-05-31T11:17:31.800249] Add path to classpath: /usr/lib64/syslog-ng/java-modules/http.jar; [2016-05-31T11:17:31.800477] Add path to classpath: /usr/lib64/syslog-ng/java-modules/httpasyncclient-4.1.1.jar; [2016-05-31T11:17:31.800684] Add path to classpath: /usr/lib64/syslog-ng/java-modules/httpclient-4.5.2.jar; [2016-05-31T11:17:31.800940] Add path to classpath: /usr/lib64/syslog-ng/java-modules/httpcore-4.4.4.jar; [2016-05-31T11:17:31.801215] Add path to classpath: /usr/lib64/syslog-ng/java-modules/httpcore-nio-4.4.4.jar; [2016-05-31T11:17:31.801449] Add path to classpath: /usr/lib64/syslog-ng/java-modules/jest-2.0.2.jar; [2016-05-31T11:17:31.801667] Add path to classpath: /usr/lib64/syslog-ng/java-modules/jest-common-2.0.2.jar; [2016-05-31T11:17:31.801932] Add path to classpath: /usr/lib64/syslog-ng/java-modules/kafka.jar; [2016-05-31T11:17:31.802119] Add path to classpath: /usr/lib64/syslog-ng/java-modules/log4j-1.2.16.jar; [2016-05-31T11:17:31.802353] Add path to classpath: /usr/lib64/syslog-ng/java-modules/slf4j-api-1.7.13.jar; [2016-05-31T11:17:31.802569] Add path to classpath: /usr/lib64/syslog-ng/java-modules/syslog-ng-common.jar; [2016-05-31T11:17:31.802849] Add path to classpath: /usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-31T11:17:31.803043] Add path to classpath: /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar; [2016-05-31T11:17:31.803264] Add path to classpath: /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar; [2016-05-31T11:17:31.803497] Add path to classpath: /usr/share/elasticsearch/lib/commons-cli-1.3.1.jar; [2016-05-31T11:17:31.803746] Add path to classpath: /usr/share/elasticsearch/lib/compiler-0.8.13.jar; [2016-05-31T11:17:31.803992] Add path to classpath: /usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar; [2016-05-31T11:17:31.804249] Add path to classpath: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar; [2016-05-31T11:17:31.804436] Add path to classpath: /usr/share/elasticsearch/lib/guava-18.0.jar; [2016-05-31T11:17:31.804651] Add path to classpath: /usr/share/elasticsearch/lib/hppc-0.7.1.jar; [2016-05-31T11:17:31.804877] Add path to classpath: /usr/share/elasticsearch/lib/jackson-core-2.6.6.jar; [2016-05-31T11:17:31.805109] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.6.jar; [2016-05-31T11:17:31.805383] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.6.jar; [2016-05-31T11:17:31.805663] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.6.jar; [2016-05-31T11:17:31.805908] Add path to classpath: /usr/share/elasticsearch/lib/jna-4.1.0.jar; [2016-05-31T11:17:31.806160] Add path to classpath: /usr/share/elasticsearch/lib/joda-convert-1.2.jar; [2016-05-31T11:17:31.806402] Add path to classpath: /usr/share/elasticsearch/lib/joda-time-2.8.2.jar; [2016-05-31T11:17:31.806654] Add path to classpath: /usr/share/elasticsearch/lib/jsr166e-1.1.0.jar; [2016-05-31T11:17:31.806889] Add path to classpath: /usr/share/elasticsearch/lib/jts-1.13.jar; [2016-05-31T11:17:31.807087] Add path to classpath: /usr/share/elasticsearch/lib/log4j-1.2.17.jar; [2016-05-31T11:17:31.807260] Add path to classpath: /usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar; [2016-05-31T11:17:31.807476] Add path to classpath: /usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar; [2016-05-31T11:17:31.807759] Add path to classpath: /usr/share/elasticsearch/lib/lucene-core-5.5.0.jar; [2016-05-31T11:17:31.808003] Add path to classpath: /usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar; [2016-05-31T11:17:31.808261] Add path to classpath: /usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar; [2016-05-31T11:17:31.808476] Add path to classpath: /usr/share/elasticsearch/lib/lucene-join-5.5.0.jar; [2016-05-31T11:17:31.808653] Add path to classpath: /usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar; [2016-05-31T11:17:31.808929] Add path to classpath: /usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar; [2016-05-31T11:17:31.809140] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar; [2016-05-31T11:17:31.809362] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar; [2016-05-31T11:17:31.809595] Add path to classpath: /usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar; [2016-05-31T11:17:31.809823] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar; [2016-05-31T11:17:31.810023] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar; [2016-05-31T11:17:31.810229] Add path to classpath: /usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar; [2016-05-31T11:17:31.810427] Add path to classpath: /usr/share/elasticsearch/lib/netty-3.10.5.Final.jar; [2016-05-31T11:17:31.810628] Add path to classpath: /usr/share/elasticsearch/lib/securesm-1.0.jar; [2016-05-31T11:17:31.810885] Add path to classpath: /usr/share/elasticsearch/lib/snakeyaml-1.15.jar; [2016-05-31T11:17:31.811065] Add path to classpath: /usr/share/elasticsearch/lib/spatial4j-0.5.jar; [2016-05-31T11:17:31.811279] Add path to classpath: /usr/share/elasticsearch/lib/t-digest-3.0.jar; [2016-05-31T11:17:33.037026] Add path to classpath: //usr/lib64/syslog-ng/java-modules/syslog-ng-core.jar; [2016-05-31T11:17:33.073362] Error initializing message pipeline; Im running with Elasticsearch with the following configuration: destination d_es { java( class-path("/usr/lib64/syslog-ng/java-modules/*.jar:/usr/share/elasticsearch/lib/*.jar") class-name("org.syslog_ng.elasticsearch.ElasticSearchDestination") option("index", "syslog-ng_${YEAR}.${MONTH}.${DAY}") option("type", "test") option("cluster", "elasticsearch") # option("flush_limit", "100") option( "message_template", "$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") ); }; Any hints on this ? Ivan

4 13

Syslog-NG with Elasticsearch
by Ivan Adji - Krstev 30 May '16

30 May '16

Does syslog-ng 3.8 supports the latest Elasticsearch version 2 ? Kind regards Ivan

2 3

Web Visualize logs
by Ivan Adji - Krstev 26 May '16

26 May '16

Hi all, Me again :) ! Is there any other tool, free open source that can show me the log in some kind of graphs and can make some triggers etc. Like LogAnalyzer ? I want to change the LogAnalyzer and try other different web interfaces etc. Any suggestions ? I see Elasticsearch or Kibana but, as i see it now, they use some different protocols and styles (i still can't understand how is working) and im not sure if i can integrate with syslog-ng, but also i need a free tool :( . As im using Syslog-NG with MongoDB all i need is something that can present all this log and have options for sending e-mails if we have disaster or something and can query fast all this logs. LogAnalyzers is ok but i want to see different styles. Thanks in advanced ! Ivan

3 13

Transform file path
by Alexey Vlasov 25 May '16

25 May '16

Hi, I intend to manage the distribution of the Apache log-files for each virtualhost using syslog-ng. I write the following in Apache vhost configue: <VirtualHost *> SetEnv V3WUSER w_test-l26-apache-_b8649b LogFormat "%{V3WUSER}e %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" xcombined CustomLog /var/log/apache_aux2_worker2/access_pipe.log xcombined </IfModule> as a result I get this kind of format of the log file: w_test-l26-apache-_b8649b 10.0.2.24 - - [24/May/2016:12:41:33 +0300] "GET / HTTP/1.1" 200 - "http://example.com/" "ELinks (0.11.7; Linux 3.14.46-1gb-csm x86_64; 158x45)" where the first field is the unique identifier of the virtualhost. Next step I write the following in syslog-ng: source src_apache_piped_logs { pipe("/var/log/apache_aux2_worker2/access_pipe.log" keep_timestamp(no)); }; destination dst_apache_piped_logs { file("/var/log/virtwww/${V3WUSER}/access.log" template("$MSGONLY\n") template-escape(no)); }; log { source(src_apache_piped_logs); destination(dst_apache_piped_logs); }; Сonsequently I would like to have each log file of the virtualhost placed in its own catalog and the path should contain the first field of the message (${V3WUSER} = w_test-l26-apache-_b8649b) Is it actually possible? Thanks in advance for the help.

4 3

WANTED: feature owner of syslog-ng command line tool :)
by Noémi Ványi 24 May '16

24 May '16

Hello! A new feature is going to be developed, as a part of GSOC, so syslog-ng can be started as a command line tool. What could the CLI do for you? - pipelines of complex filtering, rewriting could be tested more easily, thus configuration could become simpler than before - existing logs could be transformed The format of the new command line options is not decided yet. So your help and insight is needed to design the interface and the usage of the tool that you and other people around the world would use. :) If you are intereted in contributing to the CLI, thus contributing to the syslog-ng community, please send an email to this thread. :) I am available at gitter for further info. :) Regards kvch

4 5

Syslog-NG with MongoDB
by Ivan Adji - Krstev 24 May '16

24 May '16

Hi all, What is the best practice for storing all those logs in one central environment. I have one Linux Box running Syslog-NG with LogAnalyzer and MongoDB ( for now ), and is the best way to configure and use it with MongoDB or with MariaDB ( MySQL ) ? I have once install MySQL but it was getting very slow as the logs getting bigger and bigger ( for one week ). Now i have done with MongoDB ( still testing ) but i have problem as LogAnalyzer does not show me the real pictures, i have no Date info, no Facility, no serverity, Hosts, syslogtag, i just have ProcessID. Any hints on this ? I have the following configuration on the syslog-ng.cfg: destination d_mongodb { mongodb( servers("localhost:27017") database("logs") # uri('mongodb://localhost/syslog-ng') collection("syslog") value-pairs( scope("selected-macros" "nv-pairs" "sdata") ) ); }; Kind regards Ivan

5 10

[GSoC]Finding Feature owner for "Web-based syslog-ng configuration editor"
by Raymund Ács 23 May '16

23 May '16

Hi, I am PoOwAa (http://github.com/poowaa) I could not write here before because I had some issues with subscribing. I already subscribed and unsubscribed few months ago, and I was not able to subscribe again. Finally I got my confirmation mail, so I can join to the community! :) In the summer I will work on the Web-based syslog-ng configuration editor project in GSoC 2016. Here is my proposal link: https://github.com/balabit/syslog-ng/wiki/GSoC-2016-Proposal:-Web-based-sys… This project will provide a Web Application which can create and save syslog-ng configuration files by drag&drop technique. I would like to follow the trends of 2016 in design, and give a detailed documentation. If you are interested in being the feature owner of this project, please reply in this thread. Advices from you will help us greatly and make the final project more user friendly. Best Regards, Raymund Acs University of Debrecen

1 0