- syslog-ng - lists.balabit.hu

./configure not working
by amey jain 11 Feb '16

11 Feb '16

Hi, I started building syslog-ng 3.8.0alpha0 on Ubuntu 14.04 LTS. I am having a problem while running ./configure for it. I have installed the dependencies all and it is showing an error now. Please see to attachment file and tell me whats the problem. Regards

3 2

Insider 2016-02: syslog-ng 3.8; Hadoop; Rust; Scale
by Czanik, Péter 11 Feb '16

11 Feb '16

Dear syslog-ng users, This is the 47th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS 3.8 news: disk buffer, performance and more ------------------------------------------- While syslog-ng 3.8 is still in alpha phase, it already has a nice list of new features merged or waiting to be merged. It will feature disk based buffering, performance improvements, MongoDB support rewritten, and many more. Packages from git HEAD are planned to become available in the coming weeks and will be announced on the mailing list. For details, browse commit history or download sources from https://github.com/balabit/syslog-ng The Hadoop (HDFS) destination ----------------------------- Petabytes of data are now collected into huge data lakes around the world. Hadoop is the technology enabling this. While syslog-ng was able write logs to Hadoop using some workarounds for quite some time, the new Java-based HDFS destination driver provides both better performance and more reliability. The syslog-ng HDFS driver is Hortonworks HDP certified. https://czanik.blogs.balabit.com/2016/02/filling-your-data-lake-with-log-me… Rust: simple and efficient parsers ---------------------------------- If you want to create high quality, efficient parsers easily, writing them in Rust is worth considering. This post explaines the steps to use the experimental Rust language bindings for syslog-ng. There is even some sample code to help you get started on github. Read more at https://syslog-ng.org/syslog-ng-and-rust/ Scale 14x --------- Balabit, and thus syslog-ng participated the Southern California Linux Expo this year again. The most frequent topic at the booth was ESK (Elasticsearch syslog-ng Kibana). We had some very good discussions at the syslog-ng BoF and after my presentation about syslog-ng and Big Data: https://czanik.blogs.balabit.com/2016/02/czp-scale-14x-esk-bof-drm/ Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com . To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

syslog-ng and dependency libraries for sending log messages for remote log server with TLS support
by Girish Kumar 09 Feb '16

09 Feb '16

Hi All, My requirement is to send the log messages to remote log server securely (with TLS). The project which I am implementing is on embedded systems. So the space is very important. Please let me know the latest stable version of syslog-ng which can be used for this. Also please let me know the mandatory dependency libraries required. Thanks in advance. Regards, Girish _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

4 13

Re: [syslog-ng] Module development guide
by Scheidler, Balázs 06 Feb '16

06 Feb '16

Hi, Let me try to describe what is needed to parse options from the configuration file: - curl-grammar.ym: is a yacc/bison source file that parses the curl specific portion of the configuration. It is able to generate parsers for LALR(1) languages. - curl-parser.c: contains the set of keywords that curl understands. This maps keywords found in the configuration into tokens that the grammar is able to process. A token is basically a numeric identifier. So for instance to add a "header" option, you will need this stuff: 1) define a token in the grammar, so it becomes available as a macro diff --git a/modules/curl/curl-grammar.ym b/modules/curl/curl-grammar.ym index c3aaf00..eab317c 100644 --- a/modules/curl/curl-grammar.ym +++ b/modules/curl/curl-grammar.ym @@ -47,6 +47,7 @@ %parse-param {gpointer arg} %token KW_CURL +%token KW_HEADER %type <ptr> driver %type <ptr> curl_destination If you recompile the curl module, this will make KW_HEADER to become available in curl-grammar.h 2) add a keyword to curl-parser.c This will tell the syslog-ng lexer (a component that reads the configuration file and breaks them into tokens) to map the string "header" into a symbolic token named KW_HEADER. diff --git a/modules/curl/curl-parser.c b/modules/curl/curl-parser.c index 6958a8d..b60ef0b 100644 --- a/modules/curl/curl-parser.c +++ b/modules/curl/curl-parser.c @@ -30,6 +30,7 @@ int curl_parse(CfgLexer *lexer, LogDriver **instance, gpointer arg); static CfgLexerKeyword curl_keywords[] = { { "curl", KW_CURL }, + { "header", KW_HEADER }, { NULL } }; 3) add rules to the grammar to parse this option First we need a small refactor in the curl-grammar file to save the just instantiated CurlDestinationDriver (I hand edited this hunk for clarity, so it will not apply automatically): @@ -71,7 +72,15 @@ driver ; curl_destination - : KW_CURL '(' ')' { $$ = curl_dd_new(configuration); } + : KW_CURL + { + last_driver = curl_dd_new(configuration); + } + '(' ')' { $$ = last_driver; } + ; This basically just stores the driver instance in a variable named "last_driver" (defined by the core cfg-grammar.y file). So we can later use this to call setters on. And now let's add option parsing (this time the entire hunk): @@ -71,7 +72,20 @@ driver ; curl_destination - : KW_CURL '(' ')' { $$ = curl_dd_new(configuration); } + : KW_CURL + { + last_driver = curl_dd_new(configuration); + } + '(' curl_options ')' { $$ = last_driver; } + ; + +curl_options + : curl_option curl_options + | + ; + +curl_option + : KW_HEADER '(' string string ')' { curl_dd_set_header(last_driver, $3, $4); } ; /* INCLUDE_RULES */ What this does is that we added a rule "curl_options" within the parentheses of our original "curl" rule, which is the rule that must match the configuration as we parse it. (think of it as a recursive tree). curl_options is satisfied with either an empty rule (e.g. no options) or recursing itself with a curl_option rule. Note the plural and singular names, curl_options is responsible to match a list of "curl_option"s. curl_option is responsible for parsing individual options, for now only a single rule is there, KW_HEADER, which when parsed will call curl_dd_set_header(). You can add further options like this: +curl_option + : KW_HEADER '(' string string ')' { curl_dd_set_header(last_driver, $3, $4); } + | KW_URL '(' string ')' { curl_dd_set_url(last_driver, $3); } Variables like $1, $2 and so on represent the Nth token in the rule, e.g. $3 in the KW_HEADER branch is the first string value. $4, is the 2nd. Hope this helps. You will definitely be able to find tutorials that describe both yacc/lex, however there are a number of quirks how syslog-ng applies them, primarily to make it possible to extend the base grammar with plugins. Cheers, Bazsi -- Bazsi On Mon, Feb 1, 2016 at 10:01 PM, Marc Falzon <m(a)baha.mu> wrote: > Hi Bazsi > > Sorry to bother you again, but now that I'm done prototyping the "curl" > part of my module using hardcoded values for testing I'm now facing the > "grammar/parser" problem: could you please drive me though the basics of > how this works? I tried to understand looking at other modules but I don't > understand at all. > > Basically, I'd like to be able to specify some configuration settings in > the syslog-ng configuration, such as url, some HTTP headers... It would > look like this: > > destination d_curl { > curl(url("http://logs.example.net:5140/" header("Content-Type: > application/json") header("X-Custom-Header: blah"))); > }; > > > My code is still available at > https://github.com/falzm/syslog-ng/tree/f/curl-module/modules/curl > > Thank you in advance, > > m. > > > On Saturday 30 January 2016 at 10:17, m(a)baha.mu wrote: > > > Hi > > > > Great! I should be able to get going from here, thank you for your help. > > > > Cheers, > > > > m. > > > > > On Jan 30, 2016, at 08:56, Scheidler, Balázs < > balazs.scheidler(a)balabit.com (mailto:balazs.scheidler@balabit.com)> wrote: > > > > > > Hi, > > > > > > I have added code to your module, it compiles and is able to accept > messages to be sent, and prints a debug message whenever that happens. > > > > > > I didn't have too much time to clean it up, but I hope this helps to > start the ball rolling. > > > > > > https://github.com/balabit/syslog-ng/compare/f/curl-module < > https://github.com/balabit/syslog-ng/compare/f/curl-module> > > > > > > > > > Bazsi > > > > > > -- > > > Bazsi > > > > > > On Sat, Jan 23, 2016 at 9:27 PM, <m(a)baha.mu <mailto:m@baha.mu>> wrote: > > > > > > > On Jan 23, 2016, at 21:23, Scheidler, Balázs < > balazs.scheidler(a)balabit.com <mailto:balazs.scheidler@balabit.com>> wrote: > > > > > > > > If you commit your code to a branch I can help with the basic > skeleton of the parser/grammar > > > > > > > > > Thank you Balázs, my code is available here: > https://github.com/falzm/syslog-ng/tree/f/curl-module/modules/curl < > https://github.com/falzm/syslog-ng/tree/f/curl-module/modules/curl> > > > > > > m. > > > > > > > > > > > > > > On Jan 23, 2016 5:54 PM, "Marc Falzon" <m(a)baha.mu <mailto:m@baha.mu>> > wrote: > > > > I successfully generated a module template using the > `create_plugin.sh (http://create_plugin.sh)` script, however I don't have > the slightest idea on how to get started from here (I mean for the 'plugin > glue', I can manage the cURL part)... The most problematic step for me is > the whole parser/grammar thing, to which I believe to involve YACC/Bison > parsing but I don't know how it works and it's too big of a learning curve > for what I want to achieve at this point... Since the Gitbook doesn't > provide any useful information on module development at the moment, is > there any way to get me started on this topic? I'm not a developer > (sysadmin), and trying to learn from existing module doesn't quite cut it. > > > > > > > > Thank you, > > > > > > > > m. > > > > > > > > > > > > On Friday 22 January 2016 at 11:35, Tibor Benke wrote: > > > > > > > > > There is also a shell script which generates a basic destination > skeleton. You can find its source on this brach: > https://github.com/juhaszviktor/syslog-ng/tree/f/plugin-creator < > https://github.com/juhaszviktor/syslog-ng/tree/f/plugin-creator> . Check > the dev-utils/plugin_skeleton_creator directory and this commit for the > usage: > https://github.com/juhaszviktor/syslog-ng/commit/89361d4d1817560ce8c9062093… > < > https://github.com/juhaszviktor/syslog-ng/commit/89361d4d1817560ce8c9062093… > > > > > > > > > > > > 2016-01-22 11:33 GMT+01:00 Marc Falzon <m(a)baha.mu <mailto: > m(a)baha.mu> (mailto:m@baha.mu <mailto:m@baha.mu>)>: > > > > > > Hi Bazsi, > > > > > > > > > > > > > > > > > > On Friday 22 January 2016 at 11:28, Scheidler, Balázs wrote: > > > > > > > > > > > > > not really, but maybe I would read the "official" modules > source code instead. and of course you can ask questions on the mailing > list. :) > > > > > > > > > > > > > > > > > > Yes that's what I've started to do, but a proper documentation > is preferable, saves time and avoid confusion ;) > > > > > > > > > > > > > if you want to create a destination, the easiest is to create > a threaded destination, where the output is an independent thread, that can > use a synchronous API. Of course this is going to be slower than using an > asynchronous implementation, but that usually is good enough, > > > > > > > > > > > > Thank you for the hint. Given my use case (HTTP destination) I > think I'm safe to go with an asynchronous implementation since each request > is stateless, no connections/locks involved. > > > > > > > > > > > > m. > > > > > > > > > > > > > > > > > > > > On Fri, Jan 22, 2016 at 11:03 AM, Marc Falzon <m(a)baha.mu > <mailto:m@baha.mu> (mailto:m@baha.mu <mailto:m@baha.mu>) (mailto:m@baha.mu > <mailto:m@baha.mu>)> wrote: > > > > > > > > Hi > > > > > > > > > > > > > > > > Is there any documentation on how to develop new syslog-ng > module using the native C API, besides reading the incubator modules source > code? > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > m. > > > > > > > > > > > > > > > > > > > > > > > > > ______________________________________________________________________________ > > > > > > > > Member info: > https://lists.balabit.hu/mailman/listinfo/syslog-ng < > https://lists.balabit.hu/mailman/listinfo/syslog-ng> > > > > > > > > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng < > http://www.balabit.com/support/documentation/?product=syslog-ng> > > > > > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq < > http://www.balabit.com/wiki/syslog-ng-faq> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ______________________________________________________________________________ > > > > > > > Member info: > https://lists.balabit.hu/mailman/listinfo/syslog-ng < > https://lists.balabit.hu/mailman/listinfo/syslog-ng> > > > > > > > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng < > http://www.balabit.com/support/documentation/?product=syslog-ng> > > > > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq < > http://www.balabit.com/wiki/syslog-ng-faq> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ______________________________________________________________________________ > > > > > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > <https://lists.balabit.hu/mailman/listinfo/syslog-ng> > > > > > > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng < > http://www.balabit.com/support/documentation/?product=syslog-ng> > > > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq < > http://www.balabit.com/wiki/syslog-ng-faq> > > > > > > > > > > > > > > > > > > > > > > > > > > ______________________________________________________________________________ > > > > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng < > https://lists.balabit.hu/mailman/listinfo/syslog-ng> > > > > > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng < > http://www.balabit.com/support/documentation/?product=syslog-ng> > > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq < > http://www.balabit.com/wiki/syslog-ng-faq> > > > > > > > > > > > > > > > > > > > > > > > > > ______________________________________________________________________________ > > > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng < > https://lists.balabit.hu/mailman/listinfo/syslog-ng> > > > > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng < > http://www.balabit.com/support/documentation/?product=syslog-ng> > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq < > http://www.balabit.com/wiki/syslog-ng-faq> > > > > > > > > > ______________________________________________________________________________ > > > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng < > https://lists.balabit.hu/mailman/listinfo/syslog-ng> > > > > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng < > http://www.balabit.com/support/documentation/?product=syslog-ng> > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq < > http://www.balabit.com/wiki/syslog-ng-faq> > > > > > > > > > > > > > ______________________________________________________________________________ > > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng < > https://lists.balabit.hu/mailman/listinfo/syslog-ng> > > > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng < > http://www.balabit.com/support/documentation/?product=syslog-ng> > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq < > http://www.balabit.com/wiki/syslog-ng-faq> > > > > > >

3 5

Mutual certification
by Ivan Adji - Krstev 04 Feb '16

04 Feb '16

Hi all, I have follow the link <https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-…> for providing mutual authentication. I have create CA on a server and certificates for the server and client. On the server site have put: =========================================== source s_sys { system(); #unix-stream("/dev/log"); internal(); network( port(6514) # tcp(port(5140)); # file("/proc/kmsg" log_prefix("kernel: ")); transport("tls") tls( key_file("/etc/syslog-ng/cert.d/serverkey.pem") cert_file("/etc/syslog-ng/cert.d/servercert.pem") ca_dir("/etc/syslog-ng/ca.d")) ); }; i have define d_mysql etc. log { source(s_sys); destination(d_mysql); }; ============================================================= on the client site i have: destination tls_destination { network("x.x.x.x" port(6514)); transport("tls") tls( ca_dir("/etc/syslog-ng/ca.d") key_file("/etc/syslog-ng/cert.d/clientkey.pem") cert_file("/etc/syslog-ng/cert.d/clientcert.pem") ) }; log { source(s_sys); destination(tls_destination); }; ================================================================== And when i restart the syslog-ng on a client site i have the following error: /etc/init.d/syslog-ng restart Stopping syslog-ng: [FAILED] Error parsing destination, destination plugin network not found in /etc/syslog-ng/syslog-ng.conf at line 45, column 5: network("x.x.x.x" port(6514)); ^^^^^^^ syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng syslog-ng --version syslog-ng 3.2.5 Installer-Version: 3.2.5 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#9d4bea28198bd731df1a61e980a2af5b88d81116 Compile-Date: Jul 25 2014 15:20:50 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: off Enable-SQL: on Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off So can someone tell me what i'm doing wrong ? Kind regards Ivan

2 2

syslog-ng and dependency libraries for sending log messages for remote log server with TLS support
by Girish Kumar 04 Feb '16

04 Feb '16

Hi All, My requirement is to send the log messages to remote log server securely (with TLS). The project which I am implementing is on embedded systems. So the space is very important. Please let me know the latest stable version of syslog-ng which can be used for this. Also please let me know the mandatory dependency libraries required. Whether I can use openssl library for TLS support Thanks in advance. Regards, Girish

1 0

Module development guide
by Marc Falzon 30 Jan '16

30 Jan '16

Hi Is there any documentation on how to develop new syslog-ng module using the native C API, besides reading the incubator modules source code? Cheers, m.

5 17

License check failure
by Juhász, Viktor 25 Jan '16

25 Jan '16

Hi, I really want to keep the license policy, but I don't know, and can't find any description, what if I just moved files to one directory to another directory without any modifications? For example I just moved the logmsg.c from lib to lib/logmsg directory, and the license check fails in travis. What should I do in this case? BR, Viktor Juhász

2 2

Re: [syslog-ng] Dependency on default-jre for Debian package
by Laszlo Budai 22 Jan '16

22 Jan '16

Hi, thank you for sharing this issue(and for suggesting a solution to it). In the next two weeks I'm going to work on the packaging a little bit and I'll fix it. regards, Laszlo Budai Sent from my HTC ----- Reply message ----- From: "Marc Falzon" <m(a)baha.mu> To: <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] Dependency on default-jre for Debian package Date: Fri, Jan 22, 2016 10:22 Hi I'm trying to install the unofficial syslog-ng 3.7.2 Debian packages from http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… however I have a problem with the Java-related packages, since they have a package dependency on `default-jre`, and I'm using an Oracle JRE packaged using the `make-jpkg` utility which generates a Debian package providing `java-browser-plugin, java-runtime, java-runtime-headless, java-virtual-machine, java2-runtime, java2-runtime-headless, java6-runtime, java6-runtime-headless, java7-runtime, java7-runtime-headless, java8-runtime, java8-runtime-headless`, but not `default-jre`. Is it possible to modify those packages so that they depend on `java-runtime` instead? The `default-jre` packages also provide `java-runtime`. Thank you, m. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

2 1

Re: [syslog-ng] Module development guide
by Laszlo Budai 22 Jan '16

22 Jan '16

Hi, and we have an unmerged PR that could help implementing a threaded dst. : https://github.com/balabit/syslog-ng-gitbook/pull/52 We should unify these contents... L. Sent from my HTC ----- Reply message ----- From: "Marc Falzon" <m(a)baha.mu> To: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] Module development guide Date: Fri, Jan 22, 2016 11:42 On Friday 22 January 2016 at 11:35, Tibor Benke wrote: > There is also a shell script which generates a basic destination skeleton. You can find its source on this brach: https://github.com/juhaszviktor/syslog-ng/tree/f/plugin-creator . Check the dev-utils/plugin_skeleton_creator directory and this commit for the usage: https://github.com/juhaszviktor/syslog-ng/commit/89361d4d1817560ce8c9062093… Awesome, thank you! m. > > 2016-01-22 11:33 GMT+01:00 Marc Falzon <m(a)baha.mu (mailto:m@baha.mu)>: > > Hi Bazsi, > > > > > > On Friday 22 January 2016 at 11:28, Scheidler, Balázs wrote: > > > > > not really, but maybe I would read the "official" modules source code instead. and of course you can ask questions on the mailing list. :) > > > > > > Yes that's what I've started to do, but a proper documentation is preferable, saves time and avoid confusion ;) > > > > > if you want to create a destination, the easiest is to create a threaded destination, where the output is an independent thread, that can use a synchronous API. Of course this is going to be slower than using an asynchronous implementation, but that usually is good enough, > > > > Thank you for the hint. Given my use case (HTTP destination) I think I'm safe to go with an asynchronous implementation since each request is stateless, no connections/locks involved. > > > > m. > > > > > > > > On Fri, Jan 22, 2016 at 11:03 AM, Marc Falzon <m(a)baha.mu (mailto:m@baha.mu) (mailto:m@baha.mu)> wrote: > > > > Hi > > > > > > > > Is there any documentation on how to develop new syslog-ng module using the native C API, besides reading the incubator modules source code? > > > > > > > > Cheers, > > > > > > > > m. > > > > > > > > > > > > ______________________________________________________________________________ > > > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > > > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > > > > > > > > > > ______________________________________________________________________________ > > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > > > > > > > > ______________________________________________________________________________ > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

Dependency on default-jre for Debian package
by Marc Falzon 22 Jan '16

22 Jan '16

Hi I'm trying to install the unofficial syslog-ng 3.7.2 Debian packages from http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… however I have a problem with the Java-related packages, since they have a package dependency on `default-jre`, and I'm using an Oracle JRE packaged using the `make-jpkg` utility which generates a Debian package providing `java-browser-plugin, java-runtime, java-runtime-headless, java-virtual-machine, java2-runtime, java2-runtime-headless, java6-runtime, java6-runtime-headless, java7-runtime, java7-runtime-headless, java8-runtime, java8-runtime-headless`, but not `default-jre`. Is it possible to modify those packages so that they depend on `java-runtime` instead? The `default-jre` packages also provide `java-runtime`. Thank you, m.

1 0

"Syslog connection closed" but socket not closed
by Patrick Hemmer 21 Jan '16

21 Jan '16

We recently saw an issue where syslog-ng (version 3.6.4 on FreeBSD 10.1) was configured with a `syslog()` source, received an invalid frame, shut down the connection, but the socket remained open. I was looking through the documented fixes in the versions since the one we're using (3.6.4), but nothing looks related. 2016-01-21T07:44:21-05:00 iad1gweb01.ecom.chewy.com ERR syslog-ng[27090]: - Invalid frame header; header='' [meta sequenceId="389"] 2016-01-21T07:44:21-05:00 iad1gweb01.ecom.chewy.com NOTICE syslog-ng[27090]: - Syslog connection closed; fd='19', client='AF_INET(127.0.0.1:59317)', local='AF_INET(127.0.0.1:601)' [meta sequenceId="390"] # netstat -an|grep 59317 tcp4 81660 0 127.0.0.1.601 127.0.0.1.59317 ESTABLISHED tcp4 0 48923 127.0.0.1.59317 127.0.0.1.601 ESTABLISHED From the netstat output (the buffer sizes), it looks like syslog-ng stopped reading from the socket, but didn't close it, and the application on the other end kept trying to write to it. This is just my theory anyway. I can open a bug on github, but I wanted to bring it up here first. Thanks -Patrick

1 0

RFC5425 framing and log_msg_size
by Blake Day 21 Jan '16

21 Jan '16

Hi all, I’m wondering why syslog-ng disconnects clients that try to send log messages exceeding log_msg_size instead of just gracefully accepting and truncating the message? I took a quick glance at the code, and it appears to only affect clients using RFC5425 framing. Here is the relevant code: if (self->frame_len > self->super.options->max_msg_size) { msg_error("Incoming frame larger than log_msg_size()", evt_tag_int("log_msg_size", self->super.options->max_msg_size), evt_tag_int("frame_length", self->frame_len), NULL); return LPS_ERROR; } Here are excerpts from my logs showing the problem: 2016-01-20T11:27:28-05:00 [HOST] ERR syslog-ng[87462]: - Incoming frame larger than log_msg_size(); log_msg_size='16384', frame_length='76218' [meta sequenceId="6560"] 2016-01-20T11:27:28-05:00 [HOST] NOTICE syslog-ng[87462]: - Syslog connection closed; fd='26', client='AF_INET(127.0.0.1:52627)', local='AF_INET(127.0.0.1:601)' [meta sequenceId="6561"]

2 1

Re: [syslog-ng] patterndb and smtp to() issue
by Laszlo Budai 21 Jan '16

21 Jan '16

Hi, templateable afsmtp to() is available from version 3.7.1 . Sent from my HTC ----- Reply message ----- From: "Fabien Wernli" <wernli(a)in2p3.fr> To: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] patterndb and smtp to() issue Date: Thu, Jan 21, 2016 06:55 Hi Alexandre, Templateable smtp/to() and co. was added only recently. I'll check later in the day on which version it appeared. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

2 1

patterndb and smtp to() issue
by Alexandre DEPREZ 21 Jan '16

21 Jan '16

Hi all, I'm using the pattern-db to extract values from a firewall's log. This far, everything's working great. The log looks something like this: Jan 20 2016 21:48:45: %ASA-7-746012: user-identity: Add IP-User mapping 10.10.99.7 - LOCAL\alex Succeeded - VPN user Using pdbtool and matching the log against the xml pattern file, this is showing me good results : :$pdbtool match -P "%ASA-7-746012" -M "user-identity: Add IP-User mapping 10.10.99.7 - LOCAL\alex Succeeded - VPN user" -p /etc/syslog-ng/patterndb.d/vpn-parser-up.xml -D -c Pattern matching part: user-identity: Add IP-User mapping @IPv4:VPN_IP=10.10.99.7@ - LOCAL\ @STRING:VPN_USER=alex@ Succeeded - VPN user Matching part: user-identity: Add IP-User mapping 10.10.99.7 - LOCAL\alex Succeeded - VPN user Values: MESSAGE=user-identity: Add IP-User mapping 10.10.99.7 - LOCAL\alex Succeeded - VPN user PROGRAM=%ASA-7-746012 .classifier.class=vpn.access_log .classifier.rule_id=019045a7383c252e57c20435ae5bf86c VPN_IP=10.10.99.7 VPN_USER=alex TAGS= Here's the xml file <patterndb version='4' pub_date='2015-12-22'> <ruleset id='04ba26e756011614c57cf469fed7b5c0' name='%ASA-7-746012'> <pattern>%ASA-7-746012</pattern> <rules> <rule class='vpn.access_log' id='019045a7383c252e57c20435ae5bf86c' provider='alex'> <patterns> <pattern>user-identity: Add IP-User mapping @IPv4:VPN_IP@ - LOCAL\@STRING:VPN_USER@ Succeeded - VPN user</pattern> </patterns> </rule> </rules> </ruleset> </patterndb> Now, the problem lies on the destination which is using the smtp driver. destination vpn_mail_up { smtp( host("x.x.x.x") port(25) from("alex(a)x.y" "alex(a)x.y") to("${VPN_USER}(a)x.y") subject("vpn connection") body("vpn connection from ${VPN_USER} with IP: ${VPN_IP}\n") ); }; The variable is functional inside the body() but not in the to() field. Here's a dump I extracted directly from the server on the tcp session to the mail server: RCPT.TO:<${VPN_USER}(a)x.y>.. BDAT.411.. X-Mailer:.syslog-ng.3.5.6.. Date:.Wed,.20.Jan.2016.21:51:51.+0100.. From:.alex@x.y.. Message-Id:.<1453323111.149975.19608@debian>.. To:."${VPN_USER}(a)x.y".<${VPN_USER}(a)x.y>.. Subject:.vpn.connection.. .BDAT.68..vpn.connection.from.alex..with.IP:.10.10.99.7.BDAT.2.LAST.. The variable is being populated in the body message but not the recipient. Is there any chance the variable could only be used once (!?) or not being able to be used inside the to() ? Regards, Alex

2 1

Syslog-ng mutual self cert authentication
by Ivan Adji - Krstev 18 Jan '16

18 Jan '16

Hi all, I'm truing to make mutual self cert authentication with Syslog-NG and using LogAnalyzer to see the logs. I have done the following: *Server configuration:* Creating a CA making directory and put the openssl.cnf file in the directory etc. do the following command: 1. openssl req -new -x509 -keyout private/cakey.pem -out cacert.pem -days 365 -config openssl.cnf 2. openssl req -nodes -new -x509 -keyout serverkey.pem -out serverreq.pem -days 365 -config openssl.cnf 3. openssl x509 -x509toreq -in serverreq.pem -signkey serverkey.pem -out tmp.pem 4. openssl ca -config openssl.cnf -policy policy_anything -out servercert.pem -infiles tmp.pem 5. In */etc/syslog-ng/* create directory *cert.d* and *ca.d* copy the *serverkey.pem* and *servercert.pem* to *cert.d* and copy *cacert.pem* to *ca.d*. Go to *ca.d* and issue the following command *"openssl x509 -noout -hash -in cacert.pem"* and do the link *"ln -s cacert.pem 6d2962a8.0" *6. On the syslog-ng.conf file i put the following changes: *************************************** source s_sys { system(); internal(); network( port(6514) file("/proc/kmsg" log_prefix("kernel: ")); transport("tls") tls( key_file("/etc/syslog-ng/cert.d/serverkey.pem") cert_file("/etc/syslog-ng/cert.d/servercert.pem") ca_dir("/etc/syslog-ng/ca.d")) ); }; ******************************************** And restart the service ================== *Client Configuration* Creating CA making directory and put the openssl.cnf file in the directory etc. do the following command 1. openssl req -new -x509 -keyout private/cakey.pem -out cacert.pem -days 365 -config openssl.cnf 2. openssl req -nodes -new -x509 -keyout clientkey.pem -out clientreq.pem -days 365 -config openssl.cnf 3. openssl x509 -x509toreq -in clientreq.pem -signkey clientkey.pem -out tmp.pem 4. openssl ca -config openssl.cnf -policy policy_anything -out clientcert.pem -infiles tmp.pem 5. In */etc/syslog-ng/* create directory *cert.d* and *ca.d* copy the *"clientkey.pem" *and *"clientcert.pem" *to *"cert.d" *and copy *"cacert.pem" *to *"ca.d" *and issue the following command in the *ca.d* directory. *openssl x509 -noout -hash -in cacert.pem *and *ln -s cacert.pem 6d2962a8.0 *6. add the following to the *syslog-ng.conf* file ****************************************************** destination d_tcp { tcp("192.168.111.176" port(6514) tls( ca_dir("/etc/syslog-ng/ca.d") key_file("/etc/syslog-ng/cert.d/clientkey.pem") cert_file("/etc/syslog-ng/cert.d/clientcert.pem") ) ); }; *************************************************************** And finally i have the following errors :) Jan 12 11:58:02 syslogserver syslog-ng[5507]: Syslog connection accepted; fd='16', client='AF_INET(192.168.111.104:45631)', local='AF_INET(0.0.0.0:6514)' Jan 12 11:58:02 syslogserver syslog-ng[5507]: SSL error while reading stream; tls_error='SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca' Jan 12 11:58:02 syslogserver syslog-ng[5507]: I/O error occurred while reading; fd='16', error='Connection reset by peer (104)' Jan 12 11:58:02 syslogserver syslog-ng[5507]: Syslog connection closed; fd='16', client='AF_INET(192.168.111.104:45631)', local='AF_INET(0.0.0.0:6514)' Jan 12 11:58:12 syslogserver syslog-ng[5507]: Syslog connection accepted; fd='16', client='AF_INET(192.168.111.104:58937)', local='AF_INET(0.0.0.0:6514)' Jan 12 11:58:12 syslogserver syslog-ng[5507]: SSL error while reading stream; tls_error='SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca' Jan 12 11:58:12 syslogserver syslog-ng[5507]: I/O error occurred while reading; fd='16', error='Connection reset by peer (104)' Jan 12 11:58:12 syslogserver syslog-ng[5507]: Syslog connection closed; fd='16', client='AF_INET(192.168.111.104:58937)', local='AF_INET(0.0.0.0:6514)' Can someone help me with this and tell me what im doing wrong ? Thanks in advanced !

2 8

syslog-ng @ Southern California Linux Expo
by Czanik, Péter 18 Jan '16

18 Jan '16

Hi, This week I'll be at the Southern California Linux Expo in Los Angeles. If you want to talk or hear about syslog-ng, you have many chances: - Balabit booth #426 during Expo hours - syslog-ng BoF in room 104 at 7:00pm on Saturday - "syslog-ng: from raw data to Big Data" presentation in room 211, Sunday 11:30 You can also request an appointment at https://www.balabit.com/scale-trade-show-meeting and I'm happy to meet you if you are in LA also outside of Scale. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

Issue with syslog-ng and Windows Server 2012
by Artyom Gavrichenkov 15 Jan '16

15 Jan '16

Hi all, Since Windows Vista (and Server 2012), in NTFS, file size is updated not when its contents are being flushed to disk, but when the last handle to the corresponding file object is closed. More on that here: https://blogs.msdn.microsoft.com/oldnewthing/20111226-00/?p=8813 What this means in practice is that an application log file might already be quite large, but its reported file size will still be zero unless some other application will open(), read() and then close() it. And seems like it breaks syslog-ng’s source() file() handler in case the filename in file() arguments contains wildcards. Could anyone please look into it? | Artyom Gavrichenkov | gpg: fa1c670e | mailto: ximaera(a)gmail.com | xmpp: ximaera(a)gmail.com | tel. no: +7 916 515 49 58

3 2

syslog-ng ose push to master
by Scheidler, Balázs 14 Jan '16

14 Jan '16

Hi, there were some fallouts from the recent merger of #842, which I've quickly fixed up by pushing two trivial patches to master bypassing review on github. My patches compile without warnings and make check is passed, but please review them. Cheers, -- Bazsi

2 1

Insider 2016-01: syslog-ng web GUIs; Scale; GSoC; Raspberry Pi 2; SSB 4 F3
by Czanik, Péter 14 Jan '16

14 Jan '16

Dear syslog-ng users, This is the 46th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS syslog-ng web GUIs ------------------ The number and importance of logging solutions with a web interface is growing, luckily many of these are syslog-ng based. Using the Elasticsearch destination, Kibana provides an easy-to-use web interface for log messages. Read more about feeding log messages to Elasticsearch with syslog-ng with at https://czanik.blogs.balabit.com/2015/12/web-interfaces-for-your-syslog-ser… Southern California Linux Expo ------------------------------ Next week, Peter Czanik will present “syslog-ng: from raw data to Big Data” at the Southern California Linux Expo. There will be also a booth (#426) where you can talk about syslog-ng, and we also plan a syslog-ng BoF where you can share your experiences or ask your questions about syslog-ng. https://www.balabit.com/scale-trade-show-meeting Google Summer of Code --------------------- In the past four years syslog-ng participated in Google Summer of Code, where talented students spent their summers coding syslog-ng instead doing some less intellectually challenging summer jobs. Some of their results include the Redis destination or configuration management modules. Read more about our experiences and the upcoming “syslog-ng Universe” at https://syslog-ng.org/gsoc-2015-summary/ syslog-ng on the Raspberry Pi 2 ------------------------------- The performance of syslog-ng on the original Raspberry Pi was not that great. A quick test of the Raspberry Pi 2 revealed, that performance increased drastically, now you can use this $35 machine as a central log server: https://czanik.blogs.balabit.com/2016/01/syslog-ng-on-the-raspberry-pi-2/ syslog-ng Store Box 4F3 released -------------------------------- The latest version of syslog-ng Store Box (SSB), a turnkey appliance built around syslog-ng has been released. You can now search across multiple logspaces and across multiple SSB boxes from one search interface, while maintaining user profile-based access control. Read the full announcement at https://jluby.blogs.balabit.com/2015/12/15/syslog-ng-store-box-4-f3-release… Monitoring serial consoles -------------------------- The January issue of the Linux Journal contains an article from Fabien Wernli about using syslog-ng to monitor serial consoles. This month it is only available for subscribers: http://www.linuxjournal.com/content/january-2016-issue-linux-journal NEW RELEASES SSB 4 F3: https://jluby.blogs.balabit.com/2015/12/15/syslog-ng-store-box-4-f3-release… Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com . To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

Changing a value after a match with patterndb
by Mark Shetka 12 Jan '16

12 Jan '16

Is it possible to substitute a value after a pattern match? For example, in the pattern below if I match the "23" then use value "telnet", but if it is a "22", then "ssh". %SEC_LOGIN-4-LOGIN_FAILED: Login failed [user: test] [Source: 131.212.1.1] [localport: 23] [Reason: Login Authentication Failed] at 10:38:41 CST Thu Jan 23 2014 I am trying to use one pattern to match both cases and we'd prefer to have service names rather than port numbers in our db. -- Mark Shetka Infrastructure Analyst - Network Team Information Technology Systems & Services University of Minnesota - Duluth (218) 726-7682

4 6

how to enable TLS on syslog-ng 3.2.5
by Jim Zapsalis 06 Jan '16

06 Jan '16

Hi guys, How do we enable TLS on OSE version of syslog-ng version 3.2.5? Or is it simpler to enable SSL ? Regards, Jim Zapsalis Senior Platform Engineer, IS Bupa, 33 Exhibition Street, Melbourne, 3000 T +61 3 9937 4028 M 0411 779 710 E jim.zapsalis(a)bupa.com.au<mailto:jim.zapsalis@bupa.com.au> W bupa.com.au/healthandcaring<http://www.bupa.com.au/healthandcaring/index.html> [Social Media Icons-01.png]<https://www.facebook.com/BupaAustralia>[Social Media Icons-02.png]<https://mobile.twitter.com/bupaaustralia>[Social Media Icons-04.png]<https://ea.linkedin.com/company/bupa-australia>[Social Media Icons-03.png]<https://www.youtube.com/user/BupaAustralia> Bupa Australia email disclaimer: The information contained in this email and any attachments is confidential and may be subject to copyright or other intellectual property protection. If you are not the intended recipient, you are not authorized to use or disclose this information, and we request that you notify us by reply mail or telephone and delete the original message from your mail system.

3 3

Proper formatting of incoming Log4j, Jetty events
by Clayton Dukes 16 Dec '15

16 Dec '15

Hi, I'm having trouble with getting "good" fields from Jetty. I found this: http://blog.trifork.com/2010/01/14/logging-to-the-syslog-from-a-java-applic… And had my user configure their log4j with the given recommendation, but the events still come in quite poorly. Our syslog-ng template uses the following template: @${R_UNIXTIME:--}.${R_USEC}\t${HOST:--}\t${PRI:--}\t${PROGRAM:--}\t${MSGONLY}\n"); Some sample events: @1450275007.912996 host001 173 2015-12-16 16:10:07,836 DEBUG [HeartbeatGeneratorTimer] <-- sessionKey=179: out>{"method":"Heartbeat","jsonrpc":"2.0"} As you can see - this event is sending the date as the program name. Another: @1450275007.924140 host002 13 521 <174>1 2015-12-16T16:10:07+02:00 host002 /openam 1000430466 AUTHENTICATION-303 [logRecord@36733 LoginID="id=2565a24-0829-11e2-b614-001e371e7e40,ou=user,dc=opensso,dc=java,dc=net" ContextID="b1de62b91530d5303" IPAddr="10.3.109.13" LogLevel="INFO" Domain="dc=opensso,dc=java,dc=net" ModuleName="ProfileMapper|Mandate|LDAP" HostName="10.3.109.13" LoggedBy="cn=dsameuser,ou=DSAME Users,dc=opensso,dc=java,dc=net" MessageID="AUTHENTICATION-303" NameID="" TIME="2015-12-16 16:10:07"] Logout|service|mailService This one is sending the program name as an integer (521) There are many others, the program names consist of the following (out of about 100k events): - " .0.9"},"parentID" ...80) ame" at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) del" ...e.common.AuthenticationFilter.doFilter(AuthenticationFilter.java ...ee.elion.smarthome.common.AuthenticationFilter.doFilter(AuthenticationFilter.java ...eJspFile(JspServlet.java EST in ,"is_recording" iWeather"},{"id" leRecordID" me" model" ng" ngth" ...rg.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java se,"is_recording" ....servlet.JspServlet.serviceJspFile(JspServlet.java tin Does anyone have a good way to fix this? ______________________________________________________________ Clayton Dukes ______________________________________________________________

2 1

rabbitmq errors
by Al Itchon 15 Dec '15

15 Dec '15

I'm seeing an error in my log indicating that messages aren't being sent to a queue in RabbitMQ. This happens a few times a week. Here's the error: Nov 24 13:29:28 syslog09 syslog-ng[7883]: Network error while inserting into AMQP server; time_reopen='10' I have about 1000 servers sending tcp messages through a call to /dev/tcp to port on the syslog-ng server. It seems messages aren't sent to the queue for about 1 or 2 minutes, and then it resumes sending again without error. Could this be a network issue or something within syslog-ng that's limiting the messages?

4 8

Madhouse apt key is missing
by Clayton Dukes 09 Dec '15

09 Dec '15

Anyone know why https://packages.madhouse-project.org/debian/archive-key.txt is no longer available? ______________________________________________________________ Clayton Dukes ______________________________________________________________

3 5