May 2019 - syslog-ng - lists.balabit.hu

Is it possible to create Two instances of syslog-ng engine ?
by Nitish Saboo 13 May '19

13 May '19

Hi, I have installed syslog-ng on my Linux box and I am planning you use syslog-ng parser and wanted to initialise it's engine for parsing the data. so initialise() method in my Go code calls C code that is wrapper around syslog-ng C header files. So ,I am making use of cgo. Following is my Go code: func (obj Syslogparser) initialise{ X := C.CString(obj.X) defer C.free(unsafe.Pointer(X)) Y := C.CString(obj.Y) defer C.free(unsafe.Pointer(Y)) C.initialize_engine(X, Y, (C.key_value_cb)(unsafe.Pointer(C.callOnMeGo_cgo))); } And following is my C method 'initialize_engine': int initialize_engine(const gchar* X, const gchar* Y, key_value_cb cb) // cb is callback function { module_path = _module_path; app_startup(); msg_init(TRUE); configuration = cfg_new(0x0302); plugin_load_module("basicfuncs", configuration, NULL); plugin_load_module("syslogformat", configuration, NULL); pattern_db_global_init(); if(patterndb != NULL){ pattern_db_free(patterndb); } patterndb = pattern_db_new(); pattern_db_reload_ruleset(patterndb, configuration, filename); pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb); return 0; } My queries: 1)What is pattern_db_global_init, 'patterndb = pattern_db_new()', 'pattern_db_reload_ruleset', 'pattern_db_set_emit_func' doing in the following case ? 2)Can I make two independent instances of syslog-ng engines? 3)How can I call two instances of syslog-ng engines independently from my Go code? Thanks

2 2

Troubleshooting Question
by Walter Tienken 12 May '19

12 May '19

Hello All, I'd like to see if we can get some troubleshooting help with Syslog-NG OSE. Here's some background: Our environment collects logs from various network locations (F5, checkpoint, plixer, etc) and sends them to our Syslog-NG cluster through two F5 load balancers. We have four netlog boxes in round-robin that are running version 5 of the PE version of Syslog-NG on RHEL6. We also have a dev server running rhel7 and the 3.19.1 OSE edition of Syslog-NG from the COPR repository. On each of these servers, we run the Splunk Universal Forwarder that then sends the logs over to our indexer cluster. The problem we are having is that the dev server will not listen to any traffic except via localhost. We can see the traffic just fine on a TCPDUMP as it comes out of the load balancer into the dev box, but watching Syslog-NG in the foreground with -Fevd the traffic never registers at all. We can send test messages with loggen or netcat, etc, from localhost and Syslog-NG will see it and log it to disk as expected. Anywhere else it just never sees the traffic nor logs anything to disk even though we have confirmed that Syslog-NG is listening on the 9999 port to UDP with netstat. Here is our syslog-ng.conf file from the working RHEL6 boxes: https://gist.github.com/MrTink76/9ee1e88f93a313f953e4033560af463a This is the syslog-ng.conf file on the OSE box: https://gist.github.com/MrTink76/e181f4c0bf052077440d7bdfaf418e02 This is an example of our testing CONF file located in conf.d on the OSE box: https://gist.github.com/MrTink76/a0433b9e908ba683e36cb6199f9cc43f We send our test traffic to the F5 load balancer vip using UDP 9999. Like I said above, when we send on 9999 localhost with loggen or netcat, Syslog-NG sees it just fine and logs it to disk, but anywhere else it never registers nor records the test message to disk. We currently have SELinux disabled and there is no firewall running on the dev box (we see the traffic fine via tcpdump). Any help/suggestions would be greatly appreciated. Please let me know if I need to provide further information. Thanks much, Walter Tienken walter.tienken(a)asu.edu<https://ex2010.asu.edu/owa/redir.aspx?SURL=sS2_o_WV6gQ_JAkG-_VgxIDZLGj9-EeB…> Cloud and Advanced Network Engineering Services

3 4

unofficial syslog-ng packages for syslog-ng 3.21.1
by Laszlo Budai 10 May '19

10 May '19

Hi, syslog-ng 3.21.1[1] packages are available in OBS repo[2]. List of supported OSs: * Debian 8.0 * Debian 9.0 [including armv7l] * Raspbian 9.0 * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 18.04 * Ubuntu 18.10 * Ubuntu 19.04 (dependencies are still building) Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.21.1 [2] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… regards, Laszlo Budai

1 0

syslog-ng 3.21.1
by Laszlo Budai 10 May '19

10 May '19

3.21.1 Highlights * Add an alternative, native, librdkafka based kafka-c() destination in parallel of the existing Java implementation, that provides the same configuration interface. Eventually, we expect this to replace the Java one (#2496<https://github.com/balabit/syslog-ng/pull/2496>) * Add a native, http() based destination based driver for elasticsearch called elasticsearch-http(), as an alternative of the Java one. Eventually, we expect this to replace the Java implementation. (#2509<https://github.com/balabit/syslog-ng/pull/2509>) * Add the ability to automatically determine the timezone value for an incoming log entry as long as the incoming stream is close to real time and the timezone information is missing from the timestamp. Enable this function by using flags(guess-timezone) for sources and the date-parser(). (#2517<https://github.com/balabit/syslog-ng/pull/2517>, #2673<https://github.com/balabit/syslog-ng/pull/2673>) Features * syslog(): Add the ability to work with messages larger than log-msg-size() in the source driver by using the trim-large-messages(yes) option. The characters over the limit will be truncated. Previously messages longer than the limit caused the connection to be closed abruptly. (#2644<https://github.com/balabit/syslog-ng/pull/2644>) * amqp(): add support for heartbeats and the "external" authentication mechanism. (#2676<https://github.com/balabit/syslog-ng/pull/2676>, #2626<https://github.com/balabit/syslog-ng/pull/2626>) * graylog2(): add support for TLS and UDP. (#2657<https://github.com/balabit/syslog-ng/pull/2657>) * udp(): Add spoof-source-max-msglen() option to allow setting the maximum spoofed datagram size, which was hard-wired to 1024 previously. (#2535<https://github.com/balabit/syslog-ng/pull/2535>) * db-parser(): add an option program-template() that customizes the value used for matching the PROGRAM field. (#2651<https://github.com/balabit/syslog-ng/pull/2651>) * pdbtool: Add sort option to pdbtool merge (#2664<https://github.com/balabit/syslog-ng/pull/2664>) * $(implode) and $(explode): add template functions to split and join strings based on a simple separator. The exploded array is represented as a syslog-ng list that can be manipulated with the $(list-*) template functions. (#2700<https://github.com/balabit/syslog-ng/pull/2700>) * Add an --omit-empty-values option for value-pairs based destinations & template functions. (#2519<https://github.com/balabit/syslog-ng/pull/2519>) * grouping-by() parser: add sort-key() option (#2701<https://github.com/balabit/syslog-ng/pull/2701>) Support for non-syslog or non-standard formats in SCL * apache-accesslog-parser(): support for vhost:port as the first field in common/combined log formats (#2688<https://github.com/balabit/syslog-ng/pull/2688>) * Add application adapter for Junos classification (#2684<https://github.com/balabit/syslog-ng/pull/2684>) * Add parser and adapter for CheckPoint LogExporter output (#2665<https://github.com/balabit/syslog-ng/pull/2665>) Bugfixes * Fix race condition of idle timer and scheduled I/O job (#2650<https://github.com/balabit/syslog-ng/pull/2650>) * Few leaks find via sanitizer (#2696<https://github.com/balabit/syslog-ng/pull/2696>) * syslogformat: set $MSG even if the incoming message is empty (#2672<https://github.com/balabit/syslog-ng/pull/2672>) * Fix double-free error in logproto unit tests (#2662<https://github.com/balabit/syslog-ng/pull/2662>) * groupingby: identical persist name (#2659<https://github.com/balabit/syslog-ng/pull/2659>) * stats: deindex pruned counters/clusters (#2648<https://github.com/balabit/syslog-ng/pull/2648>) * Type hinting should not accept empty values (#2639<https://github.com/balabit/syslog-ng/pull/2639>) * app-parser, pseudofile: fix crash with grammar error (#2640<https://github.com/balabit/syslog-ng/pull/2640>) * python: set_timestamp normalization (#2643<https://github.com/balabit/syslog-ng/pull/2643>) * db-parser: fix memory leak (#2652<https://github.com/balabit/syslog-ng/pull/2652>) * grouping-by: use after free, memory leak, missing init calls of filters (#2655<https://github.com/balabit/syslog-ng/pull/2655>) * amqp: fixing double connect (#2660<https://github.com/balabit/syslog-ng/pull/2660>) * old style definition warning fixes (#2680<https://github.com/balabit/syslog-ng/pull/2680>) * Fix "!=" filter (#2683<https://github.com/balabit/syslog-ng/pull/2683>) * dbparser: fix memleak (#2706<https://github.com/balabit/syslog-ng/pull/2706>) * nondumpable-allocator: fixing mmap error handling (#2666<https://github.com/balabit/syslog-ng/pull/2666>) * Fix timeutils warning (#2604<https://github.com/balabit/syslog-ng/pull/2604>) * Fix old style include statement compatibility (#2600<https://github.com/balabit/syslog-ng/pull/2600>) * Fix config revert (threaded destinations) (#2596<https://github.com/balabit/syslog-ng/pull/2596>) * Add warning on old style include statement (#2592<https://github.com/balabit/syslog-ng/pull/2592>) Other changes * cfg-parser: add aliases for yesno (#2671<https://github.com/balabit/syslog-ng/pull/2671>) * Include json-c in the dist tarball (#2590<https://github.com/balabit/syslog-ng/pull/2590>) * cmake: disable_all_modules support (#2647<https://github.com/balabit/syslog-ng/pull/2647>) * Cmake clang sanitizer (#2562<https://github.com/balabit/syslog-ng/pull/2562>) * timeutils refactor (#2483<https://github.com/balabit/syslog-ng/pull/2483>) * Expedite threaded flush at reload (#2656<https://github.com/balabit/syslog-ng/pull/2656>) * elasticsearch2: Added deprecation warning (#2628<https://github.com/balabit/syslog-ng/pull/2628>) * Astyle fixes (#2624<https://github.com/balabit/syslog-ng/pull/2624>) * Force C99 with GNU (#2623<https://github.com/balabit/syslog-ng/pull/2623>) * Make rewording and other small edits to README (#2608<https://github.com/balabit/syslog-ng/pull/2608>) * Port tests to Criterion (#2607<https://github.com/balabit/syslog-ng/pull/2607>, #2661<https://github.com/balabit/syslog-ng/pull/2661>, #2621<https://github.com/balabit/syslog-ng/pull/2621>, #2620<https://github.com/balabit/syslog-ng/pull/2620>, #2619<https://github.com/balabit/syslog-ng/pull/2619>, #2618<https://github.com/balabit/syslog-ng/pull/2618>, #2617<https://github.com/balabit/syslog-ng/pull/2617>, #2616<https://github.com/balabit/syslog-ng/pull/2616>, #2615<https://github.com/balabit/syslog-ng/pull/2615>, #2599<https://github.com/balabit/syslog-ng/pull/2599>, #2594<https://github.com/balabit/syslog-ng/pull/2594>, #2593<https://github.com/balabit/syslog-ng/pull/2593>, #2591<https://github.com/balabit/syslog-ng/pull/2591>, #2586<https://github.com/balabit/syslog-ng/pull/2586>, #2584<https://github.com/balabit/syslog-ng/pull/2584>, #2583<https://github.com/balabit/syslog-ng/pull/2583>) * test_reliable_backlog: fix random failure (#2668<https://github.com/balabit/syslog-ng/pull/2668>) * Fix unit test with function pointer dereference in case of ASLR, Criterion (#2669<https://github.com/balabit/syslog-ng/pull/2669>) * test-stats-query: fix unit test (#2603<https://github.com/balabit/syslog-ng/pull/2603>) Notes to the developers * Version from git describe (#2627<https://github.com/balabit/syslog-ng/pull/2627>) * light: example-msg-generator support (#2571<https://github.com/balabit/syslog-ng/pull/2571>) * light: test app parser applications (#2686<https://github.com/balabit/syslog-ng/pull/2686>) * light: Switch to native logger (#2546<https://github.com/balabit/syslog-ng/pull/2546>) * light: Remove SetupTestcase() dependency (#2587<https://github.com/balabit/syslog-ng/pull/2587>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Chris Spencer, David Liew, Fabien Wernli, Gabor Nagy, Laszlo Budai, Laszlo Szemere, Layne, László Várady, Mehul Prajapati, Nik Ambrosch, Parth Wazurkar, Péter Kókai, Terez Nemes, Victor Ma, Zoltan Pallagi. https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.21.1 [https://avatars1.githubusercontent.com/u/986981?s=400&v=4]<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.21.1> balabit/syslog-ng<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.21.1> syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL. - balabit/syslog-ng github.com

1 0

feature request idea
by Faine, Mark R. (MSFC-IS40)[NICS] 09 May '19

09 May '19

How hard would it be to make --syntax-only accept a path to a configuration file as an argument? This would be useful to me since I use an Ansible template module to install the file and it has a validate parameter but it requires an argument for the filename. It could of course still function normally If no path to a configuration file was provided. Thanks, -Mark Mark Faine System Administrator SAIC/NICS 215 Wynn Dr. 5065 Huntsville, AL 35805 256-961-1295 (Desk) 256-617-4861 (Work Cell)

2 2

UDP alerts in syslog
by Swordfish Binary 09 May '19

09 May '19

Hi, I am having syslog running all udp/514 on 8 machines and we receive 40000 EPS in a given point in time at each of the 8 locations. We have virtual IP's in all the 8 boxes. we have syslog version 3.5.6. Questions - Besides a packet capture, is there anything in RHEL (built-in preferred) that provides a counter of UDP packets attempted and not just those that are received (see comments on *netstat -su* output below)? - Is it possible to receive all packets sent yet still get UDP errors? - What is a reasonable amount of syslog EPS to expect with this hardware? - Are there any other tweaks to the Kernel or Syslog-ng to make? Environment: - RHEL 7.3 - 16 cores - 64GB Memory - 5TB SAS 15k disk Errors: Udp: 1540062710 packets received 257 packets to unknown port received. 306945112 packet receive errors 1378189992 packets sent 306945112 receive buffer errors 101 send buffer errors - The "packets received" is how many were accepted by the listening application and NOT how many were attempted - The "packets to unknown port" is what came in for the application when the application was down or not listening (e.g. during a restart) - The "packet receive errors" clearly indicates an error; HOWEVER, it is not a one-for-one packets-to-error. You can have many more errors than packets were sent, indicating it is possible for a single packet to generate multiple errors. Tuning Steps These are similar to what was done; however, multiple values were tried so the numbers below are not exactly what is in production now: We tried turning off the udp/514 forwarding to the other applications but we did not see a noticeable drop in errors kernel net.ipv4.udp_rmem_min = 131072 net.ipv4.udp_wmem_min = 131072 net.core.netdev_max_backlog=2000 net.core.rmem_max=67108864 syslog-ng options { sync (5000); time_reopen (10); time_reap(5); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); log_fifo_size (536870912); stats_freq(60); flush_lines(500); flush_timeout(10000); };

5 4

GSoC Improve Developer Guide Project
by Victor Ma 09 May '19

09 May '19

Hi, I will be working on a plugin programming guide for this year's Google Summer of Code. It will cover source drivers, parsers, filter functions, and template functions. Each section will come with a hopefully-useful-on-its-own example plugin to aid in the explanation. My proposal can be found here: https://github.com/balabit/syslog-ng/wiki/GSoC-2019-Proposal:-Improve-Devel… I am a rising junior at the University of Toronto. I plan to graduate with a major in computer science and minors in philosophy and creative writing. Up-and-coming logger, Victor Ma

1 0

Information regarding Syslog-ng
by Nitish Saboo 08 May '19

08 May '19

Hi, I am not sure if this is the right forum to ask these generic questions.Let me know if I have to post these questions on some other forum to get the correct information. 1) What is the latest syslog-ng version ? 2) Does syslog-ng has an enterprise edition , I mean a licensed version or It has only community edition ? 3)Does the latest version contain all the bug-fixes ? Thanks, Nitish

2 1

Syslog-ng options bad_hostname & check_hostname
by Delon Lee Di Lun 08 May '19

08 May '19

Hi, Anybody used the syslog-ng options bad_hostname & check_hostname? How does that works? I have syslog-ng listening on UDP, and found that there are some gibberish logs in the folder. Was thinking if using the above two options would help reduce the gibberish. Yours Sincerely, Delon

2 4

Google Summer Of Code Student
by Mehul Prajapati 07 May '19

07 May '19

Hi All, I'm Mehul Prajapati, a graduate student in Masters of Applied Computer Science program at Concordia University, Canada. My project in GSoC 2019 is - Improve Debun tool collected information. I have 6+ years of professional work experience in C, Linux Bash Shell, and Python programming. I am an active contributor of syslog-ng project. -- Regards, Mehul Prajapati Embedded Software Developer Website: https://mehul-m-prajapati.github.io/ Email: mehulprajapati2802(a)gmail.com Mobile: +1 514-431-2331

1 0