- syslog-ng - lists.balabit.hu

Systemd service times out for syslog-ng 3.16.1
by Ronald Fenner 16 May '19

16 May '19

I'm trying to switch from an init.d script to a system service and having problems getting the service to run. The syslog-ng service gets timed out and never starts up. My service looks like this: [Unit] Description="Syslog-ng an enhanced syslog" Documentation=man:syslog-ng(8) After=network.target ConditionFileNotEmpty=/opt/syslog-ng/etc/syslog-ng.conf [Service] Type=notify ExecStart=/opt/syslog-ng/sbin/syslog-ng --no-caps -p /var/run/syslogd.pid -rdvt ExecReload=/bin/kill -HUP $MAINPID StandardOutput=journal StandardError=journal Restart=on-failure Environment=LD_LIBRARY_PATH=/usr/lib/jvm/java/jre/lib/amd64/server:$LD_LIBRARY_PATH [Install] WantedBy=multi-user.target My config looks like this: ############################################################################# # Default syslog-ng.conf.partial file which collects all local logs into a # single file called /var/log/messages. # @version: 3.16 @module "mod-java" @include "scl.conf" @include "conf.d/gc-parsers.conf" source s_internal {internal();}; source s_rtl_stream { unix-stream("/var/gamecircus/log/rtl-stream.sock" flags(no-parse) log_msg_size(1000000)); }; # Backup for when syslog-ng crashes and doesn't restart queues the messages source s_rtl_backup { file("/var/gamecircus/log/rtl-backup.log" flags(no-parse) log_msg_size(1000000)); }; destination d_kafka_unstructured { kafka ( client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka/libs/") kafka-bootstrap-servers(<redacted>) topic("syslog-ng-`DeployedEnvironment`") ); }; destination d_kafka_structured { kafka ( client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka/libs/") kafka-bootstrap-servers(<redacted>) topic("${topic}.${environment}") template("$(format-json --scope nv_pairs --exclude MESSAGE)\n") ); }; destination d_syslog_ng { file("/var/log/syslog-ng"); }; parser p_json { json-parser(); }; log { source(s_internal); destination(d_syslog_ng); }; log { source(s_rtl_stream); parser(p_json); destination(d_kafka_structured); }; log { source(s_rtl_backup); parser(p_json); destination(d_kafka_structured); }; # Build script will add stuff below for the specific build. ############################################################################### The logging that I turned on in the service above isbn't helping. Here's the last little bit May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.217091] Add path to classpath: /opt/kafka/libs/kafka-streams-0.11.0.0.jar; May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.217248] Add path to classpath: /opt/kafka/libs/rocksdbjni-5.0.1.jar; May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.217309] Add path to classpath: /opt/kafka/libs/kafka-streams-examples-0.11.0.0.jar; May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.361062] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/syslog-ng-core.jar; May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.365718] Running application hooks; hook='1' May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.365930] Running application hooks; hook='3' May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.366137] syslog-ng starting up; version='3.16.1' May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.366480] Worker thread started; driver='d_kafka_structured#0' May 16 15:11:43 ip-172-30-6-13.ec2.internal systemd[1]: syslog-ng.service start operation timed out. Terminating. May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.273058] Running application hooks; hook='4' May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.273074] syslog-ng shutting down; version='3.16.1' May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.376882] Worker thread finished; driver='d_kafka_structured#0' May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.377030] Closing log transport fd; fd='92' May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.377186] Java machine free; May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.378858] Running application hooks; hook='5' May 16 15:11:43 ip-172-30-6-13.ec2.internal systemd[1]: Failed to start "Syslog-ng an enhanced syslog". May 16 15:11:43 ip-172-30-6-13.ec2.internal systemd[1]: Unit syslog-ng.service entered failed state. May 16 15:11:43 ip-172-30-6-13.ec2.internal systemd[1]: syslog-ng.service failed. Any suggestions on what me causing it to timeout? I'm new to Ssytemd but if I understand the notify type it would suggest that syslog-ng isn't notifying systemd that it's started. I'm not sure what would be holding it up. Any help is much appreciated. Ronald Fenner Network Architect Game Circus LLC. rfenner(a)gamecircus.com

2 5

Rest2Syslog
by C B 15 May '19

15 May '19

hello all, just wanted to share with you a new python syslog-ng plugin I was working on to support ingestion of any REST API into any syslog destination. this is off course open source and available here - https://github.com/chenbekor/Rest2Syslog I'll be happy to receive any feedback/suggestion. Also - if any of you guys know of a public page with all available plugins/extensions please let me know so I can add this nice project to the list. Regards, Chen.

2 1

Re: [syslog-ng] Troubleshooting Question
by Walter Tienken 14 May '19

14 May '19

Hi, Yes, the em1 interface is for our front-end connectivity and the em2 interface is for the F5 LB where traffic is sent through. By dual stack, I assume you mean IPV4/IPV6 ... in that case, no, only IPV4. It looks like syslog-ng is listening on all interfaces: Here is the output of netstat -ulp: https://gist.github.com/MrTink76/ade770cfe6d497aa50fe820a9c438f54 The part that's throwing me off is "distinct" ... shouldn't that be 9999? You see it is listening on all interfaces and the netstat -anp | grep 9999 here as well: https://gist.github.com/MrTink76/7741d90831fe4f331abe4630f3a8b29e Thanks much, Walter Tienken walter.tienken(a)asu.edu On 5/12/19, 8:56 AM, "Fabien Wernli" <wernli(a)in2p3.fr> wrote: Does your box have multiple interfaces? Does it run a dual-stack? If yes, does the program (nc or syslogng) listen on the correct interface? (netstat -ulp)

2 1

How to create two independent patternDB instances in syslog-ng
by Nitish Saboo 13 May '19

13 May '19

I understand PatternDB is a syslog-ng pluginand its dynamic library can be found in as libdbparser.so. I went through the following tests in ' https://github.com/balabit/syslog-ng/blob/master/modules/dbparser/patterndb… ' I want to create two independent pattern_db instances having two different ruleset. My queries: 1) How can I create two independent instances of patternDB having two different ruleset ? 2)How can I get the handle of these two independent PatternDB instances ? Can someone please help me on this ? Thanks

1 0

Is it possible to create Two instances of syslog-ng engine ?
by Nitish Saboo 13 May '19

13 May '19

Hi, I have installed syslog-ng on my Linux box and I am planning you use syslog-ng parser and wanted to initialise it's engine for parsing the data. so initialise() method in my Go code calls C code that is wrapper around syslog-ng C header files. So ,I am making use of cgo. Following is my Go code: func (obj Syslogparser) initialise{ X := C.CString(obj.X) defer C.free(unsafe.Pointer(X)) Y := C.CString(obj.Y) defer C.free(unsafe.Pointer(Y)) C.initialize_engine(X, Y, (C.key_value_cb)(unsafe.Pointer(C.callOnMeGo_cgo))); } And following is my C method 'initialize_engine': int initialize_engine(const gchar* X, const gchar* Y, key_value_cb cb) // cb is callback function { module_path = _module_path; app_startup(); msg_init(TRUE); configuration = cfg_new(0x0302); plugin_load_module("basicfuncs", configuration, NULL); plugin_load_module("syslogformat", configuration, NULL); pattern_db_global_init(); if(patterndb != NULL){ pattern_db_free(patterndb); } patterndb = pattern_db_new(); pattern_db_reload_ruleset(patterndb, configuration, filename); pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb); return 0; } My queries: 1)What is pattern_db_global_init, 'patterndb = pattern_db_new()', 'pattern_db_reload_ruleset', 'pattern_db_set_emit_func' doing in the following case ? 2)Can I make two independent instances of syslog-ng engines? 3)How can I call two instances of syslog-ng engines independently from my Go code? Thanks

2 2

Troubleshooting Question
by Walter Tienken 12 May '19

12 May '19

Hello All, I'd like to see if we can get some troubleshooting help with Syslog-NG OSE. Here's some background: Our environment collects logs from various network locations (F5, checkpoint, plixer, etc) and sends them to our Syslog-NG cluster through two F5 load balancers. We have four netlog boxes in round-robin that are running version 5 of the PE version of Syslog-NG on RHEL6. We also have a dev server running rhel7 and the 3.19.1 OSE edition of Syslog-NG from the COPR repository. On each of these servers, we run the Splunk Universal Forwarder that then sends the logs over to our indexer cluster. The problem we are having is that the dev server will not listen to any traffic except via localhost. We can see the traffic just fine on a TCPDUMP as it comes out of the load balancer into the dev box, but watching Syslog-NG in the foreground with -Fevd the traffic never registers at all. We can send test messages with loggen or netcat, etc, from localhost and Syslog-NG will see it and log it to disk as expected. Anywhere else it just never sees the traffic nor logs anything to disk even though we have confirmed that Syslog-NG is listening on the 9999 port to UDP with netstat. Here is our syslog-ng.conf file from the working RHEL6 boxes: https://gist.github.com/MrTink76/9ee1e88f93a313f953e4033560af463a This is the syslog-ng.conf file on the OSE box: https://gist.github.com/MrTink76/e181f4c0bf052077440d7bdfaf418e02 This is an example of our testing CONF file located in conf.d on the OSE box: https://gist.github.com/MrTink76/a0433b9e908ba683e36cb6199f9cc43f We send our test traffic to the F5 load balancer vip using UDP 9999. Like I said above, when we send on 9999 localhost with loggen or netcat, Syslog-NG sees it just fine and logs it to disk, but anywhere else it never registers nor records the test message to disk. We currently have SELinux disabled and there is no firewall running on the dev box (we see the traffic fine via tcpdump). Any help/suggestions would be greatly appreciated. Please let me know if I need to provide further information. Thanks much, Walter Tienken walter.tienken(a)asu.edu<https://ex2010.asu.edu/owa/redir.aspx?SURL=sS2_o_WV6gQ_JAkG-_VgxIDZLGj9-EeB…> Cloud and Advanced Network Engineering Services

3 4

unofficial syslog-ng packages for syslog-ng 3.21.1
by Laszlo Budai 10 May '19

10 May '19

Hi, syslog-ng 3.21.1[1] packages are available in OBS repo[2]. List of supported OSs: * Debian 8.0 * Debian 9.0 [including armv7l] * Raspbian 9.0 * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 18.04 * Ubuntu 18.10 * Ubuntu 19.04 (dependencies are still building) Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.21.1 [2] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… regards, Laszlo Budai

1 0

syslog-ng 3.21.1
by Laszlo Budai 10 May '19

10 May '19

3.21.1 Highlights * Add an alternative, native, librdkafka based kafka-c() destination in parallel of the existing Java implementation, that provides the same configuration interface. Eventually, we expect this to replace the Java one (#2496<https://github.com/balabit/syslog-ng/pull/2496>) * Add a native, http() based destination based driver for elasticsearch called elasticsearch-http(), as an alternative of the Java one. Eventually, we expect this to replace the Java implementation. (#2509<https://github.com/balabit/syslog-ng/pull/2509>) * Add the ability to automatically determine the timezone value for an incoming log entry as long as the incoming stream is close to real time and the timezone information is missing from the timestamp. Enable this function by using flags(guess-timezone) for sources and the date-parser(). (#2517<https://github.com/balabit/syslog-ng/pull/2517>, #2673<https://github.com/balabit/syslog-ng/pull/2673>) Features * syslog(): Add the ability to work with messages larger than log-msg-size() in the source driver by using the trim-large-messages(yes) option. The characters over the limit will be truncated. Previously messages longer than the limit caused the connection to be closed abruptly. (#2644<https://github.com/balabit/syslog-ng/pull/2644>) * amqp(): add support for heartbeats and the "external" authentication mechanism. (#2676<https://github.com/balabit/syslog-ng/pull/2676>, #2626<https://github.com/balabit/syslog-ng/pull/2626>) * graylog2(): add support for TLS and UDP. (#2657<https://github.com/balabit/syslog-ng/pull/2657>) * udp(): Add spoof-source-max-msglen() option to allow setting the maximum spoofed datagram size, which was hard-wired to 1024 previously. (#2535<https://github.com/balabit/syslog-ng/pull/2535>) * db-parser(): add an option program-template() that customizes the value used for matching the PROGRAM field. (#2651<https://github.com/balabit/syslog-ng/pull/2651>) * pdbtool: Add sort option to pdbtool merge (#2664<https://github.com/balabit/syslog-ng/pull/2664>) * $(implode) and $(explode): add template functions to split and join strings based on a simple separator. The exploded array is represented as a syslog-ng list that can be manipulated with the $(list-*) template functions. (#2700<https://github.com/balabit/syslog-ng/pull/2700>) * Add an --omit-empty-values option for value-pairs based destinations & template functions. (#2519<https://github.com/balabit/syslog-ng/pull/2519>) * grouping-by() parser: add sort-key() option (#2701<https://github.com/balabit/syslog-ng/pull/2701>) Support for non-syslog or non-standard formats in SCL * apache-accesslog-parser(): support for vhost:port as the first field in common/combined log formats (#2688<https://github.com/balabit/syslog-ng/pull/2688>) * Add application adapter for Junos classification (#2684<https://github.com/balabit/syslog-ng/pull/2684>) * Add parser and adapter for CheckPoint LogExporter output (#2665<https://github.com/balabit/syslog-ng/pull/2665>) Bugfixes * Fix race condition of idle timer and scheduled I/O job (#2650<https://github.com/balabit/syslog-ng/pull/2650>) * Few leaks find via sanitizer (#2696<https://github.com/balabit/syslog-ng/pull/2696>) * syslogformat: set $MSG even if the incoming message is empty (#2672<https://github.com/balabit/syslog-ng/pull/2672>) * Fix double-free error in logproto unit tests (#2662<https://github.com/balabit/syslog-ng/pull/2662>) * groupingby: identical persist name (#2659<https://github.com/balabit/syslog-ng/pull/2659>) * stats: deindex pruned counters/clusters (#2648<https://github.com/balabit/syslog-ng/pull/2648>) * Type hinting should not accept empty values (#2639<https://github.com/balabit/syslog-ng/pull/2639>) * app-parser, pseudofile: fix crash with grammar error (#2640<https://github.com/balabit/syslog-ng/pull/2640>) * python: set_timestamp normalization (#2643<https://github.com/balabit/syslog-ng/pull/2643>) * db-parser: fix memory leak (#2652<https://github.com/balabit/syslog-ng/pull/2652>) * grouping-by: use after free, memory leak, missing init calls of filters (#2655<https://github.com/balabit/syslog-ng/pull/2655>) * amqp: fixing double connect (#2660<https://github.com/balabit/syslog-ng/pull/2660>) * old style definition warning fixes (#2680<https://github.com/balabit/syslog-ng/pull/2680>) * Fix "!=" filter (#2683<https://github.com/balabit/syslog-ng/pull/2683>) * dbparser: fix memleak (#2706<https://github.com/balabit/syslog-ng/pull/2706>) * nondumpable-allocator: fixing mmap error handling (#2666<https://github.com/balabit/syslog-ng/pull/2666>) * Fix timeutils warning (#2604<https://github.com/balabit/syslog-ng/pull/2604>) * Fix old style include statement compatibility (#2600<https://github.com/balabit/syslog-ng/pull/2600>) * Fix config revert (threaded destinations) (#2596<https://github.com/balabit/syslog-ng/pull/2596>) * Add warning on old style include statement (#2592<https://github.com/balabit/syslog-ng/pull/2592>) Other changes * cfg-parser: add aliases for yesno (#2671<https://github.com/balabit/syslog-ng/pull/2671>) * Include json-c in the dist tarball (#2590<https://github.com/balabit/syslog-ng/pull/2590>) * cmake: disable_all_modules support (#2647<https://github.com/balabit/syslog-ng/pull/2647>) * Cmake clang sanitizer (#2562<https://github.com/balabit/syslog-ng/pull/2562>) * timeutils refactor (#2483<https://github.com/balabit/syslog-ng/pull/2483>) * Expedite threaded flush at reload (#2656<https://github.com/balabit/syslog-ng/pull/2656>) * elasticsearch2: Added deprecation warning (#2628<https://github.com/balabit/syslog-ng/pull/2628>) * Astyle fixes (#2624<https://github.com/balabit/syslog-ng/pull/2624>) * Force C99 with GNU (#2623<https://github.com/balabit/syslog-ng/pull/2623>) * Make rewording and other small edits to README (#2608<https://github.com/balabit/syslog-ng/pull/2608>) * Port tests to Criterion (#2607<https://github.com/balabit/syslog-ng/pull/2607>, #2661<https://github.com/balabit/syslog-ng/pull/2661>, #2621<https://github.com/balabit/syslog-ng/pull/2621>, #2620<https://github.com/balabit/syslog-ng/pull/2620>, #2619<https://github.com/balabit/syslog-ng/pull/2619>, #2618<https://github.com/balabit/syslog-ng/pull/2618>, #2617<https://github.com/balabit/syslog-ng/pull/2617>, #2616<https://github.com/balabit/syslog-ng/pull/2616>, #2615<https://github.com/balabit/syslog-ng/pull/2615>, #2599<https://github.com/balabit/syslog-ng/pull/2599>, #2594<https://github.com/balabit/syslog-ng/pull/2594>, #2593<https://github.com/balabit/syslog-ng/pull/2593>, #2591<https://github.com/balabit/syslog-ng/pull/2591>, #2586<https://github.com/balabit/syslog-ng/pull/2586>, #2584<https://github.com/balabit/syslog-ng/pull/2584>, #2583<https://github.com/balabit/syslog-ng/pull/2583>) * test_reliable_backlog: fix random failure (#2668<https://github.com/balabit/syslog-ng/pull/2668>) * Fix unit test with function pointer dereference in case of ASLR, Criterion (#2669<https://github.com/balabit/syslog-ng/pull/2669>) * test-stats-query: fix unit test (#2603<https://github.com/balabit/syslog-ng/pull/2603>) Notes to the developers * Version from git describe (#2627<https://github.com/balabit/syslog-ng/pull/2627>) * light: example-msg-generator support (#2571<https://github.com/balabit/syslog-ng/pull/2571>) * light: test app parser applications (#2686<https://github.com/balabit/syslog-ng/pull/2686>) * light: Switch to native logger (#2546<https://github.com/balabit/syslog-ng/pull/2546>) * light: Remove SetupTestcase() dependency (#2587<https://github.com/balabit/syslog-ng/pull/2587>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Chris Spencer, David Liew, Fabien Wernli, Gabor Nagy, Laszlo Budai, Laszlo Szemere, Layne, László Várady, Mehul Prajapati, Nik Ambrosch, Parth Wazurkar, Péter Kókai, Terez Nemes, Victor Ma, Zoltan Pallagi. https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.21.1 [https://avatars1.githubusercontent.com/u/986981?s=400&v=4]<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.21.1> balabit/syslog-ng<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.21.1> syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL. - balabit/syslog-ng github.com

1 0

feature request idea
by Faine, Mark R. (MSFC-IS40)[NICS] 09 May '19

09 May '19

How hard would it be to make --syntax-only accept a path to a configuration file as an argument? This would be useful to me since I use an Ansible template module to install the file and it has a validate parameter but it requires an argument for the filename. It could of course still function normally If no path to a configuration file was provided. Thanks, -Mark Mark Faine System Administrator SAIC/NICS 215 Wynn Dr. 5065 Huntsville, AL 35805 256-961-1295 (Desk) 256-617-4861 (Work Cell)

2 2

UDP alerts in syslog
by Swordfish Binary 09 May '19

09 May '19

Hi, I am having syslog running all udp/514 on 8 machines and we receive 40000 EPS in a given point in time at each of the 8 locations. We have virtual IP's in all the 8 boxes. we have syslog version 3.5.6. Questions - Besides a packet capture, is there anything in RHEL (built-in preferred) that provides a counter of UDP packets attempted and not just those that are received (see comments on *netstat -su* output below)? - Is it possible to receive all packets sent yet still get UDP errors? - What is a reasonable amount of syslog EPS to expect with this hardware? - Are there any other tweaks to the Kernel or Syslog-ng to make? Environment: - RHEL 7.3 - 16 cores - 64GB Memory - 5TB SAS 15k disk Errors: Udp: 1540062710 packets received 257 packets to unknown port received. 306945112 packet receive errors 1378189992 packets sent 306945112 receive buffer errors 101 send buffer errors - The "packets received" is how many were accepted by the listening application and NOT how many were attempted - The "packets to unknown port" is what came in for the application when the application was down or not listening (e.g. during a restart) - The "packet receive errors" clearly indicates an error; HOWEVER, it is not a one-for-one packets-to-error. You can have many more errors than packets were sent, indicating it is possible for a single packet to generate multiple errors. Tuning Steps These are similar to what was done; however, multiple values were tried so the numbers below are not exactly what is in production now: We tried turning off the udp/514 forwarding to the other applications but we did not see a noticeable drop in errors kernel net.ipv4.udp_rmem_min = 131072 net.ipv4.udp_wmem_min = 131072 net.core.netdev_max_backlog=2000 net.core.rmem_max=67108864 syslog-ng options { sync (5000); time_reopen (10); time_reap(5); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); log_fifo_size (536870912); stats_freq(60); flush_lines(500); flush_timeout(10000); };

5 4