August 2019 - syslog-ng - lists.balabit.hu

syslog-ng 3.22.1 consuming a lot of RAM on amqp or kafka destinations
by joao reis 24 Oct '19

24 Oct '19

Hi, I'm having high RAM consuming problems using both amqp or kafka destinations. I'm using patterndb (without any context definition) some basic filters. Changing the destination to "file" or "program" type, the high RAM problem doesn't happen. I tested these backends only in this version of syslog-ng and I don't know if it is a version specific problem. The backends are up and running and receiving correctly from syslog-ng. Here are my configurations: destination d_amqp { amqp( vhost("/email_log") host("127.0.0.1") port(5672) exchange("maillog") username("maillog") password("maillog") disk-buffer( mem-buf-size(10000) disk-buf-size(2000000) reliable(yes) dir("/tmp/disk-buffer") ) exchange-type("fanout") routing-key("") persistent(yes) value-pairs( scope("nv-pairs") ) ); }; destination d_kafka { kafka( bootstrap-servers("127.0.0.1:9092") topic("MYTOPIC") ); }; I added the disk-buffer option to see if it gets any better, but without any good result; Here the syslog-ng-ctl output, when configured to amqp: [root@logserver3 conf.d]# syslog-ng-ctl stats SourceName;SourceId;SourceInstance;State;Type;Number global;payload_reallocs;;a;processed;3635299 src.journald;s_sys#0;journal;a;processed;542 src.journald;s_sys#0;journal;a;stamp;1566648214 global;sdata_updates;;a;processed;0 global;scratch_buffers_bytes;;a;queued;512 source;s_host;;a;processed;5440633 destination;d_boot;;a;processed;0 destination;d_kern;;a;processed;9 source;s_sys;;a;processed;612 dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;dropped;0 dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;processed;2691676 dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;queued;0 dst.amqp;d_amqp#0;amqp,/email_log,127.0.0.1,5672,maillog,fanout;a;written;2691676 src.internal;s_sys#1;;a;processed;70 src.internal;s_sys#1;;a;stamp;1566648339 global;msg_clones;;a;processed;2691634 global;internal_queue_length;;a;processed;0 destination;d_spol;;a;processed;0 destination;d_mlal;;a;processed;0 center;;received;a;processed;5441245 destination;d_mesg;;a;processed;531 destination;d_mail;;a;processed;0 destination;d_auth;;a;processed;29 destination;d_cron;;a;processed;52 destination;d_amqp;;a;processed;2691676 global;scratch_buffers_count;;a;queued;5970004541496 center;;queued;a;processed;2692297 Thanks in advance.

2 3

Regular expressions in rewrite
by Faine, Mark R. (MSFC-IS40)[NICS] 06 Sep '19

06 Sep '19

Is there a way to use regular expressions when setting a value in a rewrite rule? Something like: rewrite r_rewrite_set_host{ subst("^[a-z]+\-([a-z]+).*$", "$1", value("HOST")); set("HOST" value("location") condition( filter(f_filter_hostnames) ) ); }; But without changing the actual log message. The goal is to create a variable from a regular expression of the host field in the message that I can then use in a destination path. Thanks, -Mark Mark Faine System Administrator SAIC/NICS 215 Wynn Dr. 5065 Huntsville, AL 35805 256-961-1295 (Desk) 256-617-4861 (Work Cell)

4 4

syslog-ng: unofficial 3.23 Debian/Ubuntu packages
by Laszlo Budai 26 Aug '19

26 Aug '19

Hi, syslog-ng 3.23.1[1] packages are available in OBS repo[2]. List of supported OSs: * Debian 8.0 * Debian 9.0 [including armv7l] * Raspbian 9.0 * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 18.04 * Ubuntu 18.10 Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.23.1 [2] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… regards, Laszlo Budai [Twitter]<https://twitter.com/Budai_L> [LinkedIn] <https://www.linkedin.com/in/laszlo-budai-55243a3/> [Github] <https://github.com/lbudai>

1 0

syslog-ng-3.23
by Laszlo Budai 26 Aug '19

26 Aug '19

3.23.1 ====== ## Features * The `redis()` destination now handles any number of command parameters. (#2816) * The `format()` option of `date-parser()` supports lists. From now on, a single `date-parser()` instance is able to process different date formats, making it easy to catch on when some programs change the way they log information. (#2779) * Add relocation support for `disk-buffer()`. The `relocate` subcommand of `dqtool` can be used to move a single or multiple queue files. (#2855) * `file(), pipe()`: The `time-reap()` option now can be set or disabled for each destination separately. Use `time-reap(0)` to disable closing idle destination files. (#2798) * `syslog-ng-ctl`: Add `list-files` subcommand to print files present in the current configuration. (#2797) ## Bugfixes * Fix minor memory leaks (#2868) * Add global context to standalone parsers (#2876) * Fix heap usage after free in cfg_run_parser_with_main_context (#2884) * Fix g_thread_init call order issue with older glibs (#2853) * SNMP destination: fixing statistics format (#2854) * eventlog: flush escaped_buffer when full (#2837) * dbparser: remove unnecessary lock (#2838) * dbparser: fix crash when context times out in the middle of another rule (#2832) * radix: fix grouping in PCRE (#2808) * add-contextual-data: make filters config plugin aware (#2886) * Undefined warning regression (#2829) ## Other changes * Disable `time-reap()` on non-templated filenames by default (#2798) * The `--preprocess-into` command line flag accepts `-`, and writes the preprocessed configuration to stdout. (#2869) * Add information on environment variables passed to the confgen script (#2888) * Light: minor fixes (#2867, #2844) * python: use malloc_debug for python unit tests (#2866) * Travis: verbose unit test output (#2851) * Travis cleanup (#2809) * filter: add unit tests to `filter-op` (#2835) * Fix clang/gcc diagnostic differences (#2810) * Fix leak in stats test (#2874) * cmake, autotools: -Wundef to enable-extra-warning (#2806) ## Notes to the developers * Check whether commit messages are properly formatted (#2803, #2807) * gitignore: tags file and build directory (#2794) ## Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Gabor Nagy, Laszlo Budai, Laszlo Szemere, László Várady, Mark Bonsack, Mehul Prajapati, Péter Kókai, Romain Tartière, Zoltan Pallagi. https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.23.1

1 0

IP WHITE LIST
by Onur GURSOY 21 Aug '19

21 Aug '19

Hello Guys, Nowadays, im finding whitelist method for IP. Is it possible ? How can i allow IP which is in whitelist ? So Many Thanks, My Best Regards. -- Onur GÜRSOY R&D Engineer in Embedded Systems Master Student at Gebze Institute Of Technology Department Of Electronic Engineering GSM : 0(545) 764 7653 e-mail: onurgursoygyte(a)gmail.com

2 1

Dynamic collection name in MongoDb
by PC LP 15 Aug '19

15 Aug '19

Hi, I am retrieving the value for *$M_ADDRESS* from the Syslog message using patterndb. $M_ADDRESS is a unique value. Based on that I have to create the collection in MongoDB dynamically. If I provide the value like below it is not recognizing collection("$M_ADDRESS") Is there any alternate way to create it? Please advise. Regards, PCLP.

2 1

if condition
by Alexey Vlasov 15 Aug '19

15 Aug '19

Hi, I would like to optimize my syslong-ng.conf file thus it could be used on some boxes without any changing. Now I have a few modifications of syslong-ng.conf on each server to copy logs to remote box. For example, group of hosts A* ============================== A1-machine (it is its hostname) has: destination clnt { udp("remote-box-a" port(514)); }; log { source(src); filter(f_messages); destination(clnt); }; ============================== ============================== A2-machine has: destination clnt { udp("remote-box-a" port(514)); }; log { source(src); filter(f_messages); destination(clnt); }; ============================== and group of hosts B* ============================== B1-machine has: destination clnt { udp("remote-box-b" port(514)); }; log { source(src); filter(f_messages); destination(clnt); }; ============================== ============================== B2-machine has: destination clnt { udp("remote-box-b" port(514)); }; log { source(src); filter(f_messages); destination(clnt); }; ============================== Is there possible to write something like that If (${HOSTNAME} ~ ^A) use remote-box-a If (${HOSTNAME} ~ ^B) use remote-box-b Thanks in advance!

2 1

3.22.1: understanding the location of the python's part
by Sergey 14 Aug '19

14 Aug '19

Hello I see that in 3.22 has changed the location of the python's part (relative 3.20): architecture independent files moved to architecture specific directory (lib64 on x86_64). For example, rhel spec 3.20 (syslog-ng-3.20.1/packaging/rhel/syslog-ng.spec) %files python /usr/lib/python2.7/site-packages/syslogng-1.0-py2.7.egg-info %dir /usr/lib/python2.7/site-packages/syslogng/ %dir /usr/lib/python2.7/site-packages/syslogng/debuggercli/ /usr/lib/python2.7/site-packages/syslogng/* /usr/lib/python2.7/site-packages/syslogng/debuggercli/* %{_libdir}/%{name}/libmod-python.so The same in 3.22 %files python %{_libdir}/%{name}/python/syslogng-1.0-py%{py_ver}.egg-info %{_libdir}/%{name}/python/syslogng/* %{_libdir}/%{name}/libmod-python.so Is the architecture specific file libmod-python.so only? Or am I missing something? -- Regards, Sergey

2 1

Parse message fields for use as columns in MySQL
by Allen Olivas 09 Aug '19

09 Aug '19

Hello, I'm really scratching my head trying to make this work and thought maybe the community has experienced this before. I'm collecting logs from Wazuh and Syslog-NG. Those logs are sent from my Wazuh server with Sylog-NG configured to send to my MySQL server. The Syslog-ng.conf file on the MySQL server is configured with a destination to mysql. In the declared the destination and list out columns and values. # MySQL define destination destination d_mysql { sql( type(mysql) username("syslog") password("xxxxxxx") database("syslog") host("127.0.0.1") table("logs") columns("host", "id", "location", "facility", "rule", "priority", "level", "tag", "datetime", "program", "msg") values("$HOST", "$ID", "$LOCATION","$FACILITY", "$RULE", "$PRIORITY", "$LEVEL", "$TAG","$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC","$PROGRAM", "$MSG") indexes("datetime", "host", "id", "location", "rule") ); }; So here's the problem. The Message data contains information like Rule and Location that really equate to the Wazuh Rule and Location = the Wazuh Agent that's reporting it. I had hoped "location" column would populate with the Location date from the message. Same with Rule. SO my question to the community is how on earth do I parse the data in the message field to populate columns (existing or new)? Any thoughts, guidance, recommendations are greatly appreciated. Thanks, Allen Olivas InfoDefense Office: (972) 848-7910 Email: allen.olivas(a)infodefense.com Toll Free: (877) INFODEFENSE www.infodefense.com<https://l.shatrk.com/r/e/DblvLSPvKY2IxMPE?r=https://app.salesforceiq.com/r?…>

2 5

Syslog-ng stops writing until restarted
by Faine, Mark R. (MSFC-IS40)[NICS] 07 Aug '19

07 Aug '19

I have a strange problem where logs are definitely coming in over tcp and udp however they are not written, well not until I restart the server. Whenever I restart the logs are written but shortly after starting they stop until the server is restarted again. Any ideas on where I can look? Thanks, -Mark

2 2