- syslog-ng - lists.balabit.hu

Syslog-ng relay: how to delete or modify special character ^M?
by alain.villefranque＠orange.com 06 Dec '17

06 Dec '17

Hi all, I have an issue with the character Carriage Return (^M) inserted by the Syslog source machine. I try to substitute or rewrite this special character with syslog-ng relay, but it is impossible to modify it, it seems this char is invisible for the program. I've tried to modified it with either with ^M or 0xD char with no success. Example of received message: Dec 4 13:35:23 conf01 2017-12-04 13:35:23,561 Level="INFO" Name="support.sip" Message="Sending SIP response" Src-address="80.12.yy.xx" Src-port="5061" Dst-address="161.105.yy.xx" Dst-port="38509" Transport="TLS" Detail="^MSIP/2.0 403 Forbidden^MVia: SIP/2.0/TLS 161.105.150.12:38509;alias;branch=z9hG4bK.MEgSOM8O4;rport=38509;received=161.105.150.12^MFrom: <sip:SondeSQS_001!@int.ovp.orange-business.com>;tag=5kYhVMAyi^MTo: sip:SondeSQS_001!@int.ovp.orange-business.com;tag=aynkBKUjt0pXHzNv^MCSeq: 25 REGISTER^MCall-ID: Vhihsb~BhQ^MAllow: INVITE,ACK,OPTIONS,CANCEL,BYE,REGISTER,INFO,SUBSCRIBE,NOTIFY,MESSAGE^MSupported: categoryList,adhoclist,sdp-anat,replaces^MContent-Length: 0^M^M" I'd like to suppress ^M or replace it with a "space" char. Is there any specific action to do in order to modify ^M special character ? Regards Alain _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.

2 4

OSE 3.13.2
by Budai, László 06 Dec '17

06 Dec '17

Hi, the new release contains the following fixes : - Missing manpages from release tarball (#1793 <https://github.com/balabit/syslog-ng/issues/1793>) - Package syslog-ng-mod-json is removed from (#1794 <https://github.com/balabit/syslog-ng/pull/1794>) - Drop syslog-ng-abi virtual packages (#1797 <https://github.com/balabit/syslog-ng/pull/1797>) regards, Laszlo Budai

1 0

syslog-ng 3.13.1
by Budai, László 05 Dec '17

05 Dec '17

3.13.1Features - Add app-parser() framework (automatic parsing of log messages) (#1689 <https://github.com/balabit/syslog-ng/pull/1689>) - Support microseconds in Riemann destination (#1710 <https://github.com/balabit/syslog-ng/pull/1710>) - Add osquery destination as an SCL plugin (#1728 <https://github.com/balabit/syslog-ng/pull/1728>) - Add network load balancer destination (#1706 <https://github.com/balabit/syslog-ng/pull/1706>) - Add possibility to only signal re-open of file handles (SIGUSR1) (#1530 <https://github.com/balabit/syslog-ng/pull/1530>) - It is possible from now to limit the number of registered dynamic counters (#1743 <https://github.com/balabit/syslog-ng/pull/1743>) - Add $(binary) template function (#1679 <https://github.com/balabit/syslog-ng/pull/1679>) - Add experimental transport for transferring messages in whole between syslog-ng instances (EWMM) (#1689 <https://github.com/balabit/syslog-ng/pull/1689>) - Docker based build and debian package generation (#1783 <https://github.com/balabit/syslog-ng/pull/1783>) - Add auto-parse(yes/no) to app-paser(), system() and default-network-drivers() (#1788 <https://github.com/balabit/syslog-ng/pull/1788>) - Add Graylog2 destination and $(format-gelf) template function (#1680 <https://github.com/balabit/syslog-ng/pull/1680>) Bugfixes - Exit when a read fails on an included config file instead of starting up with an empty configuration. (#1721 <https://github.com/balabit/syslog-ng/pull/1721>) - Fix double free (#1720 <https://github.com/balabit/syslog-ng/issues/1720>) - Add missing discarded counter to groupingby (#1748 <https://github.com/balabit/syslog-ng/pull/1748>) - Fix a reference leak in Python destination (#1716 <https://github.com/balabit/syslog-ng/pull/1716>) - Fix timezone issue in snmptrapd parser (#1746 <https://github.com/balabit/syslog-ng/pull/1746>) - Fix potential crash in stdin driver (#1741 <https://github.com/balabit/syslog-ng/pull/1741>) - Fix a crash when initializing new config fails for socket with keep_alive off (#1723 <https://github.com/balabit/syslog-ng/pull/1723>) - Fix filter evaluation in case of contexts with multiple elements (#1718 <https://github.com/balabit/syslog-ng/pull/1718>) - Various grouping-by fixes (#1718 <https://github.com/balabit/syslog-ng/pull/1718>) - Fix potential use after free around dns-cache during shutdown (#1666 <https://github.com/balabit/syslog-ng/pull/1666>) - Fix access to indirect values within Java destination (#1732 <https://github.com/balabit/syslog-ng/pull/1732>) - Fix a crash in affile (#1725 <https://github.com/balabit/syslog-ng/pull/1725>) - Fix a memory leak (#1724 <https://github.com/balabit/syslog-ng/pull/1724>) - Fix a crash when getent is used empty group (#1691 <https://github.com/balabit/syslog-ng/pull/1691>) - Fix jvm-options() (#1704 <https://github.com/balabit/syslog-ng/pull/1704>) - Fix a crash in Python language binding (#1694 <https://github.com/balabit/syslog-ng/pull/1694>) - Fix a crash in afmongodb (#1765 <https://github.com/balabit/syslog-ng/pull/1765>) - Fix a memory leak in afmongodb (#1766 <https://github.com/balabit/syslog-ng/pull/1766>) - Fix name-to-GID calculation in the $(getent) template function (#1764 <https://github.com/balabit/syslog-ng/pull/1764>) - Fix a crash when redis is configured without the command() option ( #1767 <https://github.com/balabit/syslog-ng/pull/1767>) - Fix a race condition in kv-parser() (#1789 <https://github.com/balabit/syslog-ng/pull/1789>) Other changes - Cleanup diskq related warning messages (#1752 <https://github.com/balabit/syslog-ng/pull/1752>) - Provide tls block for tls options in amqp(), http(), riemann() destination drivers (#1715 <https://github.com/balabit/syslog-ng/pull/1715>) - It it possible from now to register blocks and generators as plugins ( #1657 <https://github.com/balabit/syslog-ng/pull/1657>) - Drop compatiblity with configurations below 3.0 (#1709 <https://github.com/balabit/syslog-ng/pull/1709>) - Do not change permissions of a file by default (#1782 <https://github.com/balabit/syslog-ng/pull/1782>) - Allow source files to specify permissions locally (#1782 <https://github.com/balabit/syslog-ng/pull/1782>) - Minor performance improvement (#1729 <https://github.com/balabit/syslog-ng/pull/1729>) - The current config version can be queried with "--version" (#1740 <https://github.com/balabit/syslog-ng/pull/1740>) - Increase the performance of kv-parser() (#1789 <https://github.com/balabit/syslog-ng/pull/1789>) Notes to the developers - Change configure default option for jsonc and mongoc from auto to internal (#1735 <https://github.com/balabit/syslog-ng/pull/1735>) - Disable ASLR when running unit tests (#1753 <https://github.com/balabit/syslog-ng/pull/1753>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szalay, Balazs Scheidler, Gabor Nagy, Jakub Jankowski, Janos Szigetvari, Laszlo Budai, Laszlo Varady, Laszlo Szemere, Marton Illes, Mate Farkas, Peter Kokai, Pontus Andersson, Sam Stephenson, Sebastian Roland, Viktor Juhasz, Zoltan Pallagi.

2 4

syslog-ng-3.13: unofficial Debian packages: still building, be careful when updating
by Budai, László 04 Dec '17

04 Dec '17

Hi, packages will be available also for Ubuntu 17.10. There is a major change in packaging: syslog-ng-mod-json is part of syslog-ng-core from now (due to the new app-parser and Enterprise Wide Message Model). regards, Laszlo Budai

1 0

Timezone
by Sebastian Roland 01 Dec '17

01 Dec '17

Hi, after reading the admin guide and playing around with different setups several times I'm still struggling to fully understand the timezone functionality of syslog-ng. How are time-zone() / recv-time-zone() and send-time-zone() related? Some notes I made during investigation: * Logging through syslog() function logs in old BSD syslog format which does not contain a timezone. recv-time-zone() is utilized to assign a timezone. If no value has been specified the local time zone is used. * According to the admin guide send-time-zone() is only used when the timezone is not specified otherwise. This didn't turned out to be true. Example: Syslog server a sends via syslog protocol over tcp (timezone is part of the message) to server b. setting send-time-zone(x) on server b changes the timezone (and timestamp) in the destination file to the time in timezone x. If send-time-zone() is not set at all nothing happens although the admin guide states that the default is to use the local timezone. IMHO no change should be applied to the message. Note that keep- timestamp(yes) is set on server b. * If both time-zone() and send-time-zone() are set globally time-zone() overrides send-time-zone() * time-zone() can be set globally and on drivers. Specific settings overrides global config. The confusing part is the behavior when a timestamp is already set within an incoming message and send-time-zone() is explicitly set (with keep-timestamp(yes)). Is it actually intended that send-time-zone() changes a timestamp? Shouldn't the logic be that recv-time-zone() and send-time-zone() are only relevant when there is no tz offset available and a default one needs to be set for receiving and sending respectively and time-zone() is used to actually convert to a different timezone? If I'm getting something fundamentally wrong please advice. Kind regards Sebastian

2 1

testing fix for file source ownership and permissionso
by Jim Segrave 01 Dec '17

01 Dec '17

> hi, > I've just posted a pull request that I didn't have time to test: can you pls test if this fixes the issue for you? > https://github.com/balabit/syslog-ng/pull/1782 <https://github.com/balabit/syslog-ng/pull/1782> While my original patch worked on 3.6.5, we actually need some of the features in 3.12. I have problems with autogen/automake being unhappy with the versions of the autotool chain I have: running autogen.sh: Running autogen in 'lib/ivykis'... libtoolize: You should add the contents of the following files to 'aclocal.m4': libtoolize: '/usr/local/share/aclocal/libtool.m4' libtoolize: '/usr/local/share/aclocal/ltoptions.m4' libtoolize: '/usr/local/share/aclocal/ltversion.m4' libtoolize: Consider adding 'AC_CONFIG_MACRO_DIRS([m4])' to configure.ac, libtoolize: and rerunning libtoolize and aclocal. libtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am. (catting the suggested files onto aclocal.m4 does not remove these warnings) However configure does finish, but make bombs out with: /bin/sh ../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I.. -D_GNU_SOURCE -I../src/include -I../src/include -I /usr/local/src/syslog-ng -Wall -pthread -Wno-error -MT iv_avl.lo -MD -MP -MF .deps/iv_avl.Tpo -c -o iv_avl.lo iv_avl.c libtool: Version mismatch error. This is libtool 2.4.5, but the libtool: definition of this LT_INIT comes from libtool 2.4.2. libtool: You should recreate aclocal.m4 with macros from libtool 2.4.5 libtool: and run autoconf again. make[5]: *** [iv_avl.lo] Error 63 make[4]: *** [all-recursive] Error 1 make[3]: *** [all] Error 2 make[2]: *** [lib/ivykis/src/libivykis.la] Error 2 make[1]: *** [all-recursive] Error 1 make: *** [all] Error 2 I was able to get a working version from the 3.6.5 sources. When I get the 3.12.1 sources from the Centos rpm, I can run configure without running autogen, which does complete and make completes without errors. However, the resulting executable fails to start with this error message: Error parsing config, Error compiling template (Unknown template function "format-json32: included included template-function "format-cim" "$(format-json --pair @timestamp='${R_ISODATE}' --pair .=_ --key *.*)\n"; ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ I know next to nothing about the automake and friends (or m4 for that matter), so I don't really know how to address this. -- -- Jim Segrave jes(a)j-e-s.net

2 1

Testing application adapters and enterprise-wide message model for syslog-ng
by Czanik, Péter 30 Nov '17

30 Nov '17

Hi, Do you want to simplify parsing your log messages? Try the new “application adapter” and “enterprise-wide message model” frameworks in syslog-ng: you can automatically parse log messages and forward the results to another syslog-ng instance. Optionally, you can also include the original, raw message that you can forward unmodified to a SIEM system for further analysis. While these features are work in progress and might change considerably in future releases, we appreciate early testing and feedback. With the help of your comments we could cover not just our internal use cases, but also take into account the needs of the wider community. Read more about it at https://www.balabit.com/blog/application-adapters-enterprise-wide-message-m… where you can find information where to download ready to use rpm packages and how to get started. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

More on syslog-ng changing ownership and perms of source file
by Jim Segrave 30 Nov '17

30 Nov '17

I added the line: | file_dir_perm_option to the source_affile_option in ./modules/affile/affile-grammar.y:419 would allow setting ownership and permissions on source files (or opting for no-change) This would gave: source_affile_option : KW_FOLLOW_FREQ '(' LL_FLOAT ')' { affile_sd_set_follow_freq(last_driver, (long) ($3 * 1000)); } | KW_FOLLOW_FREQ '(' LL_NUMBER ')' { affile_sd_set_follow_freq(last_driver, ($3 * 1000)); } | KW_PAD_SIZE '(' LL_NUMBER ')' { ((AFFileSourceDriver *) last_driver)->pad_size = $3; } | file_dir_perm_option | multi_line_option | source_reader_option ; I did this with the syslog-ng.3.6.4 source and it now does the following: The owner group and permissions are changed only if the file source line specifies a value (not -1) for one of these items. In other words, it does not use any default value from the configuration, it simply leaves things as it found them. -- Jim Segrave jes(a)j-e-s.net

3 3

Syslog-ng relay: is it possible to modify or delete META SEQUENCE DATA added automaticaly?
by alain.villefranque＠orange.com 29 Nov '17

29 Nov '17

Hi, I experience a pb with the syslog-ng relay that is used in this way: Syslog-relay gather syslog network messages from different sources (legacy RFC 3164 msg ) and send these messages to a central syslog server. The central Syslog server accepts only message with a strict format:only value in green can be modified. <174>1 time_Norme_ISO8601 fqdn syslog-ng 007 - [meta sequenceid=xxxx][bbtlog@1009 bias="titi" platform="prod" produit="ppup" logfile="rtlog.sta" rtlog-river="rtl"] Message With success, I managed to write and modify incoming messages and store them locally in log files with the awaited server format. I used filters, rewrite_set to change the hostname and template to format the received message to the expected format. We don't use meta sequenceid, so we decided to write a non variant value 0000. template("<174>1 ${ISODATE} ${HOST} syslog-ng 007 - [meta sequenceid="0000"] [bblog@1009...] ${MSG}\n"); Now, the deal is to transfer the new line of the log file to the central syslog server. I use a "source" tail file to send the new messages and one new syslog remote destination to redirect the message to the syslog server Here is the message as depicted in TCPDUMP: inpavlog1i-ad.62163 > 10.114.181.43.syslog: SYSLOG, length: 379 Facility local5 (21), Severity info (6) Msg: 1 2017-11-29T10:17:57+01:00 inxipmgt.creteil.francetelecom.fr syslog-ng 007 - [meta sequenceId="1" sequencid="0000"][rtlog@1368 basicat="xip" platform="prod" produit="pexip" logfile="rtlog.sta" rtlog-river="rtl"] 10:17:57,686 Level="INFO" Name="administrator.system.configuration" Message="Conferencing node configuration updated." Node="conf02.int.ovp.orange-business.com"\0x0a So I have an issue with the field [meta sequenceid...] which automatically doubled/added to the previous...(0000) . If I modify the template by deleting the field [meta sequenceid="0000"], template("<174>1 ${ISODATE} ${HOST} syslog-ng 007 - [bblog@1009...] ${MSG}\n"); The result is so that the syslogng-relay add the [meta sequenceId="xxxx"] after the first meta sequence field and write the message with an upper case "I" . This is not accepted by the central server. I suspect also that the location in the frame could have an impact. Here is the tcpdump: inpavlog1i-ad.62163 > 10.114.181.43.syslog: SYSLOG, length: 379 Facility local5 (21), Severity info (6) Msg: 1 2017-11-29T10:17:57+01:00 inxipmgt.creteil.francetelecom.fr syslog-ng 007 - [rtlog@1368 basicat="xip" platform="prod" produit="pexip" logfile="rtlog.sta" rtlog-river="rtl"][meta sequenceId="1"] 10:17:57,686 Level="INFO" Name="administrator.system.configuration" Message="Conferencing node configuration updated." Node="conf02.int.ovp.orange-business.com"\0x0a As I don't use this meta field, is it possible to configure syslog-ng relay not add automatically this field [meta sequenceId="XX"] or to delete it with a command? Regards Alain _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.

1 0

syslog-ng running but not writing logs
by buckingham＠nym.hush.com 27 Nov '17

27 Nov '17

Hello forum members, Having spent a couple of weeks on this problem I thought someone might be able to help getting syslog-ng 3.9.1 to write log files on OpenBSD 6.2. The O/S was a fresh install, then syslog-ng and lastly BIND. No other packages except syslog-ng and BIND dependencies have been installed. No binaries or libs from BIND conflict or replace any from the syslog-ng installation. The .conf file was copied over from an older OpenBSD system, also running an older WORKING syslog-ng (3.1.1). The required changes to the .conf were made so that syslog-ng 3.9.1 with supervisor starts and remains running. By killing the syslog-ng process I was able to test that the supervisor process restarts syslog-ng. The rc.d script starts, stops and restarts syslog-ng as expected. /usr/local/sbin/syslog-ng -s returns no errors, indicating that the config file is sane. In /var/log the only thing it writes are syslog-ng start/stop messages. In an effort to localise the problem, a stub syslog-ng.conf was created with the following contents: @version: 3.9.1 source s_local { unix-dgram("/dev/log"); }; # destination catchall { file(/var/log/catchall); }; log { source(s_local); destination(catchall); }; The permissions and ownership on /var, /var/log, /var/run, and /etc are correct according to the OpenBSD /etc/mtree/special file. Syslog-ng starts retaining root permissions (default) and creates /dev/log if it does not exist. The syslog_ng_flags I use are: "-R /var/run/syslog_ng.persist -c /var/run/syslog_ng.ctl -p /var/run/syslog_ng.pid"Have tried running without any flags to see if that would help, but did not help. The output of syslog-ng -V is: syslog-ng 3.9.1 Installer-Version: 3.9.1 Revision: Module-Directory: /usr/local/lib/syslog-ng Module-Path: /usr/local/lib/syslog-ng Available-Modules: affile,afprog,afsocket,afsql,afuser,basicfuncs,cef,confgen,cryptofuncs,csvparser,curl,date,dbparser,disk-buffer,geoip-plugin,graphite,kvformat,linux-kmsg-format,pseudofile,system-source,add-contextual-data,json-plugin,syslogformat Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off On my hunt I foud a message on NARKIVE where a user was having the same problem with syslog-ng 3.6 on OpenBSD 5.9, there was no solution. There was a reply saying: "With the changes in 5.6 using sendsyslog(2), only syslogd picks up local syslog. Search the openbsd-ports list for syslog-ng to see some comments on it.". After a further search I turned up nothing that would help. Have also tried to compile syslog-ng 3.12 from source to see if that might help solve the problem, but no success on OpenBSD. Many thanks in advance. CB

2 2

Follow-up: syslog-ng changes owner and permissions on a file source (syslog-ng Digest, Vol 151, Issue 20)
by Jim Segrave 24 Nov '17

24 Nov '17

We tried putting a file source with owner and permissions in our syslog-ng configuration: file("/var/log/usenetmonitor/dev/cron-logs/Usenetmonitor-Warnings.log" flags(no-parse) program-override("custom-test") owner(-1) group(-1) perm(-1)); And at startup we get: Error parsing affile, syntax error, unexpected KW_OWNER, expecting KW_NORMALIZE_HOSTNAMES or KW_USE_DNS or KW_USE_FQDN or KW_DNS_CACHE in /etc/syslog-ng/conf.d/remote_syslog.conf at line 4, column 123: file("/var/log/usenetmonitor/dev/cron-logs/Usenetmonitor-Warnings.log" flags(no-parse) program-override("custom-test") owner(-1) group(-1) perm(-1)); ^^^^^ So the documentation is correct, the options are not valid on file sources. But it would be a good thing if they were -- Jim Segrave jes(a)j-e-s.net

1 0

Insider 2017-11: netdata; stats to graphite; logstash to syslog-ng;
by Czanik, Péter 23 Nov '17

23 Nov '17

Dear syslog-ng users, This is the 63rd issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Collecting netdata metrics -------------------------- netdata is a system for distributed real-time performance and health monitoring. You can use syslog-ng to collect and filter data provided by netdata and then send it to Elasticsearch for long-term storage and analysis. The aim is to send both metrics and logs to an Elasticsearch instance, and then access it via Kibana. https://www.balabit.com/blog/sending-netdata-metrics-syslog-ng-elasticsearc… syslog-ng statistics to Graphite -------------------------------- syslog-ng makes available various types of statistics. Data is available in a couple of forms: emitted regularly from the internal() source of syslog-ng or obtained using the syslog-ng-ctl utility from the command line. Due to the format that the internal source or the “stats” option of syslog-ng-ctl uses, it is not easy to send statistical data to Graphite or anywhere else. The syslog-ng-ctl utility provides a flexible “query” option, (available in recent versions of syslog-ng) which uses an easy-to-parse output format. Using the “jo” utility by @JPMens, you can convert the output to JSON, which can then be parsed by syslog-ng and forwarded to Graphite. https://www.balabit.com/blog/collecting-syslog-ng-statistics-to-graphite/ Sending logs from Logstash to syslog-ng ---------------------------------- Logstash adds a new syslog header to log messages before forwarding them to a syslog server. In the case of syslog messages, it is problematic as there will be two syslog headers in the message. Using syslog-ng for everything logging related in an Elasticsearch environment can considerably simplify your architecture. Still, there are situations, when Filebeats and Logstash are already deployed and you need some logs from Logstash in syslog-ng. Learn how you can remove the extra syslog header. https://www.balabit.com/blog/sending-logs-logstash-syslog-ng/ Your feedback and news, or tips about the next issue are welcome at documentation(a)balabit.com. To read this newsletter online, visit: https://syslog-ng.org/ Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

syslog-ng changes owner and permissions on a file source
by Jim Segrave 22 Nov '17

22 Nov '17

We are running syslog-ng on a Centos-7 server: root@usenetmonitor-fe01.am4:[~]# /usr/sbin/syslog-ng -V syslog-ng 3 (3.12.1) Installer-Version: 3.12.1 Revision: Compile-Date: Nov 3 2017 15:15:50 Module-Directory: //usr/lib64/syslog-ng Module-Path: //usr/lib64/syslog-ng Available-Modules: afuser,cef,linux-kmsg-format,sdjournal,confgen,map-value-pairs,json-plugin,affile,kvformat,tfgetent,date,afstomp,csvparser,add-contextual-data,cryptofuncs,system-source,syslogformat,afsocket,afprog,pseudofile,afamqp,snmptrapd-parser,dbparser,xml,stardate,tags-parser,graphite,basicfuncs,disk-buffer Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Systemd: on When we tried to use syslog-ng to treat a file as a source, it changes the owner and group to root and the permissions to 0664, which is undesireable, as the file must remain writeable by the apache server. This is the source definition in syslog.conf when we were testing: source s_dev { file("/var/log/usenetmonitor/dev/cron-logs/Usenetmonitor-Warnings.log" flags(no-parse) program-override("custom-test")); }; with syslog-ng stopped, here's the status of the file: root@usenetmonitor-fe01:[~]# ls -lt /var/log/usenetmonitor/soc/cron-logs/Usenetmonitor-Warnings.log -rw-r----- 1 apache apache 0 Nov 21 11:00 /var/log/usenetmonitor/soc/cron-logs/Usenetmonitor-Warnings.log As soon as syslog-ng is started it becomes root@usenetmonitor-fe01:[~]# ls -lt /var/log/usenetmonitor/soc/cron-logs/Usenetmonitor-Warnings.log -rw-r--r-- 1 root root 0 Nov 21 11:00 /var/log/usenetmonitor/soc/cron-logs/Usenetmonitor-Warnings.log Checking with strace, I find 16258 open("/var/log/usenetmonitor/dev/cron-logs/Usenetmonitor-Warnings.log", O_RDONLY|O_NOCTTY|O_NONBLOCK) = 24 Followed shortly therafter by: 16258 fchown(24, 0, -1) = 0 16258 fchown(24, -1, 0) = 0 16258 fchmod(24, 0664) = 0 (and no, fd 24 hasn't been closed and re-opened) It's unclear to me why syslog-ng would have any reason to change ownership or permissions of file sources. As there are no options for file sources to set what it would change it to, it seems highly undesireable that this happens. The only workaround is to set global options to disable changing ownership or permissions: perm(-1); owner(-1); group(-1); and then specify those options for every destination file which is a maintenance burden Is there some other way to stop syslog-ng from latering files used as sources? -- Jim Segrave jes(a)j-e-s.net

2 1

syslog-ng events in log timestamping wrong
by Contreras, Gerald (DPS) 22 Nov '17

22 Nov '17

Hi All, I am very new to syslog-ng via Cygwin. I am having trouble with MWG proxy logs where individual events inside the log are being timestamped at the rotation time (timestamping at 00:00-00:01AM). This happens intermittently so the problem isn't constant. I have verified via packet captures that the timestamp is correct up until syslog-ng begins to handle the event and write it to the log. My syslog-ng.conf is as follows =============================================================== @version: 3.2 @include "scl.conf" options { create_dirs(yes); use_dns(yes); dns_cache_hosts(/etc/hosts); dns_cache_expire(3600); chain_hostnames(0); log_fifo_size(512); perm(0644); dir_perm(0755); time_reopen(10); time_reap(360); use_fqdn(no); keep_hostname(no); stats_freq(0); }; source s_network { udp(port("514")); }; destination s_files { file("/cygdrive/d/syslog/$HOST/$HOST-$MONTH-$DAY.log" create_dirs(yes)); }; log { source(s_network); destination(s_files); }; ============================================================== Thanks. Any help or direction would be much appreciated Important Notice: If you have received this email by mistake, please advise the sender and delete the message and attachments immediately. This email, including attachments, may contain confidential, sensitive, legally privileged and/or copyright information. Any review, retransmission, dissemination or other use of this information by persons or entities other than the intended recipient is prohibited. Please consider the environment before printing this email.

3 3

Hourly Log rotate
by craig bowser 20 Nov '17

20 Nov '17

So according to https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng- ose-guide-admin/html/example-logrotate.html using this format: destination d_sorted { file("/var/log/remote/${HOST}/${YEAR}_${MONTH}_${DAY}.log" create-dirs(yes)); }; I can create logs folders and files based on the timestamp. And this is working nicely for us... With a caveat. We are getting a TON of logs, so we want to rotate hourly and archive quickly. My config is this: destination d_msg { file("/var/log/message_${YEAR} _${MONTH}_${DAY}_${HOUR}.log"); }; But instead of creating one file per hour according to the time the event is received, it is creating files based on the timestamp of the event (which, while useful for discovering and tracking down machines with time synch problems, is not so useful for managing log files on the syslog server). Is there a way to make it create files based on the time the event is received and NOT the timestamp of the event? Thanks.

2 2

Cisco ACS logs truncated
by Scot 15 Nov '17

15 Nov '17

Hi, Has anyone worked with ACS logs and solved the message header limit ? We can get syslog working but as expected the message gets truncated. Local logs on the ACS have the entire payload. Thinking there may be a way to script a log fetch or something. Thanks Scot

4 8

(no subject)
by craig bowser 15 Nov '17

15 Nov '17

I installed syslog-ng on an offline CentOS 6.x machine by first downloading all required rpms using this command: yumdownloader --resolve syslog-ng There were two files needed, that did not download and I downloaded them separately using: yumdownloader eventlog-0.2.13-1.el6.x86_64.rpm and: yumdownloader libnet-1.1.6-7.el6.x86_64.rpm These are the files I moved to the offline server: 17,684 eventlog-0.2.13-1.el6.i686.rpm 17,804 eventlog-0.2.13-1.el6.x86_64.rpm 122,940 gamin-0.1.10-9.el6.i686.rpm 1,738,964 glib2-2.28.8-9.el6.i686.rpm 4,570,040 glibc-2.12-1.209.el6_9.2.i686.rpm 117,400 libgcc-4.4.7-18.el6.i686.rpm 60,464 libnet-1.1.6-7.el6.i686.rpm 59,196 libnet-1.1.6-7.el6.x86_64.rpm 111,688 libselinux-2.0.94-7.el6.i686.rpm 309,920 libstdc++-4.4.7-18.el6.i686.rpm 161,168 nss-softokn-freebl-3.14.3-23.3.el6_8.i686.rpm 201,096 pcre-7.8-7.el6.i686.rpm 452,548 syslog-ng-3.2.5-4.el6.i686.rpm 451,092 syslog-ng-3.2.5-4.el6.x86_64.rpm 63,536 tcp_wrappers-libs-7.6-58.el6.i686.rpm 74,284 zlib-1.2.3-29.el6.i686.rpm Once I moved all these files to the offline server into their own folder, I ran: rpm –ivh *.rpm to install. I then checked the config file, made a couple of tweaks, then started syslog-ng using: service syslog-ng start I got the following errors: Plugin module not found in 'module-path'; module-path='/lib64/syslog-ng', module='afsql' Plugin module not found in 'module-path'; module-path='/lib64/syslog-ng', module='afsql' [ OK ] Syslog-ng was now running, just with these two errors. A quick google pointed to the problem, syslog-ng is looking for the sql plugin and the solution: yum install syslog-ng-libdbi But when I tried to run yumdownloader syslog-ng-libdbi I got "Nothing to download" And when I tried yum install --downloadonly syslog-ng-libdbi I got "Nothing to do" So here are my questions: 1. Will syslog-ng run just fine receiving udp and sending udp with those errors and without that module? If, so, I don’t care if I don’t get that module. 2. Is it possible to run syslog-ng and disable that module in the run command? 3. Can I remove syslog-ng and re-install disable that module on install? 4. I’d rather not recompile if I don’t have to, but I found instructions for how to do that. Any tips would be appreciated. Thanks. Craig L Bowser ____________________________ This email is measured by size. Bits and bytes may have settled during transport.

2 2

Install Issue
by Elphick, Matthew Thomas 08 Nov '17

08 Nov '17

Hello, I am trying to install syslog-ng OSE on RHEL 7 but I get the following error. Transaction check error: file /etc/logrotate.d/syslog from install of syslog-ng-3.12.1-2.el7.centos.x86_64 conflicts with file from package rsyslog-8.24.0-12.el7.x86_64 I can't remove rsyslog from the system as other people us it. What is the best way to resolve this issue? Thanks for your help. Matthew Elphick, Network Engineering Marsh & McLennan Companies Global Technology Infrastructure (MGTI) | Enterprise Networking Services Tower Place, London, EC3R 5BU O +44 (0)20 73573667 | M +44 (0)7392 122399 | matthew.elphick(a)mmc.com<mailto:matthew.elphick@mmc.com> ________________________________ Marsh Ltd. Registered in England and Wales Number: 1507274 Registered office 1 Tower Place West, Tower Place, London, EC3R 5BU. Marsh Ltd is authorised and regulated by the Financial Conduct Authority. This message and any attachments are confidential. If you have received this message in error please delete it from your system. If you require any assistance please notify the sender. Thank you.

4 6

logstash to syslog-ng
by Czanik, Péter 08 Nov '17

08 Nov '17

Hi, In the past few weeks I have seen questions about sending logs from Logstash to syslog-ng, so I tried it myself and put together a blog with simple configurations to help you: https://www.balabit.com/blog/sending-logs-logstash-syslog-ng/ Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

3 3

any limitation with in-list filter
by Gopi Joshi 07 Nov '17

07 Nov '17

I am using in-list filter to match logs for specific users , these specific users are saved in a plain text file . is there any restriction or limitation on number of lines it can match in one file ? I currently have less than 10 users to compare with but this list is going to increase . Regards Gopi Joshi

3 2

Beat to syslog-ng
by Czanik, Péter 07 Nov '17

07 Nov '17

Hi, Somehow related to my previous e-mail: I was asked if Filebeat can send logs directly into syslog-ng. None of the network destinations of Beat are supported by syslog-ng, but I did a quick check and we can read and parse the file destination of Beat. Filebeat module functionality can be practically replaced by syslog-ng and one of its parsers. But there are quite a few other Beat applications. Question: is there a practical use case for syslog-ng reading the file output of a Beat module? Do you use it anywhere? Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

2 1

Question about using multiple amqp destination
by Alicia Smith 06 Nov '17

06 Nov '17

I am trying to configure syslog-ng to send multiple json formatted logs to a rabbitmq endpoint. The consumer for the logs checks for a particular field to determine the type of log it is. However, I can't seem to figure out how to send all the different logs as the field it checks is "source" that I add using --pair within the format-json template destination d_amqp_ssh { amqp( vhost("/") host("mozdefqa2.private.scl3.mozilla.com") port(5672) exchange("eventtask") exchange-type("direct") routing-key("eventtask") body("$(format-json --scope nv_pairs --pair category=\"bro\" --pair source=\"ssh\" --pair customendpoint=\" \" --pair tags=\"bro\")") persistent(no) username("guest") password("guest") ); }; So I added another destination with the name d_amqp_conn and it's source field value is "conn" and the amqp endpoint is the same host as the ssh log. But syslog-ng fails to start with this particular setup. It seems to not like having more than one amqp destination. How would something like this typically be solved? -- Alicia Smith @phrozyn Information Security Engineer asmith(a)mozilla.com

3 10

Syslog-NG Statistics
by Schoonover, Mark E HHHH 01 Nov '17

01 Nov '17

Hello, I'm working on a Splunk dashboard for our NG PE servers. Looking at the stats from the output of /opt/syslog-ng/sbin/syslog-ng-ctl stats, this one has me puzzled: src.program,,49587758,d,processed,1 I don't know what "program" means as well as the 3rd column. I understand the remaining columns. What do they mean? Thanks! Regards, Mark Schoonover - KA6WKE Infrastructure Engineering Manager, Splunk Architect ENE : Tools, Instrumentation and Common Services Team Office: 32.8697° N, 116.9711° W Phone : 770-261-7934 Email : mark.schoonover(a)cigna.com<mailto:mark.schoonover@cigna.com> HPSM Team: ENE NMS Engineering Confidential, unpublished property of CIGNA. Do not duplicate or distribute. Use and distribution limited solely to authorized personnel. ¬© Copyright 2017 CIGNA ------------------------------------------------------------------------------ CONFIDENTIALITY NOTICE: If you have received this email in error, please immediately notify the sender by e-mail at the address shown. This email transmission may contain confidential information. This information is intended only for the use of the individual(s) or entity to whom it is intended even if addressed incorrectly. Please delete it from your files if you are not the intended recipient. Thank you for your compliance. Copyright (c) 2017 Cigna ==============================================================================

2 1

Re: [syslog-ng] Need to parse named query log entries and get just the domain portion
by Robert Webb 31 Oct '17

31 Oct '17

Evan, Thanks you for pointing me in the right direction. I was so focused on the matching starting @ "queries: info:" that I totally forgot that the first part of the actual MSG has the date and time sent with it. My pattern now takes that into account. Message: 27-Oct-2017 14:19:42.359 queries: info: client 137.75.76.138#38643 (sr.symcb.com) query: sr.symcb.com IN A +E (137.75.75.75) <pattern>@ESTRING:: @@ESTRING:: @queries: info: client @IPvANY:named.client.ip@#@NUMBER:named.client.port:@ @ESTRING:named.domain.name::@ query: @HOSTNAME:named.host.name:@ @STRING:@ @STRING:named.record.type:@ @ESTRING:named.record.flags: @(@IPvANY:named.server.ip@)</pattern> Robert From: Evan Rempel [mailto:erempel@uvic.ca] Sent: Friday, October 27, 2017 9:52 AM To: Robert Webb <rwebb(a)ropeguru.com> Subject: Re: [syslog-ng] Need to parse named query log entries and get just the domain portion In the past when I have had this problem it was in the patterndb portion just outside if what you have included. essentially the program does not match, so the rule is never attempted. This results in none of the tag-value pairs being populated. The pdbtool shows a match because the pattern is always in the correct program. Your patterndb portion is inside of a set of a config such as <patterndb version="4" pub_date="2009-09-01"> <ruleset name="named" id="some-unique-value"> <pattern>named</pattern> <rules> [ your included pattern db data ] </ruleset> </pattendb> It is the line <pattern>named</pattern> that matches the parsed program name from the syslog line to allow the rule to be parsed against the syslog message. For testing purposes I make a destination to write a json object. That way I can see everything that populated. template t_json { template("$(format-json --scope everything )\n"); template_escape(no); }; destination test.log { file("/var/syslog/test.log.$S_YEAR$S_MONTH$S_DAY.000000" template(t_json)); }; Evan. On 10/27/2017 05:22 AM, Robert Webb wrote: So I have a situation where I have a syslog feed coming in that is sending named (BIND9) query logs. On the DNS servers themselves, we are doing sink hole via a local zone file and sending the dns response back as 127.0.0.1. Since there is no way for BIND9 to give me logging of the DNS query response back to the user, I need to be able to take a query log entry, parse it to get just the domain portion, and then match the domain to a list. I am trying to do all this inside syslog-ng. I have been given some guidance to use a pattern database. I worked with this and am successfully matching my incoming query log data. Using the pdbtool, I get a full match and a list of the values I expect. But where I run into an issue after searching through all the documentation is, where can these values be used? I tried building a template against a destination, and none of the values populate. So I am stuck at this point. I have also been told that I may have to do a custom python parser instead. Any help or guidance is appreciated. Here is my rule: <rule provider="balabit" id="b0ca071d-309e-454a-8e31-721b01ef55ee" class="system"> <patterns> <pattern>queries: info: client @IPvANY:named.client.ip@#@NUMBER:named.client.port:@ @ESTRING:named.domain.name::@ query: @HOSTNAME:named.host.name:@ @STRING:@ @STRING:named.record.type:@ @ESTRING:named.record.flags: @(@IPvANY:named.server.ip@)</pattern> </patterns> <examples> <example> <test_message program="named">client 213.46.246.51#1890: query: tumtali.fegade.com IN A - (103.5.198.211)</test_message> <test_values> <test_value name="named.client.ip">213.46.246.51</test_value> <test_value name="named.client.port">1890</test_value> <test_value name="named.domain.name">tumtali.fegade.com</test_value> <test_value name="named.record.type">A</test_value> <test_value name="named.record.flags">-</test_value> <test_value name="named.server.ip">103.5.198.211</test_value> </test_values> </example> <example> <test_message program="named">client 10.187.195.186#40441: query: banister.paars.tld IN A -EDC (10.226.134.25)</test_message> <test_values> <test_value name="named.client.ip">10.187.195.186</test_value> <test_value name="named.client.port">40441</test_value> <test_value name="named.domain.name">banister.paars.tld</test_value> <test_value name="named.record.type">A</test_value> <test_value name="named.record.flags">-EDC</test_value> <test_value name="named.server.ip">10.226.134.25</test_value> </test_values> </example> <example> <test_message program="named">queries: client 192.168.102.189#58454: query: maps.googleapis.com IN A + (192.168.102.2)</test_message> <test_values> <test_value name="named.client.ip">192.168.102.189</test_value> <test_value name="named.client.port">58454</test_value> <test_value name="named.domain.name">maps.googleapis.com</test_value> <test_value name="named.record.type">A</test_value> <test_value name="named.record.flags">+</test_value> <test_value name="named.server.ip">192.168.102.2</test_value> </test_values> </example> </examples> <values> </values> <tags> <tag>named</tag> <tag>queries</tag> </tags> </rule> Here are test results: Pattern matching part: queries: info: client @IP:named.client.ip=137.75.76.138@#@NUMBER:named.client.port=38643@ @ESTRING:named.domain.name=(sr.symcb.com)@ query: @HOSTNAME:named.host.name=sr.symcb.com@ @STRING:None=IN@ @STRING:named.record.type=A@ @ESTRING:named.record.flags=+E@(@IP:named.server.ip=137.75.75.75@) Matching part: queries: info: client 137.75.76.138#38643 (sr.symcb.com) query: sr.symcb.com IN A +E (137.75.75.75) Values: MESSAGE=queries: info: client 137.75.76.138#38643 (sr.symcb.com) query: sr.symcb.com IN A +E (137.75.75.75) PROGRAM=named .classifier.class=system .classifier.rule_id=b0ca071d-309e-454a-8e31-721b01ef55ee named.client.ip=137.75.76.138 named.client.port=38643 named.domain.name=(sr.symcb.com) named.record.type=A named.server.ip=137.75.75.75 named.host.name=sr.symcb.com named.record.flags=+E TAGS=.classifier.system,named,queries

2 2

Syslog 3.5 (pipe file issue)
by Amin, Jitesh CTR (US) 29 Oct '17

29 Oct '17

Hi, We have the syslog-ng 3.5 installed on Linux OS. We are trying to use the pipe function (to pipe data to the log file). This pipe file is FIFO. One of the issues we are seeing is that, the file size of this pipe file is always ZERO and never grows and second the timestampe on the file never updates or changes. Though when we do a "cat pipefilename" command we do see contents of the file. Can you please let us know what we could be doing wrong or any feedback on how to resolve the issue? Thanks Jitesh

4 5