- syslog-ng - lists.balabit.hu

Installation issue
by hari ram 07 Dec '17

07 Dec '17

Hi I have ubuntu Server install 16.X with all dependency packages #124-Ubuntu SMP Fri Nov 10 18:29:59 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux # more /etc/debian_version stretch/sid when i try to install syslog-ng with debian package as mentioned below steps. ~# more /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/xUb… ./ # apt-get update Hit:1 http://gb.archive.ubuntu.com/ubuntu xenial InRelease Hit:2 http://gb.archive.ubuntu.com/ubuntu xenial-updates InRelease Get:3 http://gb.archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB] Ign:4 http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/xUb… ./ InRelease Hit:5 http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/xUb… ./ Release Get:6 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB] Fetched 204 kB in 0s (350 kB/s) Reading package lists... Done # apt-get install syslog-ng-core Reading package lists... Done Building dependency tree Reading state information... Done Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: syslog-ng-core : Depends: libjson-c3 (>= 0.10) but it is not installable E: Unable to correct problems, you have held broken packages. But i have already install libjason package # apt-get install libjson-c-dev Reading package lists... Done Building dependency tree Reading state information... Done libjson-c-dev is already the newest version (0.11-4ubuntu2). 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. what is the issue, how can i install latest version of syslog-ng default syslog-ng only come with 3.5.6 version, but iam looking 3.11 and above version to install any suggestions ? R!

2 2

Syslog-ng relay: how to delete or modify special character ^M?
by alain.villefranque＠orange.com 06 Dec '17

06 Dec '17

Hi all, I have an issue with the character Carriage Return (^M) inserted by the Syslog source machine. I try to substitute or rewrite this special character with syslog-ng relay, but it is impossible to modify it, it seems this char is invisible for the program. I've tried to modified it with either with ^M or 0xD char with no success. Example of received message: Dec 4 13:35:23 conf01 2017-12-04 13:35:23,561 Level="INFO" Name="support.sip" Message="Sending SIP response" Src-address="80.12.yy.xx" Src-port="5061" Dst-address="161.105.yy.xx" Dst-port="38509" Transport="TLS" Detail="^MSIP/2.0 403 Forbidden^MVia: SIP/2.0/TLS 161.105.150.12:38509;alias;branch=z9hG4bK.MEgSOM8O4;rport=38509;received=161.105.150.12^MFrom: <sip:SondeSQS_001!@int.ovp.orange-business.com>;tag=5kYhVMAyi^MTo: sip:SondeSQS_001!@int.ovp.orange-business.com;tag=aynkBKUjt0pXHzNv^MCSeq: 25 REGISTER^MCall-ID: Vhihsb~BhQ^MAllow: INVITE,ACK,OPTIONS,CANCEL,BYE,REGISTER,INFO,SUBSCRIBE,NOTIFY,MESSAGE^MSupported: categoryList,adhoclist,sdp-anat,replaces^MContent-Length: 0^M^M" I'd like to suppress ^M or replace it with a "space" char. Is there any specific action to do in order to modify ^M special character ? Regards Alain _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.

2 4

OSE 3.13.2
by Budai, László 06 Dec '17

06 Dec '17

Hi, the new release contains the following fixes : - Missing manpages from release tarball (#1793 <https://github.com/balabit/syslog-ng/issues/1793>) - Package syslog-ng-mod-json is removed from (#1794 <https://github.com/balabit/syslog-ng/pull/1794>) - Drop syslog-ng-abi virtual packages (#1797 <https://github.com/balabit/syslog-ng/pull/1797>) regards, Laszlo Budai

1 0

syslog-ng 3.13.1
by Budai, László 05 Dec '17

05 Dec '17

3.13.1Features - Add app-parser() framework (automatic parsing of log messages) (#1689 <https://github.com/balabit/syslog-ng/pull/1689>) - Support microseconds in Riemann destination (#1710 <https://github.com/balabit/syslog-ng/pull/1710>) - Add osquery destination as an SCL plugin (#1728 <https://github.com/balabit/syslog-ng/pull/1728>) - Add network load balancer destination (#1706 <https://github.com/balabit/syslog-ng/pull/1706>) - Add possibility to only signal re-open of file handles (SIGUSR1) (#1530 <https://github.com/balabit/syslog-ng/pull/1530>) - It is possible from now to limit the number of registered dynamic counters (#1743 <https://github.com/balabit/syslog-ng/pull/1743>) - Add $(binary) template function (#1679 <https://github.com/balabit/syslog-ng/pull/1679>) - Add experimental transport for transferring messages in whole between syslog-ng instances (EWMM) (#1689 <https://github.com/balabit/syslog-ng/pull/1689>) - Docker based build and debian package generation (#1783 <https://github.com/balabit/syslog-ng/pull/1783>) - Add auto-parse(yes/no) to app-paser(), system() and default-network-drivers() (#1788 <https://github.com/balabit/syslog-ng/pull/1788>) - Add Graylog2 destination and $(format-gelf) template function (#1680 <https://github.com/balabit/syslog-ng/pull/1680>) Bugfixes - Exit when a read fails on an included config file instead of starting up with an empty configuration. (#1721 <https://github.com/balabit/syslog-ng/pull/1721>) - Fix double free (#1720 <https://github.com/balabit/syslog-ng/issues/1720>) - Add missing discarded counter to groupingby (#1748 <https://github.com/balabit/syslog-ng/pull/1748>) - Fix a reference leak in Python destination (#1716 <https://github.com/balabit/syslog-ng/pull/1716>) - Fix timezone issue in snmptrapd parser (#1746 <https://github.com/balabit/syslog-ng/pull/1746>) - Fix potential crash in stdin driver (#1741 <https://github.com/balabit/syslog-ng/pull/1741>) - Fix a crash when initializing new config fails for socket with keep_alive off (#1723 <https://github.com/balabit/syslog-ng/pull/1723>) - Fix filter evaluation in case of contexts with multiple elements (#1718 <https://github.com/balabit/syslog-ng/pull/1718>) - Various grouping-by fixes (#1718 <https://github.com/balabit/syslog-ng/pull/1718>) - Fix potential use after free around dns-cache during shutdown (#1666 <https://github.com/balabit/syslog-ng/pull/1666>) - Fix access to indirect values within Java destination (#1732 <https://github.com/balabit/syslog-ng/pull/1732>) - Fix a crash in affile (#1725 <https://github.com/balabit/syslog-ng/pull/1725>) - Fix a memory leak (#1724 <https://github.com/balabit/syslog-ng/pull/1724>) - Fix a crash when getent is used empty group (#1691 <https://github.com/balabit/syslog-ng/pull/1691>) - Fix jvm-options() (#1704 <https://github.com/balabit/syslog-ng/pull/1704>) - Fix a crash in Python language binding (#1694 <https://github.com/balabit/syslog-ng/pull/1694>) - Fix a crash in afmongodb (#1765 <https://github.com/balabit/syslog-ng/pull/1765>) - Fix a memory leak in afmongodb (#1766 <https://github.com/balabit/syslog-ng/pull/1766>) - Fix name-to-GID calculation in the $(getent) template function (#1764 <https://github.com/balabit/syslog-ng/pull/1764>) - Fix a crash when redis is configured without the command() option ( #1767 <https://github.com/balabit/syslog-ng/pull/1767>) - Fix a race condition in kv-parser() (#1789 <https://github.com/balabit/syslog-ng/pull/1789>) Other changes - Cleanup diskq related warning messages (#1752 <https://github.com/balabit/syslog-ng/pull/1752>) - Provide tls block for tls options in amqp(), http(), riemann() destination drivers (#1715 <https://github.com/balabit/syslog-ng/pull/1715>) - It it possible from now to register blocks and generators as plugins ( #1657 <https://github.com/balabit/syslog-ng/pull/1657>) - Drop compatiblity with configurations below 3.0 (#1709 <https://github.com/balabit/syslog-ng/pull/1709>) - Do not change permissions of a file by default (#1782 <https://github.com/balabit/syslog-ng/pull/1782>) - Allow source files to specify permissions locally (#1782 <https://github.com/balabit/syslog-ng/pull/1782>) - Minor performance improvement (#1729 <https://github.com/balabit/syslog-ng/pull/1729>) - The current config version can be queried with "--version" (#1740 <https://github.com/balabit/syslog-ng/pull/1740>) - Increase the performance of kv-parser() (#1789 <https://github.com/balabit/syslog-ng/pull/1789>) Notes to the developers - Change configure default option for jsonc and mongoc from auto to internal (#1735 <https://github.com/balabit/syslog-ng/pull/1735>) - Disable ASLR when running unit tests (#1753 <https://github.com/balabit/syslog-ng/pull/1753>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szalay, Balazs Scheidler, Gabor Nagy, Jakub Jankowski, Janos Szigetvari, Laszlo Budai, Laszlo Varady, Laszlo Szemere, Marton Illes, Mate Farkas, Peter Kokai, Pontus Andersson, Sam Stephenson, Sebastian Roland, Viktor Juhasz, Zoltan Pallagi.

2 4

syslog-ng-3.13: unofficial Debian packages: still building, be careful when updating
by Budai, László 04 Dec '17

04 Dec '17

Hi, packages will be available also for Ubuntu 17.10. There is a major change in packaging: syslog-ng-mod-json is part of syslog-ng-core from now (due to the new app-parser and Enterprise Wide Message Model). regards, Laszlo Budai

1 0

Timezone
by Sebastian Roland 01 Dec '17

01 Dec '17

Hi, after reading the admin guide and playing around with different setups several times I'm still struggling to fully understand the timezone functionality of syslog-ng. How are time-zone() / recv-time-zone() and send-time-zone() related? Some notes I made during investigation: * Logging through syslog() function logs in old BSD syslog format which does not contain a timezone. recv-time-zone() is utilized to assign a timezone. If no value has been specified the local time zone is used. * According to the admin guide send-time-zone() is only used when the timezone is not specified otherwise. This didn't turned out to be true. Example: Syslog server a sends via syslog protocol over tcp (timezone is part of the message) to server b. setting send-time-zone(x) on server b changes the timezone (and timestamp) in the destination file to the time in timezone x. If send-time-zone() is not set at all nothing happens although the admin guide states that the default is to use the local timezone. IMHO no change should be applied to the message. Note that keep- timestamp(yes) is set on server b. * If both time-zone() and send-time-zone() are set globally time-zone() overrides send-time-zone() * time-zone() can be set globally and on drivers. Specific settings overrides global config. The confusing part is the behavior when a timestamp is already set within an incoming message and send-time-zone() is explicitly set (with keep-timestamp(yes)). Is it actually intended that send-time-zone() changes a timestamp? Shouldn't the logic be that recv-time-zone() and send-time-zone() are only relevant when there is no tz offset available and a default one needs to be set for receiving and sending respectively and time-zone() is used to actually convert to a different timezone? If I'm getting something fundamentally wrong please advice. Kind regards Sebastian

2 1

testing fix for file source ownership and permissionso
by Jim Segrave 01 Dec '17

01 Dec '17

> hi, > I've just posted a pull request that I didn't have time to test: can you pls test if this fixes the issue for you? > https://github.com/balabit/syslog-ng/pull/1782 <https://github.com/balabit/syslog-ng/pull/1782> While my original patch worked on 3.6.5, we actually need some of the features in 3.12. I have problems with autogen/automake being unhappy with the versions of the autotool chain I have: running autogen.sh: Running autogen in 'lib/ivykis'... libtoolize: You should add the contents of the following files to 'aclocal.m4': libtoolize: '/usr/local/share/aclocal/libtool.m4' libtoolize: '/usr/local/share/aclocal/ltoptions.m4' libtoolize: '/usr/local/share/aclocal/ltversion.m4' libtoolize: Consider adding 'AC_CONFIG_MACRO_DIRS([m4])' to configure.ac, libtoolize: and rerunning libtoolize and aclocal. libtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am. (catting the suggested files onto aclocal.m4 does not remove these warnings) However configure does finish, but make bombs out with: /bin/sh ../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I.. -D_GNU_SOURCE -I../src/include -I../src/include -I /usr/local/src/syslog-ng -Wall -pthread -Wno-error -MT iv_avl.lo -MD -MP -MF .deps/iv_avl.Tpo -c -o iv_avl.lo iv_avl.c libtool: Version mismatch error. This is libtool 2.4.5, but the libtool: definition of this LT_INIT comes from libtool 2.4.2. libtool: You should recreate aclocal.m4 with macros from libtool 2.4.5 libtool: and run autoconf again. make[5]: *** [iv_avl.lo] Error 63 make[4]: *** [all-recursive] Error 1 make[3]: *** [all] Error 2 make[2]: *** [lib/ivykis/src/libivykis.la] Error 2 make[1]: *** [all-recursive] Error 1 make: *** [all] Error 2 I was able to get a working version from the 3.6.5 sources. When I get the 3.12.1 sources from the Centos rpm, I can run configure without running autogen, which does complete and make completes without errors. However, the resulting executable fails to start with this error message: Error parsing config, Error compiling template (Unknown template function "format-json32: included included template-function "format-cim" "$(format-json --pair @timestamp='${R_ISODATE}' --pair .=_ --key *.*)\n"; ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ I know next to nothing about the automake and friends (or m4 for that matter), so I don't really know how to address this. -- -- Jim Segrave jes(a)j-e-s.net

2 1

Testing application adapters and enterprise-wide message model for syslog-ng
by Czanik, Péter 30 Nov '17

30 Nov '17

Hi, Do you want to simplify parsing your log messages? Try the new “application adapter” and “enterprise-wide message model” frameworks in syslog-ng: you can automatically parse log messages and forward the results to another syslog-ng instance. Optionally, you can also include the original, raw message that you can forward unmodified to a SIEM system for further analysis. While these features are work in progress and might change considerably in future releases, we appreciate early testing and feedback. With the help of your comments we could cover not just our internal use cases, but also take into account the needs of the wider community. Read more about it at https://www.balabit.com/blog/application-adapters-enterprise-wide-message-m… where you can find information where to download ready to use rpm packages and how to get started. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

More on syslog-ng changing ownership and perms of source file
by Jim Segrave 30 Nov '17

30 Nov '17

I added the line: | file_dir_perm_option to the source_affile_option in ./modules/affile/affile-grammar.y:419 would allow setting ownership and permissions on source files (or opting for no-change) This would gave: source_affile_option : KW_FOLLOW_FREQ '(' LL_FLOAT ')' { affile_sd_set_follow_freq(last_driver, (long) ($3 * 1000)); } | KW_FOLLOW_FREQ '(' LL_NUMBER ')' { affile_sd_set_follow_freq(last_driver, ($3 * 1000)); } | KW_PAD_SIZE '(' LL_NUMBER ')' { ((AFFileSourceDriver *) last_driver)->pad_size = $3; } | file_dir_perm_option | multi_line_option | source_reader_option ; I did this with the syslog-ng.3.6.4 source and it now does the following: The owner group and permissions are changed only if the file source line specifies a value (not -1) for one of these items. In other words, it does not use any default value from the configuration, it simply leaves things as it found them. -- Jim Segrave jes(a)j-e-s.net

3 3

Syslog-ng relay: is it possible to modify or delete META SEQUENCE DATA added automaticaly?
by alain.villefranque＠orange.com 29 Nov '17

29 Nov '17

Hi, I experience a pb with the syslog-ng relay that is used in this way: Syslog-relay gather syslog network messages from different sources (legacy RFC 3164 msg ) and send these messages to a central syslog server. The central Syslog server accepts only message with a strict format:only value in green can be modified. <174>1 time_Norme_ISO8601 fqdn syslog-ng 007 - [meta sequenceid=xxxx][bbtlog@1009 bias="titi" platform="prod" produit="ppup" logfile="rtlog.sta" rtlog-river="rtl"] Message With success, I managed to write and modify incoming messages and store them locally in log files with the awaited server format. I used filters, rewrite_set to change the hostname and template to format the received message to the expected format. We don't use meta sequenceid, so we decided to write a non variant value 0000. template("<174>1 ${ISODATE} ${HOST} syslog-ng 007 - [meta sequenceid="0000"] [bblog@1009...] ${MSG}\n"); Now, the deal is to transfer the new line of the log file to the central syslog server. I use a "source" tail file to send the new messages and one new syslog remote destination to redirect the message to the syslog server Here is the message as depicted in TCPDUMP: inpavlog1i-ad.62163 > 10.114.181.43.syslog: SYSLOG, length: 379 Facility local5 (21), Severity info (6) Msg: 1 2017-11-29T10:17:57+01:00 inxipmgt.creteil.francetelecom.fr syslog-ng 007 - [meta sequenceId="1" sequencid="0000"][rtlog@1368 basicat="xip" platform="prod" produit="pexip" logfile="rtlog.sta" rtlog-river="rtl"] 10:17:57,686 Level="INFO" Name="administrator.system.configuration" Message="Conferencing node configuration updated." Node="conf02.int.ovp.orange-business.com"\0x0a So I have an issue with the field [meta sequenceid...] which automatically doubled/added to the previous...(0000) . If I modify the template by deleting the field [meta sequenceid="0000"], template("<174>1 ${ISODATE} ${HOST} syslog-ng 007 - [bblog@1009...] ${MSG}\n"); The result is so that the syslogng-relay add the [meta sequenceId="xxxx"] after the first meta sequence field and write the message with an upper case "I" . This is not accepted by the central server. I suspect also that the location in the frame could have an impact. Here is the tcpdump: inpavlog1i-ad.62163 > 10.114.181.43.syslog: SYSLOG, length: 379 Facility local5 (21), Severity info (6) Msg: 1 2017-11-29T10:17:57+01:00 inxipmgt.creteil.francetelecom.fr syslog-ng 007 - [rtlog@1368 basicat="xip" platform="prod" produit="pexip" logfile="rtlog.sta" rtlog-river="rtl"][meta sequenceId="1"] 10:17:57,686 Level="INFO" Name="administrator.system.configuration" Message="Conferencing node configuration updated." Node="conf02.int.ovp.orange-business.com"\0x0a As I don't use this meta field, is it possible to configure syslog-ng relay not add automatically this field [meta sequenceId="XX"] or to delete it with a command? Regards Alain _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.

1 0