- syslog-ng - lists.balabit.hu

create_dir and "dynamic" destinations.
by Scot Needy 21 Aug '14

21 Aug '14

Hi, I was wondering if anyone has tried to work around syslog_ng. My destinations are psudo dynamic in that I have one per subnet but I don’t want to pre-create every destination directory until a syslog message is received for that destination. I have 3 conf files (Dfilters.conf Ddestination.conf and Dlog.conf ) that are built using an API to our IPNMS (IP Network Management System). The purpose is to use syslog-ng stats to quickly report on subnets and also provide per-subnet options for reporting, forwarding … Example Destination destination d_192_168_9_0 { file(/opt/syslog-ng/logs/192_168_9_0/$YEAR$MONTH$DAY.$HOST.log);}; Example Filter filter f_192_168_9_0 { netmask(192.168.9.0/26);}; Example Log log { source(s_net); filter(f_192_168_9_0); destination(d_192_168_9_0);}; My question is can anyone think of a way to get the create_dir option to dynamically create my subnet directories as needed ? For now I Need to pre-create every directory or syslog-ng complains which causes me to create a bunch of empty directories. Thanks Scot

5 5

syslog-ng 3.5 for RHEL6
by Czanik, Péter 19 Aug '14

19 Aug '14

Hi, Recently many people asked for syslog-ng 3.5 rpms for RHEL6. What I did breaks most of the packaging rules, but works. It's a wild mixture of the current EPEL6 and EPEL7 packages. So it has most of the features of the EPEL7 package (except UDP spoofing), but has sysvinit style init script. It's installed to /usr/sbin instead of /sbin, has SSL support enabled, and uses bundled libraries instead of system libraries (as those are missing from RHEL6 and EPEL6). To use the packages, you need to enable EPEL6 repositories for dependencies ( https://fedoraproject.org/wiki/EPEL ) and also my Copr repository: http://copr.fedoraproject.org/coprs/czanik/syslog-ng35epel6/ It's ugly as hell, but survived my quick test on CentOS 6.5. Use at your own risk but let me know if you run into any trouble! Bye, -- Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

syslog-ng 3.6.0alpha3 has been released
by devel＠balabit.hu 19 Aug '14

19 Aug '14

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.6.0alpha3 SUMMARY : new alpha release DATE : Aug 19, 2014 ------------------------------------------------------------------------------ DESCRIPTION: The third alpha version of syslog-ng Open Source Edition (3.6.0alpha3) has been released. This being an alpha release, testing is most welcome, but production use is not recommended! CHANGES: 3.6.0alpha3 Released: Tue, 19 Aug 2014 14:33:00 +0200 This is the third alpha release of the upcoming syslog-ng OSE 3.6 branch. It is expected to be the last alpha release, with the first beta in about two weeks. This release contains a number of important features and bugfixes: Changed defaults ---------------- * The `flush-lines()` setting now defaults to *100*, rather than *1*, for increased speed. Features -------- * The `system()` source will now parse `@cim` marked messages as JSON, if the JSON module is available at run-time. This improves inter-operation with other software that uses the *Common Information Model*. ### Features from the [Incubator][incubator] * The `$(or)` template function that returns the first non-empty argument is now included in syslog-ng itself. * The `$(padding)` template function, to pad text with custom padding to a given length is also included. * The `$(graphite-output)` template function, to be used for sending metrics to [Graphite][graphite] was ported over from the Incubator. The `graphite()` destination SCL block is also available now, to make it even easier to talk to Graphite. * The `riemann()` destination, which allows sending metrics to the [Riemann][riemann] monitoring system was also ported over from the Incubator. [graphite]: http://graphite.wikidot.com/ [incubator]: https://github.com/balabit/syslog-ng-incubator [riemann]: http://riemann.io/ ### Threaded destinations A number of features were implemented for all threaded destinations: `amqp()`, `mongodb()`, `redis()`, `riemann()`, `smtp()` and `stomp()`. * The destinations gained support for `SEQNUM` persistence: the counter will be preserved across reloads and restarts. * A new option called `retries()` was implemented for all of these, which controls how many times a message delivery is retried before dropping it. * The `throttle()` option is now implemented, and works for all of the aforementioned destination drivers. * The message delivery loop was optimised to do less sleep/wakeup cycles, which should make the drivers not only faster, but more CPU friendly too. Bugfixes -------- * The basicfuncs module was fixed to work correctly on 32-bit architectures. * The `stored` statistics is no longer incremented by various drivers when they mean `processed`. * The type hinting feature is now more picky about what kind of type hints it accepts, allowing one to use template functions in - for example - `$(format-json)` pairs. Miscellaneous changes --------------------- * We now ship a "Contributors Guide" in the `CONTRIBUTING.md` file. Credits ------- syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessary to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Balazs Scheidler, Fabien Wernli, Gergely Nagy, Laszlo Budai, Peter Czanik, Robert Fekete, Tibor Benke, Viktor Juhasz, Viktor Tusa. DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.6.0a… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

Recommended Data base
by VMI X 19 Aug '14

19 Aug '14

Hi, Currently we're logging everything to text files for a few LAN clients. We're considering using a database instead and have a few questions to help us decide: - Would a database be a good option to replace existing text files for long term storage considering storage space? - Would mongo OR mysql be better suited for storing system logs? I understand answers to these questions can vary depending on specific use case but seeking a general recommendation to see what's typically being used and what the most stable/supported options would be. -- *Nullius In Verba*

5 6

Syslog-ng log rotation problem
by Riyas Ahamed 19 Aug '14

19 Aug '14

Hi Team, I have configured syslog-ng 3.2.5 with backend as mysql Ver 14.14 Distrib 5.6.20, I have able to fetch logs from the backend to frondend. Please help me to how to rotate syslog-ng logs on daily basis. Regards, N.B.RIAZ AHMED https://www.csscorp.com/email-disclaimer

1 0

Writing a null character at the end of TCP payload
by Nicolas Fédou 15 Aug '14

15 Aug '14

Hello, I need to send a null character at the end of a message on a tcp output... But this null is considered to be an end of string. My use case is sending a GELF message to a graylog's tcp input. GELF message is a json syntax quite simple to comply with thanks to format-json. Graylog needs a null character at the end of a GELF message in TCP but not in UDP. I tried many combination in the template with \x0, echo \x0, `global_definition_of_null`, etc... The matter is that null character when interpreted is an end of string, and is not written in the tcp message. I see truncated messages with ngrep. Like "a\x0b" gives "a" according to ngrep. I did not find any option to actually write a null character. Do you have any options ? As Graylog already have clients and librairies I belive they won't remove the need for a null character. Shall I open an issue on syslog-ng to ask for an option in templates or in format-json ? Regards, Nicolas Fédou.

3 8

insider 2014-08: EPEL; graphite; PCI DSS;
by Czanik, Péter 14 Aug '14

14 Aug '14

Dear syslog-ng users, This is the 36th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. FEATURED NEWS EPEL 7 now contains syslog-ng ----------------------------- RHEL 7 was released over a month ago and CentOS 7 not much later, but one piece of software was still missing: syslog-ng. Not any more. EPEL, which stands for Extra Packages for Enterprise Linux, is a software collection containing additional packages for Enterprise Linux and derivatives. Now its latest version, EPEL 7 also contains syslog-ng, version 3.5: http://czanik.blogs.balabit.com/2014/07/epel-7-now-contains-syslog-ng/ Introducing syslog-ng PE 5F1 ---------------------------- The latest version of the syslog-ng Premium Edition, 5F1, adds support for the popular NoSQL database MongoDB. Along with support for MongoDB, we have added support for Java Script Object Notation (JSON), a text-based open standard designed for human-readable data interchange. You can read more at http://gyp.blogs.balabit.com/2014/08/introducing-syslog-ng-premium-edition-… Performance monitoring using syslog-ng and graphite --------------------------------------------------- For most of its history, syslog-ng could only be used for collecting, processing and storing log messages. Not any more. The Redis and Riemann destinations are already a step into the direction of metrics-based monitoring, and the monitoring source combined with Graphite template support are the next. https://czanik.blogs.balabit.com/2014/07/how-to-setup-syslog-ng-quickly-for… Introducing syslog-ng store box 3F2 ----------------------------------- We recently released a new version of our log management appliance, the syslog-ng Store Box. 3F2 is the latest feature release and includes one major new feature and a major improvement to an existing one. First, we have added a RESTful API which opens up all sorts of possibilities for accessing log data in SSB. Second, we have revamped the search interface on the web-based user interface making searching and troubleshooting much easier. You can read more at http://gyp.blogs.balabit.com/2014/08/syslog-ng-store-box-3-f2-released/ Syslog-ng incubator 0.3.3 released ---------------------------------- The syslog-ng incubator is a set of tools and modules for syslog-ng, which are not (yet) available in the official release. This version of incubator works with the latest stable syslog-ng (v3.5.5+) and fixes many problems of the initial 0.3 incubator release. https://czanik.blogs.balabit.com/2014/07/syslog-ng-incubator-0-3-3-is-avail… Log management and the Verizon 2014 PCI Compliance Report --------------------------------------------------------- Recently, the eagerly anticipated Verizon Data Breach Investigations Report for 2014 was published. With more than 63,000 security incidents, 1,300 confirmed data breaches and 50 contributing global organizations, it provides the most comprehensive insight to state of IT security around the world. Drawing on data from the Data Breach Investigation Report, Verizon also publishes a lesser known but very interesting report on the state of compliance of with the Payment Card Industry Data Security Standard (PCI DSS), perhaps the most widely-adopted security standards globally. Read, what requirements PCI DSS has towards log management: https://jluby.blogs.balabit.com/2014/06/30/key-log-management-takeaways-fro… NEW RELEASES syslog-ng OSE 3.5.6: https://lists.balabit.hu/pipermail/syslog-ng-announce/2014-August/000192.ht… syslog-ng OSE 3.6 alpha2: https://lists.balabit.hu/pipermail/syslog-ng-announce/2014-August/000193.ht… syslog-ng PE 5LTS (5.0.5a): https://lists.balabit.hu/pipermail/syslog-ng-announce/2014-June/000189.html syslog-ng incubator 0.3.3 Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

connection broken every few minutes
by Simão Mata 13 Aug '14

13 Aug '14

Hello, Our syslog-ng clients are losing their connection to our central log server every few minutes. In the client I have the following log: 2014-08-12T15:44:02+00:00 wimdu-app07 syslog.notice syslog-ng[14032]: Syslog connection broken; fd='10', server='AF_INET(<our ip>:514)', time_reopen='1' 2014-08-12T15:44:03+00:00 wimdu-app07 syslog.notice syslog-ng[14032]: Syslog connection established; fd='10', server='AF_INET(<our ip>:514)', local='AF_INET(0.0.0.0:0)' I am using the following configuration on the clients: destination d_net_wimdu { syslog(<our ip> transport("tcp") port(514) log-fifo-size(2048)); }; log { source(s_src); source(s_net); destination(d_net_wimdu); }; And on the server: source s_net { syslog(ip(0.0.0.0) transport("tcp") max-connections(300) log-iw-size(3300) so_rcvbuf(8000000000) port(514)); }; log { source(s_net); destination(d_file_w); }; destination d_prog_logparser { program("< logparser_bin >" template(t_w_filetemplate) log_fifo_size(90000000) flags(no-multi-line)); }; log { source(s_net); destination(d_prog_logparser); flags(flow-control); }; Any idea how I can even debug this? Thank you -- [image: Wimdu] <http://www.wimdu.com/>*Wimdu GmbH* - Voltastraße 5, 13355 Berlin, Germany *Managing Directors* - Arne Bleckwenn, Hinrich Dreiling *Commercial Register Number* - 129773 B Berlin[image: Contact] <contact(a)wimdu.com>[image: Blog] <http://blog.wimdu.com/?wt_vi=signature>[image: FB] <http://www.facebook.com/wimdu>[image: TW] <http://www.twitter.com/wimdu>[image: G+] <http://www.google.com/+wimdu>[image: YT] <http://www.youtube.com/wimtubechannel>[image: GH] <https://github.com/wimdu>[image: LI] <http://www.linkedin.com/company/wimdu-gmbh>[image: XI] <http://www.xing.com/companies/wimdu>

3 2

syslog-ng 3.6.0alpha2 has been released
by devel＠balabit.hu 12 Aug '14

12 Aug '14

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.6.0alpha2 SUMMARY : new alpha release DATE : Aug 11, 2014 ------------------------------------------------------------------------------ DESCRIPTION: The second alpha version of syslog-ng Open Source Edition (3.6.0alpha2) has been released. This being an alpha release, testing is most welcome, but production use is not recommended! CHANGES: 3.6.0alpha2 Released: Mon, 11 Aug 2014 15:02:59 +0200 This is the second alpha release of the upcoming syslog-ng OSE 3.6 branch, with more internal changes and features compared to the previous 3.6.0alpha1 release. Above the changes in the latest 3.5.6 stable version, this release contains the following noteworthy changes: Changed defaults ---------------- * Threaded mode is now **enabled** by default. To turn it off, use `threaded(no)` in the global options section. * The versioning of the `libsyslog-ng` internal library has changed: instead of always using the current release number, we will now try to maintain ABI compatibility during the lifetime of a stable branch. Therefore, we use only the first two components of our version as the base of the library version. Another number will be part of the SONAME too, but that will only change when we break compatibility. The SONAME is currently set to `libsyslog-ng-3.6.so.0`, and will remain the same during alpha and beta releases, even when the ABI changes. We will start bumping the version after the first stable release from this branch, if needed. Features -------- * The new `systemd-syslog()` source replaces the former implicit support for the same thing. Users who use systemd are advised to use either the `system()` source, or this new one when they want to receive logs from systemd via the `/run/systemd/journal/syslog` socket. ### SMTP destination changes * The `smtp()` destination now supports a `retries()` option, which controls how many times a message delivery will be attempted before dropping it. * The destination no longer counts delivered messages as `stored`. * The templates used in the destination now honor the time-zone settings. * The driver will abort if required options (any of `to()`, `cc()`, `bcc()` and `from()`, and `subject()` and `body()`) are not set. Bugfixes -------- * The `file()` and `network()` (including `tcp()` et al) sources will now properly set the `$SOURCE` macro. * To build syslog-ng with the MongoDB destination, libmongo-client version 0.1.8+ is now required. (The internal copy has been updated accordingly.) * The UNIX credential extracting feature was ported to FreeBSD, syslog-ng now compiles, and has support for this feature. (The previous support in alpha1 was sadly incomplete.) * The `pdbtool merge` command will now generate version 4 patterndb files. Credits ------- syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessary to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Fabien Wernli, Gergely Nagy, Laszlo Budai, Peter Czanik, Tibor Benke, Viktor Juhasz, Viktor Tusa. DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.6.0a… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

2 1

syslog-ng redis destination problem
by jrhendri＠roadrunner.com 12 Aug '14

12 Aug '14

Hi (again) - This one has me a bit puzzled. I am trying to write to redis using the example from the syslog-ng documentation: destination d_local_redis { redis(command("HINCRBY" "${HOST}/programs" "${PROGRAM}" "1")); }; but I am getting "-ERR value is not an integer or out of range" I have tried this on a couple systems. This particular one is: [n0142566@VDDP13E-F1A47ED src]$ /usr/local/sbin/syslog-ng --version syslog-ng 3.5.4.1 Installer-Version: 3.5.4.1 Revision: Compile-Date: Jul 22 2014 12:30:06 Available-Modules: afsocket-notls,afprog,csvparser,system-source,dbparser,redis,syslogformat,afuser,afstomp,afmongodb,linux-kmsg-format,json-plugin,basicfuncs,afamqp,affile,cryptofuncs,afsocket,confgen Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off Enable-Pcre: off [n0142566@VDDP13E-F1A47ED src]$ And I have captured the packets to see what is being sent and see this following the TCP stream with Wireshark (pcap attached) *1 $4 ping +PONG *4 $7 HINCRBY $24 VDDP13E-F1A47ED/programs $16 ArcSightLogs_PDC $2 * -ERR value is not an integer or out of range Note: when I use ("INCR" "testkey") it works - seems like the integer increment is not being sent correctly? Is this something I am getting wrong or is it a bug ? Thanks (again) Jim

1 1