August 2014 - syslog-ng - lists.balabit.hu

Syslog-ng log rotation problem
by Riyas Ahamed 19 Aug '14

19 Aug '14

Hi Team, I have configured syslog-ng 3.2.5 with backend as mysql Ver 14.14 Distrib 5.6.20, I have able to fetch logs from the backend to frondend. Please help me to how to rotate syslog-ng logs on daily basis. Regards, N.B.RIAZ AHMED https://www.csscorp.com/email-disclaimer

1 0

Writing a null character at the end of TCP payload
by Nicolas Fédou 15 Aug '14

15 Aug '14

Hello, I need to send a null character at the end of a message on a tcp output... But this null is considered to be an end of string. My use case is sending a GELF message to a graylog's tcp input. GELF message is a json syntax quite simple to comply with thanks to format-json. Graylog needs a null character at the end of a GELF message in TCP but not in UDP. I tried many combination in the template with \x0, echo \x0, `global_definition_of_null`, etc... The matter is that null character when interpreted is an end of string, and is not written in the tcp message. I see truncated messages with ngrep. Like "a\x0b" gives "a" according to ngrep. I did not find any option to actually write a null character. Do you have any options ? As Graylog already have clients and librairies I belive they won't remove the need for a null character. Shall I open an issue on syslog-ng to ask for an option in templates or in format-json ? Regards, Nicolas Fédou.

3 8

insider 2014-08: EPEL; graphite; PCI DSS;
by Czanik, Péter 14 Aug '14

14 Aug '14

Dear syslog-ng users, This is the 36th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. FEATURED NEWS EPEL 7 now contains syslog-ng ----------------------------- RHEL 7 was released over a month ago and CentOS 7 not much later, but one piece of software was still missing: syslog-ng. Not any more. EPEL, which stands for Extra Packages for Enterprise Linux, is a software collection containing additional packages for Enterprise Linux and derivatives. Now its latest version, EPEL 7 also contains syslog-ng, version 3.5: http://czanik.blogs.balabit.com/2014/07/epel-7-now-contains-syslog-ng/ Introducing syslog-ng PE 5F1 ---------------------------- The latest version of the syslog-ng Premium Edition, 5F1, adds support for the popular NoSQL database MongoDB. Along with support for MongoDB, we have added support for Java Script Object Notation (JSON), a text-based open standard designed for human-readable data interchange. You can read more at http://gyp.blogs.balabit.com/2014/08/introducing-syslog-ng-premium-edition-… Performance monitoring using syslog-ng and graphite --------------------------------------------------- For most of its history, syslog-ng could only be used for collecting, processing and storing log messages. Not any more. The Redis and Riemann destinations are already a step into the direction of metrics-based monitoring, and the monitoring source combined with Graphite template support are the next. https://czanik.blogs.balabit.com/2014/07/how-to-setup-syslog-ng-quickly-for… Introducing syslog-ng store box 3F2 ----------------------------------- We recently released a new version of our log management appliance, the syslog-ng Store Box. 3F2 is the latest feature release and includes one major new feature and a major improvement to an existing one. First, we have added a RESTful API which opens up all sorts of possibilities for accessing log data in SSB. Second, we have revamped the search interface on the web-based user interface making searching and troubleshooting much easier. You can read more at http://gyp.blogs.balabit.com/2014/08/syslog-ng-store-box-3-f2-released/ Syslog-ng incubator 0.3.3 released ---------------------------------- The syslog-ng incubator is a set of tools and modules for syslog-ng, which are not (yet) available in the official release. This version of incubator works with the latest stable syslog-ng (v3.5.5+) and fixes many problems of the initial 0.3 incubator release. https://czanik.blogs.balabit.com/2014/07/syslog-ng-incubator-0-3-3-is-avail… Log management and the Verizon 2014 PCI Compliance Report --------------------------------------------------------- Recently, the eagerly anticipated Verizon Data Breach Investigations Report for 2014 was published. With more than 63,000 security incidents, 1,300 confirmed data breaches and 50 contributing global organizations, it provides the most comprehensive insight to state of IT security around the world. Drawing on data from the Data Breach Investigation Report, Verizon also publishes a lesser known but very interesting report on the state of compliance of with the Payment Card Industry Data Security Standard (PCI DSS), perhaps the most widely-adopted security standards globally. Read, what requirements PCI DSS has towards log management: https://jluby.blogs.balabit.com/2014/06/30/key-log-management-takeaways-fro… NEW RELEASES syslog-ng OSE 3.5.6: https://lists.balabit.hu/pipermail/syslog-ng-announce/2014-August/000192.ht… syslog-ng OSE 3.6 alpha2: https://lists.balabit.hu/pipermail/syslog-ng-announce/2014-August/000193.ht… syslog-ng PE 5LTS (5.0.5a): https://lists.balabit.hu/pipermail/syslog-ng-announce/2014-June/000189.html syslog-ng incubator 0.3.3 Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

connection broken every few minutes
by Simão Mata 13 Aug '14

13 Aug '14

Hello, Our syslog-ng clients are losing their connection to our central log server every few minutes. In the client I have the following log: 2014-08-12T15:44:02+00:00 wimdu-app07 syslog.notice syslog-ng[14032]: Syslog connection broken; fd='10', server='AF_INET(<our ip>:514)', time_reopen='1' 2014-08-12T15:44:03+00:00 wimdu-app07 syslog.notice syslog-ng[14032]: Syslog connection established; fd='10', server='AF_INET(<our ip>:514)', local='AF_INET(0.0.0.0:0)' I am using the following configuration on the clients: destination d_net_wimdu { syslog(<our ip> transport("tcp") port(514) log-fifo-size(2048)); }; log { source(s_src); source(s_net); destination(d_net_wimdu); }; And on the server: source s_net { syslog(ip(0.0.0.0) transport("tcp") max-connections(300) log-iw-size(3300) so_rcvbuf(8000000000) port(514)); }; log { source(s_net); destination(d_file_w); }; destination d_prog_logparser { program("< logparser_bin >" template(t_w_filetemplate) log_fifo_size(90000000) flags(no-multi-line)); }; log { source(s_net); destination(d_prog_logparser); flags(flow-control); }; Any idea how I can even debug this? Thank you -- [image: Wimdu] <http://www.wimdu.com/>*Wimdu GmbH* - Voltastraße 5, 13355 Berlin, Germany *Managing Directors* - Arne Bleckwenn, Hinrich Dreiling *Commercial Register Number* - 129773 B Berlin[image: Contact] <contact(a)wimdu.com>[image: Blog] <http://blog.wimdu.com/?wt_vi=signature>[image: FB] <http://www.facebook.com/wimdu>[image: TW] <http://www.twitter.com/wimdu>[image: G+] <http://www.google.com/+wimdu>[image: YT] <http://www.youtube.com/wimtubechannel>[image: GH] <https://github.com/wimdu>[image: LI] <http://www.linkedin.com/company/wimdu-gmbh>[image: XI] <http://www.xing.com/companies/wimdu>

3 2

syslog-ng 3.6.0alpha2 has been released
by devel＠balabit.hu 12 Aug '14

12 Aug '14

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.6.0alpha2 SUMMARY : new alpha release DATE : Aug 11, 2014 ------------------------------------------------------------------------------ DESCRIPTION: The second alpha version of syslog-ng Open Source Edition (3.6.0alpha2) has been released. This being an alpha release, testing is most welcome, but production use is not recommended! CHANGES: 3.6.0alpha2 Released: Mon, 11 Aug 2014 15:02:59 +0200 This is the second alpha release of the upcoming syslog-ng OSE 3.6 branch, with more internal changes and features compared to the previous 3.6.0alpha1 release. Above the changes in the latest 3.5.6 stable version, this release contains the following noteworthy changes: Changed defaults ---------------- * Threaded mode is now **enabled** by default. To turn it off, use `threaded(no)` in the global options section. * The versioning of the `libsyslog-ng` internal library has changed: instead of always using the current release number, we will now try to maintain ABI compatibility during the lifetime of a stable branch. Therefore, we use only the first two components of our version as the base of the library version. Another number will be part of the SONAME too, but that will only change when we break compatibility. The SONAME is currently set to `libsyslog-ng-3.6.so.0`, and will remain the same during alpha and beta releases, even when the ABI changes. We will start bumping the version after the first stable release from this branch, if needed. Features -------- * The new `systemd-syslog()` source replaces the former implicit support for the same thing. Users who use systemd are advised to use either the `system()` source, or this new one when they want to receive logs from systemd via the `/run/systemd/journal/syslog` socket. ### SMTP destination changes * The `smtp()` destination now supports a `retries()` option, which controls how many times a message delivery will be attempted before dropping it. * The destination no longer counts delivered messages as `stored`. * The templates used in the destination now honor the time-zone settings. * The driver will abort if required options (any of `to()`, `cc()`, `bcc()` and `from()`, and `subject()` and `body()`) are not set. Bugfixes -------- * The `file()` and `network()` (including `tcp()` et al) sources will now properly set the `$SOURCE` macro. * To build syslog-ng with the MongoDB destination, libmongo-client version 0.1.8+ is now required. (The internal copy has been updated accordingly.) * The UNIX credential extracting feature was ported to FreeBSD, syslog-ng now compiles, and has support for this feature. (The previous support in alpha1 was sadly incomplete.) * The `pdbtool merge` command will now generate version 4 patterndb files. Credits ------- syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessary to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Fabien Wernli, Gergely Nagy, Laszlo Budai, Peter Czanik, Tibor Benke, Viktor Juhasz, Viktor Tusa. DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.6.0a… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

2 1

syslog-ng redis destination problem
by jrhendri＠roadrunner.com 12 Aug '14

12 Aug '14

Hi (again) - This one has me a bit puzzled. I am trying to write to redis using the example from the syslog-ng documentation: destination d_local_redis { redis(command("HINCRBY" "${HOST}/programs" "${PROGRAM}" "1")); }; but I am getting "-ERR value is not an integer or out of range" I have tried this on a couple systems. This particular one is: [n0142566@VDDP13E-F1A47ED src]$ /usr/local/sbin/syslog-ng --version syslog-ng 3.5.4.1 Installer-Version: 3.5.4.1 Revision: Compile-Date: Jul 22 2014 12:30:06 Available-Modules: afsocket-notls,afprog,csvparser,system-source,dbparser,redis,syslogformat,afuser,afstomp,afmongodb,linux-kmsg-format,json-plugin,basicfuncs,afamqp,affile,cryptofuncs,afsocket,confgen Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off Enable-Pcre: off [n0142566@VDDP13E-F1A47ED src]$ And I have captured the packets to see what is being sent and see this following the TCP stream with Wireshark (pcap attached) *1 $4 ping +PONG *4 $7 HINCRBY $24 VDDP13E-F1A47ED/programs $16 ArcSightLogs_PDC $2 * -ERR value is not an integer or out of range Note: when I use ("INCR" "testkey") it works - seems like the integer increment is not being sent correctly? Is this something I am getting wrong or is it a bug ? Thanks (again) Jim

1 1

How to log messages generated by C program and log file rotation
by Jean Faye 07 Aug '14

07 Aug '14

confirm a6d843f7ad7d1dbf1fefb6f12432e54941a680a9 Hi all, I want to use syslog-ng to log the messages generated by my application implemented in C language. I added this in the code: char *log="rtcd"; printf("[%s] RTC adjustement\n",__func__); openlog(log, LOG_PID, LOG_LOCAL0); syslog(LOG_DEBUG, "[FIJ] RTC adjustement"); closelog(); For me, according to the syslog-ng file, the files /var/log/ldb/GENTrace.log, /var/log/ldb/SUTrace.log, /var/log/ldb/WANTrace.log and /var/log/ldb/CPLTrace.log must be created and must contain the syslog message. But I got no messages in my destination files. You can see the content of my syslog-ng.conf file bellow. Is it the right way to log the messages sent by C program? What can explain that I got no messages in the destinations files? Concerning log file rotation, How can we manage it using syslog-ng? For example I want to have a destination file with a size maximum = 2Mo and if the size is greater than the max size, I have to save the current one and create a new one. On my system I can have max 4 files (4 x 2Mo). How can I manage this kind of rotation? Thanks in advance. Best regards, Ismael Jean FAYE @version: 3.5 # # Syslog-ng configuration file, compatible with default Debian syslogd # installation. Originally written by anonymous (I can't find his name) # Revised, and rewrited by me (SZALAY Attila <sasa(a)debian.org>) # First, set some global options. options { chain_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no); owner("root"); group("adm"); perm(0640); stats_freq(0); bad_hostname("^gconfd$");create_dirs(yes); }; ######################## # Sources ######################## source s_mysource { pipe("/tmp/pipe" pad_size(2048)); }; ######################## # Destinations ######################## destination d_GEN { file("/var/log/ldb/GENTrace.log"); }; destination d_SU { file("/var/log/ldb/SUTrace.log"); }; destination d_WAN { file("/var/log/ldb/WANTrace.log"); }; destination d_CPL { file("/var/log/ldb/CPLTrace.log"); }; ######################## # Filters ######################## #filter f_GEN { #facility(local0) and filter(nom_du_composant_applicatif); #facility(local0) and filter(f_debug); #facility(local0); #}; filter f_SU { #facility(local0) and filter(nom_du_composant_applicatif); #facility(local0) and filter(f_debug); facility(local0); }; filter f_WAN { #facility(local0) and filter(nom_du_composant_applicatif); #facility(local0) and filter(f_debug); facility(local0); }; filter f_CPL { #facility(local0) and filter(nom_du_composant_applicatif); #facility(local0) and filter(f_debug); facility(local0); }; ######################## # Log paths ######################## #log { source(s_mysource); filter(f_GEN); destination(d_GEN); }; log { source(s_mysource); filter(f_SU); destination(d_SU); }; log { source(s_mysource); filter(f_WAN); destination(d_WAN); }; log { source(s_mysource); filter(f_CPL); destination(d_CPL); };

3 5

Re: [syslog-ng] FW: Syslog Problem
by Jim Hendrick 06 Aug '14

06 Aug '14

I agree with Evan. You need to know which events are actually being received. The simplified configuration I sent should provide that and the you can add back the filters and database destination. Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Evan Rempel <erempel(a)uvic.ca> Date:08/06/2014 11:22 AM (GMT-05:00) To: syslog-ng(a)lists.balabit.hu Subject: Re: [syslog-ng] FW: Syslog Problem > Could you please tell me how to check under load mysql silently drop records There is no record that this occurs. You need to have an authoritative source of what should be in the database, and then look to see what is actually in the database. > And could you please tell me what is the solution for this problem? To be honest, I wasn't very interested in the solution. The database should not have don that to begin with. Depending on load and version of mysql the problem would go away, or return. I never found a solution because I didn't look to hard for one. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

FW: Syslog Problem
by Riyas Ahamed 06 Aug '14

06 Aug '14

Hi, I have configured a syslog-ng server to capture network logs. I can able to fetch network logs of three devices. But in fourth network device I can able to see packets are get polling into the syslog server in port 514 by using tcpdump command but I cannot find the network device in syslogng front end. Syslog OS : CentOS release 6.5 (Final) Syslog Version : syslog-ng 3.2.5 Along with this mail I have attached configuration file of syslogng server. Please help me to resolve this issue. Regards, N.B.RIAZ AHMED (9047166496) https://www.csscorp.com/email-disclaimer

4 8

syslog-ng 3.5.6 has been released
by devel＠balabit.hu 05 Aug '14

05 Aug '14

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.5.6 SUMMARY : new stable release DATE : Aug 5, 2014 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.5.6) has been released. For latest fixes in the 3.5.x feature branch you are recommended to upgrade to this version. CHANGES: 3.5.6 Mon, 4 Aug 2014 13:39:42 +0200 This is the sixth bug-fix release for the 3.5.x series. Upgrading from earlier versions is highly recommended, as the changes in this release are very small, yet, also very important for most platforms and workloads. Bugfixes ======== * A major memory leak was fixed in the value-pairs framework, which affects $(format-json), MongoDB, AMQP and more. The leak was supposed to be fixed in 3.5.5, but due to a merging mistake, it was missed. * The configure script now detects Linux capabilities properly, there is no need to use --disable-linux-caps on non-Linux platforms anymore. * The pdbtool merge command will now generate version 4 patterndb files, instead of version 3. Credits ======= syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessary to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Fabien Wernli Gergely Nagy Peter Czanik DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.5.6 The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0