- syslog-ng - lists.balabit.hu

Syslog proxy/relay encapsulates
by Michael Yacc 07 Sep '14

07 Sep '14

Dear all, I'm trying to configure syslog-ng as relay/proxy server. Seems to be simple, but syslog server recieves encapsulated original syslog messages within outgoing "syslog-relay" messages. Is there any way to configure syslog-ng (relay) to just forward orignal messages to syslog server? Syslog-ng version used: 3.3.9 *syslog-client$* logger "hello from syslog client" on syslog-relay I added the following configuration: ######## sources ############### source s_relay_port { udp(ip(0.0.0.0) port(514)); }; ######## destinations ############### destination d_syslog_server { syslog("syslog-server" transport("udp") port(514)); }; ######## logging ############### log { source(s_relay_port); destination(d_syslog_server); }; *syslog-server$* tail -F /var/log/syslog Sep 6 23:53:28 syslog-relay-ip 1 2014-09-06T23:53:28+03:00 syslog-client-ip 1 - - - 2014-09-06T23:53:28+03:00 syslog-client-hostname root - - [meta sequenceId="38"] hello from syslog client With best regards, Michael Yacc

2 1

Individual Logs Files to each Forward to Different server/port?
by wiskbroom＠hotmail.com 05 Sep '14

05 Sep '14

Greetings! Is is possible to configure a syslog-ng client to forward logs to a syslog-ng server based on file? I am thinking of the following as an example: destination named-LOGS { udp(ip(192.168.1.100) port(555)); }; source named { file("/var/log/named/bind.log" log_prefix("BIND-LOGS")); }; log { source(named); destination(named-LOGS); }; destination dhcpd-LOGS { udp(ip(192.168.1.100) port(556)); }; source dhcpd { file("/var/log/dhcpd/dhcp.log" log_prefix("DHCPD-LOGS")); }; log { source(dhcpd); destination(dhcpd-LOGS); }; In this example, I am sending each to the same destination IP address, although that is configurable, but each log file to a different port, and with a different log_prefix as well. Does this OK,or is their a simpler way? Many thanks, .vp Vadim Anatoly Pushkin -- The Ukranian Stallion --

1 0

syslog-ng 3.6.0beta1 has been released
by devel＠balabit.hu 05 Sep '14

05 Sep '14

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.6.0beta1 SUMMARY : new beta release DATE : Sep 5, 2014 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.6.0beta1) has been released. Being the first beta release, we would be very grateful for any feedback, to stabilize the release further. CHANGES: 3.6.0beta1 Wed, 3 Sep 2014 14:40:14 +0200 This is the first beta release of the upcoming syslog-ng OSE 3.6 branch. Compared to the alphas, this release contains a moderate amount of new functionality and bugfixes. Further releases will focus on stability and bugfixes. Features -------- * One can now use multiple elements in the `key()` and `exclude()` options of any value-pairs declaration. * A new source driver was added to the syslog-ng: `systemd-journal()`, which reads from the Journal directly, not via the syslog forwarding socket. The `system()` source defaults to using this source when systemd is detected. Bugfixes -------- * All the various crypto-related template functions now check that the desired length of the digest is not larger than the digest itself. If a larger value is requested, they will truncate it to the digest length. * The `$(geoip)` template function now works with `threaded(yes)` too. * The unix domain socket credentials code was changed to only build on Linux and FreeBSD. With this change, syslog-ng should compile again on platforms where the OS does not support this, with the feature disabled. Credits ------- syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessary to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Fabien Wernli, Gergely Nagy, Laszlo Budai, Michael Hocke, Tibor Benke, Viktor Juhasz, Viktor Tusa. DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.6.0b… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

3.6beta1 packages for openSUSE
by Czanik, Péter 04 Sep '14

04 Sep '14

Hi, My initial 3.6beta1 pacages for openSUSE and SLES 11 are ready. There was a change in systemd packaging in Factory, that I still need to figure out. I tested the new package on openSUSE 13.1 and even tested it by disabling syslog compatibility in /etc/systemd/journald.conf, and the brand new journal source seems to work fine. If you run (open)SUSE, please give these packages a try, and let us know, if you run into any trouble! Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

2 1

3.6beta1 packages for Fedora/EPEL
by Czanik, Péter 04 Sep '14

04 Sep '14

Hi, http://copr.fedoraproject.org/coprs/czanik/syslog-ng36/ I prepared packages from 3.6beta1 for all recent Fedora releases and for EPEL7. I only tested on rawhide, as there syslog compatibility is disabled in journal by default and beta1 already has a journal source. I did some basic tests, and it worked fine. Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

[Bug 275] New: lib/filter/filter-in-list. c does not compile under Solaris 10
by bugzilla＠bugzilla.balabit.com 28 Aug '14

28 Aug '14

https://bugzilla.balabit.com/show_bug.cgi?id=275 Summary: lib/filter/filter-in-list.c does not compile under Solaris 10 Product: syslog-ng Version: 3.5.x Platform: PC OS/Version: Solaris Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: michael.hocke(a)nyu.edu Type of the Report: bug Estimated Hours: 0.0 Created an attachment (id=93) --> (https://bugzilla.balabit.com/attachment.cgi?id=93) diff for misc.c and misc.h lib/filter/filter-in-list.c uses getline() that is not available in Solaris 10. I worked around it by adding code snippets to lib/misc.h and lib/misc.c that implement getdelim and getline that I took from another project. I was ablt to make it work but I assume it wasn't the most elegant solution and should be integrated into autoconfig etc. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

really need help...not writing files
by Frank Wilkinson 26 Aug '14

26 Aug '14

Please forgive me if this has already been addressed. If so will you point me to it? Syslog-ng will all of a sudden, stop writing files. I'm running syslog-ng 3.5.3 Installer-Version: 3.5.3 Revision: ssh+git://algernon@git.balabit/var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.5#master#ccb05a22408ba4c837d998b2538854d994f845a5 Compile-Date: Jan 8 2014 13:35:02 Available-Modules: afsocket,afprog,dbparser,system-source,affile,syslogformat,linux-kmsg-format,csvparser,afmongodb,afsocket-tls,confgen,afuser,afstomp,afsocket-notls,basicfuncs,cryptofuncs,afamqp Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: off Enable-Pcre: on The service status is showing running but not writing log files. We are logging udp from about 2400 devices When it dies strace shows: epoll_ctl(3, EPOLL_CTL_DEL, 10, {0, {u32=19726648, u64=19726648}}) = 0 write(110, "\1\0\0\0\0\0\0\0", 8) = 8 epoll_wait(3, {{EPOLLIN, {u32=19641320, u64=19641320}}}, 11, 3414) = 1 read(6, "\1\0\0\0\0\0\0\0", 8) = 8 futex(0x7fe1b000bd34, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fe1b000bd30, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1 futex(0x7fe21cfaad68, FUTEX_WAKE_PRIVATE, 1) = 1 fcntl(10, F_GETFD) = 0x1 (flags FD_CLOEXEC) fcntl(10, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK) setsockopt(10, SOL_SOCKET, SO_OOBINLINE, [1], 4) = 0 write(110, "\1\0\0\0\0\0\0\0", 8) = 8 epoll_ctl(3, EPOLL_CTL_ADD, 10, {0, {u32=19726648, u64=19726648}}) = 0 epoll_wait(3, {{EPOLLIN, {u32=19641320, u64=19641320}}}, 12, 3413) = 1 read(6, "\1\0\0\0\0\0\0\0", 8) = 8 futex(0x7fe1b000bd34, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fe1b000bd30, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1 futex(0x7fe21cfaad68, FUTEX_WAKE_PRIVATE, 1) = 1 futex(0x14e25a0, FUTEX_WAKE_PRIVATE, 1) = 1 epoll_wait(3, {{EPOLLIN, {u32=19641320, u64=19641320}}}, 12, 0) = 1 read(6, "\1\0\0\0\0\0\0\0", 8) = 8 futex(0x7fe1b000bd34, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fe1b000bd30, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1 futex(0x7fe21cfaad68, FUTEX_WAKE_PRIVATE, 1) = 1 write(110, "\1\0\0\0\0\0\0\0", 8) = 8 futex(0x12d02c0, FUTEX_WAIT_PRIVATE, 2, NULL) = ? ERESTARTSYS (To be restarted) --- SIGTERM (Terminated) @ 0 (0) --- write(17, "\1\0\0\0\0\0\0\0", 8) = 8 rt_sigreturn(0x7fe21cfab740) = 202 futex(0x12d02c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...> here is where I did a restart +++ killed by SIGKILL +++ top - 12:45:56 up 133 days, 23:11, 13 users, load average: 1.06, 1.13, 1.14 Tasks: 634 total, 2 running, 632 sleeping, 0 stopped, 0 zombie Cpu(s): 4.2%us, 2.0%sy, 0.0%ni, 93.6%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%s Mem: 32898840k total, 31285296k used, 1613544k free, 128188k buffers Swap: 16777212k total, 684800k used, 16092412k free, 29249028k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1631 root 20 0 1725m 104m 2892 S 28.4 0.3 10:38.46 syslog-ng 2843 root 20 0 1725m 104m 2892 S 17.5 0.3 0:04.35 syslog-ng 2795 root 20 0 1725m 104m 2892 S 15.8 0.3 0:11.99 syslog-ng 2842 root 20 0 1725m 104m 2892 S 13.9 0.3 0:02.68 syslog-ng 2793 root 20 0 1725m 104m 2892 S 13.5 0.3 0:14.54 syslog-ng 2855 root 20 0 1725m 104m 2892 R 13.5 0.3 0:00.41 syslog-ng 2776 root 20 0 1725m 104m 2892 S 12.2 0.3 0:18.57 syslog-ng 43203 root 20 0 359m 101m 10m S 11.9 0.3 15:35.10 splunkd 2794 root 20 0 1725m 104m 2892 S 9.6 0.3 0:14.62 syslog-ng 2791 root 20 0 1725m 104m 2892 S 9.2 0.3 0:11.89 syslog-ng 2697 root 20 0 1725m 104m 2892 S 6.3 0.3 0:31.74 syslog-ng 43204 root 20 0 359m 101m 10m S 4.9 0.3 8:01.72 splunkd 2825 root 20 0 1725m 104m 2892 S 2.3 0.3 0:07.73 syslog-ng 2841 root 20 0 1725m 104m 2892 S 1.6 0.3 0:03.30 syslog-ng ... Also, one other problem I have is the syslog-ng log file says: Aug 26 11:48:49 sopher1 syslog-ng[488]: Input is valid utf8, but the log message is not tagged as such, this performs worse than enabling validate-utf8 flag on input; value='758AARULOCAL01' My config specifies flags(validate-utf8): source s_udp { udp( port(514) so_rcvbuf(15000000) log_iw_size(50000) log_msg_size(65535) log_fetch_limit(50000) flags(validate-utf8));}; Frank Wilkinson (205)934-3540 w

1 0

Re: [syslog-ng] Quick set of eyeballs on this
by James Lay 25 Aug '14

25 Aug '14

On 2014-08-25 15:39, Balint Kovacs wrote: > Hi, > > just a guess, but as far as i can remember, the ASA-* part is > usually > in the program field, try the program() filter instead of message() > in > f_firewall. > > Balint > > On 2014. augusztus 25. 22:48:05 CEST, James Lay > <jlay(a)slave-tothe-box.net> wrote: > >> Can anyone see anything blatantly wrong with this? The goal is to >> syslog as usual, but to forward firewall messages to a different >> server. >> Thanks for looking all. >> >> James >> >> @version:3.3.5 >> options { >> use_dns(no); >> flush_lines(0); >> stats_freq(43200); >> }; >> >> source s_local { >> unix-stream("/dev/log"); >> udp(ip(0.0.0.0) port(514)); >> tcp(ip(0.0.0.0) port(514)); >> file("/proc/kmsg"); >> }; >> >> destination d_file { >> file("/var/log/messages"); >> }; >> >> destination d_syslogserver { udp ("x.x.x.x", port(7514)); }; >> >> filter f_syslogfilter { >> not ( >> message("0x0004") >> or message("169.254.") >> or message("192.168.") >> ); >> }; >> >> filter f_firewall { >> message("ASA-4-71005") >> >> or >> message("ASA-2-106100") >> }; >> >> log { >> source(s_local); >> filter(f_syslogfilter); >> destination(d_file); >> }; >> >> log { >> source(s_local); >> filter(f_firewall); >> destination(d_syslogserver); >> }; >> >> ------------------------- Ah thank you...I've just been testing using logger. I'll adjust and try this with program(). Thanks again. James

1 0

Quick set of eyeballs on this
by James Lay 25 Aug '14

25 Aug '14

Can anyone see anything blatantly wrong with this? The goal is to syslog as usual, but to forward firewall messages to a different server. Thanks for looking all. James @version:3.3.5 options { use_dns(no); flush_lines(0); stats_freq(43200); }; source s_local { unix-stream("/dev/log"); udp(ip(0.0.0.0) port(514)); tcp(ip(0.0.0.0) port(514)); file("/proc/kmsg"); }; destination d_file { file("/var/log/messages"); }; destination d_syslogserver { udp ("x.x.x.x", port(7514)); }; filter f_syslogfilter { not ( message("0x0004") or message("169\.254\.") or message("192\.168\.") ); }; filter f_firewall { message("ASA-4-71005") or message("ASA-2-106100") }; log { source(s_local); filter(f_syslogfilter); destination(d_file); }; log { source(s_local); filter(f_firewall); destination(d_syslogserver); };

1 0

Re: [syslog-ng] create_dir and "dynamic" destinations.
by Jim Hendrick 22 Aug '14

22 Aug '14

I would check doing a manual mkdir -p </some/long/path> as the user specified in the config. Then try as root if different. Also look at how that file system is mounted. Is it local disk or NFS ? NFS treats root differently. Also I have seen the permissions of the underlying mount point also cause problems. That is rwxr-x--- for the directory on which the NFS is mounted can cause problems for the mounted filesystem. And you won't see it unless you unmount the NFS piece. Anyway that's what I can think of right off. Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: "AVELING, BEN (BEN)" <ben.aveling(a)alcatel-lucent.com> Date:08/21/2014 6:33 PM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Cc: Subject: Re: [syslog-ng] create_dir and "dynamic" destinations. Permissions problem? Is anything reported in syslog-ng’s own logs? Regards, Ben From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Renato Bezerra Sent: Friday, 22 August 2014 05:51 To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] create_dir and "dynamic" destinations. I use this same config and work very well. Em 21/08/2014 16:44, "Balazs Scheidler" <bazsi77(a)gmail.com> escreveu: Hmm it should create the entire path. On Aug 20, 2014 5:16 PM, "Scot Needy" <scotrn(a)gmail.com> wrote: Yes, I have the create-dies() option enabled but that does not seem to work for upper level directories. Example: Only works if the 192_168_9_0 directory is pre-created. destination d_192_168_9_0 { file(/opt/syslog-ng/logs/192_168_9_0/$YEAR/$MONTH/$DAY/$HOST.log);}; On Aug 20, 2014, at 7:24 AM, Fabien Wernli <wernli(a)in2p3.fr> wrote: > Hi Scot, > > You have the `create-dirs()` option for that: > > http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.5-guid… > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

3 2

create_dir and "dynamic" destinations.
by Scot Needy 21 Aug '14

21 Aug '14

Hi, I was wondering if anyone has tried to work around syslog_ng. My destinations are psudo dynamic in that I have one per subnet but I don’t want to pre-create every destination directory until a syslog message is received for that destination. I have 3 conf files (Dfilters.conf Ddestination.conf and Dlog.conf ) that are built using an API to our IPNMS (IP Network Management System). The purpose is to use syslog-ng stats to quickly report on subnets and also provide per-subnet options for reporting, forwarding … Example Destination destination d_192_168_9_0 { file(/opt/syslog-ng/logs/192_168_9_0/$YEAR$MONTH$DAY.$HOST.log);}; Example Filter filter f_192_168_9_0 { netmask(192.168.9.0/26);}; Example Log log { source(s_net); filter(f_192_168_9_0); destination(d_192_168_9_0);}; My question is can anyone think of a way to get the create_dir option to dynamically create my subnet directories as needed ? For now I Need to pre-create every directory or syslog-ng complains which causes me to create a bunch of empty directories. Thanks Scot

5 5

syslog-ng 3.5 for RHEL6
by Czanik, Péter 19 Aug '14

19 Aug '14

Hi, Recently many people asked for syslog-ng 3.5 rpms for RHEL6. What I did breaks most of the packaging rules, but works. It's a wild mixture of the current EPEL6 and EPEL7 packages. So it has most of the features of the EPEL7 package (except UDP spoofing), but has sysvinit style init script. It's installed to /usr/sbin instead of /sbin, has SSL support enabled, and uses bundled libraries instead of system libraries (as those are missing from RHEL6 and EPEL6). To use the packages, you need to enable EPEL6 repositories for dependencies ( https://fedoraproject.org/wiki/EPEL ) and also my Copr repository: http://copr.fedoraproject.org/coprs/czanik/syslog-ng35epel6/ It's ugly as hell, but survived my quick test on CentOS 6.5. Use at your own risk but let me know if you run into any trouble! Bye, -- Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

syslog-ng 3.6.0alpha3 has been released
by devel＠balabit.hu 19 Aug '14

19 Aug '14

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.6.0alpha3 SUMMARY : new alpha release DATE : Aug 19, 2014 ------------------------------------------------------------------------------ DESCRIPTION: The third alpha version of syslog-ng Open Source Edition (3.6.0alpha3) has been released. This being an alpha release, testing is most welcome, but production use is not recommended! CHANGES: 3.6.0alpha3 Released: Tue, 19 Aug 2014 14:33:00 +0200 This is the third alpha release of the upcoming syslog-ng OSE 3.6 branch. It is expected to be the last alpha release, with the first beta in about two weeks. This release contains a number of important features and bugfixes: Changed defaults ---------------- * The `flush-lines()` setting now defaults to *100*, rather than *1*, for increased speed. Features -------- * The `system()` source will now parse `@cim` marked messages as JSON, if the JSON module is available at run-time. This improves inter-operation with other software that uses the *Common Information Model*. ### Features from the [Incubator][incubator] * The `$(or)` template function that returns the first non-empty argument is now included in syslog-ng itself. * The `$(padding)` template function, to pad text with custom padding to a given length is also included. * The `$(graphite-output)` template function, to be used for sending metrics to [Graphite][graphite] was ported over from the Incubator. The `graphite()` destination SCL block is also available now, to make it even easier to talk to Graphite. * The `riemann()` destination, which allows sending metrics to the [Riemann][riemann] monitoring system was also ported over from the Incubator. [graphite]: http://graphite.wikidot.com/ [incubator]: https://github.com/balabit/syslog-ng-incubator [riemann]: http://riemann.io/ ### Threaded destinations A number of features were implemented for all threaded destinations: `amqp()`, `mongodb()`, `redis()`, `riemann()`, `smtp()` and `stomp()`. * The destinations gained support for `SEQNUM` persistence: the counter will be preserved across reloads and restarts. * A new option called `retries()` was implemented for all of these, which controls how many times a message delivery is retried before dropping it. * The `throttle()` option is now implemented, and works for all of the aforementioned destination drivers. * The message delivery loop was optimised to do less sleep/wakeup cycles, which should make the drivers not only faster, but more CPU friendly too. Bugfixes -------- * The basicfuncs module was fixed to work correctly on 32-bit architectures. * The `stored` statistics is no longer incremented by various drivers when they mean `processed`. * The type hinting feature is now more picky about what kind of type hints it accepts, allowing one to use template functions in - for example - `$(format-json)` pairs. Miscellaneous changes --------------------- * We now ship a "Contributors Guide" in the `CONTRIBUTING.md` file. Credits ------- syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessary to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Balazs Scheidler, Fabien Wernli, Gergely Nagy, Laszlo Budai, Peter Czanik, Robert Fekete, Tibor Benke, Viktor Juhasz, Viktor Tusa. DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.6.0a… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

Recommended Data base
by VMI X 19 Aug '14

19 Aug '14

Hi, Currently we're logging everything to text files for a few LAN clients. We're considering using a database instead and have a few questions to help us decide: - Would a database be a good option to replace existing text files for long term storage considering storage space? - Would mongo OR mysql be better suited for storing system logs? I understand answers to these questions can vary depending on specific use case but seeking a general recommendation to see what's typically being used and what the most stable/supported options would be. -- *Nullius In Verba*

5 6

Syslog-ng log rotation problem
by Riyas Ahamed 19 Aug '14

19 Aug '14

Hi Team, I have configured syslog-ng 3.2.5 with backend as mysql Ver 14.14 Distrib 5.6.20, I have able to fetch logs from the backend to frondend. Please help me to how to rotate syslog-ng logs on daily basis. Regards, N.B.RIAZ AHMED https://www.csscorp.com/email-disclaimer

1 0

Writing a null character at the end of TCP payload
by Nicolas Fédou 15 Aug '14

15 Aug '14

Hello, I need to send a null character at the end of a message on a tcp output... But this null is considered to be an end of string. My use case is sending a GELF message to a graylog's tcp input. GELF message is a json syntax quite simple to comply with thanks to format-json. Graylog needs a null character at the end of a GELF message in TCP but not in UDP. I tried many combination in the template with \x0, echo \x0, `global_definition_of_null`, etc... The matter is that null character when interpreted is an end of string, and is not written in the tcp message. I see truncated messages with ngrep. Like "a\x0b" gives "a" according to ngrep. I did not find any option to actually write a null character. Do you have any options ? As Graylog already have clients and librairies I belive they won't remove the need for a null character. Shall I open an issue on syslog-ng to ask for an option in templates or in format-json ? Regards, Nicolas Fédou.

3 8

insider 2014-08: EPEL; graphite; PCI DSS;
by Czanik, Péter 14 Aug '14

14 Aug '14

Dear syslog-ng users, This is the 36th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. FEATURED NEWS EPEL 7 now contains syslog-ng ----------------------------- RHEL 7 was released over a month ago and CentOS 7 not much later, but one piece of software was still missing: syslog-ng. Not any more. EPEL, which stands for Extra Packages for Enterprise Linux, is a software collection containing additional packages for Enterprise Linux and derivatives. Now its latest version, EPEL 7 also contains syslog-ng, version 3.5: http://czanik.blogs.balabit.com/2014/07/epel-7-now-contains-syslog-ng/ Introducing syslog-ng PE 5F1 ---------------------------- The latest version of the syslog-ng Premium Edition, 5F1, adds support for the popular NoSQL database MongoDB. Along with support for MongoDB, we have added support for Java Script Object Notation (JSON), a text-based open standard designed for human-readable data interchange. You can read more at http://gyp.blogs.balabit.com/2014/08/introducing-syslog-ng-premium-edition-… Performance monitoring using syslog-ng and graphite --------------------------------------------------- For most of its history, syslog-ng could only be used for collecting, processing and storing log messages. Not any more. The Redis and Riemann destinations are already a step into the direction of metrics-based monitoring, and the monitoring source combined with Graphite template support are the next. https://czanik.blogs.balabit.com/2014/07/how-to-setup-syslog-ng-quickly-for… Introducing syslog-ng store box 3F2 ----------------------------------- We recently released a new version of our log management appliance, the syslog-ng Store Box. 3F2 is the latest feature release and includes one major new feature and a major improvement to an existing one. First, we have added a RESTful API which opens up all sorts of possibilities for accessing log data in SSB. Second, we have revamped the search interface on the web-based user interface making searching and troubleshooting much easier. You can read more at http://gyp.blogs.balabit.com/2014/08/syslog-ng-store-box-3-f2-released/ Syslog-ng incubator 0.3.3 released ---------------------------------- The syslog-ng incubator is a set of tools and modules for syslog-ng, which are not (yet) available in the official release. This version of incubator works with the latest stable syslog-ng (v3.5.5+) and fixes many problems of the initial 0.3 incubator release. https://czanik.blogs.balabit.com/2014/07/syslog-ng-incubator-0-3-3-is-avail… Log management and the Verizon 2014 PCI Compliance Report --------------------------------------------------------- Recently, the eagerly anticipated Verizon Data Breach Investigations Report for 2014 was published. With more than 63,000 security incidents, 1,300 confirmed data breaches and 50 contributing global organizations, it provides the most comprehensive insight to state of IT security around the world. Drawing on data from the Data Breach Investigation Report, Verizon also publishes a lesser known but very interesting report on the state of compliance of with the Payment Card Industry Data Security Standard (PCI DSS), perhaps the most widely-adopted security standards globally. Read, what requirements PCI DSS has towards log management: https://jluby.blogs.balabit.com/2014/06/30/key-log-management-takeaways-fro… NEW RELEASES syslog-ng OSE 3.5.6: https://lists.balabit.hu/pipermail/syslog-ng-announce/2014-August/000192.ht… syslog-ng OSE 3.6 alpha2: https://lists.balabit.hu/pipermail/syslog-ng-announce/2014-August/000193.ht… syslog-ng PE 5LTS (5.0.5a): https://lists.balabit.hu/pipermail/syslog-ng-announce/2014-June/000189.html syslog-ng incubator 0.3.3 Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

connection broken every few minutes
by Simão Mata 13 Aug '14

13 Aug '14

Hello, Our syslog-ng clients are losing their connection to our central log server every few minutes. In the client I have the following log: 2014-08-12T15:44:02+00:00 wimdu-app07 syslog.notice syslog-ng[14032]: Syslog connection broken; fd='10', server='AF_INET(<our ip>:514)', time_reopen='1' 2014-08-12T15:44:03+00:00 wimdu-app07 syslog.notice syslog-ng[14032]: Syslog connection established; fd='10', server='AF_INET(<our ip>:514)', local='AF_INET(0.0.0.0:0)' I am using the following configuration on the clients: destination d_net_wimdu { syslog(<our ip> transport("tcp") port(514) log-fifo-size(2048)); }; log { source(s_src); source(s_net); destination(d_net_wimdu); }; And on the server: source s_net { syslog(ip(0.0.0.0) transport("tcp") max-connections(300) log-iw-size(3300) so_rcvbuf(8000000000) port(514)); }; log { source(s_net); destination(d_file_w); }; destination d_prog_logparser { program("< logparser_bin >" template(t_w_filetemplate) log_fifo_size(90000000) flags(no-multi-line)); }; log { source(s_net); destination(d_prog_logparser); flags(flow-control); }; Any idea how I can even debug this? Thank you -- [image: Wimdu] <http://www.wimdu.com/>*Wimdu GmbH* - Voltastraße 5, 13355 Berlin, Germany *Managing Directors* - Arne Bleckwenn, Hinrich Dreiling *Commercial Register Number* - 129773 B Berlin[image: Contact] <contact(a)wimdu.com>[image: Blog] <http://blog.wimdu.com/?wt_vi=signature>[image: FB] <http://www.facebook.com/wimdu>[image: TW] <http://www.twitter.com/wimdu>[image: G+] <http://www.google.com/+wimdu>[image: YT] <http://www.youtube.com/wimtubechannel>[image: GH] <https://github.com/wimdu>[image: LI] <http://www.linkedin.com/company/wimdu-gmbh>[image: XI] <http://www.xing.com/companies/wimdu>

3 2

syslog-ng 3.6.0alpha2 has been released
by devel＠balabit.hu 12 Aug '14

12 Aug '14

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.6.0alpha2 SUMMARY : new alpha release DATE : Aug 11, 2014 ------------------------------------------------------------------------------ DESCRIPTION: The second alpha version of syslog-ng Open Source Edition (3.6.0alpha2) has been released. This being an alpha release, testing is most welcome, but production use is not recommended! CHANGES: 3.6.0alpha2 Released: Mon, 11 Aug 2014 15:02:59 +0200 This is the second alpha release of the upcoming syslog-ng OSE 3.6 branch, with more internal changes and features compared to the previous 3.6.0alpha1 release. Above the changes in the latest 3.5.6 stable version, this release contains the following noteworthy changes: Changed defaults ---------------- * Threaded mode is now **enabled** by default. To turn it off, use `threaded(no)` in the global options section. * The versioning of the `libsyslog-ng` internal library has changed: instead of always using the current release number, we will now try to maintain ABI compatibility during the lifetime of a stable branch. Therefore, we use only the first two components of our version as the base of the library version. Another number will be part of the SONAME too, but that will only change when we break compatibility. The SONAME is currently set to `libsyslog-ng-3.6.so.0`, and will remain the same during alpha and beta releases, even when the ABI changes. We will start bumping the version after the first stable release from this branch, if needed. Features -------- * The new `systemd-syslog()` source replaces the former implicit support for the same thing. Users who use systemd are advised to use either the `system()` source, or this new one when they want to receive logs from systemd via the `/run/systemd/journal/syslog` socket. ### SMTP destination changes * The `smtp()` destination now supports a `retries()` option, which controls how many times a message delivery will be attempted before dropping it. * The destination no longer counts delivered messages as `stored`. * The templates used in the destination now honor the time-zone settings. * The driver will abort if required options (any of `to()`, `cc()`, `bcc()` and `from()`, and `subject()` and `body()`) are not set. Bugfixes -------- * The `file()` and `network()` (including `tcp()` et al) sources will now properly set the `$SOURCE` macro. * To build syslog-ng with the MongoDB destination, libmongo-client version 0.1.8+ is now required. (The internal copy has been updated accordingly.) * The UNIX credential extracting feature was ported to FreeBSD, syslog-ng now compiles, and has support for this feature. (The previous support in alpha1 was sadly incomplete.) * The `pdbtool merge` command will now generate version 4 patterndb files. Credits ------- syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessary to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Fabien Wernli, Gergely Nagy, Laszlo Budai, Peter Czanik, Tibor Benke, Viktor Juhasz, Viktor Tusa. DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.6.0a… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

2 1

syslog-ng redis destination problem
by jrhendri＠roadrunner.com 12 Aug '14

12 Aug '14

Hi (again) - This one has me a bit puzzled. I am trying to write to redis using the example from the syslog-ng documentation: destination d_local_redis { redis(command("HINCRBY" "${HOST}/programs" "${PROGRAM}" "1")); }; but I am getting "-ERR value is not an integer or out of range" I have tried this on a couple systems. This particular one is: [n0142566@VDDP13E-F1A47ED src]$ /usr/local/sbin/syslog-ng --version syslog-ng 3.5.4.1 Installer-Version: 3.5.4.1 Revision: Compile-Date: Jul 22 2014 12:30:06 Available-Modules: afsocket-notls,afprog,csvparser,system-source,dbparser,redis,syslogformat,afuser,afstomp,afmongodb,linux-kmsg-format,json-plugin,basicfuncs,afamqp,affile,cryptofuncs,afsocket,confgen Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off Enable-Pcre: off [n0142566@VDDP13E-F1A47ED src]$ And I have captured the packets to see what is being sent and see this following the TCP stream with Wireshark (pcap attached) *1 $4 ping +PONG *4 $7 HINCRBY $24 VDDP13E-F1A47ED/programs $16 ArcSightLogs_PDC $2 * -ERR value is not an integer or out of range Note: when I use ("INCR" "testkey") it works - seems like the integer increment is not being sent correctly? Is this something I am getting wrong or is it a bug ? Thanks (again) Jim

1 1

How to log messages generated by C program and log file rotation
by Jean Faye 07 Aug '14

07 Aug '14

confirm a6d843f7ad7d1dbf1fefb6f12432e54941a680a9 Hi all, I want to use syslog-ng to log the messages generated by my application implemented in C language. I added this in the code: char *log="rtcd"; printf("[%s] RTC adjustement\n",__func__); openlog(log, LOG_PID, LOG_LOCAL0); syslog(LOG_DEBUG, "[FIJ] RTC adjustement"); closelog(); For me, according to the syslog-ng file, the files /var/log/ldb/GENTrace.log, /var/log/ldb/SUTrace.log, /var/log/ldb/WANTrace.log and /var/log/ldb/CPLTrace.log must be created and must contain the syslog message. But I got no messages in my destination files. You can see the content of my syslog-ng.conf file bellow. Is it the right way to log the messages sent by C program? What can explain that I got no messages in the destinations files? Concerning log file rotation, How can we manage it using syslog-ng? For example I want to have a destination file with a size maximum = 2Mo and if the size is greater than the max size, I have to save the current one and create a new one. On my system I can have max 4 files (4 x 2Mo). How can I manage this kind of rotation? Thanks in advance. Best regards, Ismael Jean FAYE @version: 3.5 # # Syslog-ng configuration file, compatible with default Debian syslogd # installation. Originally written by anonymous (I can't find his name) # Revised, and rewrited by me (SZALAY Attila <sasa(a)debian.org>) # First, set some global options. options { chain_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no); owner("root"); group("adm"); perm(0640); stats_freq(0); bad_hostname("^gconfd$");create_dirs(yes); }; ######################## # Sources ######################## source s_mysource { pipe("/tmp/pipe" pad_size(2048)); }; ######################## # Destinations ######################## destination d_GEN { file("/var/log/ldb/GENTrace.log"); }; destination d_SU { file("/var/log/ldb/SUTrace.log"); }; destination d_WAN { file("/var/log/ldb/WANTrace.log"); }; destination d_CPL { file("/var/log/ldb/CPLTrace.log"); }; ######################## # Filters ######################## #filter f_GEN { #facility(local0) and filter(nom_du_composant_applicatif); #facility(local0) and filter(f_debug); #facility(local0); #}; filter f_SU { #facility(local0) and filter(nom_du_composant_applicatif); #facility(local0) and filter(f_debug); facility(local0); }; filter f_WAN { #facility(local0) and filter(nom_du_composant_applicatif); #facility(local0) and filter(f_debug); facility(local0); }; filter f_CPL { #facility(local0) and filter(nom_du_composant_applicatif); #facility(local0) and filter(f_debug); facility(local0); }; ######################## # Log paths ######################## #log { source(s_mysource); filter(f_GEN); destination(d_GEN); }; log { source(s_mysource); filter(f_SU); destination(d_SU); }; log { source(s_mysource); filter(f_WAN); destination(d_WAN); }; log { source(s_mysource); filter(f_CPL); destination(d_CPL); };

3 5

Re: [syslog-ng] FW: Syslog Problem
by Jim Hendrick 06 Aug '14

06 Aug '14

I agree with Evan. You need to know which events are actually being received. The simplified configuration I sent should provide that and the you can add back the filters and database destination. Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Evan Rempel <erempel(a)uvic.ca> Date:08/06/2014 11:22 AM (GMT-05:00) To: syslog-ng(a)lists.balabit.hu Subject: Re: [syslog-ng] FW: Syslog Problem > Could you please tell me how to check under load mysql silently drop records There is no record that this occurs. You need to have an authoritative source of what should be in the database, and then look to see what is actually in the database. > And could you please tell me what is the solution for this problem? To be honest, I wasn't very interested in the solution. The database should not have don that to begin with. Depending on load and version of mysql the problem would go away, or return. I never found a solution because I didn't look to hard for one. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

FW: Syslog Problem
by Riyas Ahamed 06 Aug '14

06 Aug '14

Hi, I have configured a syslog-ng server to capture network logs. I can able to fetch network logs of three devices. But in fourth network device I can able to see packets are get polling into the syslog server in port 514 by using tcpdump command but I cannot find the network device in syslogng front end. Syslog OS : CentOS release 6.5 (Final) Syslog Version : syslog-ng 3.2.5 Along with this mail I have attached configuration file of syslogng server. Please help me to resolve this issue. Regards, N.B.RIAZ AHMED (9047166496) https://www.csscorp.com/email-disclaimer

4 8

syslog-ng 3.5.6 has been released
by devel＠balabit.hu 05 Aug '14

05 Aug '14

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.5.6 SUMMARY : new stable release DATE : Aug 5, 2014 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.5.6) has been released. For latest fixes in the 3.5.x feature branch you are recommended to upgrade to this version. CHANGES: 3.5.6 Mon, 4 Aug 2014 13:39:42 +0200 This is the sixth bug-fix release for the 3.5.x series. Upgrading from earlier versions is highly recommended, as the changes in this release are very small, yet, also very important for most platforms and workloads. Bugfixes ======== * A major memory leak was fixed in the value-pairs framework, which affects $(format-json), MongoDB, AMQP and more. The leak was supposed to be fixed in 3.5.5, but due to a merging mistake, it was missed. * The configure script now detects Linux capabilities properly, there is no need to use --disable-linux-caps on non-Linux platforms anymore. * The pdbtool merge command will now generate version 4 patterndb files, instead of version 3. Credits ======= syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessary to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Fabien Wernli Gergely Nagy Peter Czanik DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.5.6 The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

Syslog Problem
by Riyas Ahamed 05 Aug '14

05 Aug '14

Hi, I have configured a syslog-ng server to capture network logs. I can able to fetch network logs of three devices. But in fourth network device I can able to see packets are get polling into the syslog server in port 514 by using tcpdump command but I cannot find the network device in syslogng front end. Syslog OS : CentOS release 6.5 (Final) Syslog Version : syslog-ng 3.2.5 Along with this mail I have attached configuration file of syslogng server. Please help me to resolve this issue. Regards, N.B.RIAZ AHMED (9047166496) https://www.csscorp.com/email-disclaimer

1 0