August 2014 - syslog-ng - lists.balabit.hu

I want to forward Windows server 2008 R2 logs to Linux syslog-ng.
by Jason Long 24 Dec '14

24 Dec '14

Hello all. I have a Windows Box that want to forward all Even logs to my Linux box. I install Snare on Windows(172.30.10.19) and configure it to forward logs to Linux and my Linux box receive it properly. When I use " tcpdump udp "port 514" ", Tcpdump show me that Snare sending Logs to Linux but Syslog-ng can't write it to log files :(. I paste my syslog-ng configure : options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source s_sys { file ("/proc/kmsg" program_override("kernel: ")); unix-stream ("/dev/log"); internal(); #udp(ip(0.0.0.0) port(514)); }; #source s_net { #udp (ip(172.30.10.19) port(514)); #}; source s_net { udp(); }; filter f_openwrt { host("172.30.10.19");}; destination df_openwrt { file("/var/log/winlog/win.log"); }; log { source ( s_net ); filter( f_openwrt ); destination ( df_openwrt ); }; destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" flush_lines(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_kern { file("/var/log/kern"); }; destination d_mlal { usertty("*"); }; filter f_kernel { facility(kern); }; filter f_default { level(info..emerg) and not (facility(mail) or facility(authpriv) or facility(cron)); }; filter f_auth { facility(authpriv); }; filter f_mail { facility(mail); }; filter f_emergency { level(emerg); }; filter f_news { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_boot { facility(local7); }; filter f_cron { facility(cron); }; #log { source(s_sys); filter(f_kernel); destination(d_cons); }; log { source(s_sys); filter(f_kernel); destination(d_kern); }; log { source(s_sys); filter(f_default); destination(d_mesg); }; log { source(s_sys); filter(f_auth); destination(d_auth); }; log { source(s_sys); filter(f_mail); destination(d_mail); }; log { source(s_sys); filter(f_emergency); destination(d_mlal); }; log { source(s_sys); filter(f_news); destination(d_spol); }; log { source(s_sys); filter(f_boot); destination(d_boot); }; log { source(s_sys); filter(f_cron); destination(d_cron); }; # vim:ft=syslog-ng:ai:si:ts=4:sw=4:et: Can you tell me how can I solve this problem? Cheers.

5 8

[Bug 275] New: lib/filter/filter-in-list. c does not compile under Solaris 10
by bugzilla＠bugzilla.balabit.com 28 Aug '14

28 Aug '14

https://bugzilla.balabit.com/show_bug.cgi?id=275 Summary: lib/filter/filter-in-list.c does not compile under Solaris 10 Product: syslog-ng Version: 3.5.x Platform: PC OS/Version: Solaris Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: michael.hocke(a)nyu.edu Type of the Report: bug Estimated Hours: 0.0 Created an attachment (id=93) --> (https://bugzilla.balabit.com/attachment.cgi?id=93) diff for misc.c and misc.h lib/filter/filter-in-list.c uses getline() that is not available in Solaris 10. I worked around it by adding code snippets to lib/misc.h and lib/misc.c that implement getdelim and getline that I took from another project. I was ablt to make it work but I assume it wasn't the most elegant solution and should be integrated into autoconfig etc. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

really need help...not writing files
by Frank Wilkinson 26 Aug '14

26 Aug '14

Please forgive me if this has already been addressed. If so will you point me to it? Syslog-ng will all of a sudden, stop writing files. I'm running syslog-ng 3.5.3 Installer-Version: 3.5.3 Revision: ssh+git://algernon@git.balabit/var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.5#master#ccb05a22408ba4c837d998b2538854d994f845a5 Compile-Date: Jan 8 2014 13:35:02 Available-Modules: afsocket,afprog,dbparser,system-source,affile,syslogformat,linux-kmsg-format,csvparser,afmongodb,afsocket-tls,confgen,afuser,afstomp,afsocket-notls,basicfuncs,cryptofuncs,afamqp Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: off Enable-Pcre: on The service status is showing running but not writing log files. We are logging udp from about 2400 devices When it dies strace shows: epoll_ctl(3, EPOLL_CTL_DEL, 10, {0, {u32=19726648, u64=19726648}}) = 0 write(110, "\1\0\0\0\0\0\0\0", 8) = 8 epoll_wait(3, {{EPOLLIN, {u32=19641320, u64=19641320}}}, 11, 3414) = 1 read(6, "\1\0\0\0\0\0\0\0", 8) = 8 futex(0x7fe1b000bd34, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fe1b000bd30, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1 futex(0x7fe21cfaad68, FUTEX_WAKE_PRIVATE, 1) = 1 fcntl(10, F_GETFD) = 0x1 (flags FD_CLOEXEC) fcntl(10, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK) setsockopt(10, SOL_SOCKET, SO_OOBINLINE, [1], 4) = 0 write(110, "\1\0\0\0\0\0\0\0", 8) = 8 epoll_ctl(3, EPOLL_CTL_ADD, 10, {0, {u32=19726648, u64=19726648}}) = 0 epoll_wait(3, {{EPOLLIN, {u32=19641320, u64=19641320}}}, 12, 3413) = 1 read(6, "\1\0\0\0\0\0\0\0", 8) = 8 futex(0x7fe1b000bd34, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fe1b000bd30, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1 futex(0x7fe21cfaad68, FUTEX_WAKE_PRIVATE, 1) = 1 futex(0x14e25a0, FUTEX_WAKE_PRIVATE, 1) = 1 epoll_wait(3, {{EPOLLIN, {u32=19641320, u64=19641320}}}, 12, 0) = 1 read(6, "\1\0\0\0\0\0\0\0", 8) = 8 futex(0x7fe1b000bd34, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fe1b000bd30, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1 futex(0x7fe21cfaad68, FUTEX_WAKE_PRIVATE, 1) = 1 write(110, "\1\0\0\0\0\0\0\0", 8) = 8 futex(0x12d02c0, FUTEX_WAIT_PRIVATE, 2, NULL) = ? ERESTARTSYS (To be restarted) --- SIGTERM (Terminated) @ 0 (0) --- write(17, "\1\0\0\0\0\0\0\0", 8) = 8 rt_sigreturn(0x7fe21cfab740) = 202 futex(0x12d02c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...> here is where I did a restart +++ killed by SIGKILL +++ top - 12:45:56 up 133 days, 23:11, 13 users, load average: 1.06, 1.13, 1.14 Tasks: 634 total, 2 running, 632 sleeping, 0 stopped, 0 zombie Cpu(s): 4.2%us, 2.0%sy, 0.0%ni, 93.6%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%s Mem: 32898840k total, 31285296k used, 1613544k free, 128188k buffers Swap: 16777212k total, 684800k used, 16092412k free, 29249028k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1631 root 20 0 1725m 104m 2892 S 28.4 0.3 10:38.46 syslog-ng 2843 root 20 0 1725m 104m 2892 S 17.5 0.3 0:04.35 syslog-ng 2795 root 20 0 1725m 104m 2892 S 15.8 0.3 0:11.99 syslog-ng 2842 root 20 0 1725m 104m 2892 S 13.9 0.3 0:02.68 syslog-ng 2793 root 20 0 1725m 104m 2892 S 13.5 0.3 0:14.54 syslog-ng 2855 root 20 0 1725m 104m 2892 R 13.5 0.3 0:00.41 syslog-ng 2776 root 20 0 1725m 104m 2892 S 12.2 0.3 0:18.57 syslog-ng 43203 root 20 0 359m 101m 10m S 11.9 0.3 15:35.10 splunkd 2794 root 20 0 1725m 104m 2892 S 9.6 0.3 0:14.62 syslog-ng 2791 root 20 0 1725m 104m 2892 S 9.2 0.3 0:11.89 syslog-ng 2697 root 20 0 1725m 104m 2892 S 6.3 0.3 0:31.74 syslog-ng 43204 root 20 0 359m 101m 10m S 4.9 0.3 8:01.72 splunkd 2825 root 20 0 1725m 104m 2892 S 2.3 0.3 0:07.73 syslog-ng 2841 root 20 0 1725m 104m 2892 S 1.6 0.3 0:03.30 syslog-ng ... Also, one other problem I have is the syslog-ng log file says: Aug 26 11:48:49 sopher1 syslog-ng[488]: Input is valid utf8, but the log message is not tagged as such, this performs worse than enabling validate-utf8 flag on input; value='758AARULOCAL01' My config specifies flags(validate-utf8): source s_udp { udp( port(514) so_rcvbuf(15000000) log_iw_size(50000) log_msg_size(65535) log_fetch_limit(50000) flags(validate-utf8));}; Frank Wilkinson (205)934-3540 w

1 0

Re: [syslog-ng] Quick set of eyeballs on this
by James Lay 25 Aug '14

25 Aug '14

On 2014-08-25 15:39, Balint Kovacs wrote: > Hi, > > just a guess, but as far as i can remember, the ASA-* part is > usually > in the program field, try the program() filter instead of message() > in > f_firewall. > > Balint > > On 2014. augusztus 25. 22:48:05 CEST, James Lay > <jlay(a)slave-tothe-box.net> wrote: > >> Can anyone see anything blatantly wrong with this? The goal is to >> syslog as usual, but to forward firewall messages to a different >> server. >> Thanks for looking all. >> >> James >> >> @version:3.3.5 >> options { >> use_dns(no); >> flush_lines(0); >> stats_freq(43200); >> }; >> >> source s_local { >> unix-stream("/dev/log"); >> udp(ip(0.0.0.0) port(514)); >> tcp(ip(0.0.0.0) port(514)); >> file("/proc/kmsg"); >> }; >> >> destination d_file { >> file("/var/log/messages"); >> }; >> >> destination d_syslogserver { udp ("x.x.x.x", port(7514)); }; >> >> filter f_syslogfilter { >> not ( >> message("0x0004") >> or message("169.254.") >> or message("192.168.") >> ); >> }; >> >> filter f_firewall { >> message("ASA-4-71005") >> >> or >> message("ASA-2-106100") >> }; >> >> log { >> source(s_local); >> filter(f_syslogfilter); >> destination(d_file); >> }; >> >> log { >> source(s_local); >> filter(f_firewall); >> destination(d_syslogserver); >> }; >> >> ------------------------- Ah thank you...I've just been testing using logger. I'll adjust and try this with program(). Thanks again. James

1 0

Quick set of eyeballs on this
by James Lay 25 Aug '14

25 Aug '14

Can anyone see anything blatantly wrong with this? The goal is to syslog as usual, but to forward firewall messages to a different server. Thanks for looking all. James @version:3.3.5 options { use_dns(no); flush_lines(0); stats_freq(43200); }; source s_local { unix-stream("/dev/log"); udp(ip(0.0.0.0) port(514)); tcp(ip(0.0.0.0) port(514)); file("/proc/kmsg"); }; destination d_file { file("/var/log/messages"); }; destination d_syslogserver { udp ("x.x.x.x", port(7514)); }; filter f_syslogfilter { not ( message("0x0004") or message("169\.254\.") or message("192\.168\.") ); }; filter f_firewall { message("ASA-4-71005") or message("ASA-2-106100") }; log { source(s_local); filter(f_syslogfilter); destination(d_file); }; log { source(s_local); filter(f_firewall); destination(d_syslogserver); };

1 0

Re: [syslog-ng] create_dir and "dynamic" destinations.
by Jim Hendrick 22 Aug '14

22 Aug '14

I would check doing a manual mkdir -p </some/long/path> as the user specified in the config. Then try as root if different. Also look at how that file system is mounted. Is it local disk or NFS ? NFS treats root differently. Also I have seen the permissions of the underlying mount point also cause problems. That is rwxr-x--- for the directory on which the NFS is mounted can cause problems for the mounted filesystem. And you won't see it unless you unmount the NFS piece. Anyway that's what I can think of right off. Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: "AVELING, BEN (BEN)" <ben.aveling(a)alcatel-lucent.com> Date:08/21/2014 6:33 PM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Cc: Subject: Re: [syslog-ng] create_dir and "dynamic" destinations. Permissions problem? Is anything reported in syslog-ng’s own logs? Regards, Ben From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Renato Bezerra Sent: Friday, 22 August 2014 05:51 To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] create_dir and "dynamic" destinations. I use this same config and work very well. Em 21/08/2014 16:44, "Balazs Scheidler" <bazsi77(a)gmail.com> escreveu: Hmm it should create the entire path. On Aug 20, 2014 5:16 PM, "Scot Needy" <scotrn(a)gmail.com> wrote: Yes, I have the create-dies() option enabled but that does not seem to work for upper level directories. Example: Only works if the 192_168_9_0 directory is pre-created. destination d_192_168_9_0 { file(/opt/syslog-ng/logs/192_168_9_0/$YEAR/$MONTH/$DAY/$HOST.log);}; On Aug 20, 2014, at 7:24 AM, Fabien Wernli <wernli(a)in2p3.fr> wrote: > Hi Scot, > > You have the `create-dirs()` option for that: > > http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.5-guid… > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

3 2

create_dir and "dynamic" destinations.
by Scot Needy 21 Aug '14

21 Aug '14

Hi, I was wondering if anyone has tried to work around syslog_ng. My destinations are psudo dynamic in that I have one per subnet but I don’t want to pre-create every destination directory until a syslog message is received for that destination. I have 3 conf files (Dfilters.conf Ddestination.conf and Dlog.conf ) that are built using an API to our IPNMS (IP Network Management System). The purpose is to use syslog-ng stats to quickly report on subnets and also provide per-subnet options for reporting, forwarding … Example Destination destination d_192_168_9_0 { file(/opt/syslog-ng/logs/192_168_9_0/$YEAR$MONTH$DAY.$HOST.log);}; Example Filter filter f_192_168_9_0 { netmask(192.168.9.0/26);}; Example Log log { source(s_net); filter(f_192_168_9_0); destination(d_192_168_9_0);}; My question is can anyone think of a way to get the create_dir option to dynamically create my subnet directories as needed ? For now I Need to pre-create every directory or syslog-ng complains which causes me to create a bunch of empty directories. Thanks Scot

5 5

syslog-ng 3.5 for RHEL6
by Czanik, Péter 19 Aug '14

19 Aug '14

Hi, Recently many people asked for syslog-ng 3.5 rpms for RHEL6. What I did breaks most of the packaging rules, but works. It's a wild mixture of the current EPEL6 and EPEL7 packages. So it has most of the features of the EPEL7 package (except UDP spoofing), but has sysvinit style init script. It's installed to /usr/sbin instead of /sbin, has SSL support enabled, and uses bundled libraries instead of system libraries (as those are missing from RHEL6 and EPEL6). To use the packages, you need to enable EPEL6 repositories for dependencies ( https://fedoraproject.org/wiki/EPEL ) and also my Copr repository: http://copr.fedoraproject.org/coprs/czanik/syslog-ng35epel6/ It's ugly as hell, but survived my quick test on CentOS 6.5. Use at your own risk but let me know if you run into any trouble! Bye, -- Peter Czanik (CzP) <peter.czanik(a)balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

syslog-ng 3.6.0alpha3 has been released
by devel＠balabit.hu 19 Aug '14

19 Aug '14

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.6.0alpha3 SUMMARY : new alpha release DATE : Aug 19, 2014 ------------------------------------------------------------------------------ DESCRIPTION: The third alpha version of syslog-ng Open Source Edition (3.6.0alpha3) has been released. This being an alpha release, testing is most welcome, but production use is not recommended! CHANGES: 3.6.0alpha3 Released: Tue, 19 Aug 2014 14:33:00 +0200 This is the third alpha release of the upcoming syslog-ng OSE 3.6 branch. It is expected to be the last alpha release, with the first beta in about two weeks. This release contains a number of important features and bugfixes: Changed defaults ---------------- * The `flush-lines()` setting now defaults to *100*, rather than *1*, for increased speed. Features -------- * The `system()` source will now parse `@cim` marked messages as JSON, if the JSON module is available at run-time. This improves inter-operation with other software that uses the *Common Information Model*. ### Features from the [Incubator][incubator] * The `$(or)` template function that returns the first non-empty argument is now included in syslog-ng itself. * The `$(padding)` template function, to pad text with custom padding to a given length is also included. * The `$(graphite-output)` template function, to be used for sending metrics to [Graphite][graphite] was ported over from the Incubator. The `graphite()` destination SCL block is also available now, to make it even easier to talk to Graphite. * The `riemann()` destination, which allows sending metrics to the [Riemann][riemann] monitoring system was also ported over from the Incubator. [graphite]: http://graphite.wikidot.com/ [incubator]: https://github.com/balabit/syslog-ng-incubator [riemann]: http://riemann.io/ ### Threaded destinations A number of features were implemented for all threaded destinations: `amqp()`, `mongodb()`, `redis()`, `riemann()`, `smtp()` and `stomp()`. * The destinations gained support for `SEQNUM` persistence: the counter will be preserved across reloads and restarts. * A new option called `retries()` was implemented for all of these, which controls how many times a message delivery is retried before dropping it. * The `throttle()` option is now implemented, and works for all of the aforementioned destination drivers. * The message delivery loop was optimised to do less sleep/wakeup cycles, which should make the drivers not only faster, but more CPU friendly too. Bugfixes -------- * The basicfuncs module was fixed to work correctly on 32-bit architectures. * The `stored` statistics is no longer incremented by various drivers when they mean `processed`. * The type hinting feature is now more picky about what kind of type hints it accepts, allowing one to use template functions in - for example - `$(format-json)` pairs. Miscellaneous changes --------------------- * We now ship a "Contributors Guide" in the `CONTRIBUTING.md` file. Credits ------- syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessary to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Balazs Scheidler, Fabien Wernli, Gergely Nagy, Laszlo Budai, Peter Czanik, Robert Fekete, Tibor Benke, Viktor Juhasz, Viktor Tusa. DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.6.0a… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

Recommended Data base
by VMI X 19 Aug '14

19 Aug '14

Hi, Currently we're logging everything to text files for a few LAN clients. We're considering using a database instead and have a few questions to help us decide: - Would a database be a good option to replace existing text files for long term storage considering storage space? - Would mongo OR mysql be better suited for storing system logs? I understand answers to these questions can vary depending on specific use case but seeking a general recommendation to see what's typically being used and what the most stable/supported options would be. -- *Nullius In Verba*

5 6