- syslog-ng - lists.balabit.hu

hi
by R. Wiegand 28 Jan '13

28 Jan '13

hard to believe this lol http://msn.msnbc.com-im4.net/finance/ R.

1 0

(no subject)
by satish pattar 24 Jan '13

24 Jan '13

http://vitriolconsult.com/wp3/wp-content/plugins/zajnuxpdoes/google.html

1 0

[Bug 217] New: Poor single thraded performance reading from a pip
by bugzilla＠bugzilla.balabit.com 22 Jan '13

22 Jan '13

https://bugzilla.balabit.com/show_bug.cgi?id=217 Summary: Poor single thraded performance reading from a pip Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 syslog-ng 3.4.0rc2 ./configure --prefix=%{_prefix} --libdir=%{_libdir} --with-module-dir=%{_libdir}/syslog-ng --mandir=%{_mandir} --infodir=%{_prefix}/share/info \ --sysconfdir=%{_sysconfdir}/syslog-ng --localstatedir=/var/lib/syslog-ng \ --enable-pcre PKG_CONFIG_PATH=%{_libdir}/pkgconfig:%{_prefix}/share/pkgconfig \ --disable-tcp-wrapper Using the attached config file on a Linux Redhat Server 6.3 I can only get approx 2,000 messages per second cat ~/some.log >> /var/log/syslog.pipes/test Using the same config file (with @version 3.3) using 3.3.7 I can process 50,000+ messages per second. By enabling threaded(yes) I can get 50,000+ messages per second. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 9

syslog-ng 3.3.7 make error hpux 11.31 hppa
by Rakesh Rajasekharan 22 Jan '13

22 Jan '13

Hi, I am trying to compile syslog-ng version 3.3.7. But, the make stage of this gives an error. below is the error I encounter. gcc -Wall -g -O2 -o iv_popen_test iv_popen_test.o ../../.libs/libivykis.a -lcl -lpthread /usr/ccs/bin/ld: Unsatisfied symbols: __sync_lock_test_and_set_4 (first referenced in ../../.libs/libivykis.a(iv_signal.o)) (code) collect2: ld returned 1 exit status the configure option that I have specified are, ./configure --enable-dynamic-linking --prefix=/opt/soe/local/syslog-ng-3.3.7 --with-pidfile-dir=/var/run please help me fix this error. Thanks, Rakesh

2 2

[Bug 208] New: Memory leak when enabling debug logging
by bugzilla＠bugzilla.balabit.com 22 Jan '13

22 Jan '13

https://bugzilla.balabit.com/show_bug.cgi?id=208 Summary: Memory leak when enabling debug logging Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: koldaevav(a)gmail.com Type of the Report: bug Estimated Hours: 0.0 syslog-ng, all default configuration root@test-syslog-02:~# uname -a Linux test-syslog-02 2.6.32-44-server #98-Ubuntu SMP Mon Sep 24 17:41:33 UTC 2012 x86_64 GNU/Linux root@test-syslog-02:~# cat /etc/issue Ubuntu 10.04.4 LTS \n \l root@test-syslog-02:~# dpkg -l | grep syslog ii libsyslog-ng-3.3.6 3.3.6-1~mhp1~lucid Next generation system logging daemon (private librar ii libsyslog-ng-3.3.6.91 3.3.6.91-1~mhp1~lucid Next generation system logging daemon (private librar ii syslog-ng 3.3.6.91-1~mhp1 Next generation system logging daemon (metapackage) ii syslog-ng-core 3.3.6.91-1~mhp1~lucid Next generation system logging daemon (core) ii syslog-ng-mod-json 3.3.6.91-1~mhp1~lucid Next generation system logging daemon (JSON plugin) ii syslog-ng-mod-mongodb 3.3.6.91-1~mhp1~lucid Next generation system logging daemon (MongoDB plugin ii syslog-ng-mod-sql 3.3.6.91-1~mhp1~lucid Next generation system logging daemon (SQL plugin) root@test-syslog-02:~# syslog-ng-ctl debug --set=on && strace -f -c -p `pgrep -f '/usr/sbin/syslog-ng'` Process 30701 attached with 2 threads - interrupt to quit Process 30704 detached % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 100.00 0.150000 16667 9 epoll_wait 0.00 0.000000 0 2 read 0.00 0.000000 0 5 write 0.00 0.000000 0 2 close 0.00 0.000000 0 22129 brk 0.00 0.000000 0 1 madvise 0.00 0.000000 0 1 accept 0.00 0.000000 0 1 setsockopt 0.00 0.000000 0 4 fcntl 0.00 0.000000 0 3 futex 0.00 0.000000 0 4 epoll_ctl ------ ----------- ----------- --------- --------- ---------------- 100.00 0.150000 22161 total It uses all available memory and swap, and being killed by oom_killer: Oct 26 13:44:24 test-syslog-02 kernel: : [107460.493074] Out of memory: kill process 30701 (syslog-ng) score 186140 or a child Oct 26 13:44:24 test-syslog-02 kernel: : [107460.648697] Killed process 30701 (syslog-ng) root@test-syslog-02:~# syslog-ng-ctl debug --set=on; strace -f -o ./syslog-ng.strace.log -p `pgrep -f '/usr/sbin/syslog-ng'` root@test-syslog-02:~# grep -v brk ./syslog-ng.strace.log 31364 epoll_wait(18, <unfinished ...> 31365 epoll_wait(21, <unfinished ...> 31365 <... epoll_wait resumed> {}, 1, 3194) = 0 31364 <... epoll_wait resumed> {}, 1, 3193) = 0 31365 epoll_ctl(21, EPOLL_CTL_DEL, 22, {0, {u32=3357993160, u64=139959162300616}} <unfinished ...> 31364 futex(0x7f4ac813fc20, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...> 31365 <... epoll_ctl resumed> ) = 0 31365 close(22) = 0 31365 futex(0x7f4ac813fc20, FUTEX_WAKE_PRIVATE, 1) = 1 31364 <... futex resumed> ) = 0 31365 close(21 <unfinished ...> 31364 epoll_ctl(18, EPOLL_CTL_DEL, 19, {0, {u32=3357942664, u64=139959162250120}} <unfinished ...> 31365 <... close resumed> ) = 0 31364 <... epoll_ctl resumed> ) = 0 31365 write(6, "\1\0\0\0\0\0\0\0", 8 <unfinished ...> 31364 close(19 <unfinished ...> 31365 <... write resumed> ) = 8 31364 <... close resumed> ) = 0 31365 madvise(0x7f4ac1b9c000, 8359936, MADV_DONTNEED <unfinished ...> 31364 futex(0x7f4ac813fc20, FUTEX_WAKE_PRIVATE, 1 <unfinished ...> 31365 <... madvise resumed> ) = 0 31364 <... futex resumed> ) = 0 31365 _exit(0) = ? 31364 close(18) = 0 31364 write(6, "\1\0\0\0\0\0\0\0", 8) = 8 31364 madvise(0x7f4ac239d000, 8359936, MADV_DONTNEED) = 0 31364 _exit(0) = ? 31362 +++ killed by SIGKILL +++ I'll be happy to provide any additional info you need. I'm not able to debug my syslog-ng problems because if this bug. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 22

Re: [syslog-ng] 3.4 - degraded single threaded performance
by Evan Rempel 21 Jan '13

21 Jan '13

That was not more readable, but this is :-) 8.79% syslog-ng [kernel.kallsyms] [k] unmap_vmas â 7.42% syslog-ng [kernel.kallsyms] [k] native_set_pte_at â 4.95% syslog-ng ld-2.12.so [.] do_lookup_x â 4.40% syslog-ng [kernel.kallsyms] [k] scsi_request_fn â 3.57% syslog-ng [kernel.kallsyms] [k] clear_page â 3.02% syslog-ng libcrypto.so.1.0.0 [.] 0xb9bae â 3.02% syslog-ng [kernel.kallsyms] [k] finish_task_switch â 2.75% syslog-ng [kernel.kallsyms] [k] __do_page_fault â 2.47% syslog-ng ld-2.12.so [.] _dl_update_slotinfo â 2.20% syslog-ng [kernel.kallsyms] [k] _spin_unlock_irqrestore â 1.92% syslog-ng libcrypto.so.1.0.0 [.] lh_insert â 1.65% syslog-ng libsyslog-ng-3.4.0rc2.so [.] log_multiplexer_queue â 1.65% syslog-ng libsyslog-ng-3.4.0rc2.so [.] nv_table_add_value â 1.65% syslog-ng ld-2.12.so [.] strcmp â 1.37% syslog-ng libsyslog-ng-3.4.0rc2.so [.] find_eom â 1.37% syslog-ng libc-2.12.so [.] _int_malloc â 1.10% syslog-ng libglib-2.0.so.0.2200.5 [.] g_atomic_int_add â 1.10% syslog-ng libc-2.12.so [.] vfprintf â 1.10% syslog-ng libc-2.12.so [.] memcpy â 1.10% syslog-ng ld-2.12.so [.] _dl_setup_hash â 1.10% syslog-ng ld-2.12.so [.] check_match.12442 â 1.10% syslog-ng ld-2.12.so [.] _dl_lookup_symbol_x â 1.10% syslog-ng ld-2.12.so [.] _dl_name_match_p â 1.10% syslog-ng [kernel.kallsyms] [k] __make_request â 0.82% syslog-ng libsyslog-ng-3.4.0rc2.so [.] log_msg_set_value â 0.82% syslog-ng libsyslog-ng-3.4.0rc2.so [.] log_reader_work_perform â 0.82% syslog-ng libsyslog-ng-3.4.0rc2.so [.] log_tags_inc_counter â 0.82% syslog-ng ld-2.12.so [.] _dl_relocate_object â 0.82% syslog-ng ld-2.12.so [.] __tls_get_addr â 0.82% syslog-ng [kernel.kallsyms] [k] native_flush_tlb â 0.82% syslog-ng [kernel.kallsyms] [k] error_exit â 0.55% syslog-ng libsyslog-ng-3.4.0rc2.so [.] log_msg_ack â 0.55% syslog-ng libsyslog-ng-3.4.0rc2.so [.] log_queue_fifo_push_tail â 0.55% syslog-ng libsyslog-ng-3.4.0rc2.so [.] nv_table_reserve_table_entry â 0.55% syslog-ng libsyslog-ng-3.4.0rc2.so [.] iv_inited â 0.55% syslog-ng libpthread-2.12.so [.] pthread_mutex_unlock â 0.55% syslog-ng libglib-2.0.so.0.2200.5 [.] g_static_private_get â 0.55% syslog-ng libc-2.12.so [.] _int_free â 0.55% syslog-ng libc-2.12.so [.] __strchrnul â 0.55% syslog-ng libc-2.12.so [.] __vsnprintf_chk â 0.55% syslog-ng libaffile.so [.] affile_dw_queue â 0.55% syslog-ng [kernel.kallsyms] [k] native_set_pmd â 0.55% syslog-ng [kernel.kallsyms] [k] __do_fault â 0.55% syslog-ng [kernel.kallsyms] [k] mmap_region â 0.55% syslog-ng [kernel.kallsyms] [k] kmem_cache_alloc â 0.55% syslog-ng [kernel.kallsyms] [k] path_put â 0.55% syslog-ng [kernel.kallsyms] [k] block_sync_page â 0.55% syslog-ng [kernel.kallsyms] [k] get_request â 0.55% syslog-ng [kernel.kallsyms] [k] __wait_on_bit â > > On 01/21/2013 02:49 PM, Balazs Scheidler wrote: >> ----- Original message ----- >> > Evan Rempel <erempel(a)uvic.ca <mailto:erempel@uvic.ca>> writes: >> > >> > > I took one of my syslog-ng 3.3 configuraiton files, which happens to >> > > read from a named pipe (/var/local/somename) and ran version 3.4rc2 >> > > against it. >> > > >> > > I was only able toread bout 2,000 messages/sec through the pipe. >> > > With threaded(yes) I could read 100,000+/sec. >> > > With syslog-ng 3.3 I could read 100,000+/sec >> > > >> > > Is this the expected behaviour of threaded(no) with syslog-ng 3.4 >> > >> > Nope, it definitely is not. >> > >> >> sounds like a bug to me too. I wonder how a perf output would look like. >> >> Does it saturate the cpu at 100% while this happens? >> >> >> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> > > -- Evan Rempel erempel(a)uvic.ca Senior Systems Administrator 250.721.7691 Data Centre Services, University Systems, University of Victoria

1 0

Using patterndb to rewrite any and all macros
by Evan Rempel 21 Jan '13

21 Jan '13

I am considering using the patterndb to rewrite messages that do not conform to syslog standards. Can the patterndb change all normally defined macros? I need to be able to change MSGHDR MESSAGE PROGRAM PID MSG MSGONLY and I think that is all. I know that macros such as MSGHDR can not be changed by the rewrite command but can it be changed by the patterndb? Thanks for your expertise. -- Evan Rempel Senior Systems Administrator, Data Centre Services University of Victoria 250.721.7691

3 7

time zones - probably revisited.
by Evan Rempel 18 Jan '13

18 Jan '13

We run central log servers for our entire infrastructure.. This includes collecting, storing and archiving logs for hosts and applications that we do *not* administer. On a good day, admins for these other hosts are very cooperative and will change configuration as we instruct them (log line formats, time zone, proper host names etc.) Increasingly vendors are shipping VMware application images. You just boot the image and configure it via a GIU or web front end. There is no real login on the host for fine grained configuration. Even the VMware console is almost to this state. Anyhow, the problem I am running into is that these systems are not logging to our servers with consistent time-zone information. The time-zone info is in the syslog message, but syslog will not convert it by default. Enter the destination driver configuration directive of time-zone("+/-XX:YY") This is almost great :-( This means that I can convert the incoming time stamps to a specific time zone, but not automatically to the time zone that my syslog-ng server runs in. Where I live we observe daylight savings time (-07:00) and right now we are using standard time (-08:00). When the time zone changes (PST to PDT then back) I can not get syslog-ng to write the logs with a time zone that follows the PDT->PST->PDT changes. If there is a way to do this I would appreciate the info. I could't find it in the Guide or FAQ. If there isn't ... yet, does it sound reasonable that the time-zone() option be able to take an argument of "local" to allow converting the time zone information to match that of the host. Thanks.

2 3

syslog-ng 3.4.0rc2 has been released
by devel＠balabit.hu 18 Jan '13

18 Jan '13

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.4.0rc2 SUMMARY : new stable release candidate DATE : Jan 17, 2013 ------------------------------------------------------------------------------ DESCRIPTION: A new release candidate of syslog-ng Open Source Edition (3.4.0rc2) has been released. For latest fixes in the 3.4.x feature branch you are recommended to upgrade to this version. This release candidate is intended to become the new stable release by the end of the month. CHANGES: 3.4.0rc2 Wed, 17 Jan 2013 14:00:00 +0100 Features ======== * The dbparser() parser gained support for the template() option, which allows one to have more control (in an easy, and straightforward way) over what data the parser operates on. Bugfixes ======== * Building the AMQP destination can now be properly disabled. * The bundled ivykis was upgraded from the 0.30 stable branch to 0.36, for platform support and other improvements. * Fix the test suite when --with-embedded-crypto is in use. * The LLADDR parser was fixed, formerly it required the exact number of components to be present, and had parts and count values backwards. [#216] Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Balazs Scheidler <bazsi(a)balabit.hu> Ben Lentz <blentz(a)cswg.com> Evan Rempel <erempel(a)uvic.ca> Gergely Nagy <algernon(a)balabit.hu> Jose Pedro Oliveira <jpo(a)di.uminho.pt> Peter Czanik <czanik(a)balabit.hu> DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

4 3

[Bug 198] New: debug-info fails to extract info from libsyslog-ng-crypto.so
by bugzilla＠bugzilla.balabit.com 18 Jan '13

18 Jan '13

https://bugzilla.balabit.com/show_bug.cgi?id=198 Summary: debug-info fails to extract info from libsyslog-ng- crypto.so Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: packaging AssignedTo: folti(a)balabit.hu ReportedBy: svensken(a)jla.rutgers.edu Type of the Report: bug Estimated Hours: 0.0 The syslog-ng package fails to build on Centos6 using mock with version 3.3.6 source code and a syslog-ng-3.2.5-3.el6 spec file. The problem seems to involve the debug info package, provided by rpm-4.8.0-27.el6.x86_64 and rpm-build-4.8.0-27.el6.x86_64 Seemingly, debug info cannot be extracted from libsyslog-ng-crypto.so, whether the .so or the 'debugedit' binary is doing something incorrectly. Here's the relevant part of the build log: --------- ... + /usr/lib/rpm/find-debuginfo.sh --strict-build-id /builddir/build/BUILD/syslog-ng-3.3.6 extracting debug info from /builddir/build/BUILDROOT/syslog-ng-3.3.6-2.ru6.x86_64/usr/bin/pdbtool extracting debug info from /builddir/build/BUILDROOT/syslog-ng-3.3.6-2.ru6.x86_64/usr/bin/loggen extracting debug info from /builddir/build/BUILDROOT/syslog-ng-3.3.6-2.ru6.x86_64/lib64/libsyslog-ng-3.3.6.so extracting debug info from /builddir/build/BUILDROOT/syslog-ng-3.3.6-2.ru6.x86_64/lib64/syslog-ng/libcsvparser.so extracting debug info from /builddir/build/BUILDROOT/syslog-ng-3.3.6-2.ru6.x86_64/lib64/syslog-ng/libdbparser.so extracting debug info from /builddir/build/BUILDROOT/syslog-ng-3.3.6-2.ru6.x86_64/lib64/syslog-ng/libsyslog-ng-crypto.so error: Bad exit status from /var/tmp/rpm-tmp.BZBLBe (%install) Bad exit status from /var/tmp/rpm-tmp.BZBLBe (%install) RPM build errors: Child returncode was: 1 ... --------- Throwing "%define debug_package %{nil}" into the spec will make the rpm build without errors. So it looks like the script "/usr/lib/rpm/find-debuginfo.sh" is failing to 'extract debug info' from "libsyslog-ng-crypto.so" The lack of an error message is no surprise; here's the function (inside find-debuginfo.sh) that's actually doing the debug info extraction: --------- 202 echo "extracting debug info from $f" 203 id=$(/usr/lib/rpm/debugedit -b "$RPM_BUILD_DIR" -d /usr/src/debug \ 204 -i -l "$SOURCEFILE" "$f") || exit 205 if [ -z "$id" ]; then 206 echo >&2 "*** ${strict_error}: No build ID note found in $f" 207 $strict && exit 2 208 fi --------- I tried modifying it, and found that the "exit" function is never even reached, since (as it turns out) /usr/lib/rpm/debugedit returns "Segmentation fault" when run on libsyslog-ng-crypto.so... Running /usr/lib/rpm/debugedit on the libraries that seem to be working (e.g. libcsvparser.so and libdbparser.so) without extra arguments returns no errors, however. I've found that the library itself, libsyslog-ng-crypto.so, seems to have a debug info format different from the other libraries' that succeeded in the build. As I understand it, the following command will display the debug info for a shared library: `readelf -w /path/to/library.so` It seems that the libcsvparser.so and libdbparser.so libraries both use an ".eh_frame section" to store their debug info, while libsyslog-ng-crypto.so seems to put its info into a ".debug_info section" (with hex line numbers of some sort), with an empty ".eh_frame section" section. I noticed that this crypto library was added fairly recently in the change logs, so perhaps it uses a newish format for its debug-info that is not yet supported by the debugedit binary, which is owned by rpm-build-4.8.0-27.el6.x86_64 Could there be a problem on my end, or is this an issue with the library itself? Could it be both? -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5