- syslog-ng - lists.balabit.hu

syslog-ng 3.4.1 has been released
by devel＠balabit.hu 02 Feb '13

02 Feb '13

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.4.1 SUMMARY : new stable release DATE : Jan 31, 2013 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.4.1) has been released. For latest fixes in the 3.4.x feature branch you are recommended to upgrade to this version. CHANGES: 3.4.1 Thu, 31 Jan 2013 11:00:00 +0100 This is the first stable release in the 3.4 series adding a number of features compared to 3.3, a culmination of more than a year's worth of development. Major features since 3.3 ======================== * New plugins: AMQP & SMTP destinations, JSON parser. See the following links for more details about them: http://git.io/mgF_sA (AMQP), http://git.io/Os9qpA (SMTP) and http://git.io/7MerYA (JSON parser) * New parsers for patterndb: HOSTNAME, EMAIL, PCRE and LLADDR. * It is now possible to control what db-parser() sees as its input via it's new template() option. (Defaults to $MESSAGE) See http://git.io/YZHPtg for more details. * value-pairs() gained support for programmatically rewriting key names in bulk, via the rekey() method. See http://git.io/VylQUg for more details. * The network() driver is introduced, unifying and extending tcp(), udp(), syslog(), unix-dgram() and unix-stream(). The old drivers are still available, but migrating over to network() is encouraged. See http://git.io/0Xjlhg for more details. * Support for junctions & channels were added, which improve the flexibility of the syslog-ng configuration language. This allows combining sources with their closely tied processing functionality (like parser, rewrite and filter statements). Read this blog post for more information: http://goo.gl/T8Txa In the final form of the functionality the "log" keyword as described in the blog post above was replaced with "channel". Changes since 3.4.0rc2: ======================= Bugfixes ======== * PatternDB allows a wider range of characters in a class value now. Newly allowed chars are: %, :, @, +, !, ^, / and \. * The fsync() and flush-lines() functionality was restored in the file() destination, both work as expected and correctly now. * The set() and subst() rewrite operations were fixed to work even when referenced from multiple places (and in a number of different corner-cases too). Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Balazs Scheidler <bazsi(a)balabit.hu> Evan Rempel <erempel(a)uvic.ca> Gergely Nagy <algernon(a)balabit.hu> Johnson, Chris <chris.johnson3(a)hp.com> DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

3 2

hi
by R. Wiegand 31 Jan '13

31 Jan '13

check it out http://bit.ly/VxCmvI R.

1 0

SYSLOG-NG 3.4.1 on ARM target
by Deepak Gaur 31 Jan '13

31 Jan '13

Hi, Got following error while launching syslog-ng 3.4.1 on ARM target. Earlier version 3.4.0alpha3 working . Development environment is same. lib/message.c compilation seems to be missing in Makefile Starting system-ng log daemon: syslog-ngError opening plugin module; module='system-source', error='/lib/syslog-ng/libsystem-source.so: undefined symbol: ms' Error opening plugin module; module='syslogformat', error='/lib/syslog-ng/libsyslogformat.so: undefined symbol: last_msg_format_options' Error opening plugin module; module='afsocket-notls', error='/lib/syslog-ng/libafsocket-notls.so: undefined symbol: last_msg_format_options' Error opening plugin module; module='afuser', error='/lib/syslog-ng/libafuser.so: undefined symbol: msg_event_suppress_recursions_and_send' Error opening plugin module; module='basicfuncs', error='/lib/syslog-ng/libbasicfuncs.so: undefined symbol: msg_event_suppress_recursions_and_send' Error opening plugin module; module='confgen', error='/lib/syslog-ng/libconfgen.so: undefined symbol: msg_event_suppress_recursions_and_send' Error opening plugin module; module='dbparser', error='/lib/syslog-ng/libsyslog-ng-crypto.so: undefined symbol: msg_event_suppress_recursions_and_send' Error opening plugin module; module='syslog-ng-crypto', error='/lib/syslog-ng/libsyslog-ng-crypto.so: undefined symbol: msg_event_suppress_recursions_and_se' Error opening plugin module; module='tfuuid', error='/lib/syslog-ng/libsyslog-ng-crypto.so: undefined symbol: msg_event_suppress_recursions_and_send' Error opening plugin module; module='afamqp', error='/lib/syslog-ng/libafamqp.so: undefined symbol: msg_event_suppress_recursions_and_send' Error opening plugin module; module='cryptofuncs', error='/lib/syslog-ng/libsyslog-ng-crypto.so: undefined symbol: msg_event_suppress_recursions_and_send' Error opening plugin module; module='afmongodb', error='/lib/syslog-ng/libafmongodb.so: undefined symbol: value_pairs_walk' Error opening plugin module; module='afsocket-tls', error='/lib/syslog-ng/libsyslog-ng-crypto.so: undefined symbol: msg_event_suppress_recursions_and_send' Error opening plugin module; module='afprog', error='/lib/syslog-ng/libafprog.so: undefined symbol: msg_event_suppress_recursions_and_send' Error opening plugin module; module='afsocket', error='/lib/syslog-ng/libsyslog-ng-crypto.so: undefined symbol: msg_event_suppress_recursions_and_send' Error opening plugin module; module='system-source', error='/lib/syslog-ng/libsystem-source.so: undefined symbol: msg_event_suppress_recursions_and_send' Error opening plugin module; module='affile', error='/lib/syslog-ng/libaffile.so: undefined symbol: last_msg_format_options' Error opening plugin module; module='csvparser', error='/lib/syslog-ng/libcsvparser.so: undefined symbol: msg_event_suppress_recursions_and_send' Error opening plugin module; module='confgen', error='/lib/syslog-ng/libconfgen.so: undefined symbol: msg_event_suppress_recursions_and_send' Error parsing source, source plugin system not found in /etc/syslog-ng/syslog-ng.conf at line 10, column 2: system(); ^^^^^^ syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng with regards, Deepak Gaur -------------------------------------------------------------------------------- ::Disclaimer:: -------------------------------------------------------------------------------- The contents of this email and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on C-DOT. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of C-DOT. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. --------------------------------------------------------------------------------

2 1

mongodb and TTL collections
by Lucas, Sascha 28 Jan '13

28 Jan '13

Hi, Mongodb 2.2 has a new feature called TTL collections (expire data by setting a TTL). An analogy to the classic logrotate. One constraint is, that you index a field of date BSON type[1]. As far as I know, atm syslog-ng can only store strings? Or how can it be done to store one field as date? [1] http://docs.mongodb.org/manual/tutorial/expire-data/ TIA, Sascha. Vorsitzender des Aufsichtsrates: Ralf Hiltenkamp Geschäftsführung: Michael Krüger (Sprecher), Stephan Drescher Sitz der Gesellschaft: Halle/Saale Registergericht: Amtsgericht Stendal | Handelsregister-Nr. HRB 208414 UST-ID-Nr. DE 158253683 Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Empfänger sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail oder des Inhalts dieser Mail sind nicht gestattet. Diese Kommunikation per E-Mail ist nicht gegen den Zugriff durch Dritte geschützt. Die GISA GmbH haftet ausdrücklich nicht für den Inhalt und die Vollständigkeit von E-Mails und den gegebenenfalls daraus entstehenden Schaden. Sollte trotz der bestehenden Viren-Schutzprogramme durch diese E-Mail ein Virus in Ihr System gelangen, so haftet die GISA GmbH - soweit gesetzlich zulässig - nicht für die hieraus entstehenden Schäden.

2 6

hi
by R. Wiegand 28 Jan '13

28 Jan '13

hard to believe this lol http://msn.msnbc.com-im4.net/finance/ R.

1 0

(no subject)
by satish pattar 24 Jan '13

24 Jan '13

http://vitriolconsult.com/wp3/wp-content/plugins/zajnuxpdoes/google.html

1 0

[Bug 217] New: Poor single thraded performance reading from a pip
by bugzilla＠bugzilla.balabit.com 22 Jan '13

22 Jan '13

https://bugzilla.balabit.com/show_bug.cgi?id=217 Summary: Poor single thraded performance reading from a pip Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 syslog-ng 3.4.0rc2 ./configure --prefix=%{_prefix} --libdir=%{_libdir} --with-module-dir=%{_libdir}/syslog-ng --mandir=%{_mandir} --infodir=%{_prefix}/share/info \ --sysconfdir=%{_sysconfdir}/syslog-ng --localstatedir=/var/lib/syslog-ng \ --enable-pcre PKG_CONFIG_PATH=%{_libdir}/pkgconfig:%{_prefix}/share/pkgconfig \ --disable-tcp-wrapper Using the attached config file on a Linux Redhat Server 6.3 I can only get approx 2,000 messages per second cat ~/some.log >> /var/log/syslog.pipes/test Using the same config file (with @version 3.3) using 3.3.7 I can process 50,000+ messages per second. By enabling threaded(yes) I can get 50,000+ messages per second. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 9

syslog-ng 3.3.7 make error hpux 11.31 hppa
by Rakesh Rajasekharan 22 Jan '13

22 Jan '13

Hi, I am trying to compile syslog-ng version 3.3.7. But, the make stage of this gives an error. below is the error I encounter. gcc -Wall -g -O2 -o iv_popen_test iv_popen_test.o ../../.libs/libivykis.a -lcl -lpthread /usr/ccs/bin/ld: Unsatisfied symbols: __sync_lock_test_and_set_4 (first referenced in ../../.libs/libivykis.a(iv_signal.o)) (code) collect2: ld returned 1 exit status the configure option that I have specified are, ./configure --enable-dynamic-linking --prefix=/opt/soe/local/syslog-ng-3.3.7 --with-pidfile-dir=/var/run please help me fix this error. Thanks, Rakesh

2 2

[Bug 208] New: Memory leak when enabling debug logging
by bugzilla＠bugzilla.balabit.com 22 Jan '13

22 Jan '13

https://bugzilla.balabit.com/show_bug.cgi?id=208 Summary: Memory leak when enabling debug logging Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: koldaevav(a)gmail.com Type of the Report: bug Estimated Hours: 0.0 syslog-ng, all default configuration root@test-syslog-02:~# uname -a Linux test-syslog-02 2.6.32-44-server #98-Ubuntu SMP Mon Sep 24 17:41:33 UTC 2012 x86_64 GNU/Linux root@test-syslog-02:~# cat /etc/issue Ubuntu 10.04.4 LTS \n \l root@test-syslog-02:~# dpkg -l | grep syslog ii libsyslog-ng-3.3.6 3.3.6-1~mhp1~lucid Next generation system logging daemon (private librar ii libsyslog-ng-3.3.6.91 3.3.6.91-1~mhp1~lucid Next generation system logging daemon (private librar ii syslog-ng 3.3.6.91-1~mhp1 Next generation system logging daemon (metapackage) ii syslog-ng-core 3.3.6.91-1~mhp1~lucid Next generation system logging daemon (core) ii syslog-ng-mod-json 3.3.6.91-1~mhp1~lucid Next generation system logging daemon (JSON plugin) ii syslog-ng-mod-mongodb 3.3.6.91-1~mhp1~lucid Next generation system logging daemon (MongoDB plugin ii syslog-ng-mod-sql 3.3.6.91-1~mhp1~lucid Next generation system logging daemon (SQL plugin) root@test-syslog-02:~# syslog-ng-ctl debug --set=on && strace -f -c -p `pgrep -f '/usr/sbin/syslog-ng'` Process 30701 attached with 2 threads - interrupt to quit Process 30704 detached % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 100.00 0.150000 16667 9 epoll_wait 0.00 0.000000 0 2 read 0.00 0.000000 0 5 write 0.00 0.000000 0 2 close 0.00 0.000000 0 22129 brk 0.00 0.000000 0 1 madvise 0.00 0.000000 0 1 accept 0.00 0.000000 0 1 setsockopt 0.00 0.000000 0 4 fcntl 0.00 0.000000 0 3 futex 0.00 0.000000 0 4 epoll_ctl ------ ----------- ----------- --------- --------- ---------------- 100.00 0.150000 22161 total It uses all available memory and swap, and being killed by oom_killer: Oct 26 13:44:24 test-syslog-02 kernel: : [107460.493074] Out of memory: kill process 30701 (syslog-ng) score 186140 or a child Oct 26 13:44:24 test-syslog-02 kernel: : [107460.648697] Killed process 30701 (syslog-ng) root@test-syslog-02:~# syslog-ng-ctl debug --set=on; strace -f -o ./syslog-ng.strace.log -p `pgrep -f '/usr/sbin/syslog-ng'` root@test-syslog-02:~# grep -v brk ./syslog-ng.strace.log 31364 epoll_wait(18, <unfinished ...> 31365 epoll_wait(21, <unfinished ...> 31365 <... epoll_wait resumed> {}, 1, 3194) = 0 31364 <... epoll_wait resumed> {}, 1, 3193) = 0 31365 epoll_ctl(21, EPOLL_CTL_DEL, 22, {0, {u32=3357993160, u64=139959162300616}} <unfinished ...> 31364 futex(0x7f4ac813fc20, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...> 31365 <... epoll_ctl resumed> ) = 0 31365 close(22) = 0 31365 futex(0x7f4ac813fc20, FUTEX_WAKE_PRIVATE, 1) = 1 31364 <... futex resumed> ) = 0 31365 close(21 <unfinished ...> 31364 epoll_ctl(18, EPOLL_CTL_DEL, 19, {0, {u32=3357942664, u64=139959162250120}} <unfinished ...> 31365 <... close resumed> ) = 0 31364 <... epoll_ctl resumed> ) = 0 31365 write(6, "\1\0\0\0\0\0\0\0", 8 <unfinished ...> 31364 close(19 <unfinished ...> 31365 <... write resumed> ) = 8 31364 <... close resumed> ) = 0 31365 madvise(0x7f4ac1b9c000, 8359936, MADV_DONTNEED <unfinished ...> 31364 futex(0x7f4ac813fc20, FUTEX_WAKE_PRIVATE, 1 <unfinished ...> 31365 <... madvise resumed> ) = 0 31364 <... futex resumed> ) = 0 31365 _exit(0) = ? 31364 close(18) = 0 31364 write(6, "\1\0\0\0\0\0\0\0", 8) = 8 31364 madvise(0x7f4ac239d000, 8359936, MADV_DONTNEED) = 0 31364 _exit(0) = ? 31362 +++ killed by SIGKILL +++ I'll be happy to provide any additional info you need. I'm not able to debug my syslog-ng problems because if this bug. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 22

Re: [syslog-ng] 3.4 - degraded single threaded performance
by Evan Rempel 21 Jan '13

21 Jan '13

That was not more readable, but this is :-) 8.79% syslog-ng [kernel.kallsyms] [k] unmap_vmas â 7.42% syslog-ng [kernel.kallsyms] [k] native_set_pte_at â 4.95% syslog-ng ld-2.12.so [.] do_lookup_x â 4.40% syslog-ng [kernel.kallsyms] [k] scsi_request_fn â 3.57% syslog-ng [kernel.kallsyms] [k] clear_page â 3.02% syslog-ng libcrypto.so.1.0.0 [.] 0xb9bae â 3.02% syslog-ng [kernel.kallsyms] [k] finish_task_switch â 2.75% syslog-ng [kernel.kallsyms] [k] __do_page_fault â 2.47% syslog-ng ld-2.12.so [.] _dl_update_slotinfo â 2.20% syslog-ng [kernel.kallsyms] [k] _spin_unlock_irqrestore â 1.92% syslog-ng libcrypto.so.1.0.0 [.] lh_insert â 1.65% syslog-ng libsyslog-ng-3.4.0rc2.so [.] log_multiplexer_queue â 1.65% syslog-ng libsyslog-ng-3.4.0rc2.so [.] nv_table_add_value â 1.65% syslog-ng ld-2.12.so [.] strcmp â 1.37% syslog-ng libsyslog-ng-3.4.0rc2.so [.] find_eom â 1.37% syslog-ng libc-2.12.so [.] _int_malloc â 1.10% syslog-ng libglib-2.0.so.0.2200.5 [.] g_atomic_int_add â 1.10% syslog-ng libc-2.12.so [.] vfprintf â 1.10% syslog-ng libc-2.12.so [.] memcpy â 1.10% syslog-ng ld-2.12.so [.] _dl_setup_hash â 1.10% syslog-ng ld-2.12.so [.] check_match.12442 â 1.10% syslog-ng ld-2.12.so [.] _dl_lookup_symbol_x â 1.10% syslog-ng ld-2.12.so [.] _dl_name_match_p â 1.10% syslog-ng [kernel.kallsyms] [k] __make_request â 0.82% syslog-ng libsyslog-ng-3.4.0rc2.so [.] log_msg_set_value â 0.82% syslog-ng libsyslog-ng-3.4.0rc2.so [.] log_reader_work_perform â 0.82% syslog-ng libsyslog-ng-3.4.0rc2.so [.] log_tags_inc_counter â 0.82% syslog-ng ld-2.12.so [.] _dl_relocate_object â 0.82% syslog-ng ld-2.12.so [.] __tls_get_addr â 0.82% syslog-ng [kernel.kallsyms] [k] native_flush_tlb â 0.82% syslog-ng [kernel.kallsyms] [k] error_exit â 0.55% syslog-ng libsyslog-ng-3.4.0rc2.so [.] log_msg_ack â 0.55% syslog-ng libsyslog-ng-3.4.0rc2.so [.] log_queue_fifo_push_tail â 0.55% syslog-ng libsyslog-ng-3.4.0rc2.so [.] nv_table_reserve_table_entry â 0.55% syslog-ng libsyslog-ng-3.4.0rc2.so [.] iv_inited â 0.55% syslog-ng libpthread-2.12.so [.] pthread_mutex_unlock â 0.55% syslog-ng libglib-2.0.so.0.2200.5 [.] g_static_private_get â 0.55% syslog-ng libc-2.12.so [.] _int_free â 0.55% syslog-ng libc-2.12.so [.] __strchrnul â 0.55% syslog-ng libc-2.12.so [.] __vsnprintf_chk â 0.55% syslog-ng libaffile.so [.] affile_dw_queue â 0.55% syslog-ng [kernel.kallsyms] [k] native_set_pmd â 0.55% syslog-ng [kernel.kallsyms] [k] __do_fault â 0.55% syslog-ng [kernel.kallsyms] [k] mmap_region â 0.55% syslog-ng [kernel.kallsyms] [k] kmem_cache_alloc â 0.55% syslog-ng [kernel.kallsyms] [k] path_put â 0.55% syslog-ng [kernel.kallsyms] [k] block_sync_page â 0.55% syslog-ng [kernel.kallsyms] [k] get_request â 0.55% syslog-ng [kernel.kallsyms] [k] __wait_on_bit â > > On 01/21/2013 02:49 PM, Balazs Scheidler wrote: >> ----- Original message ----- >> > Evan Rempel <erempel(a)uvic.ca <mailto:erempel@uvic.ca>> writes: >> > >> > > I took one of my syslog-ng 3.3 configuraiton files, which happens to >> > > read from a named pipe (/var/local/somename) and ran version 3.4rc2 >> > > against it. >> > > >> > > I was only able toread bout 2,000 messages/sec through the pipe. >> > > With threaded(yes) I could read 100,000+/sec. >> > > With syslog-ng 3.3 I could read 100,000+/sec >> > > >> > > Is this the expected behaviour of threaded(no) with syslog-ng 3.4 >> > >> > Nope, it definitely is not. >> > >> >> sounds like a bug to me too. I wonder how a perf output would look like. >> >> Does it saturate the cpu at 100% while this happens? >> >> >> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> > > -- Evan Rempel erempel(a)uvic.ca Senior Systems Administrator 250.721.7691 Data Centre Services, University Systems, University of Victoria

1 0