Using patterndb to rewrite any and all macros
I am considering using the patterndb to rewrite messages that do not conform to syslog standards. Can the patterndb change all normally defined macros? I need to be able to change MSGHDR MESSAGE PROGRAM PID MSG MSGONLY and I think that is all. I know that macros such as MSGHDR can not be changed by the rewrite command but can it be changed by the patterndb? Thanks for your expertise. -- Evan Rempel Senior Systems Administrator, Data Centre Services University of Victoria 250.721.7691
----- Original message -----
I am considering using the patterndb to rewrite messages that do not conform to syslog standards.
Can the patterndb change all normally defined macros? I need to be able to change MSGHDR MESSAGE PROGRAM PID MSG MSGONLY and I think that is all. I know that macros such as MSGHDR can not be changed by the rewrite command but can it be changed by the patterndb?
there are two kinds of 'macros' within syslog-ng (and I'm considering a name change because of that). some of them are like registers, contain a given value, can be changed at will. I plan to name these properties in the future. others are basically derived values, where syslog-ng programmatically creates the value when needed. these cannot be changed. these would be called read-only properties. having this naming scheme, here's an explanation MSGHDR - this is read only, its contents are affected by PROGRAM, PID. If any of those is changed, MSGHDR will automatically change. (there was a bug in this a couple of years ago, but 3.3 should be ok) MESSAGE - read write PROGRAM - read write PID - read write MSG - alias to MESSAGE MSGONLY - read only, but is the same as MESSAGE in recent versions. patterndb is able to change read write proprties, but cannot change read only ones, similar to how rewrite rules behave.
Thanks for your expertise. -- Evan Rempel Senior Systems Administrator, Data Centre Services University of Victoria 250.721.7691 ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Balazs Scheidler <bazsi77@gmail.com> writes:
----- Original message -----
I am considering using the patterndb to rewrite messages that do not conform to syslog standards.
Can the patterndb change all normally defined macros? I need to be able to change MSGHDR MESSAGE PROGRAM PID MSG MSGONLY and I think that is all. I know that macros such as MSGHDR can not be changed by the rewrite command but can it be changed by the patterndb?
there are two kinds of 'macros' within syslog-ng (and I'm considering a name change because of that). some of them are like registers, contain a given value, can be changed at will. I plan to name these properties in the future.
others are basically derived values, where syslog-ng programmatically creates the value when needed. these cannot be changed. these would be called read-only properties.
Wouldn't it be clearer if the read-only values would be turned into template functions? Or at least, make them appear like one? Then the whole confusion could be avoided, as far as I see. (Though, for backwards compatibility, they'd still work as 'ordinary' read-only macros) Another option would be to trigger updating these select macros whenever the values that they depend on change, but only when they weren't explicitly set. But this would add a noticable cost to paths we may not want that cost on... -- |8]
I took one of my syslog-ng 3.3 configuraiton files, which happens to read from a named pipe (/var/local/somename) and ran version 3.4rc2 against it. I was only able toread bout 2,000 messages/sec through the pipe. With threaded(yes) I could read 100,000+/sec. With syslog-ng 3.3 I could read 100,000+/sec Is this the expected behaviour of threaded(no) with syslog-ng 3.4 I just wanted to ask prior to created the minimal config and lgging a bug report. Evan.
Evan Rempel <erempel@uvic.ca> writes:
I took one of my syslog-ng 3.3 configuraiton files, which happens to read from a named pipe (/var/local/somename) and ran version 3.4rc2 against it.
I was only able toread bout 2,000 messages/sec through the pipe. With threaded(yes) I could read 100,000+/sec. With syslog-ng 3.3 I could read 100,000+/sec
Is this the expected behaviour of threaded(no) with syslog-ng 3.4
Nope, it definitely is not. -- |8]
----- Original message -----
Evan Rempel <erempel@uvic.ca> writes:
I took one of my syslog-ng 3.3 configuraiton files, which happens to read from a named pipe (/var/local/somename) and ran version 3.4rc2 against it.
I was only able toread bout 2,000 messages/sec through the pipe. With threaded(yes) I could read 100,000+/sec. With syslog-ng 3.3 I could read 100,000+/sec
Is this the expected behaviour of threaded(no) with syslog-ng 3.4
Nope, it definitely is not.
sounds like a bug to me too. I wonder how a perf output would look like. Does it saturate the cpu at 100% while this happens?
No, is seems to be around 2% CPU, but often in the D state (uninterruptable sleep) which is usually waiting on IO, but I'm not sure of the exact interaction with pipes. Evan. perf 8.79% syslog-ng [kernel.kallsyms] [k] unmap_vmas ◆ 7.42% syslog-ng [kernel.kallsyms] [k] native_set_pte_at ▒ 4.95% syslog-ng ld-2.12.so [.] do_lookup_x ▒ 4.40% syslog-ng [kernel.kallsyms] [k] scsi_request_fn ▒ 3.57% syslog-ng [kernel.kallsyms] [k] clear_page ▒ 3.02% syslog-ng libcrypto.so.1.0.0 [.] 0xb9bae ▒ 3.02% syslog-ng [kernel.kallsyms] [k] finish_task_switch ▒ 2.75% syslog-ng [kernel.kallsyms] [k] __do_page_fault ▒ 2.47% syslog-ng ld-2.12.so [.] _dl_update_slotinfo ▒ 2.20% syslog-ng [kernel.kallsyms] [k] _spin_unlock_irqrestore ▒ 1.92% syslog-ng libcrypto.so.1.0.0 [.] lh_insert ▒ 1.65% syslog-ng libsyslog-ng-3.4.0rc2.so [.] log_multiplexer_queue ▒ 1.65% syslog-ng libsyslog-ng-3.4.0rc2.so [.] nv_table_add_value ▒ 1.65% syslog-ng ld-2.12.so [.] strcmp ▒ 1.37% syslog-ng libsyslog-ng-3.4.0rc2.so [.] find_eom ▒ 1.37% syslog-ng libc-2.12.so [.] _int_malloc ▒ 1.10% syslog-ng libglib-2.0.so.0.2200.5 [.] g_atomic_int_add ▒ 1.10% syslog-ng libc-2.12.so [.] vfprintf ▒ 1.10% syslog-ng libc-2.12.so [.] memcpy ▒ 1.10% syslog-ng ld-2.12.so [.] _dl_setup_hash ▒ 1.10% syslog-ng ld-2.12.so [.] check_match.12442 ▒ 1.10% syslog-ng ld-2.12.so [.] _dl_lookup_symbol_x ▒ 1.10% syslog-ng ld-2.12.so [.] _dl_name_match_p ▒ 1.10% syslog-ng [kernel.kallsyms] [k] __make_request ▒ 0.82% syslog-ng libsyslog-ng-3.4.0rc2.so [.] log_msg_set_value ▒ 0.82% syslog-ng libsyslog-ng-3.4.0rc2.so [.] log_reader_work_perform ▒ 0.82% syslog-ng libsyslog-ng-3.4.0rc2.so [.] log_tags_inc_counter ▒ 0.82% syslog-ng ld-2.12.so [.] _dl_relocate_object ▒ 0.82% syslog-ng ld-2.12.so [.] __tls_get_addr ▒ 0.82% syslog-ng [kernel.kallsyms] [k] native_flush_tlb ▒ 0.82% syslog-ng [kernel.kallsyms] [k] error_exit ▒ 0.55% syslog-ng libsyslog-ng-3.4.0rc2.so [.] log_msg_ack ▒ 0.55% syslog-ng libsyslog-ng-3.4.0rc2.so [.] log_queue_fifo_push_tail ▒ 0.55% syslog-ng libsyslog-ng-3.4.0rc2.so [.] nv_table_reserve_table_entry ▒ 0.55% syslog-ng libsyslog-ng-3.4.0rc2.so [.] iv_inited ▒ 0.55% syslog-ng libpthread-2.12.so [.] pthread_mutex_unlock ▒ 0.55% syslog-ng libglib-2.0.so.0.2200.5 [.] g_static_private_get ▒ 0.55% syslog-ng libc-2.12.so [.] _int_free ▒ 0.55% syslog-ng libc-2.12.so [.] __strchrnul ▒ 0.55% syslog-ng libc-2.12.so [.] __vsnprintf_chk ▒ 0.55% syslog-ng libaffile.so [.] affile_dw_queue ▒ 0.55% syslog-ng [kernel.kallsyms] [k] native_set_pmd ▒ 0.55% syslog-ng [kernel.kallsyms] [k] __do_fault ▒ 0.55% syslog-ng [kernel.kallsyms] [k] mmap_region ▒ 0.55% syslog-ng [kernel.kallsyms] [k] kmem_cache_alloc ▒ 0.55% syslog-ng [kernel.kallsyms] [k] path_put ▒ 0.55% syslog-ng [kernel.kallsyms] [k] block_sync_page ▒ 0.55% syslog-ng [kernel.kallsyms] [k] get_request ▒ 0.55% syslog-ng [kernel.kallsyms] [k] __wait_on_bit ▒ Is that what you wanted to see. On 01/21/2013 02:49 PM, Balazs Scheidler wrote:
----- Original message -----
Evan Rempel <erempel@uvic.ca <mailto:erempel@uvic.ca>> writes:
I took one of my syslog-ng 3.3 configuraiton files, which happens to read from a named pipe (/var/local/somename) and ran version 3.4rc2 against it.
I was only able toread bout 2,000 messages/sec through the pipe. With threaded(yes) I could read 100,000+/sec. With syslog-ng 3.3 I could read 100,000+/sec
Is this the expected behaviour of threaded(no) with syslog-ng 3.4
Nope, it definitely is not.
sounds like a bug to me too. I wonder how a perf output would look like.
Does it saturate the cpu at 100% while this happens?
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 Data Centre Services, University Systems, University of Victoria
Sorry, this is more readable.. 2.47% syslog-ng ld-2.12.so [.] _dl_update_slotinfo ▒ 2.20% syslog-ng [kernel.kallsyms] [k] _spin_unlock_irqrestore ▒ 1.92% syslog-ng libcrypto.so.1.0.0 [.] lh_insert ▒ 1.65% syslog-ng libsyslog-ng-3.4.0rc2.so [.] log_multiplexer_queue ▒ 1.65% syslog-ng libsyslog-ng-3.4.0rc2.so [.] nv_table_add_value ▒ 1.65% syslog-ng ld-2.12.so [.] strcmp ▒ 1.37% syslog-ng libsyslog-ng-3.4.0rc2.so [.] find_eom ▒ 1.37% syslog-ng libc-2.12.so [.] _int_malloc ▒ 1.10% syslog-ng libglib-2.0.so.0.2200.5 [.] g_atomic_int_add ▒ 1.10% syslog-ng libc-2.12.so [.] vfprintf ▒ 1.10% syslog-ng libc-2.12.so [.] memcpy ▒ 1.10% syslog-ng ld-2.12.so [.] _dl_setup_hash ▒ 1.10% syslog-ng ld-2.12.so [.] check_match.12442 ▒ 1.10% syslog-ng ld-2.12.so [.] _dl_lookup_symbol_x ▒ 1.10% syslog-ng ld-2.12.so [.] _dl_name_match_p ▒ 1.10% syslog-ng [kernel.kallsyms] [k] __make_request ▒ 0.82% syslog-ng libsyslog-ng-3.4.0rc2.so [.] log_msg_set_value ▒ 0.82% syslog-ng libsyslog-ng-3.4.0rc2.so [.] log_reader_work_perform ▒ 0.82% syslog-ng libsyslog-ng-3.4.0rc2.so [.] log_tags_inc_counter ▒ 0.82% syslog-ng ld-2.12.so [.] _dl_relocate_object ▒ 0.82% syslog-ng ld-2.12.so [.] __tls_get_addr ▒ 0.82% syslog-ng [kernel.kallsyms] [k] native_flush_tlb ▒ 0.82% syslog-ng [kernel.kallsyms] [k] error_exit ▒ 0.55% syslog-ng libsyslog-ng-3.4.0rc2.so [.] log_msg_ack ▒ 0.55% syslog-ng libsyslog-ng-3.4.0rc2.so [.] log_queue_fifo_push_tail ▒ 0.55% syslog-ng libsyslog-ng-3.4.0rc2.so [.] nv_table_reserve_table_entry ▒ 0.55% syslog-ng libsyslog-ng-3.4.0rc2.so [.] iv_inited ▒ 0.55% syslog-ng libpthread-2.12.so [.] pthread_mutex_unlock ▒ 0.55% syslog-ng libglib-2.0.so.0.2200.5 [.] g_static_private_get ▒ 0.55% syslog-ng libc-2.12.so [.] _int_free ▒ 0.55% syslog-ng libc-2.12.so [.] __strchrnul ▒ 0.55% syslog-ng libc-2.12.so [.] __vsnprintf_chk ▒ 0.55% syslog-ng libaffile.so [.] affile_dw_queue ▒ 0.55% syslog-ng [kernel.kallsyms] [k] native_set_pmd ▒ 0.55% syslog-ng [kernel.kallsyms] [k] __do_fault ▒ 0.55% syslog-ng [kernel.kallsyms] [k] mmap_region ▒ 0.55% syslog-ng [kernel.kallsyms] [k] kmem_cache_alloc ▒ 0.55% syslog-ng [kernel.kallsyms] [k] path_put ▒ 0.55% syslog-ng [kernel.kallsyms] [k] block_sync_page ▒ 0.55% syslog-ng [kernel.kallsyms] [k] get_request ▒ 0.55% syslog-ng [kernel.kallsyms] [k] __wait_on_bit ▒ On 01/21/2013 02:49 PM, Balazs Scheidler wrote:
----- Original message -----
Evan Rempel <erempel@uvic.ca <mailto:erempel@uvic.ca>> writes:
I took one of my syslog-ng 3.3 configuraiton files, which happens to read from a named pipe (/var/local/somename) and ran version 3.4rc2 against it.
I was only able toread bout 2,000 messages/sec through the pipe. With threaded(yes) I could read 100,000+/sec. With syslog-ng 3.3 I could read 100,000+/sec
Is this the expected behaviour of threaded(no) with syslog-ng 3.4
Nope, it definitely is not.
sounds like a bug to me too. I wonder how a perf output would look like.
Does it saturate the cpu at 100% while this happens?
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 Data Centre Services, University Systems, University of Victoria
participants (3)
-
Balazs Scheidler
-
Evan Rempel
-
Gergely Nagy