- syslog-ng - lists.balabit.hu

[Bug 278] New: Many memory leak problems happen when do configuration reloading
by bugzilla＠bugzilla.balabit.com 03 Jun '14

03 Jun '14

https://bugzilla.balabit.com/show_bug.cgi?id=278 Summary: Many memory leak problems happen when do configuration reloading Product: syslog-ng Version: 3.5.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: xufeng.zhang(a)windriver.com Type of the Report: --- Estimated Hours: 0.0 Created an attachment (id=95) --> (https://bugzilla.balabit.com/attachment.cgi?id=95) the config and script files to reproduce this problem I found many memory leak problems on syslog-ng_3.5.4.1, I have resolved some of them, but there are still some others need to be resolved: 1). mutex problem such as: ==1077== 40 bytes in 1 blocks are definitely lost in loss record 514 of 636 ==1077== at 0x4A05F58: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==1077== by 0x36882478F4: g_malloc (in /lib64/libglib-2.0.so.0.2600.0) ==1077== by 0x36886024FE: ??? (in /lib64/libgthread-2.0.so.0.2600.0) ==1077== by 0x3688266B23: g_static_mutex_get_mutex_impl (in /lib64/libglib-2.0.so.0.2600.0) ==1077== by 0x523D83C: affile_dw_queue (affile-dest.c:263) ==1077== by 0x523D05B: affile_dd_queue (logpipe.h:320) ==1077== by 0x3688E32150: log_multiplexer_queue (logpipe.h:320) patch Fix-the-memory-leak-problem-for-mutex.patch could resolve them. 2). memory leak when HAVE_ENVIRON is defined: ==1077== 1,258 (208 direct, 1,050 indirect) bytes in 1 blocks are definitely lost in loss record 617 of 636 ==1077== at 0x4A05F58: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==1077== by 0x36882478F4: g_malloc (in /lib64/libglib-2.0.so.0.2600.0) ==1077== by 0x3688E2F5FF: g_process_set_argv_space (gprocess.c:502) ==1077== by 0x40164C: main (main.c:196) patch Fix-the-memory-leak-problem-when-HAVE_ENVIRON-defined.patch could resolve them. 3). memory leak problem for unconsumed item during configuration reloading: ==1077== 36,864 bytes in 45 blocks are definitely lost in loss record 636 of 636 ==1077== at 0x4A05F58: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==1077== by 0x36882478F4: g_malloc (in /lib64/libglib-2.0.so.0.2600.0) ==1077== by 0x3688E3D178: log_writer_flush (logwriter.c:973) ==1077== by 0x3688E3D2D2: log_writer_deinit (logwriter.c:1169) ==1077== by 0x523C09A: affile_dw_deinit (logpipe.h:268) ==1077== by 0x523C93B: affile_dd_deinit (logpipe.h:268) ==1077== by 0x3688E2A4FE: cfg_tree_stop (logpipe.h:268) ==1077== by 0x3688E3DB9F: main_loop_reload_config_apply (mainloop.c:540) ==1077== by 0x3688E644DA: iv_signal_event (iv_signal.c:170) ==1077== by 0x3688E62F28: iv_event_raw_got_event (iv_event_raw_posix.c:89) ==1077== by 0x3688E635F1: iv_fd_poll_and_run (iv_fd.c:163) ==1077== by 0x3688E63D73: iv_main (iv_main_posix.c:117) patch logwriter-still-free-the-unconsumed-item.patch 4). The remaining memory leak problems need to be resolved, and they are not in 3.4.2 version: ==1077== 24 bytes in 1 blocks are definitely lost in loss record 483 of 636 ==1077== at 0x4A04961: calloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==1077== by 0x3688247819: g_malloc0 (in /lib64/libglib-2.0.so.0.2600.0) ==1077== by 0x5023BCA: socket_options_new (socket-options.c:90) ==1077== by 0x502568A: afunix_sd_new_instance (afunix-source.c:241) ==1077== by 0x502691A: afsocket_parse (afsocket-grammar.y:537) ==1077== by 0x3688E440D1: plugin_parse_config (cfg-parser.h:83) ==1077== by 0x3688E4F0BC: main_parse (cfg-grammar.y:549) ==1077== by 0x3688E26DC2: cfg_run_parser (cfg-parser.h:83) ==1077== by 0x3688E26ED5: cfg_read_config (cfg.c:382) ==1077== by 0x3688E3DAE5: main_loop_reload_config_initiate (mainloop.c:611) ==1077== by 0x3688E644DA: iv_signal_event (iv_signal.c:170) ==1077== by 0x3688E62F28: iv_event_raw_got_event (iv_event_raw_posix.c:89) ==1077== ==1077== 24 bytes in 1 blocks are definitely lost in loss record 484 of 636 ==1077== at 0x4A04961: calloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==1077== by 0x3688247819: g_malloc0 (in /lib64/libglib-2.0.so.0.2600.0) ==1077== by 0x5023BCA: socket_options_new (socket-options.c:90) ==1077== by 0x502568A: afunix_sd_new_instance (afunix-source.c:241) ==1077== by 0x502691A: afsocket_parse (afsocket-grammar.y:537) ==1077== by 0x3688E440D1: plugin_parse_config (cfg-parser.h:83) ==1077== by 0x3688E4F0BC: main_parse (cfg-grammar.y:549) ==1077== by 0x3688E26DC2: cfg_run_parser (cfg-parser.h:83) ==1077== by 0x3688E26ED5: cfg_read_config (cfg.c:382) ==1077== by 0x3688E3E42E: main_loop_init (mainloop.c:729) ==1077== by 0x401774: main (main.c:246) ==1077== 64 bytes in 1 blocks are definitely lost in loss record 531 of 636 ==1077== at 0x4A04961: calloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==1077== by 0x3688247819: g_malloc0 (in /lib64/libglib-2.0.so.0.2600.0) ==1077== by 0x5025A81: transport_mapper_unix_new_instance (transport-mapper-unix.c:40) ==1077== by 0x5025B04: transport_mapper_unix_dgram_new (transport-mapper-unix.c:51) ==1077== by 0x50257C8: afunix_sd_new_dgram (afunix-source.c:262) ==1077== by 0x502691A: afsocket_parse (afsocket-grammar.y:537) ==1077== by 0x3688E440D1: plugin_parse_config (cfg-parser.h:83) ==1077== by 0x3688E4F0BC: main_parse (cfg-grammar.y:549) ==1077== by 0x3688E26DC2: cfg_run_parser (cfg-parser.h:83) ==1077== by 0x3688E26ED5: cfg_read_config (cfg.c:382) ==1077== by 0x3688E3DAE5: main_loop_reload_config_initiate (mainloop.c:611) ==1077== by 0x3688E644DA: iv_signal_event (iv_signal.c:170) ==1077== ==1077== 64 bytes in 1 blocks are definitely lost in loss record 532 of 636 ==1077== at 0x4A04961: calloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==1077== by 0x3688247819: g_malloc0 (in /lib64/libglib-2.0.so.0.2600.0) ==1077== by 0x5025A81: transport_mapper_unix_new_instance (transport-mapper-unix.c:40) ==1077== by 0x5025B04: transport_mapper_unix_dgram_new (transport-mapper-unix.c:51) ==1077== by 0x50257C8: afunix_sd_new_dgram (afunix-source.c:262) ==1077== by 0x502691A: afsocket_parse (afsocket-grammar.y:537) ==1077== by 0x3688E440D1: plugin_parse_config (cfg-parser.h:83) ==1077== by 0x3688E4F0BC: main_parse (cfg-grammar.y:549) ==1077== by 0x3688E26DC2: cfg_run_parser (cfg-parser.h:83) ==1077== by 0x3688E26ED5: cfg_read_config (cfg.c:382) ==1077== by 0x3688E3E42E: main_loop_init (mainloop.c:729) ==1077== by 0x401774: main (main.c:246) ==1077== 96 (24 direct, 72 indirect) bytes in 1 blocks are definitely lost in loss record 541 of 636 ==1077== at 0x4A05F58: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==1077== by 0x36882478F4: g_malloc (in /lib64/libglib-2.0.so.0.2600.0) ==1077== by 0x368825CC99: g_slice_alloc (in /lib64/libglib-2.0.so.0.2600.0) ==1077== by 0x368823D14D: g_list_prepend (in /lib64/libglib-2.0.so.0.2600.0) ==1077== by 0x3688E5DE0A: log_template_add_macro_elem (templates.c:778) ==1077== by 0x3688E5EC74: log_template_compile (templates.c:1232) ==1077== by 0x3688E507AF: main_parse (cfg-grammar.y:741) ==1077== by 0x3688E26DC2: cfg_run_parser (cfg-parser.h:83) ==1077== by 0x3688E26ED5: cfg_read_config (cfg.c:382) ==1077== by 0x3688E3DAE5: main_loop_reload_config_initiate (mainloop.c:611) ==1077== by 0x3688E644DA: iv_signal_event (iv_signal.c:170) ==1077== by 0x3688E62F28: iv_event_raw_got_event (iv_event_raw_posix.c:89) ==1077== 144 (48 direct, 96 indirect) bytes in 2 blocks are definitely lost in loss record 556 of 636 ==1077== at 0x4A05F58: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==1077== by 0x36882478F4: g_malloc (in /lib64/libglib-2.0.so.0.2600.0) ==1077== by 0x368825CC99: g_slice_alloc (in /lib64/libglib-2.0.so.0.2600.0) ==1077== by 0x368823D14D: g_list_prepend (in /lib64/libglib-2.0.so.0.2600.0) ==1077== by 0x3688E5DE0A: log_template_add_macro_elem (templates.c:778) ==1077== by 0x3688E5E661: log_template_compiler_process_token (templates.c:1129) ==1077== by 0x3688E5EBD8: log_template_compile (templates.c:1222) ==1077== by 0x3688E507AF: main_parse (cfg-grammar.y:741) ==1077== by 0x3688E26DC2: cfg_run_parser (cfg-parser.h:83) ==1077== by 0x3688E26ED5: cfg_read_config (cfg.c:382) ==1077== by 0x3688E3E42E: main_loop_init (mainloop.c:729) ==1077== by 0x401774: main (main.c:246) ==1077== 1,008 bytes in 42 blocks are definitely lost in loss record 606 of 636 ==1077== at 0x4A05F58: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==1077== by 0x36882478F4: g_malloc (in /lib64/libglib-2.0.so.0.2600.0) ==1077== by 0x368825CC99: g_slice_alloc (in /lib64/libglib-2.0.so.0.2600.0) ==1077== by 0x368823D14D: g_list_prepend (in /lib64/libglib-2.0.so.0.2600.0) ==1077== by 0x3688E5DE0A: log_template_add_macro_elem (templates.c:778) ==1077== by 0x3688E5EC74: log_template_compile (templates.c:1232) ==1077== by 0x523BC1E: affile_dd_new_instance (affile-dest.c:706) ==1077== by 0x523BCA8: affile_dd_new (affile-dest.c:722) ==1077== by 0x523EEED: affile_parse (affile-grammar.y:490) ==1077== by 0x3688E440D1: plugin_parse_config (cfg-parser.h:83) ==1077== by 0x3688E4FFCA: main_parse (cfg-grammar.y:641) ==1077== by 0x3688E26DC2: cfg_run_parser (cfg-parser.h:83) ==1077== 176 bytes in 1 blocks are definitely lost in loss record 563 of 636 ==1077== at 0x4A04961: calloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==1077== by 0x3688247819: g_malloc0 (in /lib64/libglib-2.0.so.0.2600.0) ==1077== by 0x3688E44299: poll_fd_events_new (poll-fd-events.c:77) ==1077== by 0x5022582: afsocket_sc_init (afsocket-source.c:91) ==1077== by 0x5021FC7: afsocket_sd_process_connection (logpipe.h:253) ==1077== by 0x5022B6D: afsocket_sd_init_method (afsocket-source.c:540) ==1077== by 0x5025855: afunix_sd_init (afunix-source.c:223) ==1077== by 0x3688E2BAF2: cfg_tree_start (logpipe.h:253) ==1077== by 0x3688E2716A: cfg_init (cfg.c:218) ==1077== by 0x3688E3E48C: main_loop_init (mainloop.c:527) ==1077== by 0x401774: main (main.c:246) ==1077== ==1077== 176 bytes in 1 blocks are definitely lost in loss record 564 of 636 ==1077== at 0x4A04961: calloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==1077== by 0x3688247819: g_malloc0 (in /lib64/libglib-2.0.so.0.2600.0) ==1077== by 0x3688E44299: poll_fd_events_new (poll-fd-events.c:77) ==1077== by 0x523AC56: affile_sd_construct_poll_events (affile-source.c:195) ==1077== by 0x523B696: affile_sd_init (affile-source.c:375) ==1077== by 0x3688E2BAF2: cfg_tree_start (logpipe.h:253) ==1077== by 0x3688E2716A: cfg_init (cfg.c:218) ==1077== by 0x3688E3E48C: main_loop_init (mainloop.c:527) ==1077== by 0x401774: main (main.c:246) ==1077== 352 bytes in 2 blocks are definitely lost in loss record 576 of 636 ==1077== at 0x4A04961: calloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==1077== by 0x3688247819: g_malloc0 (in /lib64/libglib-2.0.so.0.2600.0) ==1077== by 0x3688E44299: poll_fd_events_new (poll-fd-events.c:77) ==1077== by 0x523AC56: affile_sd_construct_poll_events (affile-source.c:195) ==1077== by 0x523B696: affile_sd_init (affile-source.c:375) ==1077== by 0x3688E2BAF2: cfg_tree_start (logpipe.h:253) ==1077== by 0x3688E2716A: cfg_init (cfg.c:218) ==1077== by 0x3688E3DBBE: main_loop_reload_config_apply (mainloop.c:543) ==1077== by 0x3688E644DA: iv_signal_event (iv_signal.c:170) ==1077== by 0x3688E62F28: iv_event_raw_got_event (iv_event_raw_posix.c:89) ==1077== by 0x3688E635F1: iv_fd_poll_and_run (iv_fd.c:163) ==1077== by 0x3688E63D73: iv_main (iv_main_posix.c:117) When udp connection is used, I found three new additional memory leaks(I have applied the attached three patches): ==25796== 3,072 bytes in 3 blocks are definitely lost in loss record 616 of 622 ==25796== at 0x4A05F58: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==25796== by 0x3B742478F4: g_malloc (in /lib64/libglib-2.0.so.0.2600.0) ==25796== by 0x3B74E3D1D8: log_writer_flush (logwriter.c:974) ==25796== by 0x3B74E3D2BC: log_writer_work_perform (logwriter.c:163) ==25796== by 0x3B74E3D327: log_writer_io_flush_output (logwriter.c:243) ==25796== by 0x3B74E63666: iv_fd_poll_and_run (iv_fd.c:167) ==25796== by 0x3B74E63E13: iv_main (iv_main_posix.c:117) ==25796== by 0x3B74E3D7E6: main_loop_run (mainloop.c:788) ==25796== by 0x4017A0: main (main.c:267) ==25796== 20 bytes in 2 blocks are definitely lost in loss record 17 of 622 ==25796== at 0x4A05F58: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==25796== by 0x3B6FA7C6B1: strdup (strdup.c:43) ==25796== by 0x3B74E4D082: _cfg_lexer_lex (cfg-lex.l:205) ==25796== by 0x3B74E293A1: cfg_lexer_lex (cfg-lexer.c:759) ==25796== by 0x3B74E5C7A6: rewrite_expr_parse (rewrite-expr-grammar.c:2824) ==25796== by 0x3B74E4EDD2: T.102 (cfg-parser.h:83) ==25796== by 0x3B74E5016D: main_parse (cfg-grammar.y:606) ==25796== by 0x3B74E26DC2: cfg_run_parser (cfg-parser.h:83) ==25796== by 0x3B74E26ED5: cfg_read_config (cfg.c:382) ==25796== by 0x3B74E3E4CE: main_loop_init (mainloop.c:729) ==25796== by 0x401774: main (main.c:246) ==25796== 40 bytes in 4 blocks are definitely lost in loss record 514 of 622 ==25796== at 0x4A05F58: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==25796== by 0x3B6FA7C6B1: strdup (strdup.c:43) ==25796== by 0x3B74E4D082: _cfg_lexer_lex (cfg-lex.l:205) ==25796== by 0x3B74E293A1: cfg_lexer_lex (cfg-lexer.c:759) ==25796== by 0x3B74E5C7A6: rewrite_expr_parse (rewrite-expr-grammar.c:2824) ==25796== by 0x3B74E4EDD2: T.102 (cfg-parser.h:83) ==25796== by 0x3B74E5016D: main_parse (cfg-grammar.y:606) ==25796== by 0x3B74E26DC2: cfg_run_parser (cfg-parser.h:83) ==25796== by 0x3B74E26ED5: cfg_read_config (cfg.c:382) ==25796== by 0x3B74E3DB85: main_loop_reload_config_initiate (mainloop.c:611) ==25796== by 0x3B74E6457A: iv_signal_event (iv_signal.c:170) ==25796== by 0x3B74E62FC8: iv_event_raw_got_event (iv_event_raw_posix.c:89) -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 15

New puppet module for patterndb
by Fabien Wernli 30 May '14

30 May '14

Hi, I'd like to annouce the availability of our patterndb puppet module, which is almost feature complete. Before submitting this to puppetlabs, I'd much appreciate puppet/syslog-ng users to test it and post feedback either here, or on github. Thanks for your time! https://github.com/ccin2p3/puppet-patterndb Cheers

1 0

Question on behavior of unix-stream destination when data is returned...
by Robert Nickel 29 May '14

29 May '14

So, I have the following configuration to send logs out to a unix domain socket: source s_tls { syslog( ip( 0.0.0.0 ) port( 6514 ) transport( "tls" ) tls( ca_dir( "/path/to/cacerts" ) key_file( "/path/to/private/key" ) cert_file( "/path/to/certificate" ) ) ); }; destination d_sock { unix-stream( "/var/tmp/itsa.socket" ); }; log { source( s_tls ); destination( d_sock ); }; Every time a message writes to the socket, the following logs show up: EOF occurred while idle; fd='12' Syslog connection broken; fd='12', server='AF_UNIX(/var/tmp/itsa.socket)', time_reopen='60' Digging into the code of the socket listening service, it is apparent that the socket returns messages on receipt of the message. Essentially, every time it gets a well-formatted message, it returns 'Parse message success' or 'Parse message failed'. Disabling all of these messages makes the socket work fine but it will break other code that uses this same socket. Is this expected behavior? Is there a way to get syslog-ng to ignore the returned information on the socket? Thanks! --Robert

1 0

[Bug 281] New: Feature - allow for longer program name
by bugzilla＠bugzilla.balabit.com 29 May '14

29 May '14

https://bugzilla.balabit.com/show_bug.cgi?id=281 Summary: Feature - allow for longer program name Product: syslog-ng Version: 3.5.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 Given the specification for the structured syslog protocol (RFC 5424) allows for an application name to be 48 characters and that the process ID is separate and it is permitted to be 128 characters, it seems unreasonable to truncate the program/PID combination to 32 characters. I realize that the 32 character limit is in the RFC 3164 document, but other syslog systems (rsyslog for example) do not enforce this limit. I propose that you permit the program and PID together (the TAG) to be up to 48 characters is length. Specifically, I am having problems with the very common crond TAG entries of the form run-parts(/etc/cron.hourly)[19482] run-parts(/etc/cron.monthly)[10274] which are being truncated by the syslog-ng server. Thanks, -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 211] New: dbparser pattern order breaks validation
by bugzilla＠bugzilla.balabit.com 29 May '14

29 May '14

https://bugzilla.balabit.com/show_bug.cgi?id=211 Summary: dbparser pattern order breaks validation Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 If two log lines share a common leading portion such as; xlog: backup pg_xlog/000000010000014700000076 xlog: backup pg_xlog/000000010000014700000076 failed then the order of the patterns in the pattern database xml file is important. For example, if the two patterns used are; xlog: backup pg_xlog/@SET:xid:0123456789ABCDEF@ xlog: backup pg_xlog/@SET:xid:0123456789ABCDEF@ failed and the shorter pattern is first, it will match BOTH of the above lines because it matches the leading portion of both lines. If the patterns are entered with the longer pattern first, then the longer pattern will fail on the shorter message, allowing the match to fall through to the shorter pattern matcher, but will match the longer message. I expect the behavior of the dbparser to match with the pattern containing the MOST total static content that also matches all of the parsing macros. Some method where the order is not important need to be used. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 4

Suppress almost-identical messages?
by Evade Flow 23 May '14

23 May '14

I'm trying to clean up the ad hoc logging in a collection of applications started on an embedded device using the daemon utility. Example: /usr/bin/daemon --name=myapp --output=user.debug --chdir=/MyApp -- /MyApp/bin/myapp There are 10 different applications started this way, written by different individuals/teams, and our syslog-ng.conf file currently looks something like this: @version:3.3.6 source s_local { system(); internal(); }; destination d_localfile { file("/var/log/messages" suppress(30)); }; log { source(s_local); destination(d_localfile); }; Because the --output=user.debug option to the daemon utility will cause the applications' stdout/stderr to be redirected to syslog, I'm a little worried about 'rogue' printf()'s filling up the logs with stuff like: Jan 1 00:00:54 (none) myapp: Battery voltage is 11.07 volts Jan 1 00:00:54 (none) myapp: Battery voltage is 11.10 volts Jan 1 00:00:54 (none) myapp: Battery voltage is 11.07 volts Jan 1 00:00:54 (none) myapp: Battery voltage is 11.14 volts Jan 1 00:00:54 (none) myapp: Battery voltage is 11.13 volts Jan 1 00:00:55 (none) myapp: Battery voltage is 11.09 volts Jan 1 00:00:55 (none) myapp: Battery voltage is 11.14 volts ... Does syslog-ng support suppression of almost-but-not-quite identical messages? It would be nice to see something like this in the logs: Jan 1 00:55:11 myapp: Battery voltage is 11.07 volts Jan 1 00:55:14 myapp: Last message 'Battery voltage is 1' repeated with nearly identical content 418 times, suppressed by syslog-ng on flahblargle A few of my colleagues have suggested that this is probably more trouble than it's worth, and we should just fix the spammy app(s) rather than try to 'outsmart' them. And I think I agree with them, but... it was my idea to move to syslog-ng, so there'll be a bit more egg on my face than theirs if we accidentally allow a rogue printf() in a hot loop somewhere to escape into production. `:-} Any advice/insight much appreciated, thanks!

4 6

Email
by Anwar El fatayri 22 May '14

22 May '14

syslog-ng(a)lists.balabit.hu El Fatayri Anwar

1 0

Handing over syslog-ng maintenance
by Balazs Scheidler 21 May '14

21 May '14

Dear syslog-ng users, I would like to announce that Gergely Nagy (aka @algernoone on Twitter and algernon on github), is taking over the maintenance of syslog-ng master from me. He was maintaining the older stable versions for a few years now, was also active in creating syslog-ng binaries and started the syslog-ng Incubator, and is also behind the Google Summer of Code effort. I have a lot of distractions in my personal life, and although I don't want to retreat from the syslog-ng community entirely, I wanted to leave syslog-ng in Gergely's able hands. Happy logging, Cheers, Bazsi -- Bazsi

4 3

insider 2014-05: GsoC; perl & python; web interfaces; protecting logs;
by Peter Czanik 15 May '14

15 May '14

Dear syslog-ng users, This is the 34th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. FEATURED NEWS Four GsoC students are working on syslog-ng ------------------------------------------- Thanks to Google, there are four students working on extending syslog-ng with new features during the summer. These are features, which were often requested on the mailing list or at different conferences: - integration with configuration management systems - ZMQ transport, both source and destination - AMQP source driver - TLS support for the mongodb destination Read more about GSoC at https://algernon.blogs.balabit.com/2014/04/gsoc2014-syslog-ng-accepted-proj… Python and Perl support in incubator ------------------------------------ It is still only available in git, as it needs some more polish, but the syslog-ng incubator gained Perl and Python support during the last month. Both the perl and python destinations use the value-pairs framework to get data transferred from syslog-ng to the script, and thus, work differently from the Lua destination. With value-pairs, one can select what parts of the message will be transferred to the script. The script will need to have a queue function (settable with the queue-func() option), which will receive a hash-map of values. Additionally, one can set an init and a deinit function too, to run whenever the driver starts or shuts down. Check it out and let us know your experiences! It's available athttps://github.com/balabit/syslog-ng-incubator Web based user interfaces for syslog-ng --------------------------------------- One of the most popular BalaBit blogs is about syslog-ng web based graphical user interfaces (web GUIs). It's already three years old, and many things have changed. At that time, only a single Logging as a Service solution was available, now a new one pops up regularly. Also, there were a lot less logging-related GUIs, so some not strictly syslog-ng related solutions were included as well. You can read an updated version of the blog, focusing on syslog-ng based solutions at https://czanik.blogs.balabit.com/2014/05/web-based-user-interfaces-for-sysl… Protecting log data against targeted attacks -------------------------------------------- BalaBit has been saying that SIEM and other analytic tools are only as good as the underlying data. Attackers are also aware of this, and often target log management and SIEMs to hide their presence. Read this blog post for some logging best practices and how syslog-ng can help to secure your logging infrastructure: https://jluby.blogs.balabit.com/2014/05/06/protecting-log-data-against-targ… NEW RELEASES * Check git if you are impatient :) Your feedback and news tips about the next issue is welcome at _documentation(a)balabit.com <mailto:documentation@balabit.com>_To read this newsletter on-line, visit: _http://insider.blogs.balabit.com/_ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik

1 0

weird disk usage problems -- syslog-ng/ELSA
by Russell Fulton 11 May '14

11 May '14

HI Folks It would seem that the issue here is with syslog-ng but it manifested itself in the context of an ELSA instance so I am sending this to both lists, apologies to those who get it twice. I just discovered that my ELSA machine had run out of disk some 18 hours ago. /dev/mapper/datavg-d01 985G 943G 0 100% /d01 [rful011@itslogprd05 ~]$ sudo du -sh /d01/* 671G /d01/data 16K /d01/lost+found 43G /d01/mysql But these figures don’t match — there is about 250GB of disk unaccounted for (although I did not pick up this discrepancy at first — instead I went looking in ELSA node.log to try and figure out why logs were not getting deleted). After failing to find any evidence of problems in the logs I then stopped syslog-ng and set about moving the tmp/buffers to another disk. I copied them but when I went to delete the originals they had vanished! I was dazed and confused ;) until I realised that the usage of /d01 had dropped to 77% which allowed cron.pl to process all the waiting buffers and delete them. It would seem that syslog-ng was sitting on some very large files that had been unlinked from the directory (so du did not see them) stopping syslog-ng would have closed the files allowing the disk to be released. Hmmm…. with syslog-ng it might have been fifos (the ELSA syslog-ng config uses fifos to deliver logs to the ELSA perl scripts) but they should not have had buffers that size. Has anyone seen anything like this before or have any idea what might trigger this issue? Russell

2 1