[Bug 187] New: Syslog-ng truncates SDATA PARAM VALUE to 256 chars
https://bugzilla.balabit.com/show_bug.cgi?id=187 Summary: Syslog-ng truncates SDATA PARAM VALUE to 256 chars Product: syslog-ng Version: 3.2.x Platform: All OS/Version: All Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi@balabit.hu ReportedBy: cloun-rulez@yandex.ru QAContact: cloun-rulez@yandex.ru Type of the Report: enhancement Estimated Hours: 0.0 As you can see there are no limitations of SDATA PARAM VALUE length in rfc5424 http://tools.ietf.org/html/rfc5424 As I have saw in tests/unit/test_msgparse.c this is default behavior of syslog-ng. // too long sdata value gets truncated testcase("<132>1 2006-10-29T01:59:59.156+01:00 mymachine evntslog - - [a i=\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\"] An application event log entry...", LP_SYSLOG_PROTOCOL, NULL, 132, // pri 1162083599, 156000, 3600, // timestamp (sec/usec/zone) "mymachine", // host "evntslog", //app "An application event log entry...", // msg "[a i=\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\"]", //sd_str "",//processid "",//msgid expected_sd_pairs_test_5b ); I'm trying to increase limit of truncation but still haven't success. Need help. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=187 Gergely Nagy <algernon@balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |algernon@balabit.hu --- Comment #1 from Gergely Nagy <algernon@balabit.hu> 2012-07-11 13:48:41 --- This limitation is still present in both 3.3 and 3.4. I tried increasing the limit in 3.3, from 256 bytes to 4096, and changing the size of sd_param_value in log_msg_parse_sd (modules/syslogformat/syslog-format.c) did the trick. Nevertheless, the best would be to make the limit configurable at run-time, and allow for unlimited length too (but still default to 256). The hard part is that this function gets called often, so littering it with dynamic memory allocation would have a serious impact on performance. However, we could use scratch-buffers for the task, I believe. I'll try to find some time to explore this option. Nevertheless, this is a change I wouldn't want to do in 3.3 at this point, but rather try to solve it for 3.4 instead. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=187 --- Comment #2 from Balazs Scheidler <bazsi@balabit.hu> 2012-07-24 09:17:46 --- (In reply to comment #1)
This limitation is still present in both 3.3 and 3.4. I tried increasing the limit in 3.3, from 256 bytes to 4096, and changing the size of sd_param_value in log_msg_parse_sd (modules/syslogformat/syslog-format.c) did the trick.
Nevertheless, the best would be to make the limit configurable at run-time, and allow for unlimited length too (but still default to 256). The hard part is that this function gets called often, so littering it with dynamic memory allocation would have a serious impact on performance.
However, we could use scratch-buffers for the task, I believe. I'll try to find some time to explore this option. Nevertheless, this is a change I wouldn't want to do in 3.3 at this point, but rather try to solve it for 3.4 instead.
alloca() would not cause serious performance degradation while still keeping the size of the SDATA value dynamic. The stack size is 256k by default, so using a maximum limit of 64k could be realistic. Would that be enough in your use-case? Can you please post more details what you'd like to transmit in your SDATA section? -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=187 --- Comment #3 from Gergely Nagy <algernon@balabit.hu> 2012-07-24 12:07:18 --- (In reply to comment #2)
alloca() would not cause serious performance degradation while still keeping the size of the SDATA value dynamic. The stack size is 256k by default, so using a maximum limit of 64k could be realistic.
Would that be enough in your use-case? Can you please post more details what you'd like to transmit in your SDATA section?
I was thinking whether scratch-buffers could be used in this case? That would have some minor performance impact in corner cases when the buffer needs to grow, but otherwise it shouldn't, and would allow pretty much infinite length. I'm not entirely sure scratch-buffers are usable in this case, though. But if they are, they sound like a better approach than alloca() to me. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=187 --- Comment #4 from Sergey <cloun-rulez@yandex.ru> 2012-07-25 12:13:28 --- I'm increase as Gergely Nagy said but not to 4096 but to 64k. On my not extreme flow of logs in 200KB/s max I've not seen any degradation of performance. My use cases with largest SDATA VALUE PARAMS is: 1) Transmit url-encoded stack trace of python application (2-3KB usually enough) 2) Transmit url-encoded user monitoring statistic. Ala log.info('user have signed in', jsonData={user_id:id, ...}) Here I have url-encoded json struct in sdata value. ~10 kb usually enough for different monitored events. (In reply to comment #2)
(In reply to comment #1)
This limitation is still present in both 3.3 and 3.4. I tried increasing the limit in 3.3, from 256 bytes to 4096, and changing the size of sd_param_value in log_msg_parse_sd (modules/syslogformat/syslog-format.c) did the trick.
Nevertheless, the best would be to make the limit configurable at run-time, and allow for unlimited length too (but still default to 256). The hard part is that this function gets called often, so littering it with dynamic memory allocation would have a serious impact on performance.
However, we could use scratch-buffers for the task, I believe. I'll try to find some time to explore this option. Nevertheless, this is a change I wouldn't want to do in 3.3 at this point, but rather try to solve it for 3.4 instead.
alloca() would not cause serious performance degradation while still keeping the size of the SDATA value dynamic. The stack size is 256k by default, so using a maximum limit of 64k could be realistic.
Would that be enough in your use-case? Can you please post more details what you'd like to transmit in your SDATA section?
-- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.balabit.com/show_bug.cgi?id=187 Balazs Scheidler <bazsi@balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution| |FIXED Status|NEW |RESOLVED --- Comment #5 from Balazs Scheidler <bazsi@balabit.hu> 2012-07-27 07:46:23 --- I think bumping the default limit to 64k should suffice, no real need to make this absolutely unbounded, as this is the first actual request to increase the size. The patches below make this runtime configurable from the API side, even though it is not (yet) a configuration setting. Instead of alloca/scratch-buffers it uses a gcc extension, if that proves unportable we can always change that. I don't like allocating too much data from the stack, but I think 64k should be safe. The default stack limit on my x86_64 laptop is 8MB, I remember that being around 2MB on x86. commit bf69c41f696e7d94e424a40f5979bd5efcb33179 Author: Balazs Scheidler <bazsi@balabit.hu> Date: Fri Jul 27 07:42:18 2012 +0200 msg-format: make sd_param_value maximum length adjustable via API Although this option is not (yet) introduced in the configuration file, the unit tests also try to cover the 'too-long-sdata' case, which is inpractical in case the new 64k default is to be overflown. It might make sense to make this value actually controllable by the configuration file, but for now I think it is enough to simply bump the default limit to 64k. This patch also fixes the test_msgparse unit test so it passes again. Signed-off-by: Balazs Scheidler <bazsi@balabit.hu> commit ab9c013f7f06cbb024cae52905659450ddbdfaf1 Author: Balazs Scheidler <bazsi@balabit.hu> Date: Wed Jul 25 18:39:45 2012 +0200 syslog-format: increase the limit for SDATA value length to 64k Reported-By: Sergey <cloun-rulez@yandex.ru> Signed-off-by: Balazs Scheidler <bazsi@balabit.hu> -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
participants (1)
-
bugzilla＠bugzilla.balabit.com