Writing a null character at the end of TCP payload
Hello, I need to send a null character at the end of a message on a tcp output... But this null is considered to be an end of string. My use case is sending a GELF message to a graylog's tcp input. GELF message is a json syntax quite simple to comply with thanks to format-json. Graylog needs a null character at the end of a GELF message in TCP but not in UDP. I tried many combination in the template with \x0, echo \x0, `global_definition_of_null`, etc... The matter is that null character when interpreted is an end of string, and is not written in the tcp message. I see truncated messages with ngrep. Like "a\x0b" gives "a" according to ngrep. I did not find any option to actually write a null character. Do you have any options ? As Graylog already have clients and librairies I belive they won't remove the need for a null character. Shall I open an issue on syslog-ng to ask for an option in templates or in format-json ? Regards, Nicolas Fédou.
Hi, Adding a NUL via the template is difficult as the template compiler works with zero terminated string, and even though the lexer allows the use of \x0, this will indicate the end-of-the-string like you have seen it. Adding a zero byte would be possible by writing a template function $(format-gelf) and that probably could sit within the json module, reusing the infrastructure there. On the other hand, I've checked the GELF specification, and I can't see the NUL byte being required. http://graylog2.org/gelf#specs Are you sure this is the issue? Bazsi On Thu, Aug 7, 2014 at 11:09 AM, Nicolas Fédou <nicolas.fedou@gmail.com> wrote:
Hello, I need to send a null character at the end of a message on a tcp output... But this null is considered to be an end of string.
My use case is sending a GELF message to a graylog's tcp input. GELF message is a json syntax quite simple to comply with thanks to format-json. Graylog needs a null character at the end of a GELF message in TCP but not in UDP.
I tried many combination in the template with \x0, echo \x0, `global_definition_of_null`, etc... The matter is that null character when interpreted is an end of string, and is not written in the tcp message. I see truncated messages with ngrep. Like "a\x0b" gives "a" according to ngrep.
I did not find any option to actually write a null character. Do you have any options ?
As Graylog already have clients and librairies I belive they won't remove the need for a null character. Shall I open an issue on syslog-ng to ask for an option in templates or in format-json ?
Regards, Nicolas Fédou.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
Yes, I am pretty sure as I have tested it (I posted the answer) : http://serverfault.com/questions/591758/send-echo-message-to-graylog2-via-ge... After finding a clue in Graylog's issues : https://github.com/Graylog2/graylog2-server/issues/127#issuecomment-17563306 Now, a function like template-escape() that may be called "template-null-ending()" or any other may be easyer to deliver. Gelf has mandatory fields, a compression option and seems to read only the last GELF message per tcp frames as : { gelf 1 }\x0{ gelf 2 }\x0 Graylog shows only "gelf 2" So, it forces the use of flush-lines(1). 2014-08-07 18:20 GMT+02:00 Balazs Scheidler <bazsi77@gmail.com>:
Hi,
Adding a NUL via the template is difficult as the template compiler works with zero terminated string, and even though the lexer allows the use of \x0, this will indicate the end-of-the-string like you have seen it.
Adding a zero byte would be possible by writing a template function $(format-gelf) and that probably could sit within the json module, reusing the infrastructure there.
On the other hand, I've checked the GELF specification, and I can't see the NUL byte being required.
http://graylog2.org/gelf#specs
Are you sure this is the issue?
Bazsi
On Thu, Aug 7, 2014 at 11:09 AM, Nicolas Fédou <nicolas.fedou@gmail.com> wrote:
Hello, I need to send a null character at the end of a message on a tcp output... But this null is considered to be an end of string.
My use case is sending a GELF message to a graylog's tcp input. GELF message is a json syntax quite simple to comply with thanks to format-json. Graylog needs a null character at the end of a GELF message in TCP but not in UDP.
I tried many combination in the template with \x0, echo \x0, `global_definition_of_null`, etc... The matter is that null character when interpreted is an end of string, and is not written in the tcp message. I see truncated messages with ngrep. Like "a\x0b" gives "a" according to ngrep.
I did not find any option to actually write a null character. Do you have any options ?
As Graylog already have clients and librairies I belive they won't remove the need for a null character. Shall I open an issue on syslog-ng to ask for an option in templates or in format-json ?
Regards, Nicolas Fédou.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, I would prefer the creation of $(format-gelf) which would solve both the NUL issue and makes it easier for someone else to create GELF output. Or alternatively, add an option to $(format-json) to add a suffix (and perhaps a prefix) in front of and after the resulting string. That would be useful for other purposes as well, and then $(format-gelf) could be implemented by my latest patches that enable the definition of template functions within the configuration file. E.g. template-function "format-gelf" "$(format-json --suffix "\x0" <format-json arguments to produce gelf>)" destination d_graylog { network("1.2.3.4" transport(tcp) template("$(format-gelf)"); }; }; What do you think? On Thu, Aug 7, 2014 at 8:02 PM, Nicolas Fédou <nicolas.fedou@gmail.com> wrote:
Yes, I am pretty sure as I have tested it (I posted the answer) :
http://serverfault.com/questions/591758/send-echo-message-to-graylog2-via-ge... After finding a clue in Graylog's issues :
https://github.com/Graylog2/graylog2-server/issues/127#issuecomment-17563306
Now, a function like template-escape() that may be called "template-null-ending()" or any other may be easyer to deliver. Gelf has mandatory fields, a compression option and seems to read only the last GELF message per tcp frames as : { gelf 1 }\x0{ gelf 2 }\x0 Graylog shows only "gelf 2" So, it forces the use of flush-lines(1).
2014-08-07 18:20 GMT+02:00 Balazs Scheidler <bazsi77@gmail.com>:
Hi,
Adding a NUL via the template is difficult as the template compiler works with zero terminated string, and even though the lexer allows the use of \x0, this will indicate the end-of-the-string like you have seen it.
Adding a zero byte would be possible by writing a template function $(format-gelf) and that probably could sit within the json module, reusing the infrastructure there.
On the other hand, I've checked the GELF specification, and I can't see the NUL byte being required.
http://graylog2.org/gelf#specs
Are you sure this is the issue?
Bazsi
On Thu, Aug 7, 2014 at 11:09 AM, Nicolas Fédou <nicolas.fedou@gmail.com> wrote:
Hello, I need to send a null character at the end of a message on a tcp output... But this null is considered to be an end of string.
My use case is sending a GELF message to a graylog's tcp input. GELF message is a json syntax quite simple to comply with thanks to format-json. Graylog needs a null character at the end of a GELF message in TCP but not in UDP.
I tried many combination in the template with \x0, echo \x0, `global_definition_of_null`, etc... The matter is that null character when interpreted is an end of string, and is not written in the tcp message. I see truncated messages with ngrep. Like "a\x0b" gives "a" according to ngrep.
I did not find any option to actually write a null character. Do you have any options ?
As Graylog already have clients and librairies I belive they won't remove the need for a null character. Shall I open an issue on syslog-ng to ask for an option in templates or in format-json ?
Regards, Nicolas Fédou.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
Hi, It seems really good to me. I think you have made up the answer we need. Thank you. 2014-08-10 21:19 GMT+02:00 Balazs Scheidler <bazsi77@gmail.com>:
Hi,
I would prefer the creation of $(format-gelf) which would solve both the NUL issue and makes it easier for someone else to create GELF output.
Or alternatively, add an option to $(format-json) to add a suffix (and perhaps a prefix) in front of and after the resulting string. That would be useful for other purposes as well, and then $(format-gelf) could be implemented by my latest patches that enable the definition of template functions within the configuration file.
E.g.
template-function "format-gelf" "$(format-json --suffix "\x0" <format-json arguments to produce gelf>)"
destination d_graylog { network("1.2.3.4" transport(tcp) template("$(format-gelf)"); }; };
What do you think?
On Thu, Aug 7, 2014 at 8:02 PM, Nicolas Fédou <nicolas.fedou@gmail.com> wrote:
Yes, I am pretty sure as I have tested it (I posted the answer) :
http://serverfault.com/questions/591758/send-echo-message-to-graylog2-via-ge... After finding a clue in Graylog's issues :
https://github.com/Graylog2/graylog2-server/issues/127#issuecomment-17563306
Now, a function like template-escape() that may be called "template-null-ending()" or any other may be easyer to deliver. Gelf has mandatory fields, a compression option and seems to read only the last GELF message per tcp frames as : { gelf 1 }\x0{ gelf 2 }\x0 Graylog shows only "gelf 2" So, it forces the use of flush-lines(1).
2014-08-07 18:20 GMT+02:00 Balazs Scheidler <bazsi77@gmail.com>:
Hi,
Adding a NUL via the template is difficult as the template compiler works with zero terminated string, and even though the lexer allows the use of \x0, this will indicate the end-of-the-string like you have seen it.
Adding a zero byte would be possible by writing a template function $(format-gelf) and that probably could sit within the json module, reusing the infrastructure there.
On the other hand, I've checked the GELF specification, and I can't see the NUL byte being required.
http://graylog2.org/gelf#specs
Are you sure this is the issue?
Bazsi
On Thu, Aug 7, 2014 at 11:09 AM, Nicolas Fédou <nicolas.fedou@gmail.com> wrote:
Hello, I need to send a null character at the end of a message on a tcp output... But this null is considered to be an end of string.
My use case is sending a GELF message to a graylog's tcp input. GELF message is a json syntax quite simple to comply with thanks to format-json. Graylog needs a null character at the end of a GELF message in TCP but not in UDP.
I tried many combination in the template with \x0, echo \x0, `global_definition_of_null`, etc... The matter is that null character when interpreted is an end of string, and is not written in the tcp message. I see truncated messages with ngrep. Like "a\x0b" gives "a" according to ngrep.
I did not find any option to actually write a null character. Do you have any options ?
As Graylog already have clients and librairies I belive they won't remove the need for a null character. Shall I open an issue on syslog-ng to ask for an option in templates or in format-json ?
Regards, Nicolas Fédou.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Balazs Scheidler <bazsi77@gmail.com> writes:
I would prefer the creation of $(format-gelf) which would solve both the NUL issue and makes it easier for someone else to create GELF output.
FWIW, since there is no GELF C library out there, I started hacking on one (which would also support chunking, and would provide an API we can plug into LogWriter, so we won't have to make it a LogThreadedDestDriver). It will take a while to finish it, because I'm playing with getting the API right.
Or alternatively, add an option to $(format-json) to add a suffix (and perhaps a prefix) in front of and after the resulting string. That would be useful for other purposes as well, and then $(format-gelf) could be implemented by my latest patches that enable the definition of template functions within the configuration file.
E.g.
template-function "format-gelf" "$(format-json --suffix "\x0" <format-json arguments to produce gelf>)"
destination d_graylog { network("1.2.3.4" transport(tcp) template("$(format-gelf)"); }; };
How about $(char 0) as a template function? That can be reused outside of format-json. -- |8]
Hi, Both ways answer my needs. I prefer to let a commiter/contributor choose the beter solution for the future of Syslog-ng. By the way, I shall ask a feature un the bug tracking, or you already have one ? Regards, Nicolas. Le 11 août 2014 09:01, "Gergely Nagy" <algernon@balabit.hu> a écrit :
Balazs Scheidler <bazsi77@gmail.com> writes:
I would prefer the creation of $(format-gelf) which would solve both the NUL issue and makes it easier for someone else to create GELF output.
FWIW, since there is no GELF C library out there, I started hacking on one (which would also support chunking, and would provide an API we can plug into LogWriter, so we won't have to make it a LogThreadedDestDriver). It will take a while to finish it, because I'm playing with getting the API right.
Or alternatively, add an option to $(format-json) to add a suffix (and perhaps a prefix) in front of and after the resulting string. That would be useful for other purposes as well, and then $(format-gelf) could be implemented by my latest patches that enable the definition of template functions within the configuration file.
E.g.
template-function "format-gelf" "$(format-json --suffix "\x0" <format-json arguments to produce gelf>)"
destination d_graylog { network("1.2.3.4" transport(tcp) template("$(format-gelf)"); }; };
How about $(char 0) as a template function? That can be reused outside of format-json.
-- |8]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Nicolas Fédou <nicolas.fedou@gmail.com> writes:
By the way, I shall ask a feature un the bug tracking, or you already have one ?
Please open an issue on GitHub[1], there isn't any for this feature yet. [1]: https://github.com/balabit/syslog-ng/issues -- |8]
Hello, It is done : https://github.com/balabit/syslog-ng/issues/206 Thank you. 2014-08-15 13:44 GMT+02:00 Gergely Nagy <algernon@balabit.hu>:
Nicolas Fédou <nicolas.fedou@gmail.com> writes:
By the way, I shall ask a feature un the bug tracking, or you already have one ?
Please open an issue on GitHub[1], there isn't any for this feature yet.
[1]: https://github.com/balabit/syslog-ng/issues
-- |8]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (3)
-
Balazs Scheidler
-
Gergely Nagy
-
Nicolas Fédou