[syslog-ng] Writing a null character at the end of TCP payload

[Nicolas Fédou]

Hi,
It seems really good to me.

I think you have made up the answer we need.


Thank you.



2014-08-10 21:19 GMT+02:00 Balazs Scheidler <bazsi77 at gmail.com>:

> Hi,
>
> I would prefer the creation of $(format-gelf) which would solve both the
> NUL issue and makes it easier for someone else to create GELF output.
>
> Or alternatively, add an option to $(format-json) to add a suffix (and
> perhaps a prefix) in front of and after the resulting string. That would be
> useful for other purposes as well, and then $(format-gelf) could be
> implemented by my latest patches that enable the definition of template
> functions within the configuration file.
>
> E.g.
>
> template-function "format-gelf" "$(format-json --suffix "\x0" <format-json
> arguments to produce gelf>)"
>
> destination d_graylog {
>     network("1.2.3.4" transport(tcp) template("$(format-gelf)"); };
> };
>
>
> What do you think?
>
>
>
> On Thu, Aug 7, 2014 at 8:02 PM, Nicolas Fédou <nicolas.fedou at gmail.com>
> wrote:
>
>> Yes, I am pretty sure as I have tested it (I posted the answer) :
>>
>> http://serverfault.com/questions/591758/send-echo-message-to-graylog2-via-gelf-tcp-12201-port
>> After finding a clue in Graylog's issues :
>>
>> https://github.com/Graylog2/graylog2-server/issues/127#issuecomment-17563306
>>
>>
>> Now, a function like template-escape() that may be called
>> "template-null-ending()" or any other may be easyer to deliver.
>> Gelf has mandatory fields, a compression option and seems to read only
>> the last GELF message per tcp frames as :
>> { gelf 1 }\x0{ gelf 2 }\x0
>> Graylog shows only "gelf  2"
>> So, it forces the use of flush-lines(1).
>>
>>
>>
>>
>> 2014-08-07 18:20 GMT+02:00 Balazs Scheidler <bazsi77 at gmail.com>:
>>
>> Hi,
>>>
>>> Adding a NUL via the template is difficult as the template compiler
>>> works with zero terminated string, and even though the lexer allows the use
>>> of \x0, this will indicate the end-of-the-string like you have seen it.
>>>
>>> Adding a zero byte would be possible by writing a template function
>>> $(format-gelf) and that probably could sit within the json module, reusing
>>> the infrastructure there.
>>>
>>> On the other hand, I've checked the GELF specification, and I can't see
>>> the NUL byte being required.
>>>
>>> http://graylog2.org/gelf#specs
>>>
>>> Are you sure this is the issue?
>>>
>>> Bazsi
>>>
>>>
>>> On Thu, Aug 7, 2014 at 11:09 AM, Nicolas Fédou <nicolas.fedou at gmail.com>
>>> wrote:
>>>
>>>> Hello,
>>>> I need to send a null character at the end of a message on a tcp
>>>> output...
>>>> But this null is considered to be an end of string.
>>>>
>>>> My use case is sending a GELF message to a graylog's tcp input.
>>>> GELF message is a json syntax quite simple to comply with thanks to
>>>> format-json.
>>>> Graylog needs a null character at the end of a GELF message in TCP but
>>>> not in UDP.
>>>>
>>>> I tried many combination in the template with \x0, echo \x0,
>>>> `global_definition_of_null`, etc...
>>>> The matter is that null character when interpreted is an end of string,
>>>> and is not written in the tcp message.
>>>> I see truncated messages with ngrep.
>>>> Like "a\x0b" gives "a" according to ngrep.
>>>>
>>>> I did not find any option to actually write a null character.
>>>> Do you have any options ?
>>>>
>>>> As Graylog already have clients and librairies I belive they won't
>>>> remove the need for a null character.
>>>> Shall I open an issue on syslog-ng to ask for an option in templates or
>>>> in format-json ?
>>>>
>>>> Regards, Nicolas Fédou.
>>>>
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Bazsi
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
>
> --
> Bazsi
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20140810/e9c6b9eb/attachment.htm