[syslog-ng] [PATCH 2/3] afmongodb: Support type hints

[Gergely Nagy]

Gergely Nagy <algernon at balabit.hu> writes:

> Balazs Scheidler <bazsi77 at gmail.com> writes:
>
>>> +       case TEMPLATE_TYPE_INT32:
>>> +           {
>>> +               gchar *endptr;
>>> +               gint32 i = (gint32)strtoul(value, &endptr, 10);
>>> +
>>> +               if (endptr[0] == '\0')
>>> +                   bson_append_int32 (o, name, i);
>>> +
>>> +               break;
>>> +           }
>>
>> are you sure it's a good idea to drop data for invalid formats? i'd
>> have added as strings in this case. this applies to all cases.
>
> I'm not convinced, but the reason behind dropping was that if the user
> requested an int type, putting in string is not what one would expect.
>
> Probably it would be better to propagate some kind of error to higher
> layers, and report it back, though.

After talking about this in-house, the way these kind of things should
be handled is as follows:

We'd have a global (overridable per-destination) setting called
type-hint(strict|weak|fallback).

If set to strict (default), then any typecasting error should result in
the message being discarded, and an appropriate error logged.

If set to weak, then any typecasting error should result in the value
beging discarded (but the rest of the message not), with a warning
produced aswell.

If set to fallback, then any typecasting error should result in the
value getting stored as string instead, with a warning produced aswell.

Any of these can be prefixed with "silent-", which turns the
error/warning off.

>
>>> +       case TEMPLATE_TYPE_DATETIME:
>>> +           {
>>> +               gchar *endptr;
>>> +               gint64 i = (gint64)strtoul(value, &endptr, 10) * 1000;
>>> +
>>> +               if (endptr[0] == '\0')
>>> +                   bson_append_utc_datetime (o, name, i);
>>> +
>>> +               break;
>>> +           }
>>
>> we might need to support a format with broken-down timestamp. an iso
>> stamp might easily be formatted using a template, and the log message
>> may carry such date in its payload.

I moved the parsing & casting stuff to lib/type-hint.[ch], which makes
it possible to extend type_cast_to_datetime_int() so that it recognises
not only $UNIXTIME, but other formats aswell.

This way, anything that uses these helper functions (only mongodb for
now), will support them all automatically. Some drivers (format-json,
for example) do not use the helpers though.

-- 
|8]