[syslog-ng] [PATCH (3.4)] afmongodb: Add support for datetime fields

[Gergely Nagy]

Gergely Nagy <algernon at balabit.hu> writes:

>> destination d_sql {
>>
>> 	sql(columns('id', 'now datetime', 'bar'),
>> 	    values("$ID", default, "$BAR"));
>> };
>>
>> Here, 'default' means to use the default for the column. If
>> value-pairs() could grow such a functionality, that would be great.
>
> I'm not entirely sure how and where DEFAULT would fit into
> value-pairs()...

I was thinking on the way to work, and translating DEFAULT to
value-pairs could be like:

value-pairs(pair("some.date", default()))

This would end up with VP doing pretty much nothing, and leaving the
handling of default up to the callback. That is, default would be just
another "type", and the callback would be called with an empty value,
and type set to VALUE_PAIR_TYPE_DEFAULT (or something like that). Then
the driver could figure out what to do based on the name.

>> I'd probably call this function "casting", casting the output of the
>> template to some destination specific type or function.
>>
>> This could look like this in mongodb:
>>
>>
>> mongodb(
>>         value-pairs(
>>           key("MESSAGE")
>>           pair("some.date", datetime("$UNIXTIME"))))
>
> That's an option yeah, and not even too hard to make it happen, and it
> would even be friendlier to future enhancements (such as types stored in
> nvtable). This would also make the implementation simpler, the way I did
> the type check in this patch isn't exactly nice.

I already have a PoC written in my head, which boilds down to adding a
new 'metadata' argument to the value-pairs callbacks. The reason I don't
do the conversion within vp itself, is because drivers may interpret the
various types differently (eg, datetime is different for SQL and
MongoDB).

Armed with this metadata (which, for now, will only contain type
information), the drivers can Do The Right Thing. VP could provide
typecast functions for the common types (integers, in particular).

So, the mongodb value-pairs callback would become this:

static gboolean
afmongodb_vp_process_value(const gchar *name, const gchar *prefix,
                           const gchar *value, const value_pairs_meta_t *meta,
                           gpointer *prefix_data, gpointer user_data)
{
  bson *o;

  if (prefix_data)
    o = (bson *)*prefix_data;
  else
    o = (bson *)(((gpointer *)user_data)[0]);

  if (name[0] == '.')
    {
      gchar tx_name[256];

      tx_name[0] = '_';
      strncpy(&tx_name[1], name + 1, sizeof(tx_name) - 1);
      tx_name[sizeof(tx_name) - 1] = 0;
      bson_append_string (o, tx_name, value, -1);
    }
  else
    {
      switch (meta->type)
        {
          case DATA_TYPE_INT32:
            {
              gint32 n;

              if (value_pairs_typecast_int32(value, &n))
                bson_append_int32(o, name, n);

              break;
            }
          [...]
          case DATA_TYPE_STRING:
            bson_append_string(o, name, value, -1);
            break;
        }
    }

  return FALSE;
}

This looks reasonably nice to me, and having the type in a generic
metadata argument allows us to add further meta-data to it later, if so
need be. (I already have a few ideas where this would be useful, but
more on that later)

>> +template_content
>> +        : string                               { /* simple template */ }
>> +        | LL_IDENTIFIER '(' string ')'         { /* cast, store $1 together with the template to be interpreted by the caller */
>> +        ;
>> +
>
> Mmmmhm. Neat. One thing that this is missing though, is a way to verify
> that the specified type is available, and efficient storage for it: I
> wouldn't really want to strcmp() it against N things during a
> value_pairs_foreach(), nor do I wish to defer validating the cast until
> then.
>
> Having a global list of types would perhaps be useful, even if it's not
> as flexible as it could be. (We'll need something like that for storing
> types in nvtable later anyway, as far as I see)

One thing to note here, though, is that while templates could hold
typecast data, we'll need some type stuff for nvtable too, at some
point.

In the long run, I want to be able to teach the JSON parser to also
record the data type, and pass it along, so that tcp-source ->
json-parser -> format-json would automatically preserve as much type
information as possible. Or when doing syslog()->format-json, we'd have
PID as a number by default.

-- 
|8]