[syslog-ng] Handle multiple encoding from multiple sources

[Balazs Scheidler]

On Thu, 2009-06-18 at 17:23 +0200, David Perret wrote:
> Hello,
> 
> I'm trying to understand how syslog-ng handle charset encoding.
> 
> I've seen that :
> 
> - without the "encoding("charset")" keyword in the source it just writes 
> to the output what it gets from the input.
> - with the "encoding("charset")" keyword in the source it converts what 
> it gets from the output from "charset" to UTF-8.
> 
> What is the best way to handle logs from different charset? Do I really 
> have to create a different source per charset used ? 

Currently yes, but read on.

> Is there any way to 
> say that I always want the input's charset to be converted to UTF-8 
> before writing my logs ?

I don't see any magic solution that could deduce what encoding a given
string uses before converting that to utf8. So in order to do that
conversion I need the source charset.

You need separate sources for each charset because that option
completely changes the way the incoming byte-stream is processed. For
example the NL character can be quite different from ASCII #10 if you
used UCS2 (two byte unicode what Windows uses for example), this way the
code to parse records out of the bytestream would not recognize the
record separator.

Thus it is not possible to set or discover the encoding on a per-message
basis.

What you could do to avoid lots of source() statements is to convert the
messages to utf8 on the relays/end hosts, this way the central would
only need to process utf8.

-- 
Bazsi