[syslog-ng] Encoding provlem when writing to postgres

[Balazs Scheidler]

On Wed, 2008-11-05 at 15:36 +0100, ml at bortal.de wrote:
> Hey Balazs,
> 
> i think ai am almost there yet ;)
> (see below...)
> 
> Balazs Scheidler wrote:
> >> Am 15.10.2008 um  Uhr haben Sie geschrieben:
> >>     
> >>> On Wed, 2008-10-15 at 09:53 +0200, ml at bortal.de wrote:
> >>>       
> >>>> Hello List,
> >>>>
> >>>> How can i fix this? I am running postgresql-8.1 with Debian 4.0.
> >>>>         
> >>> Probably the syslog client sends non-utf8 characters in its message.
> >>> syslog-ng 2.0 and 2.1 do not really care about the message contents.
> >>>
> >>> 3.0 has support for various encodings and utf8 validation like this:
> >>>
> >>> This assumes that the input is latin1:
> >>>
> >>> source s_net { udp(encoding("iso-8859-1")); };
> >>>
> >>> And this one enforces valid utf8 messages
> >>>
> >>> source s_net { udp(flags(validate-utf8)); };
> >>>
> >>> I was planning to add another flag, but this is not yet implemented:
> >>>
> >>> source s_net { udp(flags(force-utf8)); };
> >>>
> >>> Which would enforce valid utf8 sequences by changing the input.
> >>>
> >>> As an alternative don't use utf8 as the character encoding of your
> >>> database, use latin1, that'll permit any kind of data in the database.
> >>>
> >>>       
> syslog-ng -F -v -f /etc/syslog-ng.conf -e
> Configuration file has no version number, assuming syslog-ng 2.1 format. 
> Please add @version: maj.min to the beginning of the file;
> WARNING: global: the default value of chain_hostnames is changing to 
> 'no' in version 3.0, please update your configuration accordingly;
> Your configuration file uses an obsoleted keyword, please update your 
> configuration; keyword='sync', change='flush_lines'
> Your configuration file uses an obsoleted keyword, please update your 
> configuration; keyword='stats', change='stats_freq'
> WARNING: input: sources do not remove new-line characters from messages 
> by default in version 3.0, please add 'no-multi-line' flag to your 
> configuration if you want to retain this functionality;
> WARNING: file source: default value of follow_freq in file sources is 
> changing in 3.0 to '1' for all files except /proc/kmsg;
> syntax error in /etc/syslog-ng.conf at line 14.
> 
> 
> Line 14 and 15 look like this:
> -------------------------------------------------------------
> source s_net { udp(encoding("iso-8859-1")); };
> source s_net { udp(flags(validate-utf8)); };
> 
> I also tried to add: @version: 2.9 to the config file, but this did not 
> work.
> 

As I wrote in my previous mail, the current 2.9 snapshots are equal to
the upcoming 3.0 version, I just wanted to "reserve" the 3.0.0 release.
But since this is more confusing than it should be, I change the version
number in the current git tree to 3.0, so the nightly snapshots will be
named 3.0+date+time.

The @version: stanza should include the version number "3.0", but it is
not related to your problem. I'll copy-paste your configuration in my
working directory and check what the problem might be.

> 
-- 
Bazsi