[syslog-ng] Encoding provlem when writing to postgres

[Balazs Scheidler]

On Wed, 2008-11-05 at 16:25 +0100, Balazs Scheidler wrote:
> On Wed, 2008-11-05 at 15:36 +0100, ml at bortal.de wrote:
> > Hey Balazs,
> > 
> > i think ai am almost there yet ;)
> > (see below...)
> > 
> > Balazs Scheidler wrote:
> > >> Am 15.10.2008 um  Uhr haben Sie geschrieben:
> > >>     
> > >>> On Wed, 2008-10-15 at 09:53 +0200, ml at bortal.de wrote:
> > >>>       
> > >>>> Hello List,
> > >>>>
> > >>>> How can i fix this? I am running postgresql-8.1 with Debian 4.0.
> > >>>>         
> > >>> Probably the syslog client sends non-utf8 characters in its message.
> > >>> syslog-ng 2.0 and 2.1 do not really care about the message contents.
> > >>>
> > >>> 3.0 has support for various encodings and utf8 validation like this:
> > >>>
> > >>> This assumes that the input is latin1:
> > >>>
> > >>> source s_net { udp(encoding("iso-8859-1")); };
> > >>>
> > >>> And this one enforces valid utf8 messages
> > >>>
> > >>> source s_net { udp(flags(validate-utf8)); };
> > >>>
> > >>> I was planning to add another flag, but this is not yet implemented:
> > >>>
> > >>> source s_net { udp(flags(force-utf8)); };
> > >>>
> > >>> Which would enforce valid utf8 sequences by changing the input.
> > >>>
> > >>> As an alternative don't use utf8 as the character encoding of your
> > >>> database, use latin1, that'll permit any kind of data in the database.
> > >>>
> > >>>       
> > syslog-ng -F -v -f /etc/syslog-ng.conf -e
> > Configuration file has no version number, assuming syslog-ng 2.1 format. 
> > Please add @version: maj.min to the beginning of the file;
> > WARNING: global: the default value of chain_hostnames is changing to 
> > 'no' in version 3.0, please update your configuration accordingly;
> > Your configuration file uses an obsoleted keyword, please update your 
> > configuration; keyword='sync', change='flush_lines'
> > Your configuration file uses an obsoleted keyword, please update your 
> > configuration; keyword='stats', change='stats_freq'
> > WARNING: input: sources do not remove new-line characters from messages 
> > by default in version 3.0, please add 'no-multi-line' flag to your 
> > configuration if you want to retain this functionality;
> > WARNING: file source: default value of follow_freq in file sources is 
> > changing in 3.0 to '1' for all files except /proc/kmsg;
> > syntax error in /etc/syslog-ng.conf at line 14.
> > 
> > 
> > Line 14 and 15 look like this:
> > -------------------------------------------------------------
> > source s_net { udp(encoding("iso-8859-1")); };
> > source s_net { udp(flags(validate-utf8)); };
> > 
> > I also tried to add: @version: 2.9 to the config file, but this did not 
> > work.
> > 
> 
> As I wrote in my previous mail, the current 2.9 snapshots are equal to
> the upcoming 3.0 version, I just wanted to "reserve" the 3.0.0 release.
> But since this is more confusing than it should be, I change the version
> number in the current git tree to 3.0, so the nightly snapshots will be
> named 3.0+date+time.
> 
> The @version: stanza should include the version number "3.0", but it is
> not related to your problem. I'll copy-paste your configuration in my
> working directory and check what the problem might be.
> 
> > 

The problem is almost certainly not at either of the quoted lines above,
for me it parses this configuration all-right. If you send me your
complete configuration, I'll check where the problem is. Alternatively
you can always use the -y command line option to show parser debug
messages, but this needs the  --enable-debug  configuration option.


-- 
Bazsi