Re: [syslog-ng] Encoding provlem when writing to postgres
Hi, where can i get syslog-ng 3.0? I cant find it on http://www.balabit.com/downloads/files/syslog-ng/sources/ Thanks, Mario Am 15.10.2008 um Uhr haben Sie geschrieben:
On Wed, 2008-10-15 at 09:53 +0200, ml@bortal.de wrote:
Hello List,
How can i fix this? I am running postgresql-8.1 with Debian 4.0.
Probably the syslog client sends non-utf8 characters in its message. syslog-ng 2.0 and 2.1 do not really care about the message contents.
3.0 has support for various encodings and utf8 validation like this:
This assumes that the input is latin1:
source s_net { udp(encoding("iso-8859-1")); };
And this one enforces valid utf8 messages
source s_net { udp(flags(validate-utf8)); };
I was planning to add another flag, but this is not yet implemented:
source s_net { udp(flags(force-utf8)); };
Which would enforce valid utf8 sequences by changing the input.
As an alternative don't use utf8 as the character encoding of your database, use latin1, that'll permit any kind of data in the database.
On Thu, 2008-10-16 at 08:46 +0200, ml@bortal.de wrote:
Hi,
where can i get syslog-ng 3.0?
I cant find it on http://www.balabit.com/downloads/files/syslog-ng/sources/
I've just sent out an announcement that it's available as a git repository. No formal releases have been made yet.
Thanks, Mario
Am 15.10.2008 um Uhr haben Sie geschrieben:
On Wed, 2008-10-15 at 09:53 +0200, ml@bortal.de wrote:
Hello List,
How can i fix this? I am running postgresql-8.1 with Debian 4.0.
Probably the syslog client sends non-utf8 characters in its message. syslog-ng 2.0 and 2.1 do not really care about the message contents.
3.0 has support for various encodings and utf8 validation like this:
This assumes that the input is latin1:
source s_net { udp(encoding("iso-8859-1")); };
And this one enforces valid utf8 messages
source s_net { udp(flags(validate-utf8)); };
I was planning to add another flag, but this is not yet implemented:
source s_net { udp(flags(force-utf8)); };
Which would enforce valid utf8 sequences by changing the input.
As an alternative don't use utf8 as the character encoding of your database, use latin1, that'll permit any kind of data in the database.
-- Bazsi
Hey Balazs, i think ai am almost there yet ;) (see below...) Balazs Scheidler wrote:
Am 15.10.2008 um Uhr haben Sie geschrieben:
On Wed, 2008-10-15 at 09:53 +0200, ml@bortal.de wrote:
Hello List,
How can i fix this? I am running postgresql-8.1 with Debian 4.0.
Probably the syslog client sends non-utf8 characters in its message. syslog-ng 2.0 and 2.1 do not really care about the message contents.
3.0 has support for various encodings and utf8 validation like this:
This assumes that the input is latin1:
source s_net { udp(encoding("iso-8859-1")); };
And this one enforces valid utf8 messages
source s_net { udp(flags(validate-utf8)); };
I was planning to add another flag, but this is not yet implemented:
source s_net { udp(flags(force-utf8)); };
Which would enforce valid utf8 sequences by changing the input.
As an alternative don't use utf8 as the character encoding of your database, use latin1, that'll permit any kind of data in the database.
syslog-ng -F -v -f /etc/syslog-ng.conf -e Configuration file has no version number, assuming syslog-ng 2.1 format. Please add @version: maj.min to the beginning of the file; WARNING: global: the default value of chain_hostnames is changing to 'no' in version 3.0, please update your configuration accordingly; Your configuration file uses an obsoleted keyword, please update your configuration; keyword='sync', change='flush_lines' Your configuration file uses an obsoleted keyword, please update your configuration; keyword='stats', change='stats_freq' WARNING: input: sources do not remove new-line characters from messages by default in version 3.0, please add 'no-multi-line' flag to your configuration if you want to retain this functionality; WARNING: file source: default value of follow_freq in file sources is changing in 3.0 to '1' for all files except /proc/kmsg; syntax error in /etc/syslog-ng.conf at line 14.
Line 14 and 15 look like this: ------------------------------------------------------------- source s_net { udp(encoding("iso-8859-1")); }; source s_net { udp(flags(validate-utf8)); }; I also tried to add: @version: 2.9 to the config file, but this did not work. Any idea? Thanks, Mario
Is 3.0 open source available? I could not find it on the download page, Thanks Joseph -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of ml@bortal.de Sent: Wednesday, November 05, 2008 8:36 AM To: Syslog-ng users' and developers' mailing list Cc: Balazs Scheidler Subject: Re: [syslog-ng] Encoding provlem when writing to postgres Hey Balazs, i think ai am almost there yet ;) (see below...) Balazs Scheidler wrote:
Am 15.10.2008 um Uhr haben Sie geschrieben:
On Wed, 2008-10-15 at 09:53 +0200, ml@bortal.de wrote:
Hello List,
How can i fix this? I am running postgresql-8.1 with Debian 4.0.
Probably the syslog client sends non-utf8 characters in its message. syslog-ng 2.0 and 2.1 do not really care about the message contents.
3.0 has support for various encodings and utf8 validation like this:
This assumes that the input is latin1:
source s_net { udp(encoding("iso-8859-1")); };
And this one enforces valid utf8 messages
source s_net { udp(flags(validate-utf8)); };
I was planning to add another flag, but this is not yet implemented:
source s_net { udp(flags(force-utf8)); };
Which would enforce valid utf8 sequences by changing the input.
As an alternative don't use utf8 as the character encoding of your database, use latin1, that'll permit any kind of data in the database.
syslog-ng -F -v -f /etc/syslog-ng.conf -e Configuration file has no version number, assuming syslog-ng 2.1 format. Please add @version: maj.min to the beginning of the file; WARNING: global: the default value of chain_hostnames is changing to 'no' in version 3.0, please update your configuration accordingly; Your configuration file uses an obsoleted keyword, please update your configuration; keyword='sync', change='flush_lines' Your configuration file uses an obsoleted keyword, please update your configuration; keyword='stats', change='stats_freq' WARNING: input: sources do not remove new-line characters from messages by default in version 3.0, please add 'no-multi-line' flag to your configuration if you want to retain this functionality; WARNING: file source: default value of follow_freq in file sources is changing in 3.0 to '1' for all files except /proc/kmsg; syntax error in /etc/syslog-ng.conf at line 14.
Line 14 and 15 look like this: ------------------------------------------------------------- source s_net { udp(encoding("iso-8859-1")); }; source s_net { udp(flags(validate-utf8)); }; I also tried to add: @version: 2.9 to the config file, but this did not work. Any idea? Thanks, Mario ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the originator of the message. This footer also confirms that this e-mail message has been scanned for the presence of computer viruses. Any views expressed in this message are those of the individual sender, except where the sender specifies and with authority, states them to be the views of Iowa Student Loan.
Check out: http://www.balabit.com/downloads/files/syslog-ng/sources/ Regards, Mario Joseph Pietras wrote:
Is 3.0 open source available? I could not find it on the download page, Thanks Joseph
-----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of ml@bortal.de Sent: Wednesday, November 05, 2008 8:36 AM To: Syslog-ng users' and developers' mailing list Cc: Balazs Scheidler Subject: Re: [syslog-ng] Encoding provlem when writing to postgres
Hey Balazs,
i think ai am almost there yet ;) (see below...)
Balazs Scheidler wrote:
Am 15.10.2008 um Uhr haben Sie geschrieben:
On Wed, 2008-10-15 at 09:53 +0200, ml@bortal.de wrote:
Hello List,
How can i fix this? I am running postgresql-8.1 with Debian 4.0.
Probably the syslog client sends non-utf8 characters in its message. syslog-ng 2.0 and 2.1 do not really care about the message contents.
3.0 has support for various encodings and utf8 validation like this:
This assumes that the input is latin1:
source s_net { udp(encoding("iso-8859-1")); };
And this one enforces valid utf8 messages
source s_net { udp(flags(validate-utf8)); };
I was planning to add another flag, but this is not yet implemented:
source s_net { udp(flags(force-utf8)); };
Which would enforce valid utf8 sequences by changing the input.
As an alternative don't use utf8 as the character encoding of your database, use latin1, that'll permit any kind of data in the database.
syslog-ng -F -v -f /etc/syslog-ng.conf -e Configuration file has no version number, assuming syslog-ng 2.1 format. Please add @version: maj.min to the beginning of the file; WARNING: global: the default value of chain_hostnames is changing to 'no' in version 3.0, please update your configuration accordingly; Your configuration file uses an obsoleted keyword, please update your configuration; keyword='sync', change='flush_lines' Your configuration file uses an obsoleted keyword, please update your configuration; keyword='stats', change='stats_freq' WARNING: input: sources do not remove new-line characters from messages by default in version 3.0, please add 'no-multi-line' flag to your configuration if you want to retain this functionality; WARNING: file source: default value of follow_freq in file sources is changing in 3.0 to '1' for all files except /proc/kmsg; syntax error in /etc/syslog-ng.conf at line 14.
Line 14 and 15 look like this: ------------------------------------------------------------- source s_net { udp(encoding("iso-8859-1")); }; source s_net { udp(flags(validate-utf8)); };
I also tried to add: @version: 2.9 to the config file, but this did not work.
Any idea?
Thanks, Mario
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the originator of the message. This footer also confirms that this e-mail message has been scanned for the presence of computer viruses.
Any views expressed in this message are those of the individual sender, except where the sender specifies and with authority, states them to be the views of Iowa Student Loan.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Wed, 2008-11-05 at 08:48 -0600, Joseph Pietras wrote:
Is 3.0 open source available? I could not find it on the download page, Thanks Joseph
Yes, as nightly snapshots, with the version number "2.9", but the only reason it has 2.9 as version number as it was not finalized yet. But you are right, this adds to the confusion, I change it to 3.0 and the first official release will be 3.0.1. The nightly snapshots can be found at: http://www.balabit.com/downloads/files/syslog-ng/sources/3.0/src-snapshot
-----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of ml@bortal.de Sent: Wednesday, November 05, 2008 8:36 AM To: Syslog-ng users' and developers' mailing list Cc: Balazs Scheidler Subject: Re: [syslog-ng] Encoding provlem when writing to postgres
Hey Balazs,
i think ai am almost there yet ;) (see below...)
Balazs Scheidler wrote:
Am 15.10.2008 um Uhr haben Sie geschrieben:
On Wed, 2008-10-15 at 09:53 +0200, ml@bortal.de wrote:
Hello List,
How can i fix this? I am running postgresql-8.1 with Debian 4.0.
Probably the syslog client sends non-utf8 characters in its message. syslog-ng 2.0 and 2.1 do not really care about the message contents.
3.0 has support for various encodings and utf8 validation like this:
This assumes that the input is latin1:
source s_net { udp(encoding("iso-8859-1")); };
And this one enforces valid utf8 messages
source s_net { udp(flags(validate-utf8)); };
I was planning to add another flag, but this is not yet implemented:
source s_net { udp(flags(force-utf8)); };
Which would enforce valid utf8 sequences by changing the input.
As an alternative don't use utf8 as the character encoding of your database, use latin1, that'll permit any kind of data in the database.
syslog-ng -F -v -f /etc/syslog-ng.conf -e Configuration file has no version number, assuming syslog-ng 2.1 format. Please add @version: maj.min to the beginning of the file; WARNING: global: the default value of chain_hostnames is changing to 'no' in version 3.0, please update your configuration accordingly; Your configuration file uses an obsoleted keyword, please update your configuration; keyword='sync', change='flush_lines' Your configuration file uses an obsoleted keyword, please update your configuration; keyword='stats', change='stats_freq' WARNING: input: sources do not remove new-line characters from messages by default in version 3.0, please add 'no-multi-line' flag to your configuration if you want to retain this functionality; WARNING: file source: default value of follow_freq in file sources is changing in 3.0 to '1' for all files except /proc/kmsg; syntax error in /etc/syslog-ng.conf at line 14.
Line 14 and 15 look like this: ------------------------------------------------------------- source s_net { udp(encoding("iso-8859-1")); }; source s_net { udp(flags(validate-utf8)); };
I also tried to add: @version: 2.9 to the config file, but this did not work.
Any idea?
Thanks, Mario
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the originator of the message. This footer also confirms that this e-mail message has been scanned for the presence of computer viruses.
Any views expressed in this message are those of the individual sender, except where the sender specifies and with authority, states them to be the views of Iowa Student Loan.
-- Bazsi
On Wed, 2008-11-05 at 15:36 +0100, ml@bortal.de wrote:
Hey Balazs,
i think ai am almost there yet ;) (see below...)
Balazs Scheidler wrote:
Am 15.10.2008 um Uhr haben Sie geschrieben:
On Wed, 2008-10-15 at 09:53 +0200, ml@bortal.de wrote:
Hello List,
How can i fix this? I am running postgresql-8.1 with Debian 4.0.
Probably the syslog client sends non-utf8 characters in its message. syslog-ng 2.0 and 2.1 do not really care about the message contents.
3.0 has support for various encodings and utf8 validation like this:
This assumes that the input is latin1:
source s_net { udp(encoding("iso-8859-1")); };
And this one enforces valid utf8 messages
source s_net { udp(flags(validate-utf8)); };
I was planning to add another flag, but this is not yet implemented:
source s_net { udp(flags(force-utf8)); };
Which would enforce valid utf8 sequences by changing the input.
As an alternative don't use utf8 as the character encoding of your database, use latin1, that'll permit any kind of data in the database.
syslog-ng -F -v -f /etc/syslog-ng.conf -e Configuration file has no version number, assuming syslog-ng 2.1 format. Please add @version: maj.min to the beginning of the file; WARNING: global: the default value of chain_hostnames is changing to 'no' in version 3.0, please update your configuration accordingly; Your configuration file uses an obsoleted keyword, please update your configuration; keyword='sync', change='flush_lines' Your configuration file uses an obsoleted keyword, please update your configuration; keyword='stats', change='stats_freq' WARNING: input: sources do not remove new-line characters from messages by default in version 3.0, please add 'no-multi-line' flag to your configuration if you want to retain this functionality; WARNING: file source: default value of follow_freq in file sources is changing in 3.0 to '1' for all files except /proc/kmsg; syntax error in /etc/syslog-ng.conf at line 14.
Line 14 and 15 look like this: ------------------------------------------------------------- source s_net { udp(encoding("iso-8859-1")); }; source s_net { udp(flags(validate-utf8)); };
I also tried to add: @version: 2.9 to the config file, but this did not work.
As I wrote in my previous mail, the current 2.9 snapshots are equal to the upcoming 3.0 version, I just wanted to "reserve" the 3.0.0 release. But since this is more confusing than it should be, I change the version number in the current git tree to 3.0, so the nightly snapshots will be named 3.0+date+time. The @version: stanza should include the version number "3.0", but it is not related to your problem. I'll copy-paste your configuration in my working directory and check what the problem might be.
-- Bazsi
On Wed, 2008-11-05 at 16:25 +0100, Balazs Scheidler wrote:
On Wed, 2008-11-05 at 15:36 +0100, ml@bortal.de wrote:
Hey Balazs,
i think ai am almost there yet ;) (see below...)
Balazs Scheidler wrote:
Am 15.10.2008 um Uhr haben Sie geschrieben:
On Wed, 2008-10-15 at 09:53 +0200, ml@bortal.de wrote:
Hello List,
How can i fix this? I am running postgresql-8.1 with Debian 4.0.
Probably the syslog client sends non-utf8 characters in its message. syslog-ng 2.0 and 2.1 do not really care about the message contents.
3.0 has support for various encodings and utf8 validation like this:
This assumes that the input is latin1:
source s_net { udp(encoding("iso-8859-1")); };
And this one enforces valid utf8 messages
source s_net { udp(flags(validate-utf8)); };
I was planning to add another flag, but this is not yet implemented:
source s_net { udp(flags(force-utf8)); };
Which would enforce valid utf8 sequences by changing the input.
As an alternative don't use utf8 as the character encoding of your database, use latin1, that'll permit any kind of data in the database.
syslog-ng -F -v -f /etc/syslog-ng.conf -e Configuration file has no version number, assuming syslog-ng 2.1 format. Please add @version: maj.min to the beginning of the file; WARNING: global: the default value of chain_hostnames is changing to 'no' in version 3.0, please update your configuration accordingly; Your configuration file uses an obsoleted keyword, please update your configuration; keyword='sync', change='flush_lines' Your configuration file uses an obsoleted keyword, please update your configuration; keyword='stats', change='stats_freq' WARNING: input: sources do not remove new-line characters from messages by default in version 3.0, please add 'no-multi-line' flag to your configuration if you want to retain this functionality; WARNING: file source: default value of follow_freq in file sources is changing in 3.0 to '1' for all files except /proc/kmsg; syntax error in /etc/syslog-ng.conf at line 14.
Line 14 and 15 look like this: ------------------------------------------------------------- source s_net { udp(encoding("iso-8859-1")); }; source s_net { udp(flags(validate-utf8)); };
I also tried to add: @version: 2.9 to the config file, but this did not work.
As I wrote in my previous mail, the current 2.9 snapshots are equal to the upcoming 3.0 version, I just wanted to "reserve" the 3.0.0 release. But since this is more confusing than it should be, I change the version number in the current git tree to 3.0, so the nightly snapshots will be named 3.0+date+time.
The @version: stanza should include the version number "3.0", but it is not related to your problem. I'll copy-paste your configuration in my working directory and check what the problem might be.
The problem is almost certainly not at either of the quoted lines above, for me it parses this configuration all-right. If you send me your complete configuration, I'll check where the problem is. Alternatively you can always use the -y command line option to show parser debug messages, but this needs the --enable-debug configuration option. -- Bazsi
participants (3)
-
Balazs Scheidler
-
Joseph Pietras
-
ml＠bortal.de