SEO

Ben Trewern wrote:
> I'd like to make a few points on this issue.
>
> 1. This problem should be mentioned in the FAQs as largely as possible as
> it is difficult to rectify if you have fallen into this trap.
>
> 2. If you do have data in SQL_ASCII the old ODBC driver worked, PgAdmin III
> works, Delphi and zeoslib works, I understand why there may be a problem but
> is it not possible to make the new 8.x work?
>
> 3. Correct me if I'm wrong but SQL_ASCII is PostgreSQL's default encoding.
> If this isn't sorted out then we'll see lots more of these messages for
> help.
>
> 4. I'd like to disagree with your "DO NOT USE ASCII FOR NON-ASCII DATA" as
> if you read any of Tom Lanes many messages on the subject. Here's a quote:
>
> "The SQL_ASCII setting isn't an
> encoding, really; it's a declaration of ignorance. In this setting
> the server will just store and regurgitate whatever character strings
> you send it. This will work fine, more or less, if all your clients
> use exactly the same encoding and you don't care about functions like
> upper()/lower()"

Mind the *if*.
You'll always create mess when mixing drivers/apps. Ignoring proper
encodings is always non-standard, so don't expect drivers to support it,
and ask for support.
Server encoding is for fixing this issue, and all drivers are obeying
this. So if you want a guarantee to have a working configuration, DO NOT
USE ASCII FOR NON-ASCII DATA.

Regards,
Andreas

---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend

On Fri, Aug 19, 2005 at 10:35:28AM +0200, Andreas Pflug wrote:
> Ben Trewern wrote:
> >I'd like to make a few points on this issue.
> >
> >2. If you do have data in SQL_ASCII the old ODBC driver worked, PgAdmin
> >III works, Delphi and zeoslib works, I understand why there may be a
> >problem but is it not possible to make the new 8.x work?
> >
> >
> >4. I'd like to disagree with your "DO NOT USE ASCII FOR NON-ASCII DATA" as
> >if you read any of Tom Lanes many messages on the subject. Here's a quote:
> >
> >"The SQL_ASCII setting isn't an
> >encoding, really; it's a declaration of ignorance. In this setting
> >the server will just store and regurgitate whatever character strings
> >you send it. This will work fine, more or less, if all your clients
> >use exactly the same encoding and you don't care about functions like
> >upper()/lower()"

If SQL_ASCII is/was equivalent to "ignoring encoding", then it
looks/looked pretty misnamed! For instance if someone wants some UTF16
from/to ASCII then some conversion is definitely needed and
ASCII==ignorance is then a bug.

Encoding ignorance should rather be called SQL_BINARY. A BINARY setting
for strings makes sense, just like when transfering text files using
FTP: you just don't trust FTP for encodings and use it like a
filesystem. BINARY just means that: "don't mess-up with encodings and
let something else deal with the issue".

Of course you cannot then use upper() or whatever if you don't want to
reveal the encoding you use...


> Mind the *if*. You'll always create mess when mixing drivers/apps.

I guess some people knew what they did and simply did not mixed
driver/apps, or in a way they mastered and that worked.


> Ignoring proper encodings is always non-standard,

> so don't expect drivers to support it, and ask for support.

Well while reading at the complaints it seems this BINARY mode was
there before (by "accident"?), even if misnamed... so the BINARY mode
recently disappeared? No surprise some people become angry :-)


> Server encoding is for fixing this issue, and all drivers are obeying
> this. So if you want a guarantee to have a working configuration, DO NOT
> USE ASCII FOR NON-ASCII DATA.

Looks like people fixed issues by themselves before, and Postgres
recent fixing does not interact nicely with theirs?


PS: BTW "unicode" is not one encoding but many different ones.


---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

Looks like a lot of thoughts on this.

I would just like to say (as one who had to convert their data) thank so
much to every one who has worked so hard on the current driver.

I understand if folks are upset because of change, but keep in mind this is
an open source effort and my hat goes off to the accomplishment. The old
driver may of handles aschii in a less imposing way, but it also did not
work.

Maybe one of those needing a plain aschii odbc driver that deals with
encoding (or doesn't depending on the point of view) can make a version of
the current driver that will allow the user to not need specify and give it
a name like genericaschii or something (or add it as a connect string
param?).

I for one am very happy the work was done and seems to be working well. I
did have a major error occur during my conversion (because of human error
doing a backup restore at midnight), but it was still worth the end result.

You can not expect everyone to agree (any change is always hard to accept).
My users hate change even if it is more flexibility. I can agree it was a
major hassle for me to have to deal with a change, but we have been told the
driver functionality changed, if you want to use the driver on non aschii
then make sure the data base reflects the proper coding. I provided examples
of how to convert from one to the other, so bite the bullet and fix your
issues. In the end having upper etc working will and should be the way you
want it.

Joel Fradkin




---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly

Marc Herbert wrote:

>If SQL_ASCII is/was equivalent to "ignoring encoding", then it
>looks/looked pretty misnamed!
>
It's not. It should be used for ASCII only, but the database system will
not barf if you offer it a byte with the upper bit set. You're simply on
your own.

>Encoding ignorance should rather be called SQL_BINARY. A BINARY setting
>for strings makes sense, just like when transfering text files using
>FTP: you just don't trust FTP for encodings and use it like a
>filesystem. BINARY just means that: "don't mess-up with encodings and
>let something else deal with the issue".
>
>
No, binary would include 0x00 which is definitely *not* a character but
the string terminator. If SQL_ASCII would be implemented nowadays, there
probably would be a check for the upper bit cleared, and have it
rejected otherwise. But since this part is really really old, this can't
be changed without breaking zillions of old apps that used to ignore
proper storage encoding.

>I guess some people knew what they did and simply did not mixed
>driver/apps, or in a way they mastered and that worked.
>
>
The latter, with the obvious chance to break if the next app accesses
the data. This is certainly not the design goal of a RDBMS.

>Well while reading at the complaints it seems this BINARY mode was
>there before (by "accident"?),
>
No.

>Looks like people fixed issues by themselves before,
>
They didn't fix anything, they worked around the wrong chosen server
encoding. I perfectly understand this, because initially I did the same
mistake.

> and Postgres
>recent fixing does not interact nicely with theirs?
>
>
Automatically choosing the right client encoding and properly converting
in the driver did (and maybe still has) bugs, but fixing these will
certainly support the rules as proper design requires it, not
ill-designed apps.

>PS: BTW "unicode" is not one encoding but many different ones.
>
>
Doesn't matter. Always means the current Unicode for the system: in the
backend UTF-8, on Win32 UCS16, Linux UCS32 or UTF-8 dependent on
interface definition. The *driver* has to take care of the proper
conversion, *if* it is instructed correctly (i.e. correct server encoding)

Regards,
Andreas


---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend

On Fri, Aug 19, 2005 at 04:11:48PM +0200, Andreas Pflug wrote:
> Marc Herbert wrote:
>
> >If SQL_ASCII is/was equivalent to "ignoring encoding", then it
> >looks/looked pretty misnamed!
> >
> It's not. It should be used for ASCII only, but the database system will
> not barf if you offer it a byte with the upper bit set. You're simply on
> your own.

Well this still looks like what I called a "BINARY/don't touch it"
accidental mode.


> >Encoding ignorance should rather be called SQL_BINARY. A BINARY setting
> >for strings makes sense, just like when transfering text files using
> >FTP: you just don't trust FTP for encodings and use it like a
> >filesystem. BINARY just means that: "don't mess-up with encodings and
> >let something else deal with the issue".
> >

> No, binary would include 0x00

This seems irrelevant to me, see below.


> which is definitely *not* a character but the string terminator.

Not everyone in the world uses 0x00 as a string terminator. C does,
Postgres also, but Java does not and I don't think databases standards
and even less encoding standards say anything about this (please prove
me wrong, I'd really like to have a definitive answer on this).

It just tried to insert a string into hsqldb using JDBC and it worked
perfectly fine. Postgres JDBC driver is also "strings with
null-character"-ready, so this seems to be only a limitation of
Postgres.

By the way many ODBC function calls ask for the length of string
arguments, _optionally_ being SQL_NTS (Null-Terminated String). So it
seems some people here catered for strings with null characters even
in C!


In any case whether 0x00 is The String Terminator or not is not
relevant to the fact that there was a accidental "BINARY" string
encoding before. If we learn that 0x00 is really The Database String
Terminator, then it can also be interpreted as a terminator even in
"encoding ignorance" mode since it translates into 0x00 for every
known encoding.


> >I guess some people knew what they did and simply did not mixed
> >driver/apps, or in a way they mastered and that worked.

> The latter, with the obvious chance to break if the next app accesses
> the data. This is certainly not the design goal of a RDBMS.

There was a time, not so long ago, where every encoding-related stuff
was under-specified, every software buggy etc., so people had to cope
with it. They were probably pleased at that time to have this
accidental "BINARY" workaround available. One can easily understand
that they complain a little bit about the sudden removal of this
workaround and the unplanned migration to The Right Solution.

Of course on the other hand everyone can understand that the Postgres
developers want to get rid of this accidental BINARY string mode, and
that they are free to do what they want.


> >Well while reading at the complaints it seems this BINARY mode was
> >there before (by "accident"?),

> No.

Well, I am still waiting for some proof of the opposite (since this
0x00 stuff does not seem really related to it).

I was just reformulating Tom Lanes "SQL_ASCII ignorance" quote above,
which looked quite informed.


> >PS: BTW "unicode" is not one encoding but many different ones.

> Doesn't matter. Always means the current Unicode for the system: in the
> backend UTF-8, on Win32 UCS16, Linux UCS32 or UTF-8 dependent on
> interface definition.

Interesting. I hope this "current unicode for the system" concept is
well documented, because just saying "unicode" is not clear at all,
even if not ambiguous.

Regards,

Marc.



---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster