> Well, the psqlODBC driver apparently ran into a number of problems
with
> libpq that resulted in them not using it for their purpose.
> Given libpq primary purpose is to connect to PostgreSQL, it failing at
that is
> something that should be fixed.

I think you are forgetting, that e.g. a JDBC driver will not want to
depend on
an external C dll at all. It will want a native Java implementation
(Group 4).
Thus imho it is necessary to have a defined wire protocol, which we
have.

So if a driver needs to use the wire protocol it is imho not a problem.
If applications started using it, because they don't find a suitable
driver,
now that would be a problem.

Andreas

---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

"Zeugswetter Andreas DCP SD" <ZeugswetterA@spardat.at> writes:

> > Well, the psqlODBC driver apparently ran into a number of problems with
> > libpq that resulted in them not using it for their purpose. Given libpq
> > primary purpose is to connect to PostgreSQL, it failing at that is
> > something that should be fixed.
>
> I think you are forgetting, that e.g. a JDBC driver will not want to depend
> on an external C dll at all. It will want a native Java implementation
> (Group 4). Thus imho it is necessary to have a defined wire protocol, which
> we have.

I think you are forgetting that this is a complete nonsequitor.

Nobody suggested eliminating the defined wire protocol. Nor was anybody even
discussing JDBC. Java folks' fetish for reimplementing everything in Java is
entirely irrelevant.

--
greg


---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

Greg Stark <gsstark@MIT.EDU> writes:

> "Zeugswetter Andreas DCP SD" <ZeugswetterA@spardat.at> writes:
>
> > > Well, the psqlODBC driver apparently ran into a number of problems with
> > > libpq that resulted in them not using it for their purpose. Given libpq
> > > primary purpose is to connect to PostgreSQL, it failing at that is
> > > something that should be fixed.
> >
> > I think you are forgetting, that e.g. a JDBC driver will not want to depend
> > on an external C dll at all. It will want a native Java implementation
> > (Group 4). Thus imho it is necessary to have a defined wire protocol, which
> > we have.
>
> I think you are forgetting that this is a complete nonsequitor.

Hm, now that I've had some sleep I think I see where you're going with this.

As long as there's a defined wire protocol (and there will always be one) then
there's nothing wrong with what the psqlODBC driver is doing and having a
libpq mode that hands off small bits of the unparsed stream isn't really any
different than just having the driver read the unparsed data from the socket.

I'm not sure whether that's true or not but it's certainly a reasonable point.
Sorry for my quick response last night.


--
greg


---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match

On Fri, Apr 14, 2006 at 10:42:33AM -0400, Greg Stark wrote:
> Hm, now that I've had some sleep I think I see where you're going with this.
>
> As long as there's a defined wire protocol (and there will always be one)then
> there's nothing wrong with what the psqlODBC driver is doing and having a
> libpq mode that hands off small bits of the unparsed stream isn't really any
> different than just having the driver read the unparsed data from the socket.

Well, the main motivation for this is that when a new version of the
protocol appears, libpq will support it but psqlODBC won't. If libpq
provides a way to get these small bits of the unparsed stream in a
protocol independant way, then that problem goes away.

There are a number of other (primarily driver) projects that would
benefit from being able to bypass the PGresult structure for storing
data.

Have a nice day,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFEP7kJIB7bNG8LQkwRAkajAKCQ3bRSWK1pqQkozP96F2 kj879PIgCfYoh4
sOaAAFfEW9g1KbNjC4rAmf8=
=aVq9
-----END PGP SIGNATURE-----

Martijn van Oosterhout <kleptog@svana.org> writes:
> On Fri, Apr 14, 2006 at 10:42:33AM -0400, Greg Stark wrote:
>> As long as there's a defined wire protocol (and there will always be
>> one) then there's nothing wrong with what the psqlODBC driver is doing

> Well, the main motivation for this is that when a new version of the
> protocol appears, libpq will support it but psqlODBC won't. If libpq
> provides a way to get these small bits of the unparsed stream in a
> protocol independant way, then that problem goes away.

Greg's observation is correct, so maybe we are overthinking this
problem. A fair question to ask is whether psqlODBC would consider
going back to a non-hybrid implementation if these features did exist
in libpq.

> There are a number of other (primarily driver) projects that would
> benefit from being able to bypass the PGresult structure for storing
> data.

Please mention some specific examples. We need some examples as a
reality check.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Please mention some specific examples. We need some examples as a
> reality check.

Just took a look through a couple of Debian packages which depend on
libpq4:

libpam-pgsql: pam_pgsql.c, line 473 it uses PQgetvalue() as one would
expect, but doesn't actually save the pointer anywhere, just uses it to
do comparisons against (all it stores, apparently, is a password in the
DB).

libnss-pgsql: src/backend.c, line 228:

sptr = PQgetvalue(res, row, colnum);
slen = strlen(sptr);
if(*buflen < slen+1) {
return NSS_STATUS_TRYAGAIN;
}
strncpy(*buffer, sptr, slen);
(*buffer)[slen] = '\0';

*valptr = *buffer;

*buffer += slen + 1;
*buflen -= slen + 1;

return NSS_STATUS_SUCCESS;

That really seems to be the classic example to me. Get the data from
PQresult, store it in something else, work on it.

mapserver: mappostgis.c, starting from line 1340:

shape->values = (char **) malloc(sizeof(char *) * layer->numitems);
for(t = 0; t < layer->numitems; t++) {
temp1= (char *) PQgetvalue(query_result, 0, t);
size = PQgetlength(query_result, 0, t);
temp2 = (char *) malloc(size + 1);
memcpy(temp2, temp1, size);
temp2[size] = 0; /* null terminate it */

shape->values[t] = temp2;

}

This same code repeats in another place (1139). They also
appear to forget to PQclear() in some cases. They don't appear to
ever save the pointer returned by PQresult() for anything.

postfix: src/global/dict_pgsql.c, starting from line 349:

numcols = PQnfields(query_res);

for (expansion = i = 0; i < numrows && dict_errno == 0; i++) {
for (j = 0; j < numcols; j++) {
r = PQgetvalue(query_res, i, j);
if (db_common_expand(dict_pgsql->ctx, dict_pgsql->result_format,
r, name, result, 0)
&& dict_pgsql->expansion_limit > 0
&& ++expansion > dict_pgsql->expansion_limit) {
msg_warn("%s: %s: Expansion limit exceeded for key: '%s'",
myname, dict_pgsql->parser->name, name);
dict_errno = DICT_ERR_RETRY;
break;
}
}
}
PQclear(query_res);
r = vstring_str(result);
return ((dict_errno == 0 && *r) ? r : 0);

exim does something similar to postfix too.

It really seems unlikely that anyone keeps PQresult's around for very
long and they all seem to want to stick it into their own memory
structure. I don't know how many people would move to a new API should
one be provided though. Callbacks can be kind of a pain in the butt to
code too which makes the amount of effort required to move to using them
a bit higher too. This all means double memory usage though and that
really makes me want some kind of API that can be used to process data
as it comes in.

Another thought along these lines: Perhaps a 'PQgettuple' which can be
used to process one tuple at a time. This would be used in an ASYNC
fashion and libpq just wouldn't read/accept more than a tuple's worth
each time, which it could do into a fixed area (in general, for a
variable-length field it could default to an initial size and then only
grow it when necessary, and grow it larger than the current request by
some amount to hopefully avoid more malloc/reallocs later).

Thanks,

Stephen

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)

iD8DBQFEP9ChrzgMPqB3kigRAn+pAJ44QjT8kWmQ8cqGSIaSdr lGR2KRqwCffxKO
L88PTvKdwLfttgwcPVISzV4=
=jBMY
-----END PGP SIGNATURE-----

On Fri, Apr 14, 2006 at 11:22:23AM -0400, Tom Lane wrote:
> Greg's observation is correct, so maybe we are overthinking this
> problem. A fair question to ask is whether psqlODBC would consider
> going back to a non-hybrid implementation if these features did exist
> in libpq.

Well, it is an issue. It's not a specific problem per se that psqlODBC
implements the protocol itself. If you remember right back at the
beginning of the thread (see subject) there was the issue of users
using libpq to connect and then continuing themselves. The issue being
that the pointer from PQgetssl() wouldn't work if we had different SSL
libraries available.

Perhaps a far easier approach would be to indeed just have a hijack
interface that provides read/write over whatever protocol libpq
negotiated. Then people could write their own protocol parsers to suit
their needs while still using libpq for the connection. Have the cake
and eat it too?

Note, we would have to allow users of libpq to force the version,
otherwise libpq would connect using a version the user doesn't
understand.

> Please mention some specific examples. We need some examples as a
> reality check.

Well, psqlODBC is the obvious case. Besides that it becomes tricky. I
would think that DBI::Pg could benefit, I just don't understand the
code well enough to know if it's directly useful.

I would expect drivers in particular to benefit and some complex
applications, but if you're asking for specific examples, I don't have
any...

That doesn't change the fact that it's a nice idea, just definite
benificiaries are harder to find.

Have a nice day,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFEP9H4IB7bNG8LQkwRAr2tAJ90sa/o6P+mT5YdaCuVJfXtct4oTwCffrmC
e6eE2u+IMcSB1px0zX7CJq8=
=eofa
-----END PGP SIGNATURE-----

Martijn van Oosterhout <kleptog@svana.org> writes:
> Perhaps a far easier approach would be to indeed just have a hijack
> interface that provides read/write over whatever protocol libpq
> negotiated.

Well, there's a precedent to look at: the original implementation of
COPY mode was pretty nearly exactly that. And it sucked, and eventually
we changed it. So I'd be pretty leery of repeating the experience...

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings

On Fri, Apr 14, 2006 at 01:05:11PM -0400, Tom Lane wrote:
> Martijn van Oosterhout <kleptog@svana.org> writes:
> > Perhaps a far easier approach would be to indeed just have a hijack
> > interface that provides read/write over whatever protocol libpq
> > negotiated.
>
> Well, there's a precedent to look at: the original implementation of
> COPY mode was pretty nearly exactly that. And it sucked, and eventually
> we changed it. So I'd be pretty leery of repeating the experience...

As I remember, the main issue was with the loss of control over the
error state and recovering if stuff went wrong. In this case, once
someone hijacks a connection they can't hand it back. It only option is
to close.

It was just thinking of providing pointers to pqsecure_read/write and
maybe a few other things, but that's it.

Or was there something else?

Have a nice day,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD4DBQFEP+D6IB7bNG8LQkwRAgSZAJ99sYvK6AAWQ51KNTq7mH A03++UDwCVH64r
Q4SP6NilF78WUAk+KBJz3g==
=22cz
-----END PGP SIGNATURE-----

Stephen Frost <sfrost@snowman.net> writes:

> Another thought along these lines: Perhaps a 'PQgettuple' which can be
> used to process one tuple at a time. This would be used in an ASYNC
> fashion and libpq just wouldn't read/accept more than a tuple's worth
> each time, which it could do into a fixed area (in general, for a
> variable-length field it could default to an initial size and then only
> grow it when necessary, and grow it larger than the current request by
> some amount to hopefully avoid more malloc/reallocs later).

I know DBD::Oracle uses an interface somewhat like this but more
sophisticated. It provides a buffer and Oracle fills it with as many records
as it can.

It's blocking though (by default) and DBD::Oracle tries to adjust the size of
the buffer to keep the network pipeline full, but if the application is slow
at reading the data then the network buffers fill and it pushes back to the
database which blocks writing.

This is normally a good thing though. One of the main problems with the
current libpq interface is that if you have a very large result set it flows
in as fast as it can and the library buffers it *all*. If you're trying to
avoid forcing the user to eat millions of records at once you don't want to be
buffering them anywhere all at once. You want a constant pipeline of records
streaming out as fast as they can be processed and no faster.

--
greg


---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly