> -----Original Message-----
> From: Martijn van Oosterhout [mailto:kleptog@svana.org]
> Sent: 13 April 2006 12:34
> To: Dave Page
> Cc: pgsql-hackers@postgresql.org; Hiroshi Inoue
> Subject: Re: [HACKERS] Practical impediment to supporting
> multiple SSL libraries
>
> However, w.r.t. the copying, the pointers in get PGresult are
> in memory belonging to that result. As long as that PGresult
> hangs around, you should be able to just copy the pointers
> rather than the data? Or is this unacceptable?

It copies the data. I can't think offhand why it was implemented that
way, but then I didn't write the code (Anoop & Siva @ Pervasive did).

Anyhoo, as I've said, that approach has now been abandoned anyway in
favour of Hiroshi's, so it's him you'd need to convince to change. The
rest of us have only just started re-learning the code.

Regards, Dave

---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend

On Thu, Apr 13, 2006 at 12:48:06PM +0100, Dave Page wrote:
> Anyhoo, as I've said, that approach has now been abandoned anyway in
> favour of Hiroshi's, so it's him you'd need to convince to change. The
> rest of us have only just started re-learning the code.

Well, I quickly scanned the code in CVS to see what I could find out.
There are a few features the psqlodbc tuplereader has that libpq
doesn't.

1. It reads tuples as you go through the data. The resultset has a
cursor, it only processes the data as you request it.

2. It reads directly from the socket directly into a per-tuple
malloc()ed field.

3. It extracts per-row tids directly into a seperate array.

4. The resulting resultset can be updated and modified as well as
appended to. This requires freeing and adding rows. And committing the
result. This is probably your updatable cursors.

So in fact what you really want is libpq as a protocol decoder but want
to manage your resultset yourself. And you want to be able to let users
handle incoming data as it comes rather than waiting for the whole set.

I don't think the zero-copy is relevent, the code is not written in a
way that suggests speed was an issue. Rather I think the way you want
to use the resultset is the issue. You can't use the memory in the
PGresult because then'd you need to track which tuples were allocated
by you and which we allocated by libpq. The resulting copying is
needless, along with the fact that you double your memory usage.

In fact, can think that a number of other projects would like an
alternative. For example, a Perl module would want to load the strings
directly into blessed perl strings rather than keep a copy of the
resultset around. I think this would be a worthwhile addition to the
libpq interface.

I'll see if I can come up with a proposal (whether it'll get
implemented is another issue entirely).

Have a nice day,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFEPkhTIB7bNG8LQkwRAqjSAJ95RMzMyD+L/z9bF427OcTG7XUQHwCcC0iA
tSvonm/l4UijNo2yu2BY7Jk=
=iOhT
-----END PGP SIGNATURE-----

* Martijn van Oosterhout (kleptog@svana.org) wrote:
> Well, I quickly scanned the code in CVS to see what I could find out.

Wow, that was quick.

> So in fact what you really want is libpq as a protocol decoder but want
> to manage your resultset yourself. And you want to be able to let users
> handle incoming data as it comes rather than waiting for the whole set.

The data-as-it-comes bit could be done w/ a Postgres cursor, couldn't
it? But then you have to read through all the data using PQgetResult,
which isn't much fun.

> I don't think the zero-copy is relevent, the code is not written in a
> way that suggests speed was an issue. Rather I think the way you want
> to use the resultset is the issue. You can't use the memory in the
> PGresult because then'd you need to track which tuples were allocated
> by you and which we allocated by libpq. The resulting copying is
> needless, along with the fact that you double your memory usage.

The double memory usage definitely sucks but I really think speed would
also be greatly improved by removing the double copying and all the
function calls dealing with PQgetResult, etc...

> In fact, can think that a number of other projects would like an
> alternative. For example, a Perl module would want to load the strings
> directly into blessed perl strings rather than keep a copy of the
> resultset around. I think this would be a worthwhile addition to the
> libpq interface.

Me too.

> I'll see if I can come up with a proposal (whether it'll get
> implemented is another issue entirely).

I'd be interested in trying to help with this too..

Thanks,

Stephen

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)

iD8DBQFEPkv3rzgMPqB3kigRAva9AJ9WnU2k7QdrDs+OiHRtre PFSJyL9QCghQMU
nB/CcAtG1tKjJBAVDyl/JqE=
=yuTM
-----END PGP SIGNATURE-----