SEO

Koichi Suzuki <koichi@intellilink.co.jp> writes:
> Here're a couple of patches for PostgreSQL 64bit support. There're just
> two extension to 64bit, size of shared memory and transaction ID.

I asked originally for some experimental evidence showing any value
in having more than 2Gb of shared buffers. In the absence of any
convincing demonstration, I'm not very inclined to worry about whether
we can handle wider-than-int shared memory size.

As for the XID change, I don't think this patch accurately reflects the
size of the impact. There are a lot of things that in practice need to
be the same size as XID (CID, most obviously, but I suspect also OID).
And again, some demonstration of the performance impact would be
appropriate. Here, not only do you have to prove that widening XID
isn't a big performance hit in itself, but you also have to convince
us that it's a win compared to the existing approach of vacuuming at
least every billion transactions.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

Tom Lane wrote:
> Koichi Suzuki <koichi@intellilink.co.jp> writes:
>
>>Here're a couple of patches for PostgreSQL 64bit support. There're just
>>two extension to 64bit, size of shared memory and transaction ID.
>
>
> I asked originally for some experimental evidence showing any value
> in having more than 2Gb of shared buffers. In the absence of any
> convincing demonstration, I'm not very inclined to worry about whether
> we can handle wider-than-int shared memory size.

There is some practical evidence. Recently the number of large boxes in
the field is almost growing exponentially. Today I have heard somebody
say "this box has 'just 4 gig of ram' ".
On large installations we have already seen problems with too small
caches (= 2gb).
Surprisingly this has turned out to be a quite important issue in the
field. Tests have shown that the cache provided by the OS is a lot worse
for the database.

64-bit XIDs seem to be an overkill - the only practical impact I can see
is an even larger tuple header (this can be an issue on large boxes too
- at least compared to Oracle).

Best regards,

Hans

---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly

=?ISO-8859-1?Q?Hans-J=FCrgen_Sch=F6nig?= <postgres@cybertec.at> writes:
> There is some practical evidence. Recently the number of large boxes in
> the field is almost growing exponentially. Today I have heard somebody
> say "this box has 'just 4 gig of ram' ".
> On large installations we have already seen problems with too small
> caches (= 2gb).
> Surprisingly this has turned out to be a quite important issue in the
> field. Tests have shown that the cache provided by the OS is a lot worse
> for the database.

*What* tests? This is all handwaving :-(

What I would find credible is a set of, say, OSDL test runs, showing a
continuing increase of performance with shared_buffers right up to the
2Gb limit. Everything done to date says that you hit the point of
diminishing returns well before that.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings

Tom Lane wrote:
> =?ISO-8859-1?Q?Hans-J=FCrgen_Sch=F6nig?= <postgres@cybertec.at> writes:
>
>>There is some practical evidence. Recently the number of large boxes in
>>the field is almost growing exponentially. Today I have heard somebody
>>say "this box has 'just 4 gig of ram' ".
>>On large installations we have already seen problems with too small
>>caches (= 2gb).
>>Surprisingly this has turned out to be a quite important issue in the
>>field. Tests have shown that the cache provided by the OS is a lot worse
>>for the database.
>
>
> *What* tests? This is all handwaving :-(
>
> What I would find credible is a set of, say, OSDL test runs, showing a
> continuing increase of performance with shared_buffers right up to the
> 2Gb limit. Everything done to date says that you hit the point of
> diminishing returns well before that.
>
> regards, tom lane


well, you can easily try it on a big machine with gigs of ram and
nothing but the database running. using a very low number of shared
buffers will lead to worse performance than many shared buffers - even
if the operating system caches some disk i/O (which is done by linux if
nobody else want to have some ram). i have no public hard figures i
could post here but customers have told me that 2Q cache vs. kernel
cache is around 5-10 times faster (it varies of course).

the 2gb thing is especially important for data crunchers - not
necessarily for 'normal' OLTP databases. just assume somebody getting 5
gig of data and doing some repeated computation with this data. you
definitely don't want to go to disk in this case. people will assume
that postgresql can work with large caches ("it is a good database - why
do i get errors on startup" - this is the usual story). people rather
tend to rely on PostgreSQL than on some operating system thing .

i might have some time to provide some real hard facts to prove this but
i am a bit busy at the moment.

best regards,

hans



---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly

On Thu, Jul 07, 2005 at 06:23:40PM +0200, Hans-Jürgen Schönig wrote:
> Tom Lane wrote:
> >Koichi Suzuki <koichi@intellilink.co.jp> writes:
> >
> >>Here're a couple of patches for PostgreSQL 64bit support. There're just
> >>two extension to 64bit, size of shared memory and transaction ID.
> >
> >
> >I asked originally for some experimental evidence showing any value
> >in having more than 2Gb of shared buffers. In the absence of any
> >convincing demonstration, I'm not very inclined to worry about whether
> >we can handle wider-than-int shared memory size.
>
> There is some practical evidence. Recently the number of large boxes in
> the field is almost growing exponentially. Today I have heard somebody
> say "this box has 'just 4 gig of ram' ".

Yes, but the point is if it's a good idea to have that many shared
buffers. Is there a measurable difference between that, and leaving the
extra memory for the kernel to manage cache? Remember that there have
been measurements that showed, for previous releases, that having shared
buffers set too high was detrimental to performance. So the first thing
to do is present results showing that this is no longer true.

This could very well be the case, because of the rewriting of the buffer
manager.

--
Alvaro Herrera (<alvherre[a]alvh.no-ip.org>)
"El destino baraja y nosotros jugamos" (A. Schopenhauer)

---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match

I have some experimeltal data about this extension. I will gather it
and post hopefully this weekend.

Tom Lane wrote:
> Koichi Suzuki <koichi@intellilink.co.jp> writes:
>
>>Here're a couple of patches for PostgreSQL 64bit support. There're just
>>two extension to 64bit, size of shared memory and transaction ID.
>
>
> I asked originally for some experimental evidence showing any value
> in having more than 2Gb of shared buffers. In the absence of any
> convincing demonstration, I'm not very inclined to worry about whether
> we can handle wider-than-int shared memory size.
>
> As for the XID change, I don't think this patch accurately reflects the
> size of the impact. There are a lot of things that in practice need to
> be the same size as XID (CID, most obviously, but I suspect also OID).
> And again, some demonstration of the performance impact would be
> appropriate. Here, not only do you have to prove that widening XID
> isn't a big performance hit in itself, but you also have to convince
> us that it's a win compared to the existing approach of vacuuming at
> least every billion transactions.
>
> regards, tom lane
>


--
-------------------------------------------
Koichi Suzuki
Open Source Engineeering Departmeent,
NTT DATA Intellilink Corporation
Phone: +81-3-5566-9628 WWW: http://www.intellilink.co.jp
------------------------------------------

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

Hans-J|rgen Schvnig <postgres@cybertec.at> wrote:

> 64-bit XIDs seem to be an overkill - the only practical impact I can see
> is an even larger tuple header (this can be an issue on large boxes too
> - at least compared to Oracle).

I agreed, too. The changes of XIDs cannot be ignored because the overhead
will be 32bytes per tuple.

Avoiding overheads, can XIDs/CIDs be different bit length? For example,
can XIDs/CIDs be changed to 48/16-bit or 40/24-bit?

---
ITAGAKI Takahiro
NTT Cyber Space Laboratories


---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

On Fri, Jul 08, 2005 at 09:38:07AM +0900, ITAGAKI Takahiro wrote:
> Hans-J|rgen Schvnig <postgres@cybertec.at> wrote:
>
> > 64-bit XIDs seem to be an overkill - the only practical impact I can see
> > is an even larger tuple header (this can be an issue on large boxes too
> > - at least compared to Oracle).
>
> I agreed, too. The changes of XIDs cannot be ignored because the overhead
> will be 32bytes per tuple.
>
> Avoiding overheads, can XIDs/CIDs be different bit length? For example,
> can XIDs/CIDs be changed to 48/16-bit or 40/24-bit?

Not unless you change the definition of HeapTupleFields
(src/include/access/htup.h). Alignment concerns would probably bite you
if you changed it, anyway.

--
Alvaro Herrera (<alvherre[a]alvh.no-ip.org>)
"Ninguna manada de bestias tiene una voz tan horrible como la humana" (Orual)

---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to majordomo@postgresql.org)

Alvaro Herrera <alvherre@alvh.no-ip.org> writes:
> On Fri, Jul 08, 2005 at 09:38:07AM +0900, ITAGAKI Takahiro wrote:
>> Avoiding overheads, can XIDs/CIDs be different bit length? For example,
>> can XIDs/CIDs be changed to 48/16-bit or 40/24-bit?

> Not unless you change the definition of HeapTupleFields
> (src/include/access/htup.h). Alignment concerns would probably bite you
> if you changed it, anyway.

I don't think we could feasibly reduce the width of CID anyway; we've
already seen a few complaints of people overrunning 2^32 commands per
transaction, and surely this is a bigger rather than a lesser concern
if you are thinking of large databases.

It probably would be possible to keep CID at 32 bits and lay out the
HeapTupleHeader so that you only pay for three, not four, 64-bit
fields ... but that's still twelve bytes added per tuple.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

Hi guys,

BTW, I found the work_mem is still limited to 2GB.

If we support 64bit shared memory, we also need to support
64bit work_mem.

Thanks.
--
NAGAYASU Satoshi <nagayasus@nttdata.co.jp>




---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq