SEO

Hi,

pg_restore faied by the following operations on Windows XP.

$ createdb test
$ pgbench -i -s 1000 test
$ pg_dump -Fc test > out
$ createdb restore
$ pg_restore -d restore out
pg_restore: [custom archiver] error during file seek: Invalid argument

Win32 does not implement fseeko() and ftello(). So I think it limit to
handle a 2GB file. Is this a specification?

Regards,
--
Yoshiyuki Asaba
y-asaba@sraoss.co.jp

---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings

Hi. Asaba-san.

From: "Yoshiyuki Asaba"


> Win32 does not implement fseeko() and ftello(). So I think it limit to
> handle a 2GB file. Is this a specification?

Yes, Magnus-san suggested the problem. It is present TODO. The entire
adjustment was still difficult though I had tried it. SetFilePointer might be
able to be saved. However, I think it might be an attempt of 8.3...

Regards,
Hiroshi Saito



---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

On Fri, Dec 15, 2006 at 12:57:50AM +0900, Hiroshi Saito wrote:
>
> >Win32 does not implement fseeko() and ftello(). So I think it limit to
> >handle a 2GB file. Is this a specification?
>
> Yes, Magnus-san suggested the problem. It is present TODO. The entire
> adjustment was still difficult though I had tried it. SetFilePointer might
> be able to be saved. However, I think it might be an attempt of 8.3...

I've been looking at a fix for this, and I think I have it. The solution
looks to be to redefine off_t to 64-bit (the standard headers *always*
define it as 32-bit, and there is no way to change that - at least not
that I can find).

I have the fix made for just bin/pg_dump for now (in pg_dump.h), and I'm
testing that. (So far only on MSVC builds)

A question though - is there any *gain* from using 64-bit offsets in the
actual backend? The change could of course be done in port.h, but that
will affect the whole backend (and require a few more functions than
just fseeko/ftello to be redefined) which could have larger
consequences.

So - provided that this works after my test is completed, is the better
place to do this for just pg_dump/pg_restore, or attempt to do it for
the whole backend?

//Magnus

---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

Magnus Hagander wrote:
> On Fri, Dec 15, 2006 at 12:57:50AM +0900, Hiroshi Saito wrote:
> >
> > >Win32 does not implement fseeko() and ftello(). So I think it limit to
> > >handle a 2GB file. Is this a specification?
> >
> > Yes, Magnus-san suggested the problem. It is present TODO. The entire
> > adjustment was still difficult though I had tried it. SetFilePointer might
> > be able to be saved. However, I think it might be an attempt of 8.3...
>
> I've been looking at a fix for this, and I think I have it. The solution
> looks to be to redefine off_t to 64-bit (the standard headers *always*
> define it as 32-bit, and there is no way to change that - at least not
> that I can find).
>
> I have the fix made for just bin/pg_dump for now (in pg_dump.h), and I'm
> testing that. (So far only on MSVC builds)
>
> A question though - is there any *gain* from using 64-bit offsets in the
> actual backend? The change could of course be done in port.h, but that

No, not really. All files are kept < 1gig for the backend. We had code
for that from Berkeley, so we have just kept it.

--
Bruce Momjian bruce@momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?

http://archives.postgresql.org

Hi.

Oh, your great trust confidence.:-)

>> I have the fix made for just bin/pg_dump for now (in pg_dump.h), and I'm
>> testing that. (So far only on MSVC builds)

However, MinGW+gcc be able to be saved?
I was wishing it....

Regards,
Hiroshi Saito


---------------------------(end of broadcast)---------------------------
TIP 7: You can help support the PostgreSQL project by donating at

http://www.postgresql.org/about/donate

On Mon, Dec 18, 2006 at 09:50:12AM -0500, Bruce Momjian wrote:

> > > Yes, Magnus-san suggested the problem. It is present TODO. The entire
> > > adjustment was still difficult though I had tried it. SetFilePointer might
> > > be able to be saved. However, I think it might be an attempt of 8.3...
> >
> > I've been looking at a fix for this, and I think I have it. The solution
> > looks to be to redefine off_t to 64-bit (the standard headers *always*
> > define it as 32-bit, and there is no way to change that - at least not
> > that I can find).
> >
> > I have the fix made for just bin/pg_dump for now (in pg_dump.h), and I'm
> > testing that. (So far only on MSVC builds)
> >
> > A question though - is there any *gain* from using 64-bit offsets in the
> > actual backend? The change could of course be done in port.h, but that
>
> No, not really. All files are kept < 1gig for the backend. We had code
> for that from Berkeley, so we have just kept it.

Ok, based on this, here's a patch that *appears* to fix the problem on
Win32. I tried the suggested repro to dump a pgbench database of
appropriate size, and it does restore properly and give me the proper
amount of rows in all the tables. But that's all I've tested so far.

Also, it compiles fine on MSVC. I still haven't managed to get the MingW
build environment working properly on Win64 even for building Win32
apps, so I haven't been able to build it on MingW yet. It *should* work
since it's all standard functions, but might require further hacks. I'll
try to get around to that later, and Dave has promised to give it a
compile-try as well, but if someone wants to test that, please do

So, does it seem reasonable, and the right place to stick it? Oh, and
please do *not* apply until someone confirms it works on mingw!

//Magnus



---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend

Bruce Momjian <bruce@momjian.us> writes:
> Magnus Hagander wrote:
>> A question though - is there any *gain* from using 64-bit offsets in the
>> actual backend? The change could of course be done in port.h, but that

> No, not really. All files are kept < 1gig for the backend.

Not so: consider a backend COPY reading or writing a multi-gig table.
This will fail if the platform hasn't got largefile support.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings

Tom Lane wrote:
> Bruce Momjian <bruce@momjian.us> writes:
> > Magnus Hagander wrote:
> >> A question though - is there any *gain* from using 64-bit offsets in the
> >> actual backend? The change could of course be done in port.h, but that
>
> > No, not really. All files are kept < 1gig for the backend.
>
> Not so: consider a backend COPY reading or writing a multi-gig table.
> This will fail if the platform hasn't got largefile support.

Good point --- but do we do any seeks in COPY files? I don't think so,
so I don't see how we would benefit from large file support there.
Certainly people are dumping >2 gig files.

--
Bruce Momjian bruce@momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly

Bruce Momjian <bruce@momjian.us> writes:
> Tom Lane wrote:
>> Not so: consider a backend COPY reading or writing a multi-gig table.

> Good point --- but do we do any seeks in COPY files? I don't think so,

True, so if the problem is limited to whether we can seek or not, then
we don't need to fix the backend. Magnus said something about other
issues though?

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 7: You can help support the PostgreSQL project by donating at

http://www.postgresql.org/about/donate

Tom Lane wrote:
> Bruce Momjian <bruce@momjian.us> writes:
> > Tom Lane wrote:
> >> Not so: consider a backend COPY reading or writing a multi-gig table.
>
> > Good point --- but do we do any seeks in COPY files? I don't think so,
>
> True, so if the problem is limited to whether we can seek or not, then
> we don't need to fix the backend. Magnus said something about other
> issues though?

Yes, MinGW. I think he is researching that.

--
Bruce Momjian bruce@momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match