On Apr 18, 2006, at 2:15 PM, Tom Lane wrote:

"Thomas F. O'Connell" <tfo@sitening.com> writes:
I would've expected the RAID to protect postgres from the possibility
of data corruption, but I guess not.

Ooops :-(. It might be interesting to get pg_filedump dumps of the
corrupted pages, just to see what the failure pattern looks like.
I doubt there's much we can do about it, but you don't know till you
look. ("If we knew what it was we were doing, it wouldn't be
research.")

Any tips on turning "ERROR: invalid page header in block 34 of
relation" into a pg_filedump command that would yield something useful
or interesting? If so, I'll post the results of all three relations
known to be corrupt so far.

Also, are there any questions I could be asking vendors (Dell, LSI) that
would help sort out how the RAID contributed to corruption on disk?

--
Thomas F. O'Connell
Database Architecture and Programming
Sitening, LLC

http://www.sitening.com/
3004 B Poston Avenue
Nashville, TN 37203-1314
615-260-0005 (cell)
615-469-5150 (office)
615-469-5151 (fax)

---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match

"Thomas F. O'Connell" <tfo@sitening.com> writes:
> Any tips on turning "ERROR: invalid page header in block 34 of
> relation" into a pg_filedump command that would yield something useful
> or interesting? If so, I'll post the results of all three relations
> known to be corrupt so far.

For me, the nicest pg_filedump output format is "-i -f", but if the page
is sufficiently hosed that pg_filedump can't make any sense of it, you
may have to fall back to a raw dump, "-d". In any case, if you don't
want to dump the entire file, use "-R blkno" or "-R startblk endblk"
to constrain the dump.

BTW, last I heard the posting size limit on these lists was only about
20K. If you don't want your results to languish awhile in the
moderator's approval queue, you might want to post a link to the dump
instead of the whole thing.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings

On Apr 19, 2006, at 6:42 PM, Tom Lane wrote:

> "Thomas F. O'Connell" <tfo@sitening.com> writes:
>> Any tips on turning "ERROR: invalid page header in block 34 of
>> relation" into a pg_filedump command that would yield something
>> useful
>> or interesting? If so, I'll post the results of all three relations
>> known to be corrupt so far.
>
> For me, the nicest pg_filedump output format is "-i -f", but if the
> page
> is sufficiently hosed that pg_filedump can't make any sense of it, you
> may have to fall back to a raw dump, "-d". In any case, if you don't
> want to dump the entire file, use "-R blkno" or "-R startblk endblk"
> to constrain the dump.
>
> BTW, last I heard the posting size limit on these lists was only about
> 20K. If you don't want your results to languish awhile in the
> moderator's approval queue, you might want to post a link to the dump
> instead of the whole thing.

Unfortunately, I couldn't get the pg_filedump request through to the
list until yesterday evening (inexplicable mail delivery issue), by
which point the known points of corruption had already been fixed.

The upside is that the corruption seems to have been restricted to
three relations, all of which have been fixed. I was able to perform
a successful pg_dump.

But I'm still disappointed in the hardware bleed-through affecting
the filesystem.

--
Thomas F. O'Connell
Database Architecture and Programming
Sitening, LLC

http://www.sitening.com/
3004 B Poston Avenue
Nashville, TN 37203-1314
615-260-0005 (cell)
615-469-5150 (office)
615-469-5151 (fax)

---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match

On Thu, Apr 20, 2006 at 12:07:13PM -0500, Thomas F. O'Connell wrote:
> But I'm still disappointed in the hardware bleed-through affecting
> the filesystem.

I (well, http://stats.distributed.net/) has been bit by this before. One
issue with RAID is that it does you no good if you don't get
notification that a drive has failed. Even after that, there's still
risk of another failure before a sucessful rebuild, and of course
something can always go wrong during recovery.

In a nutshell, RAID's still no replacement for backups.
--
Jim C. Nasby, Sr. Engineering Consultant jnasby@pervasive.com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster