SEO

We've seen two recent reports:
http://archives.postgresql.org/pgsql...4/msg00008.php
http://archives.postgresql.org/pgsql...5/msg01143.php
of postmaster restart failing because the WAL contains a reference
to a page that no longer exists.

I can think of a couple of possible explanations:
1. filesystem corruption, ie the page should exist in the file but the
kernel has forgotten about it;
2. we truncated the file subsequent to the WAL record that causes
the panic.

However, neither of these theories is entirely satisfying, because
the WAL replay logic has always acted like this; why haven't we
seen similar reports ever since 7.1? And why are both of these
reports connected to btrees, when file truncation probably happens
far more often on regular tables?

But, setting those nagging doubts aside, theory #2 seems like a definite
bug that we ought to do something about.

The only really clean answer I can see is for file truncation to force a
checkpoint just before issuing the ftruncate call. That way, no WAL
records referencing the to-be-deleted pages would need to be replayed in
a subsequent crash. However, checkpoints are expensive enough to make
this solution very unattractive from a performance point of view. And
I fear it's not a 100% solution anyway: what about the PITR scenario,
where you need to replay a WAL log that was made concurrently with a
filesystem backup being taken? The backup might well include the
truncated version of the file, but you can't avoid replaying the
beginning portion of the WAL log.

Plan B is for WAL replay to always be willing to extend the file to
whatever record number is mentioned in the log, even though this
may require inventing the contents of empty pages; we trust that their
contents won't matter because they'll be truncated again later in the
replay sequence. This seems pretty messy though, especially for
indexes. The major objection to it is that it gives up error detection
in real filesystem-corruption cases: we'll just silently build an
invalid index and then try to run with it. (Still, that might be better
than refusing to start; at least you can REINDEX afterwards.)

Any thoughts?

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

Tom Lane wrote:
> Plan B is for WAL replay to always be willing to extend the file to
> whatever record number is mentioned in the log, even though this
> may require inventing the contents of empty pages; we trust that their
> contents won't matter because they'll be truncated again later in the
> replay sequence. This seems pretty messy though, especially for
> indexes. The major objection to it is that it gives up error detection
> in real filesystem-corruption cases: we'll just silently build an
> invalid index and then try to run with it. (Still, that might be better
> than refusing to start; at least you can REINDEX afterwards.)

Should we add a GUC to allow recovery in such cases, but don't mention
it in postgresql.conf? This way we could give people a recovery
solution, and also track the cases it happens, and not accidentally
trigger the recovery case.

--
Bruce Momjian | http://candle.pha.pa.us
pgman@candle.pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

On Wed, 25 May 2005 11:02:11 -0400, Tom Lane <tgl@sss.pgh.pa.us>
wrote:
>Plan B is for WAL replay to always be willing to extend the file to
>whatever record number is mentioned in the log, even though this
>may require inventing the contents of empty pages; we trust that their
>contents won't matter because they'll be truncated again later in the
>replay sequence.

Another idea: WAL replay does not apply changes to nonexistent blocks,
but it keeps a list (hash table, file, whatever) of those blocks.
When a truncate WAL record is found, all entries for blocks affected
by the truncation are removed from the list. Is it sufficient to
remember just the relation and the block number or do we need the
contents a well?

If the list is non-empty at the end of WAL replay, this is evidence of
a serious problem (file system corruption or Postgres bug).

Servus
Manfred


---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

Manfred Koizar <mkoi-pg@aon.at> writes:
> Another idea: WAL replay does not apply changes to nonexistent blocks,
> but it keeps a list (hash table, file, whatever) of those blocks.
> When a truncate WAL record is found, all entries for blocks affected
> by the truncation are removed from the list. Is it sufficient to
> remember just the relation and the block number or do we need the
> contents a well?

We don't *have* the contents ... that's exactly why it's panicking ...

> If the list is non-empty at the end of WAL replay, this is evidence of
> a serious problem (file system corruption or Postgres bug).

That seems like a good idea --- it covers the problem, and what's more,
it won't complain until after it finishes replay. Which means that if
you do get the PANIC, you can get out of it with pg_resetxlog and not
need to worry that you are throwing away whatever good data is available
from the WAL log. (This assumes that we go ahead and checkpoint out
the working buffers before we make the check for empty list.)

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

> Plan B is for WAL replay to always be willing to extend the file to
> whatever record number is mentioned in the log, even though this
> may require inventing the contents of empty pages; we trust that their
> contents won't matter because they'll be truncated again later in the
> replay sequence. This seems pretty messy though, especially for
> indexes. The major objection to it is that it gives up error detection
> in real filesystem-corruption cases: we'll just silently build an
> invalid index and then try to run with it. (Still, that might be better
> than refusing to start; at least you can REINDEX afterwards.)

You could at least log some sort of warning during the PITR process.
Anyone running a PITR not paying attention to their logs is in trouble
anyway...

Chris

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

Christopher Kings-Lynne <chriskl@familyhealth.com.au> writes:
>> Plan B is for WAL replay to always be willing to extend the file to
>> whatever record number is mentioned in the log, even though this
>> may require inventing the contents of empty pages; we trust that their
>> contents won't matter because they'll be truncated again later in the
>> replay sequence. This seems pretty messy though, especially for
>> indexes. The major objection to it is that it gives up error detection
>> in real filesystem-corruption cases: we'll just silently build an
>> invalid index and then try to run with it. (Still, that might be better
>> than refusing to start; at least you can REINDEX afterwards.)

> You could at least log some sort of warning during the PITR process.
> Anyone running a PITR not paying attention to their logs is in trouble
> anyway...

I'm more worried about the garden variety restart-after-power-failure
scenario. As long as the postmaster starts up, it's unlikely people
will inspect the postmaster log too closely. I think we have a choice
of PANICking and refusing to start, or assuming that no one will notice
that we did something dubious.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

On Wed, 25 May 2005 18:19:19 -0400, Tom Lane <tgl@sss.pgh.pa.us>
wrote:
>> but it keeps a list (hash table, file, whatever) of those blocks.
>> [...] Is it sufficient to
>> remember just the relation and the block number or do we need the
>> contents a well?
>
>We don't *have* the contents ... that's exactly why it's panicking ...

I meant the contents of the WAL record, not the original block
contents. Anyway, I think it's not needed.

Servus
Manfred


---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

Manfred Koizar <mkoi-pg@aon.at> writes:
>>> [...] Is it sufficient to
>>> remember just the relation and the block number or do we need the
>>> contents a well?

> I meant the contents of the WAL record, not the original block
> contents. Anyway, I think it's not needed.

Oh, I see. Yes, it might be worth hanging onto for debugging purposes.
If we did get a report of such a failure, I'm sure we'd wish to know
what sort of WAL record triggered it. One trusts there won't be so many
that storing 'em all is a problem ...

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly

Tom Lane wrote:
> Manfred Koizar <mkoi-pg@aon.at> writes:
>
>>>>[...] Is it sufficient to
>>>>remember just the relation and the block number or do we need the
>>>>contents a well?
>
>
>>I meant the contents of the WAL record, not the original block
>>contents. Anyway, I think it's not needed.
>
>
> Oh, I see. Yes, it might be worth hanging onto for debugging purposes.
> If we did get a report of such a failure, I'm sure we'd wish to know
> what sort of WAL record triggered it. One trusts there won't be so many
> that storing 'em all is a problem ...
>
> regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly


I guess I am having the same problem here: I am just dealing with a
truncated table after a hard kill.
The symptoms are: The storage file of the table is missing while the
system tables can still see the table.
Looking at TRUNCATE (this is the only command which could potentially
have caused this problem in my case) it seems as if the system tables
are actually changed propery before the file on disk is truncated.

My question is: What happens if the system is killed inside
rebuild_relation or inside swap_relfilenodes which is called by
rebuild_relation?

many thanks and best regards,

Hans


--
Cybertec Geschwinde u Schoenig
Schoengrabern 134, A-2020 Hollabrunn, Austria
Tel: +43/664/393 39 74
www.cybertec.at, www.postgresql.at


---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

=?ISO-8859-1?Q?Hans-J=FCrgen_Sch=F6nig?= <postgres@cybertec.at> writes:
> My question is: What happens if the system is killed inside
> rebuild_relation or inside swap_relfilenodes which is called by
> rebuild_relation?

Nothing at all, because the system catalog updates aren't committed yet,
and we haven't done anything to the relation's old physical file.

If I were you I'd be looking into whether your disk hardware honors
write ordering properly. This sounds like something allowed the
directory change to reach disk before the transaction commit WAL record
did; which is impossible if fsync is doing what it's supposed to.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org