This morning one of our production servers (8.1.1, Linux) threw a number of errors like these (different final number; first two [1663/43801] are all the same]:

2006-09-26 10:02:15.124 PDT 1387333718 ERROR: could not open segment 1 of relation 1663/43801/7274801 (target block 478216192): No such file or directory
2006-09-26 10:02:52.991 PDT 1387340456 ERROR: could not open segment 1 of relation 1663/43801/7274801 (target block 478216192): No such file or directory
2006-09-26 10:02:53.002 PDT 1387340458 ERROR: could not open segment 1 of relation 1663/43801/7274801 (target block 478216192): No such file or directory

The server never really recovered -- it kept running but was horribly slow. Rebooting seems to have cured the problem, but I am wondering what I made look into, etc.

The server in question is really a read-only beast with several hundred connections, typically, 4 gigs of RAM, and has not misbehaved before.

Any suggestions would be welcome! (Although I fear I know the answer -- disk issues.)

Thanks,

Greg Williamson
DBA
GlobeXplorer LLC

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

Hi Greg,

Is it possible that the server was shutdown using -mi switch?

cuz immediate shutdown ends causing these problems.

Regards
Talha Khan


On 9/26/06, Gregory S. Williamson <gsw@globexplorer.com> wrote:
>
> This morning one of our production servers (8.1.1, Linux) threw a number
> of errors like these (different final number; first two [1663/43801] are
> all the same]:
>
> 2006-09-26 10:02:15.124 PDT 1387333718 ERROR: could not open segment 1 of
> relation 1663/43801/7274801 (target block 478216192): No such file or
> directory
> 2006-09-26 10:02:52.991 PDT 1387340456 ERROR: could not open segment 1 of
> relation 1663/43801/7274801 (target block 478216192): No such file or
> directory
> 2006-09-26 10:02:53.002 PDT 1387340458 ERROR: could not open segment 1 of
> relation 1663/43801/7274801 (target block 478216192): No such file or
> directory
>
> The server never really recovered -- it kept running but was horribly
> slow. Rebooting seems to have cured the problem, but I am wondering what I
> made look into, etc.
>
> The server in question is really a read-only beast with several hundred
> connections, typically, 4 gigs of RAM, and has not misbehaved before.
>
> Any suggestions would be welcome! (Although I fear I know the answer --
> disk issues.)
>
> Thanks,
>
> Greg Williamson
> DBA
> GlobeXplorer LLC
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: Don't 'kill -9' the postmaster
>

Thanks for the suggestion ...no evidence of such a shutdown. The first of these errors (since 2005-10-28) was at 2006-09-26 05:37:19.157 PDT today; machine was last shut serveral weeks ago. There's been several large updates since then (sort of batch process for updates, then the server is read-only for 99.999 of the time).

Greg

-----Original Message-----
From: Talha Khan [mailto:talha.amjad@gmail.com]
Sent: Tue 9/26/2006 3:21 PM
To: Gregory S. Williamson
Cc: pgsql-admin@postgresql.org
Subject: Re: [ADMIN] Runtime error: could not open segment 1 of relation ...

Hi Greg,

Is it possible that the server was shutdown using -mi switch?

cuz immediate shutdown ends causing these problems.

Regards
Talha Khan


On 9/26/06, Gregory S. Williamson <gsw@globexplorer.com> wrote:
>
> This morning one of our production servers (8.1.1, Linux) threw a number
> of errors like these (different final number; first two [1663/43801] are
> all the same]:
>
> 2006-09-26 10:02:15.124 PDT 1387333718 ERROR: could not open segment 1 of
> relation 1663/43801/7274801 (target block 478216192): No such file or
> directory
> 2006-09-26 10:02:52.991 PDT 1387340456 ERROR: could not open segment 1 of
> relation 1663/43801/7274801 (target block 478216192): No such file or
> directory
> 2006-09-26 10:02:53.002 PDT 1387340458 ERROR: could not open segment 1 of
> relation 1663/43801/7274801 (target block 478216192): No such file or
> directory
>
> The server never really recovered -- it kept running but was horribly
> slow. Rebooting seems to have cured the problem, but I am wondering what I
> made look into, etc.
>
> The server in question is really a read-only beast with several hundred
> connections, typically, 4 gigs of RAM, and has not misbehaved before.
>
> Any suggestions would be welcome! (Although I fear I know the answer --
> disk issues.)
>
> Thanks,
>
> Greg Williamson
> DBA
> GlobeXplorer LLC
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: Don't 'kill -9' the postmaster
>



-------------------------------------------------------
Click link below if it is SPAM gsw@globexplorer.com
"https://mailscanner.globexplorer.com/dspam/dspam.cgi?signatureID=4519a6f898846491211187&user= gsw@globexplorer.com&retrain=spam&template=history &history_page=1"
!DSPAM:4519a6f898846491211187!
-------------------------------------------------------




---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings

Try reindexing your database and see if that solves your problem.

Regards
Talha Khan


On 9/26/06, Gregory S. Williamson <gsw@globexplorer.com> wrote:
>
> Thanks for the suggestion ...no evidence of such a shutdown. The first of
> these errors (since 2005-10-28) was at 2006-09-26 05:37:19.157 PDT today;
> machine was last shut serveral weeks ago. There's been several large updates
> since then (sort of batch process for updates, then the server is read-only
> for 99.999 of the time).
>
> Greg
>
> -----Original Message-----
> From: Talha Khan [mailto:talha.amjad@gmail.com]
> Sent: Tue 9/26/2006 3:21 PM
> To: Gregory S. Williamson
> Cc: pgsql-admin@postgresql.org
> Subject: Re: [ADMIN] Runtime error: could not open segment 1 of
> relation ...
>
> Hi Greg,
>
> Is it possible that the server was shutdown using -mi switch?
>
> cuz immediate shutdown ends causing these problems.
>
> Regards
> Talha Khan
>
>
> On 9/26/06, Gregory S. Williamson <gsw@globexplorer.com> wrote:
> >
> > This morning one of our production servers (8.1.1, Linux) threw a number
> > of errors like these (different final number; first two [1663/43801]
> are
> > all the same]:
> >
> > 2006-09-26 10:02:15.124 PDT 1387333718 ERROR: could not open segment 1
> of
> > relation 1663/43801/7274801 (target block 478216192): No such file or
> > directory
> > 2006-09-26 10:02:52.991 PDT 1387340456 ERROR: could not open segment 1
> of
> > relation 1663/43801/7274801 (target block 478216192): No such file or
> > directory
> > 2006-09-26 10:02:53.002 PDT 1387340458 ERROR: could not open segment 1
> of
> > relation 1663/43801/7274801 (target block 478216192): No such file or
> > directory
> >
> > The server never really recovered -- it kept running but was horribly
> > slow. Rebooting seems to have cured the problem, but I am wondering what
> I
> > made look into, etc.
> >
> > The server in question is really a read-only beast with several hundred
> > connections, typically, 4 gigs of RAM, and has not misbehaved before.
> >
> > Any suggestions would be welcome! (Although I fear I know the answer --
> > disk issues.)
> >
> > Thanks,
> >
> > Greg Williamson
> > DBA
> > GlobeXplorer LLC
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 2: Don't 'kill -9' the postmaster
> >
>
>
>
> -------------------------------------------------------
> Click link below if it is SPAM gsw@globexplorer.com
> "
> https://mailscanner.globexplorer.com...history_page=1
> "
> !DSPAM:4519a6f898846491211187!
> -------------------------------------------------------
>
>
>
>

"Gregory S. Williamson" <gsw@globexplorer.com> writes:
>> 2006-09-26 10:02:15.124 PDT 1387333718 ERROR: could not open segment 1 of
>> relation 1663/43801/7274801 (target block 478216192): No such file or
>> directory

This is most likely caused by trying to follow a bogus item pointer in
an index. If the problem went away after a restart, the bogus value
could not have been on disk but must have been in a bad in-memory copy
of an index page. Which suggests a hardware glitch ... although if you
are using contrib/ltree, I recall seeing several bug fixes go by
recently that were described as fixing potential ltree-index corruption
issues.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend

Tom,

I haven't used the ltree module ... I'll take a look at our DB and see. It's possible that this was an index problem as there was a bulk update with indexes in tact, so there would have been a lot of index shuffling, and the db was not shut down since that update.

Thanks,

Greg


-----Original Message-----
From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
Sent: Tue 9/26/2006 3:36 PM
To: Gregory S. Williamson
Cc: Talha Khan; pgsql-admin@postgresql.org
Subject: Re: [ADMIN] Runtime error: could not open segment 1 of relation ...

"Gregory S. Williamson" <gsw@globexplorer.com> writes:
>> 2006-09-26 10:02:15.124 PDT 1387333718 ERROR: could not open segment 1 of
>> relation 1663/43801/7274801 (target block 478216192): No such file or
>> directory

This is most likely caused by trying to follow a bogus item pointer in
an index. If the problem went away after a restart, the bogus value
could not have been on disk but must have been in a bad in-memory copy
of an index page. Which suggests a hardware glitch ... although if you
are using contrib/ltree, I recall seeing several bug fixes go by
recently that were described as fixing potential ltree-index corruption
issues.

regards, tom lane


-------------------------------------------------------
Click link below if it is SPAM gsw@globexplorer.com
"https://mailscanner.globexplorer.com/dspam/dspam.cgi?signatureID=4519aa77100151804284693&user =gsw@globexplorer.com&retrain=spam&template=histor y&history_page=1"
!DSPAM:4519aa77100151804284693!
-------------------------------------------------------






---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

Talha,

Seems like a plan ... once we have it out of runtime I'll be able to redo indexes, whatever.

Thanks,

Greg

-----Original Message-----
From: Talha Khan [mailto:talha.amjad@gmail.com]
Sent: Tue 9/26/2006 3:33 PM
To: Gregory S. Williamson
Cc: pgsql-admin@postgresql.org
Subject: Re: [ADMIN] Runtime error: could not open segment 1 of relation ...

Try reindexing your database and see if that solves your problem.

Regards
Talha Khan


On 9/26/06, Gregory S. Williamson <gsw@globexplorer.com> wrote:
>
> Thanks for the suggestion ...no evidence of such a shutdown. The first of
> these errors (since 2005-10-28) was at 2006-09-26 05:37:19.157 PDT today;
> machine was last shut serveral weeks ago. There's been several large updates
> since then (sort of batch process for updates, then the server is read-only
> for 99.999 of the time).
>
> Greg
>
> -----Original Message-----
> From: Talha Khan [mailto:talha.amjad@gmail.com]
> Sent: Tue 9/26/2006 3:21 PM
> To: Gregory S. Williamson
> Cc: pgsql-admin@postgresql.org
> Subject: Re: [ADMIN] Runtime error: could not open segment 1 of
> relation ...
>
> Hi Greg,
>
> Is it possible that the server was shutdown using -mi switch?
>
> cuz immediate shutdown ends causing these problems.
>
> Regards
> Talha Khan
>
>
> On 9/26/06, Gregory S. Williamson <gsw@globexplorer.com> wrote:
> >
> > This morning one of our production servers (8.1.1, Linux) threw a number
> > of errors like these (different final number; first two [1663/43801]
> are
> > all the same]:
> >
> > 2006-09-26 10:02:15.124 PDT 1387333718 ERROR: could not open segment 1
> of
> > relation 1663/43801/7274801 (target block 478216192): No such file or
> > directory
> > 2006-09-26 10:02:52.991 PDT 1387340456 ERROR: could not open segment 1
> of
> > relation 1663/43801/7274801 (target block 478216192): No such file or
> > directory
> > 2006-09-26 10:02:53.002 PDT 1387340458 ERROR: could not open segment 1
> of
> > relation 1663/43801/7274801 (target block 478216192): No such file or
> > directory
> >
> > The server never really recovered -- it kept running but was horribly
> > slow. Rebooting seems to have cured the problem, but I am wondering what
> I
> > made look into, etc.
> >
> > The server in question is really a read-only beast with several hundred
> > connections, typically, 4 gigs of RAM, and has not misbehaved before.
> >
> > Any suggestions would be welcome! (Although I fear I know the answer --
> > disk issues.)
> >
> > Thanks,
> >
> > Greg Williamson
> > DBA
> > GlobeXplorer LLC
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 2: Don't 'kill -9' the postmaster
> >
>
>
>
> -------------------------------------------------------
> Click link below if it is SPAM gsw@globexplorer.com
> "
> https://mailscanner.globexplorer.com...history_page=1
> "
>
> -------------------------------------------------------
>
>
>
>



-------------------------------------------------------
Click link below if it is SPAM gsw@globexplorer.com
"https://mailscanner.globexplorer.com/dspam/dspam.cgi?signatureID=4519a9a399831804284693&user= gsw@globexplorer.com&retrain=spam&template=history &history_page=1"
!DSPAM:4519a9a399831804284693!
-------------------------------------------------------




---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings