SEO

Hi all,

I'm looking to push 8.3 out this week, but I'm running into a
particularly nasty bus error. I'm not sure what's causing it as it
appears to be transient (or at least somewhat random), but I do know
that it bites on connection time and takes down the entire server with
it.

I'm going to try to force a crash on my test server to see if I can
find out anything more. Does anyone know how to get useful debugging
info at the time of the crash? I don't think I have things set to dump
core anywhere, but that and/or stack traces would be nice (especially
for child processes).

If anyone has any idea what's going on and could clue me in, that
would be excellent.

Peter

P.S. Here's the syslog transcript of the crash.

Apr 28 23:02:02 mitchell postgres[3442]: [3-1] LOG: connection
received: host=mitchell.cs.wisc.edu port=37588
Apr 28 23:02:02 mitchell postgres[3442]: [4-1] LOG: connection
authorized: user=postgres database=template1
Apr 28 23:02:02 mitchell postgres[3442]: [5-1] LOG: disconnection:
session time: 0:00:00.239 user=postgres database=template1
host=mitchell.cs.wisc.edu port=37588
Apr 28 23:02:02 mitchell postgres[3444]: [3-1] LOG: connection
received: host=mitchell.cs.wisc.edu port=37589
Apr 28 23:02:02 mitchell postgres[3444]: [4-1] LOG: connection
authorized: user=postgres database=564testdb
Apr 28 23:02:02 mitchell postgres[461]: [3-1] LOG: server process
(PID 3444) was terminated by signal 7: Bus error
Apr 28 23:02:02 mitchell postgres[461]: [4-1] LOG: terminating any
other active server processes
Apr 28 23:02:02 mitchell postgres[461]: [5-1] LOG: all server
processes terminated; reinitializing
Apr 28 23:02:02 mitchell postgres[3447]: [6-1] LOG: connection
received: host=mitchell.cs.wisc.edu port=37590
Apr 28 23:02:02 mitchell postgres[3447]: [7-1] FATAL: the database
system is in recovery mode
Apr 28 23:02:02 mitchell postgres[3448]: [6-1] LOG: connection
received: host=mitchell.cs.wisc.edu port=37591
Apr 28 23:02:02 mitchell postgres[3448]: [7-1] FATAL: the database
system is in recovery mode
Apr 28 23:02:23 mitchell postgres[461]: [6-1] LOG: startup process
(PID 3446) was terminated by signal 7: Bus error
Apr 28 23:02:23 mitchell postgres[461]: [7-1] LOG: aborting startup
due to startup process failure
Apr 28 23:22:15 mitchell postgres[3702]: [1-1] LOG: could not load
root certificate file "root.crt": No such file or directory
Apr 28 23:22:15 mitchell postgres[3702]: [1-2] DETAIL: Will not
verify client certificates.
Apr 28 23:22:15 mitchell postgres[3703]: [2-1] LOG: database system
was interrupted; last known up at 2008-04-28 07:02:32 CDT
Apr 28 23:22:15 mitchell postgres[3703]: [3-1] LOG: database system
was not properly shut down; automatic recovery in progress
Apr 28 23:22:15 mitchell postgres[3703]: [4-1] LOG: record with zero
length at 6/915320A0

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

"Peter Koczan" <pjkoczan@gmail.com> writes:
> I'm going to try to force a crash on my test server to see if I can
> find out anything more. Does anyone know how to get useful debugging
> info at the time of the crash? I don't think I have things set to dump
> core anywhere, but that and/or stack traces would be nice (especially
> for child processes).

Make sure the postmaster is started under ulimit -c unlimited.
On a depressingly large fraction of modern platforms, daemons are
started with ulimit -c 0 by default :-(. Try putting "ulimit -c unlimited"
into your PG init script and restarting.

regards, tom lane

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

On Tue, Apr 29, 2008 at 10:35 AM, Peter Koczan <pjkoczan@gmail.com> wrote:

>
> I'm going to try to force a crash on my test server to see if I can
> find out anything more. Does anyone know how to get useful debugging
> info at the time of the crash? I don't think I have things set to dump
> core anywhere, but that and/or stack traces would be nice (especially
> for child processes).
>

Yeah, a stack trace and if possible, a self contained test case to
reproduce the bug would help. If you are using a custom build, then
using a debug build would help a lot too.

Thanks,
Pavan

--
Pavan Deolasee
EnterpriseDB http://www.enterprisedb.com

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

On Tue, Apr 29, 2008 at 1:01 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Make sure the postmaster is started under ulimit -c unlimited.
> On a depressingly large fraction of modern platforms, daemons are
> started with ulimit -c 0 by default :-(. Try putting "ulimit -c unlimited"
> into your PG init script and restarting.

> On Tue, Apr 29, 2008 at 10:35 AM, Peter Koczan <pjkoczan@gmail.com> wrote:
> Yeah, a stack trace and if possible, a self contained test case to
> reproduce the bug would help. If you are using a custom build, then
> using a debug build would help a lot too.

So far, this problem hasn't reoccurred since I first saw it, no matter
how much I pummel my test server.

I was testing some patches about the time that this happened
(including one in the libpq backend), so there may have been an ABI
mismatch which led to the bus error.

In any case, I've set the postmaster to have ulimit -c unlimited in
case this problem reoccurs. I'll also look into using a debug build in
case this happens again. But until then, I'm confident enough to keep
pushing out 8.3.

Thanks again.
Peter

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

On Tue, Apr 29, 2008 at 8:01 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> "Peter Koczan" <pjkoczan@gmail.com> writes:
> > I'm going to try to force a crash on my test server to see if I can
> > find out anything more. Does anyone know how to get useful debugging
> > info at the time of the crash? I don't think I have things set to dump
> > core anywhere, but that and/or stack traces would be nice (especially
> > for child processes).
>
> Make sure the postmaster is started under ulimit -c unlimited.
> On a depressingly large fraction of modern platforms, daemons are
> started with ulimit -c 0 by default :-(. Try putting "ulimit -c unlimited"
> into your PG init script and restarting.
>
> regards, tom lane

Do you think a too low limit on user processes could be causing the problem?

We have had similar crashes (server process terminated with Bus
Error). This is running 8.2.5 on OS X (Tiger, intel), with about
130-140 connections, but the process count limit is 1000. Any reason
why the process limit should be vastly higher than the number of
connections?

--
Venlig Hilsen / Kind Regards

Filip Svendsen
fs@basepointmedia.com

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Filip Svendsen wrote:
> On Tue, Apr 29, 2008 at 8:01 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
>> "Peter Koczan" <pjkoczan@gmail.com> writes:
>> > I'm going to try to force a crash on my test server to see if I can
>> > find out anything more. Does anyone know how to get useful debugging
>> > info at the time of the crash? I don't think I have things set to dump
>> > core anywhere, but that and/or stack traces would be nice (especially
>> > for child processes).
>>
>> Make sure the postmaster is started under ulimit -c unlimited.
>> On a depressingly large fraction of modern platforms, daemons are
>> started with ulimit -c 0 by default :-(. Try putting "ulimit -c unlimited"
>> into your PG init script and restarting.
>>
>> regards, tom lane
>>
>
> Do you think a too low limit on user processes could be causing the problem?
>
> We have had similar crashes (server process terminated with Bus
> Error). This is running 8.2.5 on OS X (Tiger, intel), with about
> 130-140 connections, but the process count limit is 1000. Any reason
> why the process limit should be vastly higher than the number of
> connections?
>

"ulimit -c unlimited" sets the core dump size to be unlimited, not the
connection count.
That would be related to "ulimit -n" (descriptor count.)

Darren


--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Hi All,

On a newly created server I stupidly restored over the top of the
postgres system database using the GUI...(postgresql n00b) and am
wondering a couple of things.

Firstly, if this database is overwritten, is it detrimental to the
running of PostgreSQL?

Secondly is it just a simple matter of restoring it using a backup
from a different PostgreSQL instance installed elsewhere.

Regards,


Patrick



__________________________________________________ ____________________
This email, including attachments, is intended only for the addressee
and may be confidential, privileged and subject to copyright. If you
have received this email in error, please advise the sender and delete
it. If you are not the intended recipient of this email, you must not
use, copy or disclose its content to anyone. You must not copy or
communicate to others content that is confidential or subject to
copyright, unless you have the consent of the content owner.

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

> On a newly created server I stupidly restored over the top of the
> postgres system database using the GUI...(postgresql n00b) and am
> wondering a couple of things.
> Firstly, if this database is overwritten, is it detrimental to the
running of PostgreSQL?
> Secondly is it just a simple matter of restoring it using a backup from
a different PostgreSQL instance installed elsewhere.

When you say "system database", do you mean 'template1'? Or 'template0'? Or
'postgres'? Or something else?


THINK BEFORE YOU PRINT - Save paper if you don't really need to print this

*******************Confidentiality and Privilege Notice*******************

The material contained in this message is privileged and confidential to
the addressee. If you are not the addressee indicated in this message or
responsible for delivery of the message to such person, you may not copy
or deliver this message to anyone, and you should destroy it and kindly
notify the sender by reply email.

Information in this message that does not relate to the official business
of Weatherbeeta must be treated as neither given nor endorsed by Weatherbeeta.
Weatherbeeta, its employees, contractors or associates shall not be liable
for direct, indirect or consequential loss arising from transmission of this
message or any attachments
e-mail.

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

To be more clear it was the 'postgres' database.

Regards,

Patrick

-----Original Message-----
From: Phillip Smith [mailtohillip.smith@weatherbeeta.com.au]
Sent: Monday, 5 May 2008 4:22 PM
To: Patrick Roberts; pgsql-admin@postgresql.org
Subject: RE: [ADMIN] restore postgres system database 8.1.11

> On a newly created server I stupidly restored over the top of the
> postgres system database using the GUI...(postgresql n00b) and am
> wondering a couple of things.
> Firstly, if this database is overwritten, is it detrimental to the
running of PostgreSQL?
> Secondly is it just a simple matter of restoring it using a backup
from
a different PostgreSQL instance installed elsewhere.

When you say "system database", do you mean 'template1'? Or 'template0'?
Or
'postgres'? Or something else?


THINK BEFORE YOU PRINT - Save paper if you don't really need to print
this

*******************Confidentiality and Privilege
Notice*******************

The material contained in this message is privileged and confidential to
the addressee. If you are not the addressee indicated in this message
or
responsible for delivery of the message to such person, you may not copy
or deliver this message to anyone, and you should destroy it and kindly
notify the sender by reply email.

Information in this message that does not relate to the official
business
of Weatherbeeta must be treated as neither given nor endorsed by
Weatherbeeta.
Weatherbeeta, its employees, contractors or associates shall not be
liable
for direct, indirect or consequential loss arising from transmission of
this
message or any attachments
e-mail.

__________________________________________________ ____________________
This email, including attachments, is intended only for the addressee
and may be confidential, privileged and subject to copyright. If you
have received this email in error, please advise the sender and delete
it. If you are not the intended recipient of this email, you must not
use, copy or disclose its content to anyone. You must not copy or
communicate to others content that is confidential or subject to
copyright, unless you have the consent of the content owner.

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Patrick Roberts wrote:
> Hi All,
>
> On a newly created server I stupidly restored over the top of the
> postgres system database using the GUI...(postgresql n00b) and am
> wondering a couple of things.
>
> Firstly, if this database is overwritten, is it detrimental to the
> running of PostgreSQL?

The "postgres" database is basically an empty holder so that when you do
this:

psql

as the user postgres, you actually get into the system.
>
> Secondly is it just a simple matter of restoring it using a backup
> from a different PostgreSQL instance installed elsewhere.

If you are indeed speaking about the "postgres" database yes.

Sincerely,

Joshua D. Drake

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin