SEO

Hello All

We are having an issue and it appears to be with file descriptors.
We are running PostgreSQL 7.4.13 on RHEL 3.0 box. This system is running
Redhat Cluster Suite with each system running multiple postmasters. The
error has occurred on both boxes and we can't find any consistencies on
what is causing the problem. I don't see this message in dmesg or any
other log file. It only appears in this single database. There are other
databases running on the same host but from different postmasters and I
don't see the same message. A couple of things to add is that this has
happened on another machine from a different database. When this
problem has been reoccurring for a period of several days the server
get fenced by the other machine. Don't know if related but seems to
happen at the time when the descriptors error is occurring. Here is a
snippet from the log of one of the databases.

2006-11-29 14:57:51 [3823] LOG: out of file descriptors: Too many open
files; release and retry
2006-11-29 14:58:01 [3823] LOG: out of file descriptors: Too many open
files; release and retry
2006-11-29 15:00:26 [3823] LOG: out of file descriptors: Too many open
files; release and retry

--
**John Allgood**
/*/Senior Systems Administrator/*/
**Turbo Logistics - Division of Ozburn-Hessey Logistics**
**2251 Jesse Jewell Pky. NE**
**Gainesville, GA 30507******
**tel (678) 989-3051 fax (770) 531-7878**
**john@turbocorp.com** <mailto:john@turbocorp.com>
**www.turbocorp.com** <http://www.turbocorp.com>



---------------------------(end of broadcast)---------------------------
TIP 7: You can help support the PostgreSQL project by donating at

http://www.postgresql.org/about/donate

John Allgood <john@turbocorp.com> writes:
> 2006-11-29 14:57:51 [3823] LOG: out of file descriptors: Too many open
> files; release and retry

Consider reducing PG's max_files_per_process setting. Postgres itself
will usually not have a serious problem when you've run the kernel out
of file descriptors, but everything else on the machine will :-(

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

Hey Tom

I assume that if it is the kernel running out of descriptors that I
would get the messages in dmesg. This message only appears in the log
file for that database.

Thanks

Tom Lane wrote:
> John Allgood <john@turbocorp.com> writes:
>
>> 2006-11-29 14:57:51 [3823] LOG: out of file descriptors: Too many open
>> files; release and retry
>>
>
> Consider reducing PG's max_files_per_process setting. Postgres itself
> will usually not have a serious problem when you've run the kernel out
> of file descriptors, but everything else on the machine will :-(
>
> regards, tom lane
>

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

John Allgood wrote:
> Hey Tom
>
> I assume that if it is the kernel running out of descriptors that I
> would get the messages in dmesg. This message only appears in the log
> file for that database.

Yeah, the point is that you have the max_files_per_process setting
higher than what the kernel likes. So decrease it, and Postgres will
adjust itself to use less file descriptors by closing and reopening
files as needed.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

Alvaro Herrera wrote:
> John Allgood wrote:
>
>> Hey Tom
>>
>> I assume that if it is the kernel running out of descriptors that I
>> would get the messages in dmesg. This message only appears in the log
>> file for that database.
>>
>
> Yeah, the point is that you have the max_files_per_process setting
> higher than what the kernel likes. So decrease it, and Postgres will
> adjust itself to use less file descriptors by closing and reopening
> files as needed.
>
>
I really can't see that is possible file-max is set to 838860. I believe
it is based on the amount of ram on the machine and we have 8GB. ulimit
-a reports you can have 1024 open files. I thought maybe we were running
out of descriptors for the postgres user but that should not bring down
the machine. Thoughts anyone.

---------------------------(end of broadcast)---------------------------
TIP 7: You can help support the PostgreSQL project by donating at

http://www.postgresql.org/about/donate

Alvaro Herrera wrote:
> John Allgood wrote:
>> Hey Tom
>>
>> I assume that if it is the kernel running out of descriptors that I
>> would get the messages in dmesg. This message only appears in the log
>> file for that database.
>
> Yeah, the point is that you have the max_files_per_process setting
> higher than what the kernel likes. So decrease it, and Postgres will
> adjust itself to use less file descriptors by closing and reopening
> files as needed.

I work with the original poster and wanted to make sure the problem here
is clear. The 'out of file descriptors' message is coming from
Postgresql, not the kernel. Thus, it doesn't make sense to me that the
max_files_per_process setting is too high. I would think we need to
increase it so that Postgresql will stop generating these errors.

--
Until later, Geoffrey

Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety.
- Benjamin Franklin

---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

Geoffrey wrote:

> Okay, I'm just not getting it. Postgres complains that it is out of
> file descriptors. The kernel is not complaining about any such issues.
>
>
> So I should lower the max_files_per_process
> value and this will rid us of the 'out of file descriptors' error?
>
> Is it because the max_files_per_process is greater then the number of
> file descriptors that are alloted to Postgres by the kernel?

Yes. Which is kinda weird, because Postgres actually tests the number
when it starts, so that if you set the number too high, it will decrease
it according to what the kernel allows.

Maybe the test is newer than the version you are running -- what was it,
again?

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match

Maybe the whole problem is with ulimit. Ulimit sets the max files per
user at the shell and postgres might be exceeding the 1024 limit. What
would increasing this value to 2048 do?

Geoffrey wrote:
> Alvaro Herrera wrote:
>> Geoffrey wrote:
>>> Alvaro Herrera wrote:
>>>> John Allgood wrote:
>>>>> Hey Tom
>>>>>
>>>>> I assume that if it is the kernel running out of descriptors
>>>>> that I
>>>>> would get the messages in dmesg. This message only appears in the log
>>>>> file for that database.
>>>> Yeah, the point is that you have the max_files_per_process setting
>>>> higher than what the kernel likes. So decrease it, and Postgres will
>>>> adjust itself to use less file descriptors by closing and reopening
>>>> files as needed.
>>> I work with the original poster and wanted to make sure the problem
>>> here is clear.
>>
>> Yes, that was understood from the beginning.
>>
>>> The 'out of file descriptors' message is coming from Postgresql, not
>>> the kernel. Thus, it doesn't make sense to me that the
>>> max_files_per_process setting is too high. I would think we need to
>>> increase it so that Postgresql will stop generating these errors.
>>
>> No, you need to lower it so that Postgres doesn't _try_ to use as many
>> file descriptors. Read this again:
>>
>>>> So decrease it, and Postgres will
>>>> adjust itself to use less file descriptors by closing and reopening
>>>> files as needed.
>
> Okay, I'm just not getting it. Postgres complains that it is out of
> file descriptors. The kernel is not complaining about any such issues.
>
> So I should lower the max_files_per_process
> value and this will rid us of the 'out of file descriptors' error?
>
> Is it because the max_files_per_process is greater then the number of
> file descriptors that are alloted to Postgres by the kernel?
>

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

Alvaro Herrera wrote:
> Geoffrey wrote:
>
>> Okay, I'm just not getting it. Postgres complains that it is out of
>> file descriptors. The kernel is not complaining about any such issues.
>>
>>
>> So I should lower the max_files_per_process
>> value and this will rid us of the 'out of file descriptors' error?
>>
>> Is it because the max_files_per_process is greater then the number of
>> file descriptors that are alloted to Postgres by the kernel?
>
> Yes. Which is kinda weird, because Postgres actually tests the number
> when it starts, so that if you set the number too high, it will decrease
> it according to what the kernel allows.
>
> Maybe the test is newer than the version you are running -- what was it,
> again?
>
7.4.13

--
Until later, Geoffrey

Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety.
- Benjamin Franklin

---------------------------(end of broadcast)---------------------------
TIP 7: You can help support the PostgreSQL project by donating at

http://www.postgresql.org/about/donate

Bradley Kieser wrote:
> Hmm, not entirely true. You may well only see the error in the process
> itself.

You're telling me that the kernel could be running out of file
descriptors and I wouldn't see a message regarding this in the kernel
logs? I don't believe that is possible.

>
>
> Geoffrey wrote:
>> Alvaro Herrera wrote:
>>> John Allgood wrote:
>>>> Hey Tom
>>>>
>>>> I assume that if it is the kernel running out of descriptors that I
>>>> would get the messages in dmesg. This message only appears in the log
>>>> file for that database.
>>>
>>> Yeah, the point is that you have the max_files_per_process setting
>>> higher than what the kernel likes. So decrease it, and Postgres will
>>> adjust itself to use less file descriptors by closing and reopening
>>> files as needed.
>>
>> I work with the original poster and wanted to make sure the problem
>> here is clear. The 'out of file descriptors' message is coming from
>> Postgresql, not the kernel. Thus, it doesn't make sense to me that
>> the max_files_per_process setting is too high. I would think we need
>> to increase it so that Postgresql will stop generating these errors.
>>
>


--
Until later, Geoffrey

Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety.
- Benjamin Franklin

---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq