Tom Lane <tgl@sss.pgh.pa.us> wrote:

> ITAGAKI Takahiro <itagaki.takahiro@lab.ntt.co.jp> writes:
> > Attached is a revised patch. It became very simple, but I worry that
> > one magic number (BUFFERS_PER_ABSORB) is still left.
>
> Have you checked that this version of the patch fixes the problem you
> saw originally? Does the problem come back if you change
> BUFFERS_PER_ABSORB to too large a value?

The problem on my machine was resolved by this patch. I tested it and
logged the maximum of BgWriterShmem->num_requests for each checkpoint.
Test condition was:
- shared_buffers = 65536
- connections = 30
The average of maximums was 25857 and the while max was 31807.
They didn't exceed the max_requests(= 65536).

> I suspect it'd probably be sufficient to absorb requests every few times
> through the fsync loop, too, if you want to experiment with that.

In the above test, smgrsync took 50 sec for syncing 32 files. This means
absorb are requested every 1.5 sec, which is less frequent than absorbs by
normal activity of bgwriter (bgwriter_delay=200ms). So I assume absorb
requests the fsync loop would not be a problem.


BUFFERS_PER_ABSORB = 10 (absorb per 1/10 of shared_buffers) is enough at least
on my machine, but it doesn't necessarily work well in all environments.
If we need to set BUFFERS_PER_ABSORB to a reasonably value, I think the number
of active backends might be useful; for example, half of num of backends.

---
ITAGAKI Takahiro
NTT Cyber Space Laboratories



---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?

http://archives.postgresql.org

Tom Lane <tgl@sss.pgh.pa.us> wrote:

> I've applied the attached modified version of your patch. In this
> coding, absorbs are done after every 1000 buffer writes in BufferSync
> and after every 10 fsyncs in mdsync. We may need to twiddle these
> numbers but it seems at least in the right ballpark. If you have time
> to repeat your original test and see how this does, it'd be much
> appreciated.

Thank you. It worked well on my machine(*).
Undesirable behavior was not seen.

(*)
TPC-C(DBT-2)
RHEL4 U1 (2.6.9-11)
XFS, 8 S-ATA disks / 8GB memory(shmem=512MB)

---
ITAGAKI Takahiro
NTT Cyber Space Laboratories



---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match