SEO

"Mark Wong" <markwkm@gmail.com> writes:
> I saw a that a patch was committed that exposed a configure switch for
> BLCKSZ. I was hoping that I could do that same for XLOG_BLCKSZ.

Well, we certainly *could*, but what's the use-case really? The case
for varying BLCKSZ is marginal already, and I've seen none at all for
varying XLOG_BLCKSZ. Why do we need to make it easier than "edit
pg_config_manual.h"?

regards, tom lane

--
Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

Tom Lane wrote:
> "Mark Wong" <markwkm@gmail.com> writes:
>> I saw a that a patch was committed that exposed a configure switch for
>> BLCKSZ. I was hoping that I could do that same for XLOG_BLCKSZ.
>
> Well, we certainly *could*, but what's the use-case really? The case
> for varying BLCKSZ is marginal already, and I've seen none at all for
> varying XLOG_BLCKSZ. Why do we need to make it easier than "edit
> pg_config_manual.h"?

The use case I could see is for performance testing but I would concur
that it doesn't take much to modify pg_config_manual.h. In thinking
about it, this might actually be a foot gun. You have a new pg guy,
download source and think to himself..., "Hey I have a 4k block size as
formatted on my hard disk". Then all of a sudden they have an
incompatible PostgreSQL with everything else.

Sincerely,

Joshua D. Drake

>
> regards, tom lane
>


--
Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

On Fri, May 2, 2008 at 12:04 AM, Joshua D. Drake <jd@commandprompt.com> wrote:
>
> Tom Lane wrote:
>
> > "Mark Wong" <markwkm@gmail.com> writes:
> >
> > > I saw a that a patch was committed that exposed a configure switch for
> > > BLCKSZ. I was hoping that I could do that same for XLOG_BLCKSZ.
> > >
> >
> > Well, we certainly *could*, but what's the use-case really? The case
> > for varying BLCKSZ is marginal already, and I've seen none at all for
> > varying XLOG_BLCKSZ. Why do we need to make it easier than "edit
> > pg_config_manual.h"?
> >
>
> The use case I could see is for performance testing but I would concur that
> it doesn't take much to modify pg_config_manual.h. In thinking about it,
> this might actually be a foot gun. You have a new pg guy, download source
> and think to himself..., "Hey I have a 4k block size as formatted on my hard
> disk". Then all of a sudden they have an incompatible PostgreSQL with
> everything else.

As someone who has tested varying both those parameters it feels
awkward to have a configure option for one and not the other, or vice
versa. I have slightly stronger feelings for having them both as
configure options because it's easier to script, but feel a little
more strongly about having BLCKSZ and XLOG_BLCKSZ both as either
configure options or in pg_config_manual.h. To have them such that
one needs to change them in different manners makes a tad more work in
automating testing. So my case is just for ease of testing.

Regards,
Mark

--
Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

"Mark Wong" <markwkm@gmail.com> writes:
> As someone who has tested varying both those parameters it feels
> awkward to have a configure option for one and not the other, or vice
> versa. I have slightly stronger feelings for having them both as
> configure options because it's easier to script, but feel a little
> more strongly about having BLCKSZ and XLOG_BLCKSZ both as either
> configure options or in pg_config_manual.h. To have them such that
> one needs to change them in different manners makes a tad more work in
> automating testing. So my case is just for ease of testing.

Well, that's a fair point. Another issue though is whether it makes
sense for XLOG_BLCKSZ to be different from BLCKSZ at all, at least in
the default case. They are both the unit of I/O and it's not clear
why you'd want different units. Mark, has your testing shown any
indication that they really ought to be separately configurable?
I could see having the same configure switch set both of 'em.

regards, tom lane

--
Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

On Fri, May 2, 2008 at 8:50 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> "Mark Wong" <markwkm@gmail.com> writes:
>
> > As someone who has tested varying both those parameters it feels
> > awkward to have a configure option for one and not the other, or vice
> > versa. I have slightly stronger feelings for having them both as
> > configure options because it's easier to script, but feel a little
> > more strongly about having BLCKSZ and XLOG_BLCKSZ both as either
> > configure options or in pg_config_manual.h. To have them such that
> > one needs to change them in different manners makes a tad more work in
> > automating testing. So my case is just for ease of testing.
>
> Well, that's a fair point. Another issue though is whether it makes
> sense for XLOG_BLCKSZ to be different from BLCKSZ at all, at least in
> the default case. They are both the unit of I/O and it's not clear
> why you'd want different units. Mark, has your testing shown any
> indication that they really ought to be separately configurable?
> I could see having the same configure switch set both of 'em.

I still believe it makes sense to have them separated. I did have
some data, which has since been destroyed, that suggested there were
some system characterization differences for OLTP workloads with
PostgreSQL. Let's hope those disks get delivered to Portland soon.

Regards,
Mark

--
Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

"Mark Wong" <markwkm@gmail.com> writes:
> I still believe it makes sense to have them separated. I did have
> some data, which has since been destroyed, that suggested there were
> some system characterization differences for OLTP workloads with
> PostgreSQL. Let's hope those disks get delivered to Portland soon.

Fair enough. It's not that much more code to have another configure
switch --- will go do that.

If we are allowing blocksize and relation seg size to have configure
switches, seems that symmetry would demand that XLOG_SEG_SIZE be
configurable as well. Thoughts?

regards, tom lane

--
Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

On Fri, 2 May 2008 09:12:32 -0700
"Mark Wong" <markwkm@gmail.com> wrote:


> I still believe it makes sense to have them separated. I did have
> some data, which has since been destroyed, that suggested there were
> some system characterization differences for OLTP workloads with
> PostgreSQL. Let's hope those disks get delivered to Portland soon.

I have those disks.

Joshua D. Drake


>
> Regards,
> Mark
>


--
The PostgreSQL Company since 1997: http://www.commandprompt.com/
PostgreSQL Community Conference: http://www.postgresqlconference.org/
United States PostgreSQL Association: http://www.postgresql.us/
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFIG0FLATb/zqfZUUQRAsZGAKCblFWMfjrYxTHlYgSMRCYG5s+J2ACgkduf
uZZPLNEjFssXhuYubt6DLAI=
=GiNg
-----END PGP SIGNATURE-----

On Fri, 2 May 2008, Tom Lane wrote:

> The case for varying BLCKSZ is marginal already, and I've seen none at
> all for varying XLOG_BLCKSZ.

I recall someone on the performance list who felt it useful increase
XLOG_BLCKSZ to support a high-write environment with WAL shipping, just to
make sending the files over the network more efficient. Can't seem to
find a reference in the archives though.

If you look at things like the giant Sun system tests, there was
significant tuning getting all the block sizes to line up better with the
underlying hardware. I would not be surprised to discover that sort of
install gains a bit from slinging WAL files around in larger chunks as
well. They're already using small values for commit_delay just to get the
typical WAL write to be in larger blocks.

As PostgreSQL makes it way into higher throughput environments, it
wouldn't surprise me to discover more of these situations where switching
WAL segments every 16MB turns into a bottleneck. Right now, it may only
be a few people in the world, but saying "that's big enough" for an
allocation of anything usually turns out wrong if you wait long enough.

One real concern I have with making this easier to adjust is that I'd hate
to let people pick any old block size with the default wal_sync_method,
only to have them later discover they can't turn on any direct I/O write
method because they botched the alignment restrictions.

> Another issue though is whether it makes sense for XLOG_BLCKSZ to be
> different from BLCKSZ at all, at least in the default case. They are
> both the unit of I/O and it's not clear why you'd want different units.

There are lots of people who use completely different physical or logical
disk setups for the WAL disk than the regular database. That's going to
get even more varied moving forward as SSD starts getting used more, since
those devices have a very different set of block size optimization
characteristics compared with traditional RAID setups. They prefer
smaller blocks to match the underlying flash better, and you don't pay as
much of a penalty for writing that way because lining up with the spinning
disk isn't important. Someone who put one of DB/WAL on SSD and the other
on traditional disk might end up with very different DB/WAL block sizes to
match.

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD

--
Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

On Fri, May 2, 2008 at 9:16 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> "Mark Wong" <markwkm@gmail.com> writes:
>
> > I still believe it makes sense to have them separated. I did have
> > some data, which has since been destroyed, that suggested there were
> > some system characterization differences for OLTP workloads with
> > PostgreSQL. Let's hope those disks get delivered to Portland soon.
>
> Fair enough. It's not that much more code to have another configure
> switch --- will go do that.
>
> If we are allowing blocksize and relation seg size to have configure
> switches, seems that symmetry would demand that XLOG_SEG_SIZE be
> configurable as well. Thoughts?

I don't have a feel for this one, but when we get the disks set up we
can certainly test to see what effects it has.

Regards,
Mark

--
Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches

On Fri, 2008-05-02 at 12:28 -0400, Greg Smith wrote:

> As PostgreSQL makes its way into higher throughput environments, it
> wouldn't surprise me to discover more of these situations where switching
> WAL segments every 16MB turns into a bottleneck.

We already hit that issue and fixed it early in the 8.3 cycle. It was
more of a problem than the checkpoint issue because it caused hard
lock-outs while the file switches occurred. It didn't show up unless you
looked at the very detailed transaction result data because on fast
systems we are file switching every few seconds.

Not seen any gains from varying the WAL file size since then...

--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com


--
Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches