SEO

Every release we seem to have the same debates about performance issues.

In 8.0 we shipped knowing that bgwriter had serious deficiencies, plus
had no way of logging SQL statements for performance tuning. In 8.2 we
even ended up tweaking the planner *after* release.

What I don't understand is all the words about quality, yet we don't
seem to include performance as part of that. Performance always seems to
be a "feature" that can be left until the next release and it's never
the right time to fix it.

I would hope to persuade all that Performance is an integral part of
Quality, not a hindrance to it.

I've never worked on a software project where either the Users or the
Sponsors said "don't worry about performance, it can wait, but I really
love the way you coded that". Quality is very, very high with Postgres,
but we also need to include performance as one of the Top Level concerns
*and* do that without dropping the ball on other concerns. That clearly
takes time and effort to balance those concerns.

We obviously need a performance build farm and I think everyone accepts
that. We just need to do it, so that's a given and is something I hope
to be involved in.

What I would really like to persuade everybody is that performance needs
specific attention. Once we've finished integrating the code, we're in
Beta and changes seem to be more difficult then. We must give time and
attention both to measuring performance and to fixing the things we
find. Sure we've done a lot of that, and I've been very happy with that,
but recent events make me think we have lapsed back into thinking that
performance is a threat to quality. I'd love to hear people say loud and
clear that performance matters and we can't ship when we know about
fixable performance holes.

Please can we clear some space in the next release schedule for
performance, plus give some credence to the thought that performance
issues rate our attention just as much as other kinds of bugs?

Maybe we should give each Beta a name, such as "Initial Beta",
"Performance Beta", "Usability Beta" as a way of encouraging folk to
focus onto particular aspects of quality at what we consider to be
appropriate times to do so. Not sure whether thats a good idea, but I'd
love to hear about ways to include performance as one of the essential
behaviours of PostgreSQL.

Your thoughts are welcome,

--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com


---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

On Tue, Nov 27, 2007 at 05:32:49PM +0000, Simon Riggs wrote:
> What I would really like to persuade everybody is that performance needs
> specific attention.

[. . .]

> Your thoughts are welcome,

Well, one thing that might help is something of the specifics you mention.

I remember mentioning to Jan not long after he started at Afilias that we
occasionally saw strange behaviour that looked like "lock up". He was
slightly incredulous, and I didn't have time to build a repeatable test
case. So it was in the context of testing Slony that he discovered the dual
pains of buffer shuffling and checkpoint storms; this is part of what led
him to work on those problems in 8.0.

The key was to state, at the outset, "Here is the problem I want to fix."
By stating precisely and specifically what is to be fixed, the issue moves
from "performance needs" to a feature that can be implemented.

Perhaps now is the time to list some specific performance areas you want to
fix up?

A

--
Andrew Sullivan
Old sigs will return after re-constitution of blue smoke

---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

On Tue, 2007-11-27 at 10:08 -0800, Joshua D. Drake wrote:
> Simon Riggs <simon@2ndquadrant.com> wrote:

> > Maybe we should give each Beta a name, such as "Initial Beta",
> > "Performance Beta", "Usability Beta" as a way of encouraging folk to
> > focus onto particular aspects of quality at what we consider to be
> > appropriate times to do so. Not sure whether thats a good idea, but
> > I'd love to hear about ways to include performance as one of the
> > essential behaviours of PostgreSQL.
> >
> > Your thoughts are welcome,
>
> Well I think that we do take performance into account. I agree
> that we should *never* have a regression in performance from release
> to release, which is what I believe has inspired this thread.
>
> However if you look at a lot of the items that have gone into this
> release they were all about performance:

Agreed. I either initiated or assisted with most of those items; but
that's not really my point, however because those were planned
performance "features".

My thinking was about how we handle the last minute attention to detail
that ensures we have performance everywhere, not just on the main
features.

--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com


---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend

Simon Riggs wrote:
>
> We obviously need a performance build farm and I think everyone accepts
> that. We just need to do it, so that's a given and is something I hope
> to be involved in.
>
>
>

It's on my list ... Had I but world enough and time ...

Performance testing can be bolted onto the exiting buildfarm as an
option. However, performance test machines have some requirements that
pure functional/build test machines don't have: especially stability. A
standard buildfarm client can be put on almost any machine and run
happily. My main workstation runs four buildfarm members including three
in a VM, and I never notice any impact. But a performance test machine
probably needs to be dedicated to just that function. And at least some
members of the performance test machines would need to be higher end
machines. The number of people who can afford such resources is much
lower than those who can run a relatively low impact simple buildfarm
member.

Maybe we also need to talk about running clients elsewhere for
performance testing too.

We also need to talk about what would be a good set of tests to run.

One useful thing this would buy us is a time series of test results so
we could easily see sudden degradations in performance. It must have
been annoying trying to triangulate performance dropoff recently.

cheers

andrew


---------------------------(end of broadcast)---------------------------
TIP 7: You can help support the PostgreSQL project by donating at

http://www.postgresql.org/about/donate

Joshua D. Drake wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Tue, 27 Nov 2007 17:32:49 +0000
> Simon Riggs <simon@2ndquadrant.com> wrote:
>
>
> > Maybe we should give each Beta a name, such as "Initial Beta",
> > "Performance Beta", "Usability Beta" as a way of encouraging folk to
> > focus onto particular aspects of quality at what we consider to be
> > appropriate times to do so. Not sure whether thats a good idea, but
> > I'd love to hear about ways to include performance as one of the
> > essential behaviours of PostgreSQL.
>
> Well I think that we do take performance into account. I agree
> that we should *never* have a regression in performance from release
> to release, which is what I believe has inspired this thread.

Hmm. I have developed several features that have driven performance
down. Autovacuum enabled by default for one. IIRC the SELECT FOR SHARE
stuff (tuple-level share locks) also hurt performance. Savepoints
required enlarging tuple headers, which also hurt performance.

In all cases we have gotten some other benefit, be it reduced
administrative pain, or reduced lock contention, or a new feature.

(In fact I think most performance drops have been my fault.)

--
Alvaro Herrera http://www.advogato.org/person/alvherre
"Some men are heterosexual, and some are bisexual, and some
men don't think about sex at all... they become lawyers" (Woody Allen)

---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?

http://archives.postgresql.org

Alvaro Herrera <alvherre@alvh.no-ip.org> writes:
> Joshua D. Drake wrote:
>> Well I think that we do take performance into account. I agree
>> that we should *never* have a regression in performance from release
>> to release, which is what I believe has inspired this thread.

> Hmm. I have developed several features that have driven performance
> down.

Even changes that are not feature additions but intended solely to
improve performance may have corner cases where they are losses rather
than wins. I think "*never* have a regression in performance" is not
only pie-in-the-sky but would be a bad policy to adopt, because it
would mean for instance that we couldn't intentionally optimize common
cases at the expense of uncommon ones.

However, I think everybody agrees that getting blindsided by unexpected
performance dropoffs is a bad thing. We really need to reinstitute
the sort of daily (or near-daily) performance tracking that Mark Wong
used to be doing, and extend it to cover a wider variety of test cases
than just DBT-2. As an example, I'll bet that this issue of operator
lookup speed would never have been visible at all in DBT-2.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

On Tue, 2007-11-27 at 13:54 -0500, Tom Lane wrote:

> However, I think everybody agrees that getting blindsided by unexpected
> performance dropoffs is a bad thing. We really need to reinstitute
> the sort of daily (or near-daily) performance tracking that Mark Wong
> used to be doing, and extend it to cover a wider variety of test cases
> than just DBT-2. As an example, I'll bet that this issue of operator
> lookup speed would never have been visible at all in DBT-2.

Yeh, we need multiple large benchmarks run on a regular basis.

My understanding is the community has two 8-core servers to run
benchmarks on, but I'd quite like to have some details on where these
are at. One is likely to be running RHEL, one Solaris.

We also need performance regression tests, which is a slightly different
thing even if they do sound similar.

--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com


---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend

On Tue, 2007-11-27 at 13:32 -0500, Andrew Dunstan wrote:

> We also need to talk about what would be a good set of tests to run.

I think we should develop a series of performance regression tests that
can be run as an option on the buildfarm. We'd want a separate page for
that with graphs etc, as you suggest.

My vision for that is a set of tests that test very specific aspects of
code, much the same way as the regression tests attempt feature
coverage. Examples would be
- 10000 INSERTs
- 10000 INSERTs using multi-VALUEs clauses
- 100000 rows inserted by COPY
- 100000 rows inserted by CTAS

We would need a way to compare results between releases, so we can see
which aspects have regressed/improved, just as we have with the
buildfarm. That will also be food for release notes, where we can
mention all actions that are >5% faster, or anything we must regrettably
report as being slower.

Sounds like it's waiting on somebody to make the first move, so maybe I
should do that, then let everybody else chip into the framework.

Should we do this as part of core, or as a separate pgfoundry project?

--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com


---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly

On Tue, 2007-11-27 at 12:36 -0800, Joshua D. Drake wrote:

> The RHEL one as I know it, is the MyYearbook donated one. We are
> currently unaware of the status of that machine except to say it is
> currently running Gentoo.
>
> I don't know the status of the Solaris machine except that I think we
> had IO issues with it.

Might I enquire who has these machines, so I can ask them how can I get
access to them? Are they really Community systems? In what sense?

--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com


---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

On Tue, 2007-11-27 at 15:33 -0300, Alvaro Herrera wrote:
> Joshua D. Drake wrote:
> I agree
> > that we should *never* have a regression in performance from release
> > to release, which is what I believe has inspired this thread.
>
> Hmm. I have developed several features that have driven performance
> down.

I think performance reductions as a result of additional functionality
are acceptable, but we should aim to minimise them.

It's the small things that crop up along the way that must be fixed,
with the same vigour we fix other bugs.

--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com


---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly