SEO

Hello.
We have made some performance tests with DRBD and Postgresql 8.2.3. We
have two identical servers in a cluster (Dell 2950) with a partition of
100 GB managed by DRBD: once we checked Postgres keeping his data folder
in a local partition, the second time we moved the data folder in the
shared partition. The two servers are connected point to point using a
cross cable to reduce their latency.
The partition is mounted with the option noatime in order to not update
the inode access time in case of read access.
We used pgbench for the testings, creating a dabase of about 3GB with a
scale of 200. After we perfomed 10 tests for each configuration,
simulating the usage of 100 clients with 500 transactions each.

DRBD configuration:
--------------------------------------------------------------------------------------------
resource drbd0 {

protocol C;
incon-degr-cmd "halt -f";

on db-node1 {
device /dev/drbd0;
disk /dev/sda2;
address 10.0.0.201:7788;
meta-disk internal;
}

on db-node2 {
device /dev/drbd0;
disk /dev/sda2;
address 10.0.0.202:7788;
meta-disk internal;
}
syncer {
rate 700000K;
}
}
--------------------------------------------------------------------------------------------

Pgbench

--------------------------------------------------------------------------------------------
pgbench -i pgbench -s 200
pgbench -c 100 -t 500 pgbench
--------------------------------------------------------------------------------------------

The results were that the TPS (transaction per second) with Postgres
running in the local partition is almost double than the one with the DRDB:

Postgres in shared DRBD partition: 60.863324 TPS
Postgres in local partition: 122.016138 TPS

Obviously, working with the database in DRBD, we had two writes instead
of only one but we are a bit disappointed about the low results. We
would like to know if there is any way to improve the performance in
order to have a 3/4 rate instead of the 1/2 one.

We would really appreciate it if you could give us some feedback.

Thank you in advance,
Maila Fatticcioni

--
__________________________________________________ ____________
Maila Fatticcioni
__________________________________________________ ____________
Mediterranean Broadband Infrastructure s.r.l.
ITALY
__________________________________________________ ____________


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFG4Rvoi2q3wPb3FcMRAhx6AKDi9cU6/mdSDv+KnwG7k1SmujxiSgCguCsR
XI9axOcrOf0gQRLaIDL459Q=
=6C9s
-----END PGP SIGNATURE-----

Maila Fatticcioni wrote:
> Hello.
> We have made some performance tests with DRBD and Postgresql 8.2.3. We
> have two identical servers in a cluster (Dell 2950) with a partition of
> 100 GB managed by DRBD: once we checked Postgres keeping his data folder
> in a local partition, the second time we moved the data folder in the
> shared partition. The two servers are connected point to point using a
> cross cable to reduce their latency.
> The partition is mounted with the option noatime in order to not update
> the inode access time in case of read access.
> We used pgbench for the testings, creating a dabase of about 3GB with a
> scale of 200. After we perfomed 10 tests for each configuration,
> simulating the usage of 100 clients with 500 transactions each.
>
> DRBD configuration:
> --------------------------------------------------------------------------------------------
> resource drbd0 {
>
> protocol C;
> incon-degr-cmd "halt -f";
>
> on db-node1 {
> device /dev/drbd0;
> disk /dev/sda2;
> address 10.0.0.201:7788;
> meta-disk internal;
> }
>
> on db-node2 {
> device /dev/drbd0;
> disk /dev/sda2;
> address 10.0.0.202:7788;
> meta-disk internal;
> }
> syncer {
> rate 700000K;
> }
> }
> --------------------------------------------------------------------------------------------
>
> Pgbench
>
> --------------------------------------------------------------------------------------------
> pgbench -i pgbench -s 200
> pgbench -c 100 -t 500 pgbench
> --------------------------------------------------------------------------------------------
>
> The results were that the TPS (transaction per second) with Postgres
> running in the local partition is almost double than the one with the DRDB:
>
> Postgres in shared DRBD partition: 60.863324 TPS
> Postgres in local partition: 122.016138 TPS
>
> Obviously, working with the database in DRBD, we had two writes instead
> of only one but we are a bit disappointed about the low results. We
> would like to know if there is any way to improve the performance in
> order to have a 3/4 rate instead of the 1/2 one.

You seem to be limited by the speed you can fsync the WAL to the DRBD
device. Using a RAID controller with a battery-backed up cache in both
servers should help, with and without DRBD. You might find that the
difference between local and shared partition just gets bigger, but you
should get better numbers.

In 8.3, you could turn synchronous_commit=off, if you can accept the
loss of recently committed transactions in case of a crash.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

On 9/7/07, Maila Fatticcioni <mfatticcioni@mbigroup.it> wrote:
> Obviously, working with the database in DRBD, we had two writes instead
> of only one but we are a bit disappointed about the low results. We
> would like to know if there is any way to improve the performance in
> order to have a 3/4 rate instead of the 1/2 one.

Have you considered warm standby PITR? It achieves essentially the
same thing with very little overhead on the master. The only downside
relative to DRDB is you have to think about the small gap between WAL
file rotations. From what I understand, there is some new stuff
(check out skype skytools) that may help minimize this problem.

merlin

---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend

On Fri, 2007-09-07 at 11:37 +0200, Maila Fatticcioni wrote:

> protocol C;

Try protocol B instead.

--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com


---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?

http://archives.postgresql.org

Thank you very much for your ideas. I've tried to change the protocol
from C to B and I got an increase in the number of TPS: 64.555763.

Now I would like to follow the advice of Mr. Bernd Helmle and change the
value of snd-bufsize.

The servers are cross connected with a common 100 Mbit/sec Ethernet so I
think they have a bandwidth around 80 Mbit/sec (even if I haven't yet
done any test on it). A rate of 70Mb seems reasonable to me.

The two servers are in two different racks (next to each other) and they
have two power supplies connected to two different sets of UPS.

Unfortunately we cannot accept a loss of recently committed transactions
so we cannot put the synchronous_commit to off.

Regards,
Maila Fatticcioni

Simon Riggs wrote:
> On Fri, 2007-09-07 at 11:37 +0200, Maila Fatticcioni wrote:
>
>> protocol C;
>
> Try protocol B instead.
>

--
__________________________________________________ ____________
Maila Fatticcioni
__________________________________________________ ____________
Mediterranean Broadband Infrastructure s.r.l.
ITALY
__________________________________________________ ____________


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFG5qqRi2q3wPb3FcMRAgoTAKDjOICYvjdC5/tE+XDmSfJ3nYiwxgCeIV32
FYDHb+q3raS1LaF+L8dWVJc=
=2WaD
-----END PGP SIGNATURE-----

On Tue, Sep 11, 2007 at 04:47:40PM +0200, Maila Fatticcioni wrote:
> The servers are cross connected with a common 100 Mbit/sec Ethernet so I
> think they have a bandwidth around 80 Mbit/sec (even if I haven't yet
> done any test on it). A rate of 70Mb seems reasonable to me.

Umm, seriously? Unless that was a typo, you should consider very seriously to
go to gigabit; it's cheap these days, and should provide you with a very
decent speed boost if the network bandwidth is the bottleneck.

/* Steinar */
--
Homepage: http://www.sesse.net/

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

Simon Riggs schrieb:
> On Fri, 2007-09-07 at 11:37 +0200, Maila Fatticcioni wrote:
>
>> protocol C;
>
> Try protocol B instead.
>

Sure? I've always heard that there has yet to be a case found, where B
is better than C. We use DRBD with protocol C, and are quite happy with it.

---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend

On Tue, Sep 11, 2007 at 04:57:24PM +0200, Steinar H. Gunderson wrote:
> On Tue, Sep 11, 2007 at 04:47:40PM +0200, Maila Fatticcioni wrote:
> > The servers are cross connected with a common 100 Mbit/sec Ethernet so I
> > think they have a bandwidth around 80 Mbit/sec (even if I haven't yet
> > done any test on it). A rate of 70Mb seems reasonable to me.
>
> Umm, seriously? Unless that was a typo, you should consider very seriously to
> go to gigabit; it's cheap these days, and should provide you with a very
> decent speed boost if the network bandwidth is the bottleneck.

Actually, in this case, I suspect that latency will be far more critical
than overall bandwidth. I don't know if it's inherent to Gig-E, but my
limited experience has been that Gig-E has higher latency than 100mb.
--
Decibel!, aka Jim Nasby decibel@decibel.org
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.3 (FreeBSD)

iD8DBQFG5wb5dO30qud8SkgRAutVAKCMaMXLphqscoqBnuDAP5 9naWQNxgCeMBWk
Vw05+ozjqDgNS2zgAMtCNlo=
=80Jp
-----END PGP SIGNATURE-----

Hi,

Decibel! wrote:
> Actually, in this case, I suspect that latency will be far more critical
> than overall bandwidth. I don't know if it's inherent to Gig-E, but my
> limited experience has been that Gig-E has higher latency than 100mb.

I've been looking for some benchmarks, but it's rather hard to find. It
looks like people are much more concerned about throughput ?!?

However, I'd like to share some of the sites I've found, especially
regarding Fast Ethernet vs. Gigabit Ethernet:

- Ashford Computer Consulting Service benchmarked five different
gigabit ethernet adapters [1], back in 2004. For most cards they
measured between ca. 100 - 150 microseconds for a UDP round trip of a
token, a so called hot potato benchmark. Unfortunately they didn't
compare with Fast Ethernet.

- The NetPIPE project has some of it's measurements at the very bottom
of it's website [2]. Mostly for high speed and low latency links. Again,
Fast Ethernet is missing. The diagram tells the following latencies (in
microseconds):

75 10 Gigabit Ethernet
62 Gigabit Ethernet
8 Myrinet
7.5 Infini Band
4.7 Atoll
4.2 SCI

I've no explanation for the significantly better measure for gigabit
ethernet compared with the above benchmark. From their description I'm
concluding that they also measured a round-trip, but not via UDP.

The bad value for 10 Gigabit Ethernet is due to a poor Intel adapter,
which also has poor throughput. They claim that newer adapters are better.

- Finally, I've found a latency comparison between Fast vs Gigabit
Ethernet, here [3]. Figure 6, in the second third of the page shows a
NetPIPE latency benchmark between Ethernet, Fast Ethernet and Gigabit
Ethernet (additionally ATM and FDDI). It looks like Gigabit Ethernet
features slightly better latency.

From these findings I'm concluding, that commodity Ethernet hardware
has quite similar latencies, no matter if you are using Fast, Gigabit or
10 Gigabit Ethernet. If you really want to have a low latency
interconnect, you need to pay the extra bucks for specialized, low
latency networking hardware (which may still be based on 10GE, see
Myrinet's 10GE adapter).

If you know other resources, I'd be curious to know.

Regards

Markus

[1]: Ashford Computer Consulting Service, GigE benchmarks:
http://www.accs.com/p_and_p/GigaBit/conclusion.html

[2]: NetPIPE website:
http://www.scl.ameslab.gov/netpipe/

[3]: Gigabit Ethernet and Low-Cost Supercomputing
http://www.scl.ameslab.gov/Publicati...it/tr5126.html

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster