SEO

I am looking for some information about the
speed of an IBM ESS 2105-800 (shark) disk subsystem,
connected to an IBM pSeries (AIX 5.2-ML2)
with 2 Gigabit Fibre Channel (SAN Switch 2109-F32).

I tested with:

time dd if=/dev/$DEVICE of=/dev/null bs=128k count=10000

a)
DEVICE=hdisk0 # internal SCSI-disk (10000rpm):
30 MB/s

b)
DEVICE=hdisk0 # internal SCSI-disk (15000rpm):
45 MB/s

c)
DEVICE=vpath0 # ESS-LUN (SSA-disks, 15000rpm, RAID5 over an array/8-Pack):
22 MB/s

All values seem to be independent from blocksize (4k to 1024k)

Is the ESS really slower than a single SCSI-disk?

Is my test procedure wrong?

Thanks in advance.

In article <c78me8$46v$1@online.de>, Dieter Mosbach <news.eue+402@esquad.de> wrote:
> I am looking for some information about the
> speed of an IBM ESS 2105-800 (shark) disk subsystem,
> connected to an IBM pSeries (AIX 5.2-ML2)
> with 2 Gigabit Fibre Channel (SAN Switch 2109-F32).

I tested with a p660 model 6H1 (6 668 MHz RSIV processors) and a FC6227
adapter (1 Gb/sec rebadged Emulex LP7000 adapter) running AIX 5.2 with
latest patches and latest ibm2105 + sdd stuff and we have a 2105-F20
Shark, a few days ago.

I got 115 MB/sec to a jfs2 filesystem mounted on the ESS, and about 40
MB/sec to a jfs filesystem mounted on the ESS.

115 MB/sec appears to be the adapter maxing out (it's about 90% of
maximum), but achievable only with use of jfs2 filesystems.

Too bad we only have 1 Gb/sec FC adapters on the ESS side and host side.
But at least the ESS writes out to its huge cache so it can
essentially accept data as fast as the FC adapters can deliver data.

-Dan

Dan Foster wrote:

> In article <c78me8$46v$1@online.de>, Dieter Mosbach <news.eue+402@esquad.de> wrote:
>
>>I am looking for some information about the
>>speed of an IBM ESS 2105-800 (shark) disk subsystem,
>>connected to an IBM pSeries (AIX 5.2-ML2)
>>with 2 Gigabit Fibre Channel (SAN Switch 2109-F32).
>
>
> I tested with a p660 model 6H1 (6 668 MHz RSIV processors) and a FC6227
> adapter (1 Gb/sec rebadged Emulex LP7000 adapter) running AIX 5.2 with
> latest patches and latest ibm2105 + sdd stuff and we have a 2105-F20
> Shark, a few days ago.
>
> I got 115 MB/sec to a jfs2 filesystem mounted on the ESS, and about 40
> MB/sec to a jfs filesystem mounted on the ESS.

How did you test it?
With dd?
Was the filesystem on one vpath/lun or on many luns?
How big was the file?
Is there an effect caused by the AIX-cache/ESS-cache?

-Dieter

In article <c78r9i$g4o$1@online.de>, Dieter Mosbach <news.eue+402@esquad.de> wrote:
>
> How did you test it? With dd?

Yes. Same way you did. I tried a block size of 64K and 128K since that's
often the magical point (64K for me with SSA and SCSI) for optimized dd
writes due to RAID-5 stripe size, but in this case, I saw the same thing
you did -- did not see the dd block size really affect the observed
performance numbers.

> Was the filesystem on one vpath/lun or on many luns?

The filesystem was in an ESS volume group on a single vpath with a
single LUN, and was a 105.24 GB volume on its own disk group (8-pack
with 18.2 GB disks).

> How big was the file?

640 MB.

> Is there an effect caused by the AIX-cache/ESS-cache?

Yes and no.

I don't think there was any AIX buffer cache effect because I did not
run the test on the same file/filesystem more than once.

I don't remember all of the subtleties of the ESS write acknowledgement
and cache destaging process, but if I recall, it files all incoming data
into the cache then immediately acknowledges it without waiting for it
to go to disk.

It can afford to do that because it's got a large battery backing the
cache and enough emergency battery power to run full operation for five
minutes with the model F20 even when external power is completely lost.

So with the ESS end, it can basically accept data as fast as you can
deliver to it through its network connections. This is how its cache
really helps.

We had a maxed out FC adapter, so it appears the bottleneck was using 1
Gb/sec rather than 2 Gb/sec (it was an older model so it had older
adapters).

I imagine that we could possibly max out 2 Gb/sec FC adapters with our 6
processor 6H1 system, but unfortunately, don't have that on the host or
the ESS.

Given what I saw with my basic testing, I can only conclude that the jfs
filesystem was my main bottleneck for getting full performance out of
the ESS with that kind of simulated test; when I used jfs2, maxed out
the FC adapter which suggests it's capable of doing even better if I had
newer adapters.

So I had two bottlenecks: primary was jfs... then once I used jfs2,
bottleneck moved to the FC adapter.

Can you retry your test with a jfs2 fs when the ESS vpath is quiet?
That would give you a better idea of your raw available performance.

-Dan

Dieter Mosbach wrote:
> I am looking for some information about the
> speed of an IBM ESS 2105-800 (shark) disk subsystem,
> connected to an IBM pSeries (AIX 5.2-ML2)
> with 2 Gigabit Fibre Channel (SAN Switch 2109-F32).
>
> I tested with:
>
> time dd if=/dev/$DEVICE of=/dev/null bs=128k count=10000
>
> a)
> DEVICE=hdisk0 # internal SCSI-disk (10000rpm):
> 30 MB/s
>
> b)
> DEVICE=hdisk0 # internal SCSI-disk (15000rpm):
> 45 MB/s
>
> c)
> DEVICE=vpath0 # ESS-LUN (SSA-disks, 15000rpm, RAID5 over an array/8-Pack):
> 22 MB/s
>
> All values seem to be independent from blocksize (4k to 1024k)
>
> Is the ESS really slower than a single SCSI-disk?
>
> Is my test procedure wrong?
>
> Thanks in advance.

A couple of things.

When I tried to replicate this on a machine with a single disk I got a
read error trying to dd hdisk0 or rhdisk0. Perhaps neither of your
hdisk0 were rootvg?

Going past that, making a large lv on the disk, I verified that reading
from the cooked device gives about half the thruput of the raw disk at
what I remember as the maximum block size of a scsi device, 1024k.
Reading from the raw device, there is a huge difference between the
minimum and maximum block size.

Reading from the cooked device, there is a very small difference between
minimum and maximum sizes.

The only other difference in method is that I leave a window with iostat
1 running while I change around the dd in another window.

On a 43p-150 with a 30 gig drive the numbers are:

dd if=/dev/rtest of=/dev/null bs=1024k 18 M/s
512 1.3 M/s
dd if=/dev/test of=/dev/null bs=1024k 8.6 M/s
512 5.3 M/s

My other machine with two drives is running another test that won't be
done for hours, but I will try and see if I can read either of the hdisk
on that tomorrow, but I would be curious what you see on your disks
looking at them this way.

Perhaps next week I will have my old SSA up and running to look at some
of the raid issues discussed.

Cheers.

Timothy J. Bogart wrote:
> Dieter Mosbach wrote:
>
>> I am looking for some information about the
>> speed of an IBM ESS 2105-800 (shark) disk subsystem,
>> connected to an IBM pSeries (AIX 5.2-ML2)
>> with 2 Gigabit Fibre Channel (SAN Switch 2109-F32).
>>
>> I tested with:
>>
>> time dd if=/dev/$DEVICE of=/dev/null bs=128k count=10000
>>
>> a)
>> DEVICE=hdisk0 # internal SCSI-disk (10000rpm):
>> 30 MB/s
>>
>> b)
>> DEVICE=hdisk0 # internal SCSI-disk (15000rpm):
>> 45 MB/s
>>
>> c)
>> DEVICE=vpath0 # ESS-LUN (SSA-disks, 15000rpm, RAID5 over an
>> array/8-Pack):
>> 22 MB/s
>>
>> All values seem to be independent from blocksize (4k to 1024k)
>>
>> Is the ESS really slower than a single SCSI-disk?
>>
>> Is my test procedure wrong?
>>
>> Thanks in advance.
>
>
> A couple of things.
>
> When I tried to replicate this on a machine with a single disk I got a
> read error trying to dd hdisk0 or rhdisk0. Perhaps neither of your
> hdisk0 were rootvg?
>
> Going past that, making a large lv on the disk, I verified that reading
> from the cooked device gives about half the thruput of the raw disk at
> what I remember as the maximum block size of a scsi device, 1024k.
> Reading from the raw device, there is a huge difference between the
> minimum and maximum block size.
>
> Reading from the cooked device, there is a very small difference between
> minimum and maximum sizes.
>
> The only other difference in method is that I leave a window with iostat
> 1 running while I change around the dd in another window.
>
> On a 43p-150 with a 30 gig drive the numbers are:
>
> dd if=/dev/rtest of=/dev/null bs=1024k 18 M/s
> 512 1.3 M/s
> dd if=/dev/test of=/dev/null bs=1024k 8.6 M/s
> 512 5.3 M/s
>
> My other machine with two drives is running another test that won't be
> done for hours, but I will try and see if I can read either of the hdisk
> on that tomorrow, but I would be curious what you see on your disks
> looking at them this way.
>
> Perhaps next week I will have my old SSA up and running to look at some
> of the raid issues discussed.
>
> Cheers.
>


OK, on a 44p-170, I did not get the error reading the hdisks (?), and
the results were largely the same. The numbers are:

dd if=/dev/rhdisk0 of=/dev/null bs=1024k 33 M/s
512 2.4 M/s
dd if=/dev/hdisk0 of=/dev/null bs=1024k 22 M/s
512 8.1 M/s

The only difference here was I happened to already have a topas window
up to watch the results. 8-) Slightly different ratios, but still
marked differences between the raw and cooked, and across block sizes.

Cheers.

Timothy J. Bogart wrote:


>
> When I tried to replicate this on a machine with a single disk I got a
> read error trying to dd hdisk0 or rhdisk0. Perhaps neither of your
> hdisk0 were rootvg?
>
> Going past that, making a large lv on the disk, I verified that reading
> from the cooked device gives about half the thruput of the raw disk at
> what I remember as the maximum block size of a scsi device, 1024k.
> Reading from the raw device, there is a huge difference between the
> minimum and maximum block size.
>
> Reading from the cooked device, there is a very small difference between
> minimum and maximum sizes.
>
> The only other difference in method is that I leave a window with iostat
> 1 running while I change around the dd in another window.
>
> On a 43p-150 with a 30 gig drive the numbers are:
>
> dd if=/dev/rtest of=/dev/null bs=1024k 18 M/s
> 512 1.3 M/s
> dd if=/dev/test of=/dev/null bs=1024k 8.6 M/s
> 512 5.3 M/s
>
> My other machine with two drives is running another test that won't be
> done for hours, but I will try and see if I can read either of the hdisk
> on that tomorrow, but I would be curious what you see on your disks
> looking at them this way.

Ok, I did some tests on raw devices:


I) Filesize: 2000MB

a)
DEVICE=rhdisk0 # internal SCSI-disk (10000rpm):
5 MB/s @ bs=1k
66 MB/s @ bs=1024k

b)
DEVICE=rhdisk0 # internal SCSI-disk (15000rpm):
9 MB/s @ bs=1k
74 MB/s @ bs=1024k

c)
DEVICE=rvpath0 # ESS-LUN (SSA-disks, 15000rpm, RAID5 over an array/8-Pack):
3 MB/s @ bs=1k
117 MB/s @ bs=1024k


II) Filesize: 8000MB

a)
DEVICE=rhdisk0 # internal SCSI-disk (10000rpm):
5 MB/s @ bs=1k
65 MB/s @ bs=1024k

b)
DEVICE=rhdisk0 # internal SCSI-disk (15000rpm):
9 MB/s @ bs=1k
70 MB/s @ bs=1024k

c)
DEVICE=rvpath0 # ESS-LUN (SSA-disks, 15000rpm, RAID5 over an array/8-Pack):
3 MB/s @ bs=1k
38 MB/s @ bs=1024k


The very good value of I.c could be the result of the shark-cache?

I will do more tests on logical volumes and on filesystem (jfs2)

-Dieter