I have recently setup a three disk RAID-5 system for file storage.
There are three EIDE ATA 133 disks off two controllers in the machine.
I've partitioned each disk into 40G segments and used them as a
(seemingly) 12 disk array with 8 active and 4 spares. I'm using the XFS
file system on FC5 with 2.6.17 kernel with 64k RAID chunks. I've
hdparm'd the disks so throughput is on the order of 57MB/Sec/disk when
tested with hdparm -t.

Write performance across my 100BT network averages around 8.3MB/sec.
Read performance across the network is absolutely blazing (too fast to
measure accurately without a lot of work). However, the rm command
takes ~10x as long on the RAID/XFS system as it does on my ext3 SATA
disk! Unlike the nearly instantaneous removal of items on my ext3
filesystem, the RAID/XFS combo seems to take a rather long time. I
haven't seen any mention of this on USENET or web pages so I'm
wondering if I have something setup wrong or this is somehow inherent
to either the maintenance of the RAID or XFS.

For instance on a 2.3G directory holding 46 directories and 483 files,
logged in locally with nothing else going on:
(RAID XFS) time rm -rf Pink\ Floyd/

real 0m26.317s
user 0m0.004s
sys 0m0.320s

(EXT3 )
time rm -rf Pink\ Floyd/

real 0m2.237s
user 0m0.004s
sys 0m0.296s

I'm hoping someone more familiar with the setup and intrinsics of these
two systems can provide either some insight as to why this is
unavoidable or some guidance on how to proceed to fix it.

Thanks....

On Wednesday 06 September 2006 03:00, justin.seiferth@gmail.com stood up and
addressed the masses in /comp.os.linux.misc/ as follows...:

> I have recently setup a three disk RAID-5 system for file storage.
> There are three EIDE ATA 133 disks off two controllers in the machine.
> I've partitioned each disk into 40G segments and used them as a
> (seemingly) 12 disk array with 8 active and 4 spares. I'm using the XFS
> file system on FC5 with 2.6.17 kernel with 64k RAID chunks. I've
> hdparm'd the disks so throughput is on the order of 57MB/Sec/disk when
> tested with hdparm -t.
>
> [...]
>
> For instance on a 2.3G directory holding 46 directories and 483 files,
> logged in locally with nothing else going on:
> (RAID XFS) time rm -rf Pink\ Floyd/
>
> real 0m26.317s
> user 0m0.004s
> sys 0m0.320s
>
> (EXT3 )
> time rm -rf Pink\ Floyd/
>
> real 0m2.237s
> user 0m0.004s
> sys 0m0.296s
>
> I'm hoping someone more familiar with the setup and intrinsics of these
> two systems can provide either some insight as to why this is
> unavoidable or some guidance on how to proceed to fix it.

Just a guess... You're using an IDE RAID, and IDE disks are controlled by
the CPU. Most likely your RAID set-up will also be one of the type
"semi-hardware RAID".

Perhaps an "rm" on such a system - depending on the driver - provides for
far more overhead?

Like I said... Just a guess... ;-)

--
With kind regards,

*Aragorn*
(registered GNU/Linux user #223157)

justin.seiferth@gmail.com wrote:
> I have recently setup a three disk RAID-5 system for file storage.
> There are three EIDE ATA 133 disks off two controllers in the machine.

There's problem number 1. For IDE RAID, you really want each disk on
it's own controller. They don't share well, so access to the two disks
that are on the same controller will happen sequentially, not in parallel.

> I've partitioned each disk into 40G segments and used them as a
> (seemingly) 12 disk array with 8 active and 4 spares.

There's problem number 2. You don't want multiple partitions on the
same drive in same array. RAID tries to access its component drives in
parallel, but that can't happen in your setup. Instead, your setup will
require multiple (time consuming) head seeks from one partition to the
next to complete one operation.

If you're serious about performance, you should run each disk from its
own controller and not put partitions from the same drive into the same
array. Either partition the disks so they each only have one partition
-or- put their partitions into different arrays (as long as those arrays
won't be accessed simultaneously). Then you'll be impressed with the
speeds you can get across Gigabit Ethernet.

Thanks for the responses so far, however rm is the ONLY operation which
seems to suffer. Reading and writing , in my understanding would be far
more likely to saturate the controllers but only rm performance seems
to suffer.

As far as how I've partitioned the disks, I did it for a reason (see
slashdot discussion on the topic,
http://linux.slashdot.org/article.pl.../05/22/1624246.

I am using IDE but I am also using SW raid, not a HW controller in the
motherboard (the motherboard doesn't have one). The processor is a 1GHZ
Duron and seems up to the task of reading/writing. I am using separate
controllers for the disks, all on the same motherboard but different
controllers for each as there are four on the board (Promise
Ultra-ATA). The problem seems unique to rm.