SEO

Hi -

Last night I had a problem with a Solaris 9 server. A 67 Gb slice (on a 73 Gb
RAID-1 disk) failed and the slice was taken over by a hot spare. This
morning, metastat was showing the following :

# metastat d3

d3: Mirror
Submirror 0: d33
State: Okay
Submirror 1: d43
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 141161948 blocks (67 GB)

d33: Submirror of d3
State: Okay
Hot spare pool: hsp000
Size: 141161948 blocks (67 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c0t2d0s3 0 No Okay Yes c0t4d0s7

d43: Submirror of d3
State: Okay
Hot spare pool: hsp000
Size: 141161948 blocks (67 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c0t3d0s3 0 No Okay Yes

Device Relocation Information:
Device Reloc Device ID
c0t4d0 Yes id1,sd@w5005076719c8d3c6
c0t3d0 Yes id1,sd@w0010b9ff130f7a12

# metastat hsp000
hsp000: 7 hot spares
Device Status Length Reloc
c0t4d0s0 Available 2124024 blocks Yes
c0t4d0s1 Available 2124024 blocks Yes
c0t4d0s3 Available 20995464 blocks Yes
c0t4d0s4 Available 20995464 blocks Yes
c0t4d0s5 Available 48260352 blocks Yes
c0t4d0s6 Available 48260352 blocks Yes
c0t4d0s7 In use 143967768 blocks Yes

Device Relocation Information:
Device Reloc Device ID
c0t4d0 Yes id1,sd@w5005076719c8d3c6

However, the disk seems to be OK (less than a year old). No bad blocks and
the others slices on it are just fine. So before physically replacing it, I'd
like to put back the submirror on it and see how it goes. If it fails again
I'll replace it.

I'm just not 100% sure how to enable again the mirror. It would be great if
someone could confirm if that's the right way to do it:

# metadetach d3 d33 (detaching the failed submirror)
# metaclear d33 (clearing it)
# metainit d33 1 1 c0t2d0s3 -h hsp000 (creating the submirror and allocating
again the hot spare pool)
# metattach d3 d33 (attaching and synchronizing the submirror)

I'm not sure about the third step. In Sun's documentation, they suggest a
command like "metainit d33 2 1 c0t2d0s3 1 c0t3d0s3".

I'm concerned about the risk of loosing the data on the working submirror
(c0t3d0s3)...

After enabling again the submirror, the hot spare should be automatically
released, right ?

Any help would be appreciated...

Thanks,

Georges

--
Georges Tomazi - gt@diapason.com

In article <0001HW.BE22C8EA00096080F03055B0@news.diapason.com >,
Georges Tomazi <gt@diapason.com> writes:
>
> Hi -
>
> Last night I had a problem with a Solaris 9 server. A 67 Gb slice (on a 73 Gb
> RAID-1 disk) failed and the slice was taken over by a hot spare. This
> morning, metastat was showing the following :
>
> # metastat d3
....
> d33: Submirror of d3
....
> Device Start Block Dbase State Reloc Hot Spare
> c0t2d0s3 0 No Okay Yes c0t4d0s7
....

> However, the disk seems to be OK (less than a year old). No bad blocks and
> the others slices on it are just fine.

So why did disksuite fail it? It must have had some reason for doing so
that is likely to be logged somewhere. Disksuite is pretty aggressive
at failing devices (it dosn't allow many errors before chucking it out
completely) but I've always seen at least one error that explains it.

I usuall (in the cases where the disk is responsive at all) do a
format/analyze/read/repair and try replacing it just to see if it was a
single bad block. Sometimes works.

> So before physically replacing it, I'd
> like to put back the submirror on it and see how it goes. If it fails again
> I'll replace it.
>
> I'm just not 100% sure how to enable again the mirror. It would be great if
> someone could confirm if that's the right way to do it:
>
> # metadetach d3 d33 (detaching the failed submirror)
> # metaclear d33 (clearing it)
> # metainit d33 1 1 c0t2d0s3 -h hsp000 (creating the submirror and allocating
> again the hot spare pool)
> # metattach d3 d33 (attaching and synchronizing the submirror)

Seems complicated. How about just

metareplace -e d3 c0t2d0s3

> I'm not sure about the third step. In Sun's documentation, they suggest a
> command like "metainit d33 2 1 c0t2d0s3 1 c0t3d0s3".

That's a concatenation. Don't think you want that...

> I'm concerned about the risk of loosing the data on the working submirror
> (c0t3d0s3)...
>
> After enabling again the submirror, the hot spare should be automatically
> released, right ?

If the metareplace succeeds. And if the metareplace fails, the hot
spare should still be in place.

--
-Peter Tribble
MRC Rosalind Franklin Centre for Genomics Research
http://www.rfcgr.mrc.ac.uk/~ptribble/ - http://ptribble.blogspot.com/

Peter C. Tribble wrote:
> In article <0001HW.BE22C8EA00096080F03055B0@news.diapason.com >,
> Georges Tomazi <gt@diapason.com> writes:
>
>>Hi -
>>
>>Last night I had a problem with a Solaris 9 server. A 67 Gb slice (on a 73 Gb
>>RAID-1 disk) failed and the slice was taken over by a hot spare. This
>>morning, metastat was showing the following :
>>
>># metastat d3
>
> ...
>
>>d33: Submirror of d3
>
> ...
>
>> Device Start Block Dbase State Reloc Hot Spare
>> c0t2d0s3 0 No Okay Yes c0t4d0s7
>
> ...
>
>
>>However, the disk seems to be OK (less than a year old). No bad blocks and
>>the others slices on it are just fine.
>
>
> So why did disksuite fail it? It must have had some reason for doing so
> that is likely to be logged somewhere. Disksuite is pretty aggressive
> at failing devices (it dosn't allow many errors before chucking it out
> completely) but I've always seen at least one error that explains it.
>
> I usuall (in the cases where the disk is responsive at all) do a
> format/analyze/read/repair and try replacing it just to see if it was a
> single bad block. Sometimes works.
>
>
>>So before physically replacing it, I'd
>>like to put back the submirror on it and see how it goes. If it fails again
>>I'll replace it.
>>
>>I'm just not 100% sure how to enable again the mirror. It would be great if
>>someone could confirm if that's the right way to do it:
>>
>># metadetach d3 d33 (detaching the failed submirror)
>># metaclear d33 (clearing it)
>># metainit d33 1 1 c0t2d0s3 -h hsp000 (creating the submirror and allocating
>>again the hot spare pool)
>># metattach d3 d33 (attaching and synchronizing the submirror)
>
>
> Seems complicated. How about just
>
> metareplace -e d3 c0t2d0s3
>
>
>>I'm not sure about the third step. In Sun's documentation, they suggest a
>>command like "metainit d33 2 1 c0t2d0s3 1 c0t3d0s3".
>
>
> That's a concatenation. Don't think you want that...
>
>
>>I'm concerned about the risk of loosing the data on the working submirror
>>(c0t3d0s3)...
>>
>>After enabling again the submirror, the hot spare should be automatically
>>released, right ?
>
>
> If the metareplace succeeds. And if the metareplace fails, the hot
> spare should still be in place.
>

It should be in the messages file.

Peter -

On Sun, 30 Jan 2005 17:54:11 +0100, Peter C. Tribble wrote
(in article <ctj3fj$f0a$1@helium.hgmp.mrc.ac.uk>):

[...]

> So why did disksuite fail it? It must have had some reason for doing so that
> is likely to be logged somewhere. Disksuite is pretty aggressive at failing
> devices (it dosn't allow many errors before chucking it out completely) but
> I've always seen at least one error that explains it.

I found that error message in the logs:

Jan 29 22:45:40 gbr2-p40 scsi: [ID 107833 kern.warning] WARNING:
/pci@1f,0/pci@1,1/scsi@2/sd@2,0 (sd2):
Jan 29 22:45:40 gbr2-p40 SCSI transport failed: reason 'reset':
retrying command
Jan 29 22:45:48 gbr2-p40 md_stripe: [ID 641072 kern.warning] WARNING: md:
d33: read error on /dev/dsk/c0t2d0s3
Jan 29 22:45:48 gbr2-p40 md_mirror: [ID 842313 kern.info] NOTICE: md: d33:
B_FAILFAST I/O retry
Jan 29 22:46:00 gbr2-p40 md_stripe: [ID 641072 kern.warning] WARNING: md:
d33: read error on /dev/dsk/c0t2d0s3
Jan 29 22:46:00 gbr2-p40 md_mirror: [ID 104909 kern.warning] WARNING: md:
d33: /dev/dsk/c0t2d0s3 needs maintenance
Jan 29 22:46:00 gbr2-p40 md_stripe: [ID 241980 kern.notice] NOTICE: md: d33:
hotspared device /dev/dsk/c0t2d0s3 with /dev/dsk/c0t4d0s7

> I usuall (in the cases where the disk is responsive at all) do a
> format/analyze/read/repair and try replacing it just to see if it was a
> single bad block. Sometimes works.

I checked the grown defects list and it's still empty. The disk is a 73 Gb
Maxtor Atlas 10K IV bought in May 2004.

defect> prim
Extracting primary defect list...Extraction complete.
Defect List has a total of 684 defects.

defect> g
Extracting grown defects list...Extraction complete.
Defect List has a total of 0 defects.

[...]

> Seems complicated. How about just
>
> metareplace -e d3 c0t2d0s3

I tried and it worked. Thanks a lot ! Much simpler and easier than what I was
going to do.

[...]

> That's a concatenation. Don't think you want that...

Definitely not ;-)

[...]

> If the metareplace succeeds. And if the metareplace fails, the hot spare
> should still be in place.

You're right. When the metareplace started to resync the failed slice, the
hot spare switched back to "available".

So what do you think now ? Do you believe that a LVM software failure is
something possible or the drive is definitely dying ? Is it worth breaking
the mirror to reformat the disk and recreate the mirror ?

Thx again,

Georges

--
Georges Tomazi - gt@diapason.com

In article <0001HW.BE232895001FCC77F04885B0@news.diapason.com >,
Georges Tomazi <gt@diapason.com> writes:
> On Sun, 30 Jan 2005 17:54:11 +0100, Peter C. Tribble wrote
> (in article <ctj3fj$f0a$1@helium.hgmp.mrc.ac.uk>):
>
>> So why did disksuite fail it?
> error message in the logs:
>
> Jan 29 22:45:40 gbr2-p40 scsi: [ID 107833 kern.warning] WARNING:
> /pci@1f,0/pci@1,1/scsi@2/sd@2,0 (sd2):
> Jan 29 22:45:40 gbr2-p40 SCSI transport failed: reason 'reset':
> retrying command
> Jan 29 22:45:48 gbr2-p40 md_stripe: [ID 641072 kern.warning] WARNING: md:
> d33: read error on /dev/dsk/c0t2d0s3
> Jan 29 22:45:48 gbr2-p40 md_mirror: [ID 842313 kern.info] NOTICE: md: d33:
> B_FAILFAST I/O retry
> Jan 29 22:46:00 gbr2-p40 md_stripe: [ID 641072 kern.warning] WARNING: md:
> d33: read error on /dev/dsk/c0t2d0s3

Looks more like a SCSI glitch rather than a disk error as such
(although then the question is - what caused the glitch? In the absence
of any other errors it's not obvious. Does iostat -E give any clues?)

>> Seems complicated. How about just
>>
>> metareplace -e d3 c0t2d0s3
>
> I tried and it worked. Thanks a lot ! Much simpler and easier than what I was
> going to do.

Good!

> So what do you think now ? Do you believe that a LVM software failure is
> something possible or the drive is definitely dying ? Is it worth breaking
> the mirror to reformat the disk and recreate the mirror ?

As I said, I would always do a format/analyze/read test.

(The resync only does writes...so isn't a full test.)

--
-Peter Tribble
MRC Rosalind Franklin Centre for Genomics Research
http://www.rfcgr.mrc.ac.uk/~ptribble/ - http://ptribble.blogspot.com/

Peter -

On Mon, 31 Jan 2005 09:57:23 +0100, Peter C. Tribble wrote
(in article <ctkrtj$56v$1@helium.hgmp.mrc.ac.uk>):

[...]

> Looks more like a SCSI glitch rather than a disk error as such (although then

> the question is - what caused the glitch? In the absence of any other errors
> it's not obvious. Does iostat -E give any clues?)

I have a script running a stripped iostat -nE every morning at six and I did
have a few alerts but none but the last were bad enough to switch to the hot
spare.

That's the full history (s/w h/w trn tot ):

0 0 0 0 c0t2d0 (brand new disk installed on May 26)
0 1 0 1 c0t2d0 (first error on Sept 25)
5 1 0 6 c0t2d0 (Oct 27)
5 3 0 8 c0t2d0 (Jan 15)
5 6 0 11 c0t2d0 (Jan 20)
5 9 0 14 c0t2d0 (Jan 29)
5 23 6 34 c0t2d0 (switch to hot spare on Jan 30)
0 0 0 0 c0t2d0 (back to normal on Jan 31 after metareplace)

Note that I do have from time to time, on all disks, both internal and
external) software errors (5 above). I tought of a bug in LVM.

[...]

> As I said, I would always do a format/analyze/read test.
>
> (The resync only does writes...so isn't a full test.)

OK, you meant a non destructive test. I'll do that.

Thanks !

Georges

--
Georges Tomazi - gt@diapason.com