I'm using Solaris Volume Manager to mirror the root disks on a Solaris
9 server.
I only have 2 disks.
I have 1 small slices on each disk dedicated for the metadb's.
How many metadb's should I have on each disk?
I understand that 1 metadb on each disk is to few, but is 2 on each
enough or should I have 3? I've seen booth 2 and 3 sugested but no
real good explanation to why one is better then the other, could
someone please explain this to me?

/Regards Stefan Åhser

Stefan ?hser wrote:
> I'm using Solaris Volume Manager to mirror the root disks on a Solaris
> 9 server.
> I only have 2 disks.
> I have 1 small slices on each disk dedicated for the metadb's.
> How many metadb's should I have on each disk?
> I understand that 1 metadb on each disk is to few, but is 2 on each
> enough or should I have 3? I've seen booth 2 and 3 sugested but no
> real good explanation to why one is better then the other, could
> someone please explain this to me?
>
> /Regards Stefan Åhser

I did the same, however you need to make sure that when a disk fails:

remaining_dbs = (total_dbs / 2) + 1

i.e You still have 'over' half the dbs left after a failure in order for the system to use the single root mirror.

So one on each disk ( total 2) - one disk fails - remaining is 1 ( <= half ) - no good
two on each disk ( total 4) - one disk fails - remaining is 2 ( <= half ) - no good
.....

to fix this

I rummaged for an old HD, installed a "really" old quantum 211Mb disk and just made 2 dbs on that
to complement 4 dbs in different slices on the other 2 disks.

so

So two on each disk (total 6) - one disk fails - remaining is 4 ( > half ) OK system will come up.

The noise this old disk makes is awful, but thankfully only dbs are on it so very little hd action.

Don't take my word for it though - I haven't a disk failure yet so no real conclusion.

Rob

--
Rap it up for the common good
Let us enlist the neighbourhood
It's OK, I've overstood
This is a wordy rappinghood. OK, bye.

Tomtomclub, 1980.

Rob Shepherd <robshep@invalid.invalid> wrote:
> I did the same, however you need to make sure that when a disk fails:
>
> remaining_dbs = (total_dbs / 2) + 1
>
> i.e You still have 'over' half the dbs left after a failure in order for the system to use the single root mirror.

This is not correct. In order to continue using the disk you need to have
half or more of the metadb's available. ie, if you have two on each disk
and one disk fails, then you will have 2/4 left, and the system will stay
up and running.
In order to reboot you need to have >1/2 available. This is very easily
achieved by removing the stale/dead replicas before rebooting, so that
you now have 2/2 available, and the machine will boot succesfully.

The most critical rule with only two disks is to put exactly the same number
of metadb's on each disk. The exact number you use isn't as critical as
making sure you have the same number on each. Generally I'd suggest 3 copies
per disk, but 2 will work just as well.

Scott.

* Scott Howard wrote:
> This is not correct. In order to continue using the disk you need to have
> half or more of the metadb's available. ie, if you have two on each disk
> and one disk fails, then you will have 2/4 left, and the system will stay
> up and running.
> In order to reboot you need to have >1/2 available. This is very easily
> achieved by removing the stale/dead replicas before rebooting, so that
> you now have 2/2 available, and the machine will boot succesfully.

I think this is misleading. There are worse things than machines
which are running more-or-less happily but which will fail to boot
dismally, but not that many. Imagine the scenario: inexperienced
sysadmin called out at 3AM: machine is running but has logged scsi
errors on a system disk mirror. OK, so let's be safe as it's 3AM:
halt it, swap the disk, bring it back up. Oops: it won't boot, now
there's a major crisis.

So I'd always suggest making sure there are an odd number of disks
with metadb replicas on, and the same number per disk. A bad
alternative is to set the /etc/system thing which will let it boot
with half the replicas.

--tim

"Tim Bradshaw" <tfb@cley.com> wrote in message
news:ey37jnmxtzi.fsf@cley.com...
>* Scott Howard wrote:
>> This is not correct. In order to continue using the disk you need to
>> have
>> half or more of the metadb's available. ie, if you have two on each disk
>> and one disk fails, then you will have 2/4 left, and the system will stay
>> up and running.
>> In order to reboot you need to have >1/2 available. This is very easily
>> achieved by removing the stale/dead replicas before rebooting, so that
>> you now have 2/2 available, and the machine will boot succesfully.
>
> I think this is misleading. There are worse things than machines
> which are running more-or-less happily but which will fail to boot
> dismally, but not that many. Imagine the scenario: inexperienced
> sysadmin called out at 3AM: machine is running but has logged scsi
> errors on a system disk mirror. OK, so let's be safe as it's 3AM:
> halt it, swap the disk, bring it back up. Oops: it won't boot, now
> there's a major crisis.
>

And if you have two disks with 2 + 3 replicas and one with 3 replicas fails
your system panics. So you have statistically 50% chances that system with
failed disk stops working completely.

And you do not need to stop system to exchange dsik in most cases. And even
if you do - your system is already down and out of service, so it is
possible to spend extra 15 minutes to clean up stale replicas. It is still
far better than unexpected system panic without automatic reboot.

> So I'd always suggest making sure there are an odd number of disks
> with metadb replicas on, and the same number per disk. A bad
> alternative is to set the /etc/system thing which will let it boot
> with half the replicas.
>

Sure it is better to have odd number of disks but the main question is what
to do with only two disks. Many servers I'm working with have only two
system disks everything else is external (FC or like). In this case having
even number of replicas is IMHO better (from overall system availability
point of view).

regards

=arvi=

* kermit wrote:

> And if you have two disks with 2 + 3 replicas and one with 3 replicas fails
> your system panics. So you have statistically 50% chances that system with
> failed disk stops working completely.

Yes. I didn't suggest this configuration of course as it's insane, so
I'm not sure why you're stressing about it.

> Sure it is better to have odd number of disks but the main question is what
> to do with only two disks. Many servers I'm working with have only two
> system disks everything else is external (FC or like). In this case having
> even number of replicas is IMHO better (from overall system availability
> point of view).

Same number per disk is of course the only sane option. An odd number
of *disks* is a pretty good option though: an extra random disk is not
generally that much money. If you really can't afford that, and you
can't put some replicas on the data disks as well then the choice is
between the nasty won't reboot situation (and this is really nasty if
you are not expecting it), and the /etc/system `boot with only half
the replicas' setting. What I'd do is have the default system file
with this off, and another with it on (generated from the former by a
script), and documentation saying how to bring the system up using the
alternate system file (boot -a really).

--tim

On Fri, 10 Dec 2004 12:54:06 +0000, Rob Shepherd
<robshep@invalid.invalid> wrote:

>I did the same, however you need to make sure that when a disk fails:
>
>remaining_dbs = (total_dbs / 2) + 1
>
>i.e You still have 'over' half the dbs left after a failure in order for the system to use the single root mirror.
>
>So one on each disk ( total 2) - one disk fails - remaining is 1 ( <= half ) - no good
> two on each disk ( total 4) - one disk fails - remaining is 2 ( <= half ) - no good
>....
>
>to fix this
>
>I rummaged for an old HD, installed a "really" old quantum 211Mb disk and just made 2 dbs on that
>to complement 4 dbs in different slices on the other 2 disks.
>
>so
>
>So two on each disk (total 6) - one disk fails - remaining is 4 ( > half ) OK system will come up.
>
>The noise this old disk makes is awful, but thankfully only dbs are on it so very little hd action.
>
>Don't take my word for it though - I haven't a disk failure yet so no real conclusion.

Add "set md:mirrored_root_flag=1" to /etc/system. This enables the
system to boot without a majority of state databases (for example, one
disk fails!) I tested this, and it certainly works with Solaris 8 and
Disksuite 4.2.1.

http://www.sun.com/blueprints/1002/817-0407-10.pdf

Tim Bradshaw wrote:

> So I'd always suggest making sure there are an odd number of disks
> with metadb replicas on, and the same number per disk. A bad
> alternative is to set the /etc/system thing which will let it boot
> with half the replicas.

Sorry Tim, but this seems like an impractical advice. For instance,
in our lab, most of the Sun machines we have run with two disks
because... that's what fits inside. Starting with X1's, T1's, V100,
V120's, 280R's (loads), Netra 20's, 480's (yes, we have loads of
machines which fit more disks as well, but running another disk just to
have a metadb on it is impractical). In order to have another metadb,
we'd need to add extrnal boxes, which IMHO, just add to a number of
parts that can fail.

The way to handle problems is to monitor the disks, and if one fails,
get it replaced. Chances of both disks failing at once (without
anything else in the lab breaking down) are small. As pointed out, the
system will continue to run. If you halt it, you can still bring it up
to maintenance mode, recreate the new disk+metadb, and be up and running
in a couple of minutes.

If you have system with more disks, this issue becomes somewhat
irrelevant. Just follow the recommendations, and store the replicas
across disks (noting the problems that can arise if you store them
someplace which can be disconnected without a non-disk failure, for
instance, an exteranal scsi box).

/Marcin

* Marcin Dobrucki wrote:
> Sorry Tim, but this seems like an impractical advice. For
> instance, in our lab, most of the Sun machines we have run with two
> disks because... that's what fits inside. Starting with X1's,
> T1's, V100, V120's, 280R's (loads), Netra 20's, 480's (yes, we
> have loads of machines which fit more disks as well, but running
> another disk just to have a metadb on it is impractical). In order
> to have another metadb, we'd need to add extrnal boxes, which IMHO,
> just add to a number of parts that can fail.

Right, so I'd vote for the bad alternative.

--tim

Tim Bradshaw <tfb@cley.com> wrote:
> I think this is misleading. There are worse things than machines
> which are running more-or-less happily but which will fail to boot
> dismally, but not that many. Imagine the scenario: inexperienced
> sysadmin called out at 3AM: machine is running but has logged scsi
> errors on a system disk mirror. OK, so let's be safe as it's 3AM:
> halt it, swap the disk, bring it back up. Oops: it won't boot, now
> there's a major crisis.

It's not a major crisis. Plus in your root password when it asks
for it, run a metadb -d to delete the stale replicas, and reboot.
Costs a few minutes, but that's about all.

I wrote a fairly long description of this exact situation a while back -
I'll see if I can dig it up and post it.

Scott