We are well on our way and on the final few steps of migrating from
Oracle to ASE.

Our configuration is an E3500 running Solaris8, with dual A1000s. The
disks in each A1000 is mirrored to the other A1000 by Solstice
DiskSuite. Now I noticed that metainit actually says about
"Write-On-Write Problem" .... in particular for raw I/O and direct
I/O, that you need to have

set md_mirror:md_mirror_wow_flg=0x20

in your /etc/system file ... but it will degrade performance.
Question ... by how much ?
Is anyone using ASE with SDS with the above turned on ?
Or should I just use ASE's mirroring ?


Here is the snippet from "man metainit":


Write-On-Write Problem
When mirroring data in Solstice DiskSuite 4.2.1, transfers
from memory to the disks do not all occur at exactly the
same time for all sides of the mirror. If the contents of
buffers are changed while the data is in-flight to the disk
(called write-on-write), then different data can end up
being stored on each side of a mirror.

This problem can be addressed by making a private copy of
the data for mirror writes, however, doing this copy is
expensive. Another approach is to detect when memory has
been modified across a write by looking at the dirty-bit
associated with the memory page. DiskSuite 4.2.1 uses this
dirty-bit technique when it can. Unfortunately, this tech-
nique does not work for raw I/O or direct I/O. By default,
DiskSuite 4.2.1 is tuned for performance with the liability

SunOS 5.8 Last change: 11 May 2001 9

Maintenance Commands metainit(1M)

that mirrored data might be out of sync if an application
does a "write-on-write" to buffers associated with raw I/O
or direct I/O.

Note that without mirroring, you were not guaranteed what
data would actually end up on media, but multiple reads
would return the same data. With mirroring, multiple reads
may return different data. The following line can be added
to /etc/system to cause a stable copy of the buffers to be
used for all raw I/O and direct I/O write operations.

set md_mirror:md_mirror_wow_flg=0x20

Setting this flag will degrade performance.

certainly the Sybase mirroring won't benefit any performance goal.
Maybe you should do some basic benchmarking about how much performance
is lost with this md_mirror setting.
Compare the numbers with I/O to a single submirror without the md_mirror
parameter and (if you want) with Sybase mirroring.

PS where/how did you encounter the problem that showed you the
write-on-write problem ?

Ramon

jmsalvo wrote:
> We are well on our way and on the final few steps of migrating from
> Oracle to ASE.
>
> Our configuration is an E3500 running Solaris8, with dual A1000s. The
> disks in each A1000 is mirrored to the other A1000 by Solstice
> DiskSuite. Now I noticed that metainit actually says about
> "Write-On-Write Problem" .... in particular for raw I/O and direct
> I/O, that you need to have
>
> set md_mirror:md_mirror_wow_flg=0x20
>
> in your /etc/system file ... but it will degrade performance.
> Question ... by how much ?
> Is anyone using ASE with SDS with the above turned on ?
> Or should I just use ASE's mirroring ?
>
>
> Here is the snippet from "man metainit":
>
>
> Write-On-Write Problem
> When mirroring data in Solstice DiskSuite 4.2.1, transfers
> from memory to the disks do not all occur at exactly the
> same time for all sides of the mirror. If the contents of
> buffers are changed while the data is in-flight to the disk
> (called write-on-write), then different data can end up
> being stored on each side of a mirror.
>
> This problem can be addressed by making a private copy of
> the data for mirror writes, however, doing this copy is
> expensive. Another approach is to detect when memory has
> been modified across a write by looking at the dirty-bit
> associated with the memory page. DiskSuite 4.2.1 uses this
> dirty-bit technique when it can. Unfortunately, this tech-
> nique does not work for raw I/O or direct I/O. By default,
> DiskSuite 4.2.1 is tuned for performance with the liability
>
> SunOS 5.8 Last change: 11 May 2001 9
>
> Maintenance Commands metainit(1M)
>
> that mirrored data might be out of sync if an application
> does a "write-on-write" to buffers associated with raw I/O
> or direct I/O.
>
> Note that without mirroring, you were not guaranteed what
> data would actually end up on media, but multiple reads
> would return the same data. With mirroring, multiple reads
> may return different data. The following line can be added
> to /etc/system to cause a stable copy of the buffers to be
> used for all raw I/O and direct I/O write operations.
>
> set md_mirror:md_mirror_wow_flg=0x20
>
> Setting this flag will degrade performance.

--
Groetjes

Ramon Vennik

================================================== ====================
As a computing professional, I believe it would be unethical for me to
advise, recommend, or support the use (save possibly for personal
amusement) of any product that is or depends on any Microsoft product.
================================================== ====================

jmsalvo wrote:
>
> We are well on our way and on the final few steps of migrating from
> Oracle to ASE.

Congratulations.

> Our configuration is an E3500 running Solaris8, with dual A1000s. The
> disks in each A1000 is mirrored to the other A1000 by Solstice
> DiskSuite.

Given that the E3500 has internal FC-AL disks, I would have
gone with that and an external FC-AL unit to suit for the
mirroring. The i/o performance would have been a lot higher.

> Now I noticed that metainit actually says about
> "Write-On-Write Problem" .... in particular for raw I/O and direct
> I/O, that you need to have
>
> set md_mirror:md_mirror_wow_flg=0x20
>
> in your /etc/system file ... but it will degrade performance.
> Question ... by how much ?

That's something you'd need to ask Sun tech support about.
I don't expect the perforamnce impact to be really noticeable.

> Is anyone using ASE with SDS with the above turned on ?
> Or should I just use ASE's mirroring ?

I would stick to SDS's mirroring. Turn the option on (I haven't
- but then I don't have your exact combination either but I've
noticed no problems) if you are concerned but don't forget that
the A1000s have a write cache and the parallel write method
should write to each mirror at the same time. Some of the
comments in the reference seem to suggest its not applicable
to ASE either (cf. dirty-bit and memory page).

-am © 2003

Ramon Vennik <ramon@vennik.com.nocrap> wrote in message news:<3f0bb80a$0$282$ba620e4c@reader1.news.skynet. be>...
> certainly the Sybase mirroring won't benefit any performance goal.
> Maybe you should do some basic benchmarking about how much performance
> is lost with this md_mirror setting.
> Compare the numbers with I/O to a single submirror without the md_mirror
> parameter and (if you want) with Sybase mirroring.

That's exactly what I will do today.

>
> PS where/how did you encounter the problem that showed you the
> write-on-write problem ?
>

I have not yet ... but in the course of trying to find out more about
SDS, I read about "write-on-write" problem. I searched on google, saw
this:

http://dbforums.com/arch/158/2003/5/782082

Searching Sunsolve, saw this:

Bug ID: 4331413
http://sunsolve.sun.com/private-cgi/...e_ 32=4306468

jmsalvo@yahoo.com.au (jmsalvo) wrote in message news:<4d6d063b.0307091632.27585b36@posting.google. com>...
> Ramon Vennik <ramon@vennik.com.nocrap> wrote in message news:<3f0bb80a$0$282$ba620e4c@reader1.news.skynet. be>...
> > certainly the Sybase mirroring won't benefit any performance goal.
> > Maybe you should do some basic benchmarking about how much performance
> > is lost with this md_mirror setting.
> > Compare the numbers with I/O to a single submirror without the md_mirror
> > parameter and (if you want) with Sybase mirroring.
>
> That's exactly what I will do today.
>

I'll be doing the tests tonight ( combination of dd, time, and iostat
).

If worse comes to worst ... I may end up using cooked filesystems for
ASE ( hope not! ) on Solaris8 ... just to make sure that the data is
mirrored between the two A1000s that we have.

Anyway, the main case for using raw device is data integrity, since
the writes are not buffered by the OS. But doesn't using a cooked
filesystem with DSYNC to ON equivalent anyway .. acheiving the same
reliability?

If yes, the only argument left is performance ... if Solaris have
async I/O for cooked filesystem. From what I can recall, it does.

jmsalvo@yahoo.com.au (jmsalvo) wrote in message news:<4d6d063b.0307091632.27585b36@posting.google. com>...
> Ramon Vennik <ramon@vennik.com.nocrap> wrote in message news:<3f0bb80a$0$282$ba620e4c@reader1.news.skynet. be>...
> > certainly the Sybase mirroring won't benefit any performance goal.
> > Maybe you should do some basic benchmarking about how much performance
> > is lost with this md_mirror setting.
> > Compare the numbers with I/O to a single submirror without the md_mirror
> > parameter and (if you want) with Sybase mirroring.
>
> That's exactly what I will do today.
>

I read that Solaris8 Update 3 has a new implementation of Direct I/O
allows the UFS file system to perform at near raw disk performance:

http://mailman.eng.auburn.edu/piperm...ly/004459.html

http://de.sun.com/Partner/Softwarepa...on_Solaris.pdf


Yeah .. yeah ... the PDF link is about Oracle .. but it is where the
improvement in Solaris8 Update 3 is mentioned.

jmsalvo wrote:
> jmsalvo@yahoo.com.au (jmsalvo) wrote in message news:<4d6d063b.0307091632.27585b36@posting.google. com>...
>
>>Ramon Vennik <ramon@vennik.com.nocrap> wrote in message news:<3f0bb80a$0$282$ba620e4c@reader1.news.skynet. be>...
>>
>>>certainly the Sybase mirroring won't benefit any performance goal.
>>>Maybe you should do some basic benchmarking about how much performance
>>>is lost with this md_mirror setting.
>>>Compare the numbers with I/O to a single submirror without the md_mirror
>>>parameter and (if you want) with Sybase mirroring.
>>
>>That's exactly what I will do today.
>>
>
>
> I read that Solaris8 Update 3 has a new implementation of Direct I/O
> allows the UFS file system to perform at near raw disk performance:
>
> http://mailman.eng.auburn.edu/piperm...ly/004459.html
>
> http://de.sun.com/Partner/Softwarepa...on_Solaris.pdf
>
>
> Yeah .. yeah ... the PDF link is about Oracle .. but it is where the
> improvement in Solaris8 Update 3 is mentioned.


Ack! Unfortunately ... that write-on-write problem of SDS also happens
for direct I/O.

jmsalvo wrote:
> jmsalvo@yahoo.com.au (jmsalvo) wrote in message news:<4d6d063b.0307091632.27585b36@posting.google. com>...
>
>>Ramon Vennik <ramon@vennik.com.nocrap> wrote in message news:<3f0bb80a$0$282$ba620e4c@reader1.news.skynet. be>...
>>
>>>certainly the Sybase mirroring won't benefit any performance goal.
>>>Maybe you should do some basic benchmarking about how much performance
>>>is lost with this md_mirror setting.
>>>Compare the numbers with I/O to a single submirror without the md_mirror
>>>parameter and (if you want) with Sybase mirroring.
>>
>>That's exactly what I will do today.
>>
>
>
> I'll be doing the tests tonight ( combination of dd, time, and iostat
> ).
>
> If worse comes to worst ... I may end up using cooked filesystems for
> ASE ( hope not! ) on Solaris8 ... just to make sure that the data is
> mirrored between the two A1000s that we have.
>
> Anyway, the main case for using raw device is data integrity, since
> the writes are not buffered by the OS. But doesn't using a cooked
> filesystem with DSYNC to ON equivalent anyway .. acheiving the same
> reliability?
>
> If yes, the only argument left is performance ... if Solaris have
> async I/O for cooked filesystem. From what I can recall, it does.


Correct me if I am wrong here:


Cooked filesystems in general:
* Synchronous, and writes are buffered by OS

Cooked filesystems, mounted with "forcedirectio"
* Synchronous, writes are not buffered

Cooked filesystems with concurrent direct I/O:
* Asynchronous, writes are not buffered

Raw devices in general:
* Asynchronous, writes are not buffered



Now in ASE .. when a sybase device is created on a cooked filesystem
with DYNSC set to true, does that mean that it uses direct I/O ?

I suppose note ... in particular if the filesystem is not mounted with
"forcediretedio". So what does it mean to the OS when you create a
sybase device on a cooked filesystem with DSYNC set to true ?

... in particular for Solaris ?

jmsalvo wrote:

> > If worse comes to worst ... I may end up using cooked filesystems for
> > ASE ( hope not! ) on Solaris8 ... just to make sure that the data is
> > mirrored between the two A1000s that we have.
> >
> > Anyway, the main case for using raw device is data integrity, since
> > the writes are not buffered by the OS. But doesn't using a cooked
> > filesystem with DSYNC to ON equivalent anyway .. acheiving the same
> > reliability?
> >
> > If yes, the only argument left is performance ... if Solaris have
> > async I/O for cooked filesystem. From what I can recall, it does.
>
> Correct me if I am wrong here:
>
> Cooked filesystems in general:
> * Synchronous, and writes are buffered by OS
>
> Cooked filesystems, mounted with "forcedirectio"
> * Synchronous, writes are not buffered
>
> Cooked filesystems with concurrent direct I/O:
> * Asynchronous, writes are not buffered
>
> Raw devices in general:
> * Asynchronous, writes are not buffered
>
> Now in ASE .. when a sybase device is created on a cooked filesystem
> with DYNSC set to true, does that mean that it uses direct I/O ?
>
> I suppose note ... in particular if the filesystem is not mounted with
> "forcediretedio". So what does it mean to the OS when you create a
> sybase device on a cooked filesystem with DSYNC set to true ?
>
> .. in particular for Solaris ?

The open man page -

http://docs.sun.com/db/doc/816-0212/...#indexterm-173

doesn't say very much about it. I'd suggest asking in the
Solaris newsgroup.

-am © 2003

jmsalvo <jmsalvo@yahoo.com.au> wrote in message news:<bejvto$egq$1@lust.ihug.co.nz>...
> jmsalvo wrote:
> > jmsalvo@yahoo.com.au (jmsalvo) wrote in message news:<4d6d063b.0307091632.27585b36@posting.google. com>...
> >
> >>Ramon Vennik <ramon@vennik.com.nocrap> wrote in message news:<3f0bb80a$0$282$ba620e4c@reader1.news.skynet. be>...
> >>
> >>>certainly the Sybase mirroring won't benefit any performance goal.
> >>>Maybe you should do some basic benchmarking about how much performance
> >>>is lost with this md_mirror setting.
> >>>Compare the numbers with I/O to a single submirror without the md_mirror
> >>>parameter and (if you want) with Sybase mirroring.
> >>
> >>That's exactly what I will do today.
> >>
> >
> >
> > I read that Solaris8 Update 3 has a new implementation of Direct I/O
> > allows the UFS file system to perform at near raw disk performance:
> >
> > http://mailman.eng.auburn.edu/piperm...ly/004459.html
> >
> > http://de.sun.com/Partner/Softwarepa...on_Solaris.pdf
> >
> >
> > Yeah .. yeah ... the PDF link is about Oracle .. but it is where the
> > improvement in Solaris8 Update 3 is mentioned.
>
>
> Ack! Unfortunately ... that write-on-write problem of SDS also happens
> for direct I/O.


Got the response from Sun Support here in Australia....

The statement that I got from them is that, when using SDS mirroring,
don't use raw devices but use instead ufs with concurrent direct I/O
.... which will avoid the situation altogether.

However, he did mention that to take great advantage of concurrent
direct I/O, the database should use kaio instead of aio for
performance.

So that's what I have to do then ... forget about raw devices and use
concurrent direct I/O on UFS.