Unix Technical Forum

vxresize caused disk IO suspended?

This is a discussion on vxresize caused disk IO suspended? within the Sun Solaris Administration forums, part of the Solaris Operating System category; --> hi all, We have an incident today, after we resized the home volume for oracle (to hold the oracle ...


Go Back   Unix Technical Forum > Unix Operating Systems > Solaris Operating System > Sun Solaris Administration

FAQ Members List Calendar Search Today's Posts Mark Forums Read
  #1 (permalink)  
Old 01-16-2008, 10:18 AM
chao_ping
 
Posts: n/a
Default vxresize caused disk IO suspended?

hi all,
We have an incident today, after we resized the home volume for
oracle (to hold the oracle software), we see high load on unix box
(solaris 8, sun V2900 , oracle 8.1.7, veritas 3.1.1).

We just did vxresize -g oradg homevol 12g (originally it was 8GB).
During the resize time, seems all the log (our own monitoring script)
freezed and we see high load on unix box(70-90 load average), and
application do not get response from database and disconnected.

While resize data volume (hold oracle datafile/archivelog) , we never
seen that problem.

anyone have experience with that?

thx

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #2 (permalink)  
Old 01-16-2008, 10:18 AM
chao_ping
 
Posts: n/a
Default Re: vxresize caused disk IO suspended?

by the way, the volume we are expanding, is just a normal volume, no
concatation, no stripe etc.

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #3 (permalink)  
Old 01-16-2008, 10:18 AM
Darren Dunham
 
Posts: n/a
Default Re: vxresize caused disk IO suspended?

In comp.unix.solaris chao_ping <zhuchao@gmail.com> wrote:
> hi all,
> We have an incident today, after we resized the home volume for
> oracle (to hold the oracle software), we see high load on unix box
> (solaris 8, sun V2900 , oracle 8.1.7, veritas 3.1.1).


> We just did vxresize -g oradg homevol 12g (originally it was 8GB).
> During the resize time, seems all the log (our own monitoring script)
> freezed and we see high load on unix box(70-90 load average), and
> application do not get response from database and disconnected.


Seems odd. What filesystem was involved? VxFS or UFS?

--
Darren Dunham ddunham@taos.com
Senior Technical Consultant TAOS http://www.taos.com/
Got some Dr Pepper? San Francisco, CA bay area
< This line left intentionally blank to confuse you. >
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #4 (permalink)  
Old 01-16-2008, 10:18 AM
zhuchao@gmail.com
 
Posts: n/a
Default Re: vxresize caused disk IO suspended?

it is vxfs

monitoring job failed to start at time when the resize was running. it
should start at 23:01, but actually it start at 23:02:14.
and during that time, we see very high IO wait. after that minute, the
IO wait goes to normal.
db125$> cat mpstat.out_0301_230214
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys
wt idl
0 65 9 410 388 201 1822 95 329 339 0 588 27 11
27 35
1 63 9 263 113 1 1931 102 359 429 0 613 26 10
28 35
2 62 8 2080 958 853 1860 100 348 425 0 488 26 14
25 35
3 61 8 2124 996 891 1869 101 350 428 0 479 25 14
25 35
8 62 9 179 113 1 1934 103 363 427 0 588 26 11
28 35
9 63 9 153 113 1 1926 103 362 426 0 601 26 11
28 35
10 63 9 169 115 1 1936 105 364 433 0 629 26 11
28 35
11 56 8 305 1860 1785 1664 98 326 685 0 236 23 22
22 32
512 62 9 180 116 1 1978 105 369 434 0 601 26 10
29 36
513 63 9 173 136 21 1955 105 364 425 0 628 26 10
28 35
514 64 9 227 145 30 1952 105 366 430 0 658 27 10
28 35
515 64 9 180 116 1 1949 106 365 428 0 665 27 10
28 35
520 65 9 156 115 1 1924 104 360 411 0 629 27 11
28 35
521 65 9 153 134 21 1912 104 358 408 0 629 27 11
28 35
522 66 9 181 115 1 1917 105 358 411 0 650 27 11
28 35
523 66 9 166 113 1 1897 103 357 406 0 656 27 11
28 35

--at this minute, the io wait goes to mormal. it is mpstat 60 60
output.
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys
wt idl
0 510 2 2281 315 201 566 7 44 135 0 2646 9 7
8 76
1 423 2 2071 20 1 570 7 53 119 0 2256 8 6
8 78
2 407 2 2323 234 215 518 8 48 98 0 2005 12 6
9 73
3 335 1 1909 253 234 471 8 46 110 0 2065 20 6
6 68
8 355 2 2016 18 1 506 8 49 82 0 1718 11 6
9 74
9 378 1 1663 19 1 522 9 50 79 0 1621 15 5
9 71
10 447 2 2396 21 1 578 9 53 91 0 2182 9 6
8 76
11 328 3 2417 1111 1093 571 8 52 142 0 2256 8 7
9 75
512 366 2 1975 22 1 562 8 50 120 0 1920 12 5
11 72
513 346 1 2030 43 22 523 8 49 102 0 1858 14 4
7 75
514 457 1 5573 52 31 586 8 51 162 0 2317 11 7
6 77
515 382 1 2061 19 1 463 8 47 97 0 2156 15 5
6 74
520 384 2 1640 20 1 565 9 50 155 0 2277 15 5
10 71
521 305 2 2286 41 22 557 8 49 110 0 2679 10 5
8 76
522 305 1 2447 19 1 550 8 47 127 0 2195 14 5
8 73
523 353 2 3474 20 1 556 9 51 122 0 2405 11 7
9 74
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys
wt idl

vxstat job also failed to capture the io statistics during the problem
time.

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #5 (permalink)  
Old 01-16-2008, 10:18 AM
Nathan Dietsch
 
Posts: n/a
Default Re: vxresize caused disk IO suspended?

Hi Chao,

chao_ping wrote:
> hi all,
> We have an incident today, after we resized the home volume for
> oracle (to hold the oracle software), we see high load on unix box
> (solaris 8, sun V2900 , oracle 8.1.7, veritas 3.1.1).
>
> We just did vxresize -g oradg homevol 12g (originally it was 8GB).
> During the resize time, seems all the log (our own monitoring script)
> freezed and we see high load on unix box(70-90 load average), and
> application do not get response from database and disconnected.
>
> While resize data volume (hold oracle datafile/archivelog) , we never
> seen that problem.
>
> anyone have experience with that?


Quick question: Are your redologs located on that filesystem? All Oracle
IO goes through the redologs and then, after a certain time, gets
flushed out to the data and index filesystems.

The data and index filesystems can be written asynchronously, but Oracle
can not acknowledge a commit until it is written to the redo log.

Darren brings up a very good question, is the filesystem UFS or VxFS?

Kind Regards,

Nathan Dietsch

>
> thx
>

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #6 (permalink)  
Old 01-16-2008, 10:18 AM
chao_ping
 
Posts: n/a
Default Re: vxresize caused disk IO suspended?

hi,
There were no oracle redolog in that filesystem, and it was vxfs.

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
Reply


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On
Forum Jump


All times are GMT. The time now is 08:27 AM.


Powered by vBulletin® Version 3.6.5
Copyright ©2000 - 2008, Jelsoft Enterprises Ltd.
SEO by vBSEO 3.2.0
www.UnixAdminTalk.com