I'm beginning to wonder if we have an i/o bottleneck on our p630 Oracle
DB server. Looking at vmstat i was surprised to see the run queue
numbers quite high but then I remembered that 2 jobs per CPU was fine
and this is a 4-way server so moving across to CPW waiting on i/o (wa)
the number seems to be consistently high.

vmstat 2 20
System Configuration: lcpu=4 mem=8192MB
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
2 1 1471217 24148 0 5 5 118 46 0 855 22911 5148 24 5 56 15
2 0 1459870 35479 0 0 0 0 0 0 1587 16040 4106 24 7 15
54
1 0 1461205 34141 0 0 0 0 0 0 1670 12105 3964 20 3 22
55
2 0 1475169 20174 0 0 0 0 0 0 1550 14211 2881 23 6 14
57
7 0 1483321 12021 0 0 0 0 0 0 3007 28710 5895 41 7 18
34
2 0 1477516 17826 0 0 0 0 0 0 1943 14026 3564 41 4 11
44
2 0 1474878 20464 0 0 0 0 0 0 4107 25524 8263 20 7 7
67
2 0 1485275 10067 0 0 0 0 0 0 1710 11294 3212 16 4 18
62
4 0 1484883 10459 0 0 0 0 0 0 3983 30203 7638 52 8 10
30
7 0 1451573 43744 0 0 0 0 0 0 3209 29553 8087 75 8 3
13
6 0 1456513 38747 0 1 0 0 0 0 4427 35539 10320 36 7 11
46
2 0 1472992 22169 0 0 0 0 0 0 3674 32302 10218 42 8 13
38
5 0 1473987 21034 0 5 0 0 0 0 2121 103976 6846 52 15 12
20
2 0 1475032 19853 0 5 0 0 0 0 1478 20902 7709 44 5 24
26
6 0 1477729 17155 0 0 0 0 0 0 1577 8975 3655 32 4 24 40
1 1 1478807 16070 0 3 0 0 0 0 1598 14229 3777 22 6 25
47
4 0 1479379 15498 0 0 0 0 0 0 1722 10560 4265 22 3 30
45
4 0 1469500 25377 0 0 0 0 0 0 1480 10586 3310 56 3 12
29
2 0 1465774 29102 0 0 0 0 0 0 2525 15341 5204 29 5 10
56
0 0 1454459 40417 0 0 0 0 0 0 3619 37886 8715 33 11 10
46

The disk that Oracle is using is a 'raw' device ie no filesystem on it
but it seems to be very busy as show by iostat

iran:/usr/adm >iostat 2 10

System configuration: lcpu=4 disk=5

tty: tin tout avg-cpu: % user % sys % idle %
iowait
0.1 19.5 24.0 5.1 56.1
14.7
" Disk history since boot not available. "


tty: tin tout avg-cpu: % user % sys % idle %
iowait
0.0 120.0 53.2 3.6 8.4
34.7

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 7.8 109.9 21.4 0 241
hdisk1 8.7 109.9 21.4 0 241
hdisk2 0.0 0.0 0.0 0 0
hdisk3 91.2 14878.0 187.0 8072 24548
cd0 0.0 0.0 0.0 0 0

tty: tin tout avg-cpu: % user % sys % idle %
iowait
0.0 272.0 20.6 4.1 14.4
60.9

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 0.0 0.0 0.0 0 0
hdisk1 2.0 14.0 3.5 0 28
hdisk2 0.0 0.0 0.0 0 0
hdisk3 100.0 11550.0 210.0 18320 4780
cd0 0.0 0.0 0.0 0 0

tty: tin tout avg-cpu: % user % sys % idle %
iowait
0.0 272.0 56.5 4.1 11.6
27.8

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 0.0 0.0 0.0 0 0
hdisk1 3.0 32.0 6.5 4 60
hdisk2 0.0 0.0 0.0 0 0
hdisk3 86.0 10472.0 205.0 8888 12056
cd0 0.0 0.0 0.0 0 0

tty: tin tout avg-cpu: % user % sys % idle %
iowait
0.0 272.0 60.4 7.6 7.1
24.9

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 0.0 0.0 0.0 0 0
hdisk1 0.5 2.0 0.5 0 4
hdisk2 0.0 0.0 0.0 0 0
hdisk3 88.5 14954.0 179.5 18184 11724
cd0 0.0 0.0 0.0 0 0

tty: tin tout avg-cpu: % user % sys % idle %
iowait
0.0 272.0 33.9 9.6 10.5
46.0

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 11.0 46.0 11.5 68 24
hdisk1 1.5 70.0 2.0 0 140
hdisk2 0.0 0.0 0.0 0 0
hdisk3 99.0 13114.0 212.0 19552 6676
cd0 0.0 0.0 0.0 0 0

tty: tin tout avg-cpu: % user % sys % idle %
iowait
0.0 272.0 26.2 6.8 13.0
54.0

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 0.0 0.0 0.0 0 0
hdisk1 0.0 0.0 0.0 0 0
hdisk2 0.0 0.0 0.0 0 0
hdisk3 100.0 12528.0 225.5 0 25056
cd0 0.0 0.0 0.0 0 0

tty: tin tout avg-cpu: % user % sys % idle %
iowait
0.0 272.0 27.6 8.4 9.5
54.5

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 0.0 0.0 0.0 0 0
hdisk1 1.0 8.0 2.0 0 16
hdisk2 0.0 0.0 0.0 0 0
hdisk3 100.0 13930.0 249.5 24664 3196
cd0 0.0 0.0 0.0 0 0

tty: tin tout avg-cpu: % user % sys % idle %
iowait
0.0 272.0 55.2 3.0 12.9
28.9

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 0.0 0.0 0.0 0 0
hdisk1 0.0 0.0 0.0 0 0
hdisk2 0.0 0.0 0.0 0 0
hdisk3 95.5 10800.0 240.0 1824 19776
cd0 0.0 0.0 0.0 0 0

tty: tin tout avg-cpu: % user % sys % idle %
iowait
0.0 272.0 39.6 1.6 17.4
41.4

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 0.5 2.0 0.5 0 4
hdisk1 0.0 0.0 0.0 0 0
hdisk2 0.0 0.0 0.0 0 0
hdisk3 100.0 13430.0 241.0 9016 17844
cd0 0.0 0.0 0.0 0 0

Is there anything I should consider doing to improve the situation ?

TIA,
Mike.

Just my curiosity - where do you keep Oracle logs? On rootvg disks
(hdisk0 & probably hdisk1)?

Regarding your situation, if response time is degrading, I'd consider
database tuning.
Put indexes for tables that do most SELECT jobs.
Cache statements (expand SGA if necessary).
If possible, move some tables to a different disk. I've noticed that
hdisk2 is idle. If there's a filesystem already, you may create raw
type LV, and use it as a raw device for Oracle.

If hdisk3 is a SAN attached RAID5 array, and your database does many
INSERT/UPDATE operations, change it to RAID1+0.