SEO

AIX'ers!

I have a machine that have problems with performance.
Output from vmstat shows:

kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
18 0 8483035 1571613 0 0 0 0 0 0 6499 99823 13804 68 22 4 7
17 1 8482913 1571734 0 0 0 0 0 0 6778 86403 14977 78 13 6 4
8 2 8483831 1570812 0 0 0 0 0 0 6969 79401 16112 67 15 10 8
8 1 8489595 1565047 0 0 0 0 0 0 6191 209083 17598 70 18 6 6
11 1 8482764 1571877 0 0 0 0 0 0 6049 271784 15108 68 20 5 7
10 1 8482126 1572515 0 0 0 0 0 0 6051 73168 14993 57 11 18 13
12 1 8481532 1573107 0 0 0 0 0 0 6081 87441 15317 60 15 13 12
10 1 8481446 1573193 0 0 0 0 0 0 6344 75612 14889 71 14 7 7
16 1 8483620 1571017 0 0 0 0 0 0 7486 65699 16522 74 13 6 7

I believe this machine has serious problems with CPU and not with memory (columns "pi" and
"po" are always 0).

But my question is: Why is the column "us" under CPU so high instead of the column "sy"?

Anybody have other comments to the output?

We run AIX 5.2, HACMP 5 and Oracle9 RAC. (2 node cluster) The other machine has the same
problem too.


TIA!

Arne S

Arne S <apsodal@NOSPAM.online.no> wrote in message news:<40fccb2a$1@news.broadpark.no>...
> kthr memory page faults cpu
> ----- ----------- ------------------------ ------------ -----------
> r b avm fre re pi po fr sr cy in sy cs us sy id wa
> 18 0 8483035 1571613 0 0 0 0 0 0 6499 99823 13804 68 22 4 7
> 17 1 8482913 1571734 0 0 0 0 0 0 6778 86403 14977 78 13 6 4
> 8 2 8483831 1570812 0 0 0 0 0 0 6969 79401 16112 67 15 10 8
> 8 1 8489595 1565047 0 0 0 0 0 0 6191 209083 17598 70 18 6 6
> 11 1 8482764 1571877 0 0 0 0 0 0 6049 271784 15108 68 20 5 7
> 10 1 8482126 1572515 0 0 0 0 0 0 6051 73168 14993 57 11 18 13
> 12 1 8481532 1573107 0 0 0 0 0 0 6081 87441 15317 60 15 13 12
> 10 1 8481446 1573193 0 0 0 0 0 0 6344 75612 14889 71 14 7 7
> 16 1 8483620 1571017 0 0 0 0 0 0 7486 65699 16522 74 13 6 7
>
> I believe this machine has serious problems with CPU and not with memory (columns "pi" and
> "po" are always 0).
>
> But my question is: Why is the column "us" under CPU so high instead of the column "sy"?
>
> Anybody have other comments to the output?
>

us is for user processes, sys is for system processes, Oracle and HA
will come under us. System looks fine to me, no 0's under idle You
have a bit of I/O wait, but if you have very fast processors then that
is normal, as disk speeds ahve increased as fast as processor speeds.

I wouldn't be concerned about this system, yet, when you start getting
0 in the id column for a long time, then you have a CPU bottleneck.
Are users complaining of slow response ?

Steve Nottingham wrote:

> Arne S <apsodal@NOSPAM.online.no> wrote in message news:<40fccb2a$1@news.broadpark.no>...
>
>>kthr memory page faults cpu
>>----- ----------- ------------------------ ------------ -----------
>> r b avm fre re pi po fr sr cy in sy cs us sy id wa
>>18 0 8483035 1571613 0 0 0 0 0 0 6499 99823 13804 68 22 4 7
>>17 1 8482913 1571734 0 0 0 0 0 0 6778 86403 14977 78 13 6 4
>> 8 2 8483831 1570812 0 0 0 0 0 0 6969 79401 16112 67 15 10 8
>> 8 1 8489595 1565047 0 0 0 0 0 0 6191 209083 17598 70 18 6 6
>>11 1 8482764 1571877 0 0 0 0 0 0 6049 271784 15108 68 20 5 7
>>10 1 8482126 1572515 0 0 0 0 0 0 6051 73168 14993 57 11 18 13
>>12 1 8481532 1573107 0 0 0 0 0 0 6081 87441 15317 60 15 13 12
>>10 1 8481446 1573193 0 0 0 0 0 0 6344 75612 14889 71 14 7 7
>>16 1 8483620 1571017 0 0 0 0 0 0 7486 65699 16522 74 13 6 7
>>
>>I believe this machine has serious problems with CPU and not with memory (columns "pi" and
>>"po" are always 0).
>>
>>But my question is: Why is the column "us" under CPU so high instead of the column "sy"?
>>
>>Anybody have other comments to the output?
>>
>
>
> us is for user processes, sys is for system processes, Oracle and HA
> will come under us. System looks fine to me, no 0's under idle You
> have a bit of I/O wait, but if you have very fast processors then that
> is normal, as disk speeds ahve increased as fast as processor speeds.
>
> I wouldn't be concerned about this system, yet, when you start getting
> 0 in the id column for a long time, then you have a CPU bottleneck.
> Are users complaining of slow response ?
Thanks for response!

I thought if sum of "us" and "sy" was above 80%, then you had a CPU bottleneck. True or not?
I have done some measures for a week or so, and the "id" is quite low between 07:00 am and
04:00 pm (from 5% to 10%). And yes, the users complain about slow performance....
The oracle procs are stealing the most of the CPU power, and many of the oracle procs have
lots of CPU time (Oracle db procs (smon, pmon,arch... not included).

The column "r" to the left from vmstat output often shows a value above 30, is that
normal? I have even seen the value 98 in this column, I think that is too high to get good
performance?

Both of the nodes have 6 CPU's.

Arne S

Arne S <apsodal@NOSPAM.online.no> wrote in message news:<40FD2C54.60400@NOSPAM.online.no>...
> Steve Nottingham wrote:
>
> > Arne S <apsodal@NOSPAM.online.no> wrote in message news:<40fccb2a$1@news.broadpark.no>...
> >
> >>kthr memory page faults cpu
> >>----- ----------- ------------------------ ------------ -----------
> >> r b avm fre re pi po fr sr cy in sy cs us sy id wa
> >>18 0 8483035 1571613 0 0 0 0 0 0 6499 99823 13804 68 22 4 7
> >>17 1 8482913 1571734 0 0 0 0 0 0 6778 86403 14977 78 13 6 4
> >> 8 2 8483831 1570812 0 0 0 0 0 0 6969 79401 16112 67 15 10 8
> >> 8 1 8489595 1565047 0 0 0 0 0 0 6191 209083 17598 70 18 6 6
> >>11 1 8482764 1571877 0 0 0 0 0 0 6049 271784 15108 68 20 5 7
> >>10 1 8482126 1572515 0 0 0 0 0 0 6051 73168 14993 57 11 18 13
> >>12 1 8481532 1573107 0 0 0 0 0 0 6081 87441 15317 60 15 13 12
> >>10 1 8481446 1573193 0 0 0 0 0 0 6344 75612 14889 71 14 7 7
> >>16 1 8483620 1571017 0 0 0 0 0 0 7486 65699 16522 74 13 6 7
> >>
> >>I believe this machine has serious problems with CPU and not with memory (columns "pi" and
> >>"po" are always 0).
> >>
> >>But my question is: Why is the column "us" under CPU so high instead of the column "sy"?
> >>
> >>Anybody have other comments to the output?
> >>
> >
> >
> > us is for user processes, sys is for system processes, Oracle and HA
> > will come under us. System looks fine to me, no 0's under idle You
> > have a bit of I/O wait, but if you have very fast processors then that
> > is normal, as disk speeds ahve increased as fast as processor speeds.
> >
> > I wouldn't be concerned about this system, yet, when you start getting
> > 0 in the id column for a long time, then you have a CPU bottleneck.
> > Are users complaining of slow response ?
> Thanks for response!
>
> I thought if sum of "us" and "sy" was above 80%, then you had a CPU bottleneck. True or not?
> I have done some measures for a week or so, and the "id" is quite low between 07:00 am and
> 04:00 pm (from 5% to 10%). And yes, the users complain about slow performance....
> The oracle procs are stealing the most of the CPU power, and many of the oracle procs have
> lots of CPU time (Oracle db procs (smon, pmon,arch... not included).
>
> The column "r" to the left from vmstat output often shows a value above 30, is that
> normal? I have even seen the value 98 in this column, I think that is too high to get good
> performance?
>
> Both of the nodes have 6 CPU's.
>
> Arne S


The "r" column is way too high (kernel threads in the run queue). I
would expect
poor performance when this starts to exceed 2 X the number of cpu's. I
bet when
this number is 10 or less your performance is fine. Also, you have a
very large amount of "freelist" (fre). AIX would normally keep the
free list fairly low.
(maybe Oracle is using raw area's)

Post some additional info, like:
Release of Oracle (32 or 64 bit)
Operating system release, and maint level.
output from vmtune (/usr/samples/kernel/vmtune) (vmo -L command for
Aix 5.2)
we need to see settings for
minfree
maxfree
minperm
maxperm
are your filesystems for Oracle raw? jfs? jfs2?
asyncronous io server settings if using jfs(2)

Should be pretty easy to trouble shoot.

Don Davis wrote:

> Arne S <apsodal@NOSPAM.online.no> wrote in message news:<40FD2C54.60400@NOSPAM.online.no>...
>
>>Steve Nottingham wrote:
>>
>>
>>>Arne S <apsodal@NOSPAM.online.no> wrote in message news:<40fccb2a$1@news.broadpark.no>...
>>>
>>>
>>>>kthr memory page faults cpu
>>>>----- ----------- ------------------------ ------------ -----------
>>>> r b avm fre re pi po fr sr cy in sy cs us sy id wa
>>>>18 0 8483035 1571613 0 0 0 0 0 0 6499 99823 13804 68 22 4 7
>>>>17 1 8482913 1571734 0 0 0 0 0 0 6778 86403 14977 78 13 6 4
>>>> 8 2 8483831 1570812 0 0 0 0 0 0 6969 79401 16112 67 15 10 8
>>>> 8 1 8489595 1565047 0 0 0 0 0 0 6191 209083 17598 70 18 6 6
>>>>11 1 8482764 1571877 0 0 0 0 0 0 6049 271784 15108 68 20 5 7
>>>>10 1 8482126 1572515 0 0 0 0 0 0 6051 73168 14993 57 11 18 13
>>>>12 1 8481532 1573107 0 0 0 0 0 0 6081 87441 15317 60 15 13 12
>>>>10 1 8481446 1573193 0 0 0 0 0 0 6344 75612 14889 71 14 7 7
>>>>16 1 8483620 1571017 0 0 0 0 0 0 7486 65699 16522 74 13 6 7
>>>>
>>>>I believe this machine has serious problems with CPU and not with memory (columns "pi" and
>>>>"po" are always 0).
>>>>
>>>>But my question is: Why is the column "us" under CPU so high instead of the column "sy"?
>>>>
>>>>Anybody have other comments to the output?
>>>>
>>>
>>>
>>>us is for user processes, sys is for system processes, Oracle and HA
>>>will come under us. System looks fine to me, no 0's under idle You
>>>have a bit of I/O wait, but if you have very fast processors then that
>>>is normal, as disk speeds ahve increased as fast as processor speeds.
>>>
>>>I wouldn't be concerned about this system, yet, when you start getting
>>>0 in the id column for a long time, then you have a CPU bottleneck.
>>>Are users complaining of slow response ?
>>
>>Thanks for response!
>>
>>I thought if sum of "us" and "sy" was above 80%, then you had a CPU bottleneck. True or not?
>>I have done some measures for a week or so, and the "id" is quite low between 07:00 am and
>>04:00 pm (from 5% to 10%). And yes, the users complain about slow performance....
>>The oracle procs are stealing the most of the CPU power, and many of the oracle procs have
>>lots of CPU time (Oracle db procs (smon, pmon,arch... not included).
>>
>>The column "r" to the left from vmstat output often shows a value above 30, is that
>>normal? I have even seen the value 98 in this column, I think that is too high to get good
>>performance?
>>
>>Both of the nodes have 6 CPU's.
>>
>>Arne S
>
>
>
> The "r" column is way too high (kernel threads in the run queue). I
> would expect
> poor performance when this starts to exceed 2 X the number of cpu's. I
> bet when
> this number is 10 or less your performance is fine. Also, you have a
> very large amount of "freelist" (fre). AIX would normally keep the
> free list fairly low.
> (maybe Oracle is using raw area's)
>
> Post some additional info, like:
> Release of Oracle (32 or 64 bit)
> Operating system release, and maint level.
> output from vmtune (/usr/samples/kernel/vmtune) (vmo -L command for
> Aix 5.2)
> we need to see settings for
> minfree
> maxfree
> minperm
> maxperm
> are your filesystems for Oracle raw? jfs? jfs2?
> asyncronous io server settings if using jfs(2)
>
> Should be pretty easy to trouble shoot.

Output:
Release of Oracle (32 or 64 bit)
== 64-bit on Oracle9 RAC and HACMP version 4.5 with concurrent disks
Operating system release, and maint level.
== AIX 5200-02
output from vmtune (/usr/samples/kernel/vmtune) (vmo -L command for Aix 5.2)
== vmtune:
memory_frames = 12582912
pinnable_frames = 11467818
maxfree = 512
minfree = 504
minperm% = 10
minperm = 1185164
maxperm% = 20
maxperm = 2370330
strict_maxperm = 0
maxpin% = 80
maxpin = 10066330
maxclient% = 20
lrubucket = 131072
defps = 1
nokilluid = 0
numpsblks = 16187392
npskill = 126464
npswarn = 505856
v_pinshm = 0
pta_balance_threshold = 0
pagecoloring = 0
framesets = 2
mempools = 1
lgpg_size = 0
lgpg_regions = 0
num_spec_dataseg = 0
spec_dataseg_int = 512
memory_affinity = 1
htabscale = -1
force_relalias_lite = 0
relalias_percentage = 0
minpgahead = 2
maxpgahead = 256
pd_npages = 65536
maxrandwrt = 0
numclust = 1
numfsbufs = 196
sync_release_ilock = 0
lvm_bufcnt = 9
j2_minPageReadAhead = 2
j2_maxPageReadAhead = 8
j2_nBufferPerPagerDevice = 512
j2_nPagesPerWriteBehindCluster = 32
j2_maxRandomWrite = 0
j2_nRandomCluster = 0
jfs_clread_enabled = 0
jfs_use_read_lock = 1
hd_pvs_opn = 19
hd_pbuf_cnt = 2944
hd_pendqblked = 5274
psbufwaitcnt = 0
fsbufwaitcnt = 7479
rfsbufwaitcnt = 0
xpagerbufwaitcnt = 14219

== vmo -L:
NAME CUR DEF BOOT MIN MAX UNIT TYPE
DEPENDENCIES
--------------------------------------------------------------------------------
memory_frames 12M 12M 4KB pages S
--------------------------------------------------------------------------------
pinnable_frames 11199K 11199K 4KB pages S
--------------------------------------------------------------------------------
maxfree 512 128 512 16 200K 4KB pages D
minfree
memory_frames
--------------------------------------------------------------------------------
minfree 504 504 504 8 200K 4KB pages D
maxfree
memory_frames
--------------------------------------------------------------------------------
minperm% 10 20 10 1 100 % memory D
maxperm%
--------------------------------------------------------------------------------
minperm 1157K 1157K S
--------------------------------------------------------------------------------
maxperm% 20 80 20 1 100 % memory D
minperm%
maxclient%
--------------------------------------------------------------------------------
maxperm 2314K 2314K S
--------------------------------------------------------------------------------
strict_maxperm 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
maxpin% 80 80 80 1 99 % memory D
pinnable_frames
memory_frames
--------------------------------------------------------------------------------
maxpin 9830K 9830K S
--------------------------------------------------------------------------------
maxclient% 20 80 20 1 100 % memory D
maxperm%
--------------------------------------------------------------------------------
lrubucket 128K 128K 128K 64K 4KB pages D
--------------------------------------------------------------------------------
defps 1 1 1 0 1 boolean D
--------------------------------------------------------------------------------
nokilluid 0 0 0 0 4095M uid D
--------------------------------------------------------------------------------
numpsblks 15808K 15808K 4KB pages S
--------------------------------------------------------------------------------
npskill 126464 126464 126464 1 15807K 4KB pages D
--------------------------------------------------------------------------------
npswarn 494K 494K 494K 0 15807K 4KB pages D
--------------------------------------------------------------------------------
v_pinshm 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
pta_balance_threshold n/a 50 50 1 99 % pta segment R
--------------------------------------------------------------------------------
pagecoloring n/a 0 0 0 1 boolean B
--------------------------------------------------------------------------------
framesets 2 2 2 1 10 B
--------------------------------------------------------------------------------
mempools 1 1 1 1 6 B
--------------------------------------------------------------------------------
lgpg_size 0 0 0 0 256M bytes B
lgpg_regions
--------------------------------------------------------------------------------
lgpg_regions 0 0 0 0 B
lgpg_size
--------------------------------------------------------------------------------
num_spec_dataseg 0 0 0 0 B
--------------------------------------------------------------------------------
spec_dataseg_int 512 512 512 0 B
--------------------------------------------------------------------------------
memory_affinity 1 1 1 0 1 boolean B
--------------------------------------------------------------------------------
htabscale -1 -1 -1 -2 0 B
--------------------------------------------------------------------------------
force_relalias_lite 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
relalias_percentage 0 0 0 0 32767 D
--------------------------------------------------------------------------------
we need to see settings for
minfree
maxfree
minperm
maxperm
are your filesystems for Oracle raw? jfs? jfs2?
== raw devices for all 30 Oracle dabases
asyncronous io server settings if using jfs(2)
== lsattr -E -l aio0 shows (even if we don't use aio):
autoconfig available STATE to be configured at system restart True
fastpath enable State of fast path True
kprocprio 39 Server PRIORITY True
maxreqs 16384 Maximum number of REQUESTS True
maxservers 100 MAXIMUM number of servers per cpu True
minservers 50 MINIMUM number of servers True

TIA!

Arne S

"Arne S" <apsodal@NOSPAM.online.no> wrote in message
news:40FD2C54.60400@NOSPAM.online.no...
> Steve Nottingham wrote:
>
> > Arne S <apsodal@NOSPAM.online.no> wrote in message
news:<40fccb2a$1@news.broadpark.no>...
> >
> >>kthr memory page faults cpu
> >>----- ----------- ------------------------ ------------ -----------
> >> r b avm fre re pi po fr sr cy in sy cs us sy id wa
> >>18 0 8483035 1571613 0 0 0 0 0 0 6499 99823 13804 68 22 4
7
> >>17 1 8482913 1571734 0 0 0 0 0 0 6778 86403 14977 78 13 6
4
> >> 8 2 8483831 1570812 0 0 0 0 0 0 6969 79401 16112 67 15
10 8
> >> 8 1 8489595 1565047 0 0 0 0 0 0 6191 209083 17598 70 18
6 6
> >>11 1 8482764 1571877 0 0 0 0 0 0 6049 271784 15108 68 20
5 7
> >>10 1 8482126 1572515 0 0 0 0 0 0 6051 73168 14993 57 11 18
13
> >>12 1 8481532 1573107 0 0 0 0 0 0 6081 87441 15317 60 15 13
12
> >>10 1 8481446 1573193 0 0 0 0 0 0 6344 75612 14889 71 14 7
7
> >>16 1 8483620 1571017 0 0 0 0 0 0 7486 65699 16522 74 13 6
7
> >>
> >>I believe this machine has serious problems with CPU and not with memory
(columns "pi" and
> >>"po" are always 0).
> >>
> >>But my question is: Why is the column "us" under CPU so high instead of
the column "sy"?
> >>
> >>Anybody have other comments to the output?
> >>
> >
> >
> > us is for user processes, sys is for system processes, Oracle and HA
> > will come under us. System looks fine to me, no 0's under idle You
> > have a bit of I/O wait, but if you have very fast processors then that
> > is normal, as disk speeds ahve increased as fast as processor speeds.
> >
> > I wouldn't be concerned about this system, yet, when you start getting
> > 0 in the id column for a long time, then you have a CPU bottleneck.
> > Are users complaining of slow response ?
> Thanks for response!
>
> I thought if sum of "us" and "sy" was above 80%, then you had a CPU
bottleneck. True or not?
> I have done some measures for a week or so, and the "id" is quite low
between 07:00 am and
> 04:00 pm (from 5% to 10%). And yes, the users complain about slow
performance....
> The oracle procs are stealing the most of the CPU power, and many of the
oracle procs have
> lots of CPU time (Oracle db procs (smon, pmon,arch... not included).
>
> The column "r" to the left from vmstat output often shows a value above
30, is that
> normal? I have even seen the value 98 in this column, I think that is too
high to get good
> performance?
>
> Both of the nodes have 6 CPU's.
>
> Arne S

us is not for user process, it's for time the processor spent running user
instructions,
sy is for time spent running kernel instructions. a user process can
accumulate both.
80% cpu utilization does not represent a bottleneck. in fact neither does
100%. the only
relevant number here seems to the the kthr number, which corresponds to the
load
average and seems a little high.

> System looks fine to me, no 0's under idle

Yes but what is the sample time of that vmstat command ?
vmstat results are MEAN values over the sample time.
So if it is too long, say more than 15 secondes, low values of %idle
are a concern since they account for underlying repetiting zero
values.


dav'