SEO

I'm an end user working on a box with a fair amount of memory and a really
low virtual memory setting in comparision to RAM: 4096.0MB VM to 24575.8MB
RAM. Anyway, the box isn't remarkably consistant in performance from week
to week so I'm trying to help troubleshoot what is wrong. In all of the
docs that I've been looking over, the common sentement seems to be to have
your virtual memory set to at least 1.2xRAM. We are nowhere near that, but
from looking at the nmon output below (during a fairly busy point in our
update), we don't even touch the virtual memory that we have (1%).

My thoughts are that I'm barking up the wrong tree and need to move on...
would you agree?

Also, the 'DANGER' line of the Verbose Mode output is a little disturbing,
but I'm not sure what I'm looking at there... other than a really busy
disk(s). Our CPU wait is at 0, so I don't know if this is affecting us at
all.

Any help would be greatly appreciated.

Thanks in advance,
Josh

nmon v8b [H for help] Hostname= Refresh=2.0secs 16:50.51
Verbose Mode
Code Resource Stats Now Warn Danger
OK -> CPU %busy 19.7% >80% >90%
Warning -> Paging Space %free 99.0% VM<RAM <20%
Warning -> Page Faults faults 55.0 >32/s >320/s
DANGER -> Top Disk hdisk22 %busy 98.4% >40% >60%

CPU Utilisation
+-------------------------------------------------+
CPU User% Sys% Wait% Idle|0 |25 |50 |75
100|
0 22.5 0.0 0.0 77.5|UUUUUUUUUUU >
|
1 33.0 0.0 0.0 67.0|UUUUUUUUUUUUUUUU>
|
2 7.5 0.0 0.0 92.5|UUU >
|
3 55.2 0.0 0.0 44.8|UUUUUUUUUUUUUUUUUUUUUUUUUUU>
|
4 9.5 0.0 0.0 90.5|UUUU >
|
5 2.5 1.0 0.0 96.5|U >
|
6 5.5 2.0 0.0 92.5|UUs >
|
7 12.5 1.0 0.0 86.5|UUUUUU>
|
8 60.5 1.0 0.0 38.5|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUU >
|
9 6.5 0.0 0.0 93.5|UUU >
|
10 0.0 2.5 0.5 97.0|s >
|
11 9.5 0.5 0.0 90.0|UUUU >
|
12 43.0 0.0 0.0 57.0|UUUUUUUUUUUUUUUUUUUUU>
|
13 20.5 0.5 0.0 79.0|UUUUUUUUUU>
|
14 3.5 3.5 0.0 93.0|Us >
|
15 11.5 0.5 0.0 88.0|UUUUU>
|

+-------------------------------------------------+
18.9 0.8 0.0 80.3|UUUUUUUUU>
|

+-------------------------------------------------+
Memory Use Physical Virtual Paging pages/sec In Out VM
parameters
% Used 69.2% 1.0% to Paging Space 0.0 0.0 numperm
50.8%
% Free 30.8% 99.0% to File System 0.0 13.0 minperm
4.7%
MB Used 17005.1MB 40.4MB Page Scans 0.0 maxperm
9.5%
MB Free 7570.8MB 4055.6MB Page Cycles 0.0 minfree
1024
Total(MB) 24575.8MB 4096.0MB Page Reclaim 0.0 maxfree
1280

DiskName Busy Read Write |0 |25 |50 |75
100|
hdisk0 0% 0.0 0.0KB| >
|
hdisk3 0% 0.0 0.0KB|>
|
hdisk2 0% 0.0 0.0KB|>
|
hdisk1 0% 0.0 0.0KB| >
|
cd0 0% 0.0 0.0KB|>
|
cd1 0% 0.0 0.0KB|>
|
dac0 0% 0.0 8.0KB|>
|
dac1 0% 0.0 8.0KB|>
|
dac2 0% 0.0 0.0KB|>
|
dac3 0% 9032.3 6202.7KB|>
|
dac4 0% 0.0 8.0KB|>
|
dac5 0% 0.0 0.0KB|>
|
dac6 0% 0.0 24.0KB|>
|
dac7 0% 0.0 1055.1KB|>
|
dac8 0% 8105.1 10319.2K|>
|
dac9 0% 1518.7 1510.7KB|>
|
hdisk4 0% 0.0 8.0KB|
|
hdisk5 0% 0.0 8.0KB| >
|
hdisk6 0% 0.0 0.0KB|
|
hdisk7 0% 0.0 0.0KB| >
|
hdisk8 0% 0.0 0.0KB|
|
hdisk9 0% 0.0 0.0KB|
|
hdisk10 0% 0.0 0.0KB| >
|
hdisk11 0% 0.0 8.0KB| >
|
hdisk12 0% 0.0 0.0KB|>
|
hdisk13 0% 0.0 0.0KB|
|
hdisk14 0% 0.0 24.0KB|
|
hdisk15 0% 0.0 0.0KB| >
|
hdisk16 0% 0.0 0.0KB| >
|
hdisk17 0% 0.0 0.0KB| >
|
hdisk18 55% 3804.8 0.0KB|RRRRRRRRRRRRRRRRRRRRRRRRRRRR
|RRRRRRRRRRRRRRRRRRRRRRRRRRR
hdisk19 6% 0.0 1079.1KB|WWWW
|
hdisk20 26% 5227.6 6202.7KB|WWWWWWWWRRRRRR
|RR
hdisk21 0% 0.0 0.0KB|
|
hdisk22 99% 2581.8
4436.2KB|WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWRRRRRRRRR RRRRRRRR|RWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWW
hdisk23 25% 1518.7 1510.7KB|WWWWWWWRRRRRRR
|WWWRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
hdisk24 31% 5523.3 5883.0KB|WWWWWWWWRRRRRRRR
|RRRRR>RRRRRRRRRRRRRRRRRRRR
hdisk25 0% 0.0 0.0KB|>
|

In article <%X0Oa.1041$Dr4.986@newssvr22.news.prodigy.com>, Joshua McAdams <joshmcadams@sbcglobal.net> wrote:
> I'm an end user working on a box with a fair amount of memory and a really
> low virtual memory setting in comparision to RAM: 4096.0MB VM to 24575.8MB
> RAM. Anyway, the box isn't remarkably consistant in performance from week
> to week so I'm trying to help troubleshoot what is wrong. In all of the
> docs that I've been looking over, the common sentement seems to be to have
> your virtual memory set to at least 1.2xRAM. We are nowhere near that, but

Ignore the "set virtual memory to..." stuff -- you just size it to match
your expected workload. From the numbers you posted, virtual memory is
*NOT* at all a problem... just about none of it is being used at all.
(40MB out of 17,000MB of virtual memory committed) This is what I would
expect from a well written application on a large system that uses its
memory as (essentially) one large cache to offset the cost of doing disk
I/O. (Disk I/O is generally at least 1,000 times slower than memory I/O!)

There's some CPU loading from your application but it's not severe, and
there's usually 40-60% idle time.

So all perf issues usually comes from one of these three areas:

CPU usage, memory usage, I/O performance.

CPU looks fine, memory looks fine. Your I/O looks really stressed -- I see
some hot spots... such as hdisk22 where the I/O wait is 99%+, and a few
other disks with a significant enough I/O wait.

It isn't so much a case of "Getting faster disks" -- hdisk22 sounds like a
possible RAID or volume set of some sort... it's more a case of spreading
out the I/Os across more spindles (disks) rather than slamming only one
physical volume (disk) so heavily.

There are 'features' of AIX where traditionally, all heavy I/Os done to a
volume group for the single jfs (journalled filesystem) log was done on a
single disk because that's how it's configured by default... which becomes
an hot spot if you have a *lot* of small filesystem changes to the point
where the single disk holding the jfs log can't keep up with all the I/O
transactions in a timely manner. (AIX 5.2 finally fixes this better.)

To fix that, may be a possible combination of retooling application a bit
better to spread out the I/O, as well as rethinking the underlying disk
layout strategy -- maybe add a fast-write RAID cache to the controller or
to break it out to some sort of non-RAID-5 array, or whatever.

With serious I/O wait, aka 'hot spots', it is possible for the entire
performance of the application to come to a screeching halt while it waits
for data to be returned from disk (or for disk to complete a write).

So I bet this is your performance issue. You just need to look at the
actual numbers you posted and think about what it's trying to tell you,
rather than trying to fudge a predetermined conclusion. So...

(The above paragraph was meant for your system admin. I admire you having
taken the initiative to research this issue.)

-Dan

(mailed to original poster and posted to USENET news, as a courtesy.)