In article <%X0Oa.1041$Dr4.986@newssvr22.news.prodigy.com>, Joshua McAdams <joshmcadams@sbcglobal.net> wrote:
> I'm an end user working on a box with a fair amount of memory and a really
> low virtual memory setting in comparision to RAM: 4096.0MB VM to 24575.8MB
> RAM. Anyway, the box isn't remarkably consistant in performance from week
> to week so I'm trying to help troubleshoot what is wrong. In all of the
> docs that I've been looking over, the common sentement seems to be to have
> your virtual memory set to at least 1.2xRAM. We are nowhere near that, but

Ignore the "set virtual memory to..." stuff -- you just size it to match
your expected workload. From the numbers you posted, virtual memory is
*NOT* at all a problem... just about none of it is being used at all.
(40MB out of 17,000MB of virtual memory committed) This is what I would
expect from a well written application on a large system that uses its
memory as (essentially) one large cache to offset the cost of doing disk
I/O. (Disk I/O is generally at least 1,000 times slower than memory I/O!)

There's some CPU loading from your application but it's not severe, and
there's usually 40-60% idle time.

So all perf issues usually comes from one of these three areas:

CPU usage, memory usage, I/O performance.

CPU looks fine, memory looks fine. Your I/O looks really stressed -- I see
some hot spots... such as hdisk22 where the I/O wait is 99%+, and a few
other disks with a significant enough I/O wait.

It isn't so much a case of "Getting faster disks" -- hdisk22 sounds like a
possible RAID or volume set of some sort... it's more a case of spreading
out the I/Os across more spindles (disks) rather than slamming only one
physical volume (disk) so heavily.

There are 'features' of AIX where traditionally, all heavy I/Os done to a
volume group for the single jfs (journalled filesystem) log was done on a
single disk because that's how it's configured by default... which becomes
an hot spot if you have a *lot* of small filesystem changes to the point
where the single disk holding the jfs log can't keep up with all the I/O
transactions in a timely manner. (AIX 5.2 finally fixes this better.)

To fix that, may be a possible combination of retooling application a bit
better to spread out the I/O, as well as rethinking the underlying disk
layout strategy -- maybe add a fast-write RAID cache to the controller or
to break it out to some sort of non-RAID-5 array, or whatever.

With serious I/O wait, aka 'hot spots', it is possible for the entire
performance of the application to come to a screeching halt while it waits
for data to be returned from disk (or for disk to complete a write).

So I bet this is your performance issue. You just need to look at the
actual numbers you posted and think about what it's trying to tell you,
rather than trying to fudge a predetermined conclusion. So...

(The above paragraph was meant for your system admin. I admire you having
taken the initiative to research this issue.)

-Dan

(mailed to original poster and posted to USENET news, as a courtesy.)