SEO

Hi all. Newbie here. I'm not a Unix admin but I'm trying to convince
the guy that has this responsibility to investigate a possible a memory
issue on our AIX server. So I've been collecting some vmstat and sar
output over the last few days. I've been focusing on page-ins and
examples from some texts which say what is important and what is not.
Trouble is, even if I knew what to watch I wouldn't know what to get
excited about - and neither would our admin since it is new to him too.


So I have two basic questions that I'm hoping somebody can offer some
guidelines for:
1. What should I watch to determine if we're short on memory
2. What is something out of the ordinary or something to be concerned
about (is pi > 5 bad, or don't get excited until it's over 100, for
example)

I appreciate the fact that the quick answer to my questions is "It
depends". I'm just hoping somebody can tell me if what I'm seeing is
something to be concerned about.

We have a p660 with 4GB RAM, 2 CPUs. It hosts an ERP application
using Oracle and some web services for user access. Our one paging
space is also 4GB. Below is a sample of vmstat -I grabbed every 5
seconds with nothing special running, just typical activity:

kthr memory page faults cpu
time
-------- ----------- ------------------------ ------------ -----------
--------
r b p avm fre fi fo pi po fr sr in sy cs us sy id wa
hr mi se
1 0 0 597505 16917 0 3 0 0 0 0 298 4576 511 2 2 96 0
13:31:35
1 0 0 603100 11298 2 9 1 0 0 0 313 6979 861 7 3 88 1
13:31:40
0 1 0 603024 11375 1 92 0 0 0 0 379 2455 1014 2 6 89
3 13:31:45
0 0 0 603190 11207 0 3 0 0 0 0 304 2350 617 1 1 98 0
13:31:50
0 0 0 606037 8353 0 5 0 0 0 0 279 4413 1114 2 2 96
0 13:31:55
2 1 0 608709 5610 12 12 0 0 0 0 296 14467 3148 7 5 86
2 13:32:00
2 1 0 619730 2599 76 6 5 176 1695 3036 502 79307 1057 21 14 53
12 13:32:05
1 1 0 633214 67 8 5 25 410 2236 5885 691 333069 1173 23 39
1 37 13:32:10
1 1 0 630947 3749 43 5 13 56 341 632 381 100934 988 12 14 65
9 13:32:15
0 1 0 630921 3732 20 24 2 2 24 37 314 8764 1855 7 4 84
4 13:32:20
1 0 0 631010 3643 0 4 0 0 0 0 274 1724 416 0 0 99 0
13:32:25
0 0 0 630876 3768 1 6 0 0 0 0 272 1382 399 1 1 98 0
13:32:30
1 0 0 630742 3899 0 6 0 0 0 0 290 1595 519 1 1 98 0
13:32:35
0 0 0 630570 4069 0 4 0 0 0 0 288 1842 467 1 0 99 0
13:32:40
1 0 0 630571 4052 2 26 0 0 0 0 307 3143 1021 8 3 86
3 13:32:45
0 0 0 630575 4046 0 47 0 0 0 0 322 1247 396 0 3 96 0
13:32:50
1 0 0 630600 3994 4 7 0 0 0 0 294 3137 1028 11 1 88
1 13:32:55
3 0 0 628444 6097 10 4 0 0 0 0 284 2145 695 2 1 95 2
13:33:00
0 0 0 629526 5000 1 9 0 0 0 0 286 5779 578 2 2 96 0
13:33:05
1 1 0 639068 126 114 118 9 193 1123 1915 601 37815 13174 19 10
42 29 13:33:10
2 2 0 633948 126 976 2250 146 123 2109 6019 1374 12285 7024 22 11
14 53 13:33:15
2 1 0 626833 120 750 3249 2 16 2550 15039 942 7091 2967 10 12
33 45 13:33:20
2 0 0 595753 35013 1 6 3 0 769 1945 299 8583 3066 18 5 76
0 13:33:25
0 0 0 595752 35005 0 12 1 0 0 0 288 8319 1959 3 3 92
1 13:33:30
0 0 0 595752 34984 4 8 0 0 0 0 294 1418 458 1 1 98 0
13:33:35
0 0 0 595925 34797 2 5 0 0 0 0 303 2337 644 1 1 98 1
13:33:40
0 0 0 595943 34772 1 11 0 0 0 0 290 3241 948 2 1 95 1
13:33:45
1 0 0 595806 34851 11 73 0 1 0 0 378 4743 739 2 7 87 4
13:33:50
0 0 0 595806 34734 16 12 0 0 0 0 289 2368 1005 2 1 94
3 13:33:55
0 0 0 595806 34734 0 4 0 0 0 0 274 1600 335 0 0 99 0
13:34:00

Any input would be appreciated.

Greg wrote:
> So I have two basic questions that I'm hoping somebody can offer some
> guidelines for:
> 1. What should I watch to determine if we're short on memory
> 2. What is something out of the ordinary or something to be concerned
> about (is pi > 5 bad, or don't get excited until it's over 100, for
> example)

> We have a p660 with 4GB RAM, 2 CPUs. It hosts an ERP application
> using Oracle and some web services for user access. Our one paging
> space is also 4GB. Below is a sample of vmstat -I grabbed every 5
> seconds with nothing special running, just typical activity:

Greg,

First I will start by stating that I am not a AIX performance expert. I
have not taken the system performance analysis or tuning courses with
IBM, although I want to as soon as my company will send me on the
course.

On that note, there are a couple of things I noticed about the vmstat
info you published. At twopoints during the sampling, your free memory
allocation (fre) dropped to nearly zero and the CPU wait time (wa)
jumped up dramatically.

If this is typical of daily behaviour, then you defintely have a
problem. My guess is that your Oracle instances are using up all of the
available memory for some reason, which in turns causes the system to
start paging, and all the reading/writing to the disks are causing the
CPU to have to wait for the I/O to clear up.

You may want to go post this in the Oracle newsgroups and they can
suggest improvements. One thought I have is that Oracle (at least 8i
and 9i) may be spending its time trying to write archive log
information and that is affecting the performance of the system.

Try checking your alert logs in $ORACLE_HOME/rdbms/log and compare the
times that it writes a log to the times that you see the memory drop
and the CPU wait time go up. You may need an Oracle DBA rather than an
AIX Admin ;-)

Steve

On Thu, 14 Sep 2006 12:32:43 -0700, Greg wrote:

> Hi all. Newbie here. I'm not a Unix admin but I'm trying to convince the
> guy that has this responsibility to investigate a possible a memory issue
> on our AIX server. So I've been collecting some vmstat and sar output
> over the last few days. I've been focusing on page-ins and examples from
> some texts which say what is important and what is not. Trouble is, even
> if I knew what to watch I wouldn't know what to get excited about - and
> neither would our admin since it is new to him too.
>
>
> So I have two basic questions that I'm hoping somebody can offer some
> guidelines for:
> 1. What should I watch to determine if we're short on memory 2. What is
> something out of the ordinary or something to be concerned about (is pi >
> 5 bad, or don't get excited until it's over 100, for example)
>
> I appreciate the fact that the quick answer to my questions is "It
> depends". I'm just hoping somebody can tell me if what I'm seeing is
> something to be concerned about.
>
> We have a p660 with 4GB RAM, 2 CPUs. It hosts an ERP application using
> Oracle and some web services for user access. Our one paging space is
> also 4GB. Below is a sample of vmstat -I grabbed every 5 seconds with
> nothing special running, just typical activity:
>
> kthr memory page faults cpu
> time
> -------- ----------- ------------------------ ------------ -----------
> --------
> r b p avm fre fi fo pi po fr sr in sy cs us sy id wa
> hr mi se
> 1 0 0 597505 16917 0 3 0 0 0 0 298 4576 511 2 2 96 0
> 13:31:35
> 1 0 0 603100 11298 2 9 1 0 0 0 313 6979 861 7 3 88 1
> 13:31:40
> 0 1 0 603024 11375 1 92 0 0 0 0 379 2455 1014 2 6 89
> 3 13:31:45
> 0 0 0 603190 11207 0 3 0 0 0 0 304 2350 617 1 1 98 0
> 13:31:50
> 0 0 0 606037 8353 0 5 0 0 0 0 279 4413 1114 2 2 96
> 0 13:31:55
> 2 1 0 608709 5610 12 12 0 0 0 0 296 14467 3148 7 5 86
> 2 13:32:00
> 2 1 0 619730 2599 76 6 5 176 1695 3036 502 79307 1057 21 14 53
> 12 13:32:05
> 1 1 0 633214 67 8 5 25 410 2236 5885 691 333069 1173 23 39
> 1 37 13:32:10
> 1 1 0 630947 3749 43 5 13 56 341 632 381 100934 988 12 14 65
> 9 13:32:15
> 0 1 0 630921 3732 20 24 2 2 24 37 314 8764 1855 7 4 84
> 4 13:32:20
> 1 0 0 631010 3643 0 4 0 0 0 0 274 1724 416 0 0 99 0
> 13:32:25
> 0 0 0 630876 3768 1 6 0 0 0 0 272 1382 399 1 1 98 0
> 13:32:30
> 1 0 0 630742 3899 0 6 0 0 0 0 290 1595 519 1 1 98 0
> 13:32:35
> 0 0 0 630570 4069 0 4 0 0 0 0 288 1842 467 1 0 99 0
> 13:32:40
> 1 0 0 630571 4052 2 26 0 0 0 0 307 3143 1021 8 3 86
> 3 13:32:45
> 0 0 0 630575 4046 0 47 0 0 0 0 322 1247 396 0 3 96 0
> 13:32:50
> 1 0 0 630600 3994 4 7 0 0 0 0 294 3137 1028 11 1 88
> 1 13:32:55
> 3 0 0 628444 6097 10 4 0 0 0 0 284 2145 695 2 1 95 2
> 13:33:00
> 0 0 0 629526 5000 1 9 0 0 0 0 286 5779 578 2 2 96 0
> 13:33:05
> 1 1 0 639068 126 114 118 9 193 1123 1915 601 37815 13174 19 10
> 42 29 13:33:10
> 2 2 0 633948 126 976 2250 146 123 2109 6019 1374 12285 7024 22 11
> 14 53 13:33:15
> 2 1 0 626833 120 750 3249 2 16 2550 15039 942 7091 2967 10 12
> 33 45 13:33:20
> 2 0 0 595753 35013 1 6 3 0 769 1945 299 8583 3066 18 5 76
> 0 13:33:25
> 0 0 0 595752 35005 0 12 1 0 0 0 288 8319 1959 3 3 92
> 1 13:33:30
> 0 0 0 595752 34984 4 8 0 0 0 0 294 1418 458 1 1 98 0
> 13:33:35
> 0 0 0 595925 34797 2 5 0 0 0 0 303 2337 644 1 1 98 1
> 13:33:40
> 0 0 0 595943 34772 1 11 0 0 0 0 290 3241 948 2 1 95 1
> 13:33:45
> 1 0 0 595806 34851 11 73 0 1 0 0 378 4743 739 2 7 87 4
> 13:33:50
> 0 0 0 595806 34734 16 12 0 0 0 0 289 2368 1005 2 1 94
> 3 13:33:55
> 0 0 0 595806 34734 0 4 0 0 0 0 274 1600 335 0 0 99 0
> 13:34:00
>
> Any input would be appreciated.

That does look to be quite a lot of paging and these days memory is so
cheap you should aim to page very litte. Nmon (not shipped as part of AIX
but downloadable from IBM) will tell you more, but you have to be root to
run it - you say you are not the admin, do you have root?

What is your paging space occupancy - lsps -s? You don't have to be root
to run this.

IMO given what you've said your admin is deficient in his duties, if
someone reports a potential paging/memory problem to me at least I
investigate it to see if a problem really exists.

4GB doesn't sounsd much for an Oracle/ERP system to me, Oracle and every
ERP app I have seen are very memory hungry. But you give no indication of
the scale of the operation. How many users? How big is the database?

Your wait time is getting high too and it doesn't always correalate with
the paging activity. Iostat will give an indication of which disks are
hot. It's possible that some Oracle tuning is needed. Do you have a decent
DBA?

Is the application unacceptably slow?

Regards, Ian

Greg,

Here is one of our Oracle systems that is used for data entry,
accounting, inventory, and various reporting tasks like sales analysis
and stock reports.

RS/6000 p550, 8 GB Memory, 4 processors

This was designed to hold a max of 180 users and is running 6 instances
(2 production, 2 training, 2 month-end reporting). It seems to be
running Oracle 9.2 well at the present time.

Our production init.ora file had these settings if it helps at all. I'm
not an Oracle DBA so could not tell you if these will work well for
you. But it may give you something to compare to.

db_files = 80
db_file_multiblock_read_count = 1000
db_block_buffers = 45000
db_block_size=4096
shared_pool_size = 300000000
log_checkpoint_interval = 10000
processes=600
parallel_max_servers = 16
log_buffer=64000
timed_statistics=false
max_dump_file_size = 10240
transactions = 200
transactions_per_rollback_segment = 40
open_cursors=2500
optimizer_index_caching=50
dml_locks=90000

>
> Any input would be appreciated.
>

One memory problem very common on AIX/Oracle systems is "double-caching" of
database pages. If the Oracle database is on a JFS/JFS2 filesystem, the LVM will
cache the database I/O, even while Oracle is caching them in his SGA. You can
eliminate the LVM caching (and thus reclaim real memory) by either creating the
Oracle database on raw volumes, using CIO (if at the correct level of AIX and
Oracle), or tuning the LVM file caching parameters (maxperm%, maxclient%). I
believe you can find pertinent articles on Oracle Metalink, and I think I even
saw an IBM whitepaper on the subject, although it uses DB2 as an example.

--
steve stackwick
sstackwick@jasi.com
---------------------------------------------------------------------------
The social dynamics of the net are a direct consequence of the fact that
nobody has yet developed a Remote Strangulation Protocol. -- Larry Wall

Lets start with the basics .. I can see your AVM rises to somewhere
around 633948 x 4k pages, lets do the math

633948 * 4096
2596651008
2596651008/1024/1024
2476

Just under 2.5GB, you have plenty of memory left, you need to some out
your paging .. depending on you level of AIX you have some options ..
what level of AIX you running ?


You also may want to point your AIX Admin at these docs

http://www-03.ibm.com/support/techdo...9?OpenDocument
http://www-03.ibm.com/servers/aix/wh...b_perf_aix.pdf

HTH
Mark Taylor

Steve:

That's a good idea. I'll check the alert logs for Oracle. Another
sampling from vmstat shows me that every night when Oracle does it's
backup the (fre) drops and stays low and the (wa) stays high for about
40 minutes.

Thanks for the feedback.
-Greg

steven_nospam at Yahoo! Canada wrote:
> Greg wrote:
> > So I have two basic questions that I'm hoping somebody can offer some
> > guidelines for:
> > 1. What should I watch to determine if we're short on memory
> > 2. What is something out of the ordinary or something to be concerned
> > about (is pi > 5 bad, or don't get excited until it's over 100, for
> > example)
>
> > We have a p660 with 4GB RAM, 2 CPUs. It hosts an ERP application
> > using Oracle and some web services for user access. Our one paging
> > space is also 4GB. Below is a sample of vmstat -I grabbed every 5
> > seconds with nothing special running, just typical activity:
>
> Greg,
>
> First I will start by stating that I am not a AIX performance expert. I
> have not taken the system performance analysis or tuning courses with
> IBM, although I want to as soon as my company will send me on the
> course.
>
> On that note, there are a couple of things I noticed about the vmstat
> info you published. At twopoints during the sampling, your free memory
> allocation (fre) dropped to nearly zero and the CPU wait time (wa)
> jumped up dramatically.
>
> If this is typical of daily behaviour, then you defintely have a
> problem. My guess is that your Oracle instances are using up all of the
> available memory for some reason, which in turns causes the system to
> start paging, and all the reading/writing to the disks are causing the
> CPU to have to wait for the I/O to clear up.
>
> You may want to go post this in the Oracle newsgroups and they can
> suggest improvements. One thought I have is that Oracle (at least 8i
> and 9i) may be spending its time trying to write archive log
> information and that is affecting the performance of the system.
>
> Try checking your alert logs in $ORACLE_HOME/rdbms/log and compare the
> times that it writes a log to the times that you see the memory drop
> and the CPU wait time go up. You may need an Oracle DBA rather than an
> AIX Admin ;-)
>
> Steve

Ian:

I have root access and check the paging space at the top of every hour
via script. At the time of this log being posted it looked like this:

Total Paging Space Percent Used
2048MB 78%

Since then the paging space has been doubled to 4096MB.

We have about 80 active users of the system at any one time and the
database itself is about 35 GB. Users have not been reporting any
general slowness. My suspicions came from an outage a few weeks ago
caused by a javacore error file which had an "otofmemory" message in
it.

Based upon your post and a few of the others I'll be looking towards my
DBA for some tuning opportunities (no pun intended!).

Thanks for your input.
-Greg

Ian Northeast wrote:
> On Thu, 14 Sep 2006 12:32:43 -0700, Greg wrote:
>
> > Hi all. Newbie here. I'm not a Unix admin but I'm trying to convince the
> > guy that has this responsibility to investigate a possible a memory issue
> > on our AIX server. So I've been collecting some vmstat and sar output
> > over the last few days. I've been focusing on page-ins and examples from
> > some texts which say what is important and what is not. Trouble is, even
> > if I knew what to watch I wouldn't know what to get excited about - and
> > neither would our admin since it is new to him too.
> >
> >
> > So I have two basic questions that I'm hoping somebody can offer some
> > guidelines for:
> > 1. What should I watch to determine if we're short on memory 2. What is
> > something out of the ordinary or something to be concerned about (is pi >
> > 5 bad, or don't get excited until it's over 100, for example)
> >
> > I appreciate the fact that the quick answer to my questions is "It
> > depends". I'm just hoping somebody can tell me if what I'm seeing is
> > something to be concerned about.
> >
> > We have a p660 with 4GB RAM, 2 CPUs. It hosts an ERP application using
> > Oracle and some web services for user access. Our one paging space is
> > also 4GB. Below is a sample of vmstat -I grabbed every 5 seconds with
> > nothing special running, just typical activity:
> >
> > kthr memory page faults cpu
> > time
> > -------- ----------- ------------------------ ------------ -----------
> > --------
> > r b p avm fre fi fo pi po fr sr in sy cs us sy id wa
> > hr mi se
> > 1 0 0 597505 16917 0 3 0 0 0 0 298 4576 511 2 2 96 0
> > 13:31:35
> > 1 0 0 603100 11298 2 9 1 0 0 0 313 6979 861 7 3 88 1
> > 13:31:40
> > 0 1 0 603024 11375 1 92 0 0 0 0 379 2455 1014 2 6 89
> > 3 13:31:45
> > 0 0 0 603190 11207 0 3 0 0 0 0 304 2350 617 1 1 98 0
> > 13:31:50
> > 0 0 0 606037 8353 0 5 0 0 0 0 279 4413 1114 2 2 96
> > 0 13:31:55
> > 2 1 0 608709 5610 12 12 0 0 0 0 296 14467 3148 7 5 86
> > 2 13:32:00
> > 2 1 0 619730 2599 76 6 5 176 1695 3036 502 79307 1057 21 14 53
> > 12 13:32:05
> > 1 1 0 633214 67 8 5 25 410 2236 5885 691 333069 1173 23 39
> > 1 37 13:32:10
> > 1 1 0 630947 3749 43 5 13 56 341 632 381 100934 988 12 14 65
> > 9 13:32:15
> > 0 1 0 630921 3732 20 24 2 2 24 37 314 8764 1855 7 4 84
> > 4 13:32:20
> > 1 0 0 631010 3643 0 4 0 0 0 0 274 1724 416 0 0 99 0
> > 13:32:25
> > 0 0 0 630876 3768 1 6 0 0 0 0 272 1382 399 1 1 98 0
> > 13:32:30
> > 1 0 0 630742 3899 0 6 0 0 0 0 290 1595 519 1 1 98 0
> > 13:32:35
> > 0 0 0 630570 4069 0 4 0 0 0 0 288 1842 467 1 0 99 0
> > 13:32:40
> > 1 0 0 630571 4052 2 26 0 0 0 0 307 3143 1021 8 3 86
> > 3 13:32:45
> > 0 0 0 630575 4046 0 47 0 0 0 0 322 1247 396 0 3 96 0
> > 13:32:50
> > 1 0 0 630600 3994 4 7 0 0 0 0 294 3137 1028 11 1 88
> > 1 13:32:55
> > 3 0 0 628444 6097 10 4 0 0 0 0 284 2145 695 2 1 95 2
> > 13:33:00
> > 0 0 0 629526 5000 1 9 0 0 0 0 286 5779 578 2 2 96 0
> > 13:33:05
> > 1 1 0 639068 126 114 118 9 193 1123 1915 601 37815 13174 19 10
> > 42 29 13:33:10
> > 2 2 0 633948 126 976 2250 146 123 2109 6019 1374 12285 7024 22 11
> > 14 53 13:33:15
> > 2 1 0 626833 120 750 3249 2 16 2550 15039 942 7091 2967 10 12
> > 33 45 13:33:20
> > 2 0 0 595753 35013 1 6 3 0 769 1945 299 8583 3066 18 5 76
> > 0 13:33:25
> > 0 0 0 595752 35005 0 12 1 0 0 0 288 8319 1959 3 3 92
> > 1 13:33:30
> > 0 0 0 595752 34984 4 8 0 0 0 0 294 1418 458 1 1 98 0
> > 13:33:35
> > 0 0 0 595925 34797 2 5 0 0 0 0 303 2337 644 1 1 98 1
> > 13:33:40
> > 0 0 0 595943 34772 1 11 0 0 0 0 290 3241 948 2 1 95 1
> > 13:33:45
> > 1 0 0 595806 34851 11 73 0 1 0 0 378 4743 739 2 7 87 4
> > 13:33:50
> > 0 0 0 595806 34734 16 12 0 0 0 0 289 2368 1005 2 1 94
> > 3 13:33:55
> > 0 0 0 595806 34734 0 4 0 0 0 0 274 1600 335 0 0 99 0
> > 13:34:00
> >
> > Any input would be appreciated.
>
> That does look to be quite a lot of paging and these days memory is so
> cheap you should aim to page very litte. Nmon (not shipped as part of AIX
> but downloadable from IBM) will tell you more, but you have to be root to
> run it - you say you are not the admin, do you have root?
>
> What is your paging space occupancy - lsps -s? You don't have to be root
> to run this.
>
> IMO given what you've said your admin is deficient in his duties, if
> someone reports a potential paging/memory problem to me at least I
> investigate it to see if a problem really exists.
>
> 4GB doesn't sounsd much for an Oracle/ERP system to me, Oracle and every
> ERP app I have seen are very memory hungry. But you give no indication of
> the scale of the operation. How many users? How big is the database?
>
> Your wait time is getting high too and it doesn't always correalate with
> the paging activity. Iostat will give an indication of which disks are
> hot. It's possible that some Oracle tuning is needed. Do you have a decent
> DBA?
>
> Is the application unacceptably slow?
>
> Regards, Ian

Mark:

Our OS level is 5.2.0.0 or 5200-08, I'm not sure how it is typically
reported in this forum.

We'll check those articles, thanks.
-Greg

Mark Taylor wrote:
> Lets start with the basics .. I can see your AVM rises to somewhere
> around 633948 x 4k pages, lets do the math
>
> 633948 * 4096
> 2596651008
> 2596651008/1024/1024
> 2476
>
> Just under 2.5GB, you have plenty of memory left, you need to some out
> your paging .. depending on you level of AIX you have some options ..
> what level of AIX you running ?
>
>
> You also may want to point your AIX Admin at these docs
>
> http://www-03.ibm.com/support/techdo...9?OpenDocument
> http://www-03.ibm.com/servers/aix/wh...b_perf_aix.pdf
>
> HTH
> Mark Taylor

I also should have mentioned that our web services are apache 1.3,
tomcat 4.06 and RMI. All of the users access the system via these
services.

-Greg

Greg wrote:
> Hi all. Newbie here. I'm not a Unix admin but I'm trying to convince
> the guy that has this responsibility to investigate a possible a memory
> issue on our AIX server. So I've been collecting some vmstat and sar
> output over the last few days. I've been focusing on page-ins and
> examples from some texts which say what is important and what is not.
> Trouble is, even if I knew what to watch I wouldn't know what to get
> excited about - and neither would our admin since it is new to him too.
>
>
> So I have two basic questions that I'm hoping somebody can offer some
> guidelines for:
> 1. What should I watch to determine if we're short on memory
> 2. What is something out of the ordinary or something to be concerned
> about (is pi > 5 bad, or don't get excited until it's over 100, for
> example)
>
> I appreciate the fact that the quick answer to my questions is "It
> depends". I'm just hoping somebody can tell me if what I'm seeing is
> something to be concerned about.
>
> We have a p660 with 4GB RAM, 2 CPUs. It hosts an ERP application
> using Oracle and some web services for user access. Our one paging
> space is also 4GB. Below is a sample of vmstat -I grabbed every 5
> seconds with nothing special running, just typical activity:
>
> kthr memory page faults cpu
> time
> -------- ----------- ------------------------ ------------ -----------
> --------
> r b p avm fre fi fo pi po fr sr in sy cs us sy id wa
> hr mi se
> 1 0 0 597505 16917 0 3 0 0 0 0 298 4576 511 2 2 96 0
> 13:31:35
> 1 0 0 603100 11298 2 9 1 0 0 0 313 6979 861 7 3 88 1
> 13:31:40
> 0 1 0 603024 11375 1 92 0 0 0 0 379 2455 1014 2 6 89
> 3 13:31:45
> 0 0 0 603190 11207 0 3 0 0 0 0 304 2350 617 1 1 98 0
> 13:31:50
> 0 0 0 606037 8353 0 5 0 0 0 0 279 4413 1114 2 2 96
> 0 13:31:55
> 2 1 0 608709 5610 12 12 0 0 0 0 296 14467 3148 7 5 86
> 2 13:32:00
> 2 1 0 619730 2599 76 6 5 176 1695 3036 502 79307 1057 21 14 53
> 12 13:32:05
> 1 1 0 633214 67 8 5 25 410 2236 5885 691 333069 1173 23 39
> 1 37 13:32:10
> 1 1 0 630947 3749 43 5 13 56 341 632 381 100934 988 12 14 65
> 9 13:32:15
> 0 1 0 630921 3732 20 24 2 2 24 37 314 8764 1855 7 4 84
> 4 13:32:20
> 1 0 0 631010 3643 0 4 0 0 0 0 274 1724 416 0 0 99 0
> 13:32:25
> 0 0 0 630876 3768 1 6 0 0 0 0 272 1382 399 1 1 98 0
> 13:32:30
> 1 0 0 630742 3899 0 6 0 0 0 0 290 1595 519 1 1 98 0
> 13:32:35
> 0 0 0 630570 4069 0 4 0 0 0 0 288 1842 467 1 0 99 0
> 13:32:40
> 1 0 0 630571 4052 2 26 0 0 0 0 307 3143 1021 8 3 86
> 3 13:32:45
> 0 0 0 630575 4046 0 47 0 0 0 0 322 1247 396 0 3 96 0
> 13:32:50
> 1 0 0 630600 3994 4 7 0 0 0 0 294 3137 1028 11 1 88
> 1 13:32:55
> 3 0 0 628444 6097 10 4 0 0 0 0 284 2145 695 2 1 95 2
> 13:33:00
> 0 0 0 629526 5000 1 9 0 0 0 0 286 5779 578 2 2 96 0
> 13:33:05
> 1 1 0 639068 126 114 118 9 193 1123 1915 601 37815 13174 19 10
> 42 29 13:33:10
> 2 2 0 633948 126 976 2250 146 123 2109 6019 1374 12285 7024 22 11
> 14 53 13:33:15
> 2 1 0 626833 120 750 3249 2 16 2550 15039 942 7091 2967 10 12
> 33 45 13:33:20
> 2 0 0 595753 35013 1 6 3 0 769 1945 299 8583 3066 18 5 76
> 0 13:33:25
> 0 0 0 595752 35005 0 12 1 0 0 0 288 8319 1959 3 3 92
> 1 13:33:30
> 0 0 0 595752 34984 4 8 0 0 0 0 294 1418 458 1 1 98 0
> 13:33:35
> 0 0 0 595925 34797 2 5 0 0 0 0 303 2337 644 1 1 98 1
> 13:33:40
> 0 0 0 595943 34772 1 11 0 0 0 0 290 3241 948 2 1 95 1
> 13:33:45
> 1 0 0 595806 34851 11 73 0 1 0 0 378 4743 739 2 7 87 4
> 13:33:50
> 0 0 0 595806 34734 16 12 0 0 0 0 289 2368 1005 2 1 94
> 3 13:33:55
> 0 0 0 595806 34734 0 4 0 0 0 0 274 1600 335 0 0 99 0
> 13:34:00
>
> Any input would be appreciated.