Greg wrote:
> Ian:
>
> I have root access and check the paging space at the top of every hour
> via script. At the time of this log being posted it looked like this:
>
> Total Paging Space Percent Used
> 2048MB 78%
>
> Since then the paging space has been doubled to 4096MB.
>
> We have about 80 active users of the system at any one time and the
> database itself is about 35 GB. Users have not been reporting any
> general slowness. My suspicions came from an outage a few weeks ago
> caused by a javacore error file which had an "otofmemory" message in
> it.
>
> Based upon your post and a few of the others I'll be looking towards my
> DBA for some tuning opportunities (no pun intended!).
>
> Thanks for your input.
> -Greg
>
> Ian Northeast wrote:
>
>>On Thu, 14 Sep 2006 12:32:43 -0700, Greg wrote:
>>
>>
>>>Hi all. Newbie here. I'm not a Unix admin but I'm trying to convince the
>>>guy that has this responsibility to investigate a possible a memory issue
>>>on our AIX server. So I've been collecting some vmstat and sar output
>>>over the last few days. I've been focusing on page-ins and examples from
>>>some texts which say what is important and what is not. Trouble is, even
>>>if I knew what to watch I wouldn't know what to get excited about - and
>>>neither would our admin since it is new to him too.
>>>
>>>
>>>So I have two basic questions that I'm hoping somebody can offer some
>>>guidelines for:
>>>1. What should I watch to determine if we're short on memory 2. What is
>>>something out of the ordinary or something to be concerned about (is pi >
>>>5 bad, or don't get excited until it's over 100, for example)
>>>
>>>I appreciate the fact that the quick answer to my questions is "It
>>>depends". I'm just hoping somebody can tell me if what I'm seeing is
>>>something to be concerned about.
>>>
>>>We have a p660 with 4GB RAM, 2 CPUs. It hosts an ERP application using
>>>Oracle and some web services for user access. Our one paging space is
>>>also 4GB. Below is a sample of vmstat -I grabbed every 5 seconds with
>>>nothing special running, just typical activity:
>>>
>>> kthr memory page faults cpu
>>> time
>>>-------- ----------- ------------------------ ------------ -----------
>>>--------
>>> r b p avm fre fi fo pi po fr sr in sy cs us sy id wa
>>>hr mi se
>>> 1 0 0 597505 16917 0 3 0 0 0 0 298 4576 511 2 2 96 0
>>>13:31:35
>>> 1 0 0 603100 11298 2 9 1 0 0 0 313 6979 861 7 3 88 1
>>>13:31:40
>>> 0 1 0 603024 11375 1 92 0 0 0 0 379 2455 1014 2 6 89
>>>3 13:31:45
>>> 0 0 0 603190 11207 0 3 0 0 0 0 304 2350 617 1 1 98 0
>>>13:31:50
>>> 0 0 0 606037 8353 0 5 0 0 0 0 279 4413 1114 2 2 96
>>>0 13:31:55
>>> 2 1 0 608709 5610 12 12 0 0 0 0 296 14467 3148 7 5 86
>>>2 13:32:00
>>> 2 1 0 619730 2599 76 6 5 176 1695 3036 502 79307 1057 21 14 53
>>>12 13:32:05
>>> 1 1 0 633214 67 8 5 25 410 2236 5885 691 333069 1173 23 39
>>>1 37 13:32:10
>>> 1 1 0 630947 3749 43 5 13 56 341 632 381 100934 988 12 14 65
>>>9 13:32:15
>>> 0 1 0 630921 3732 20 24 2 2 24 37 314 8764 1855 7 4 84
>>>4 13:32:20
>>> 1 0 0 631010 3643 0 4 0 0 0 0 274 1724 416 0 0 99 0
>>>13:32:25
>>> 0 0 0 630876 3768 1 6 0 0 0 0 272 1382 399 1 1 98 0
>>>13:32:30
>>> 1 0 0 630742 3899 0 6 0 0 0 0 290 1595 519 1 1 98 0
>>>13:32:35
>>> 0 0 0 630570 4069 0 4 0 0 0 0 288 1842 467 1 0 99 0
>>>13:32:40
>>> 1 0 0 630571 4052 2 26 0 0 0 0 307 3143 1021 8 3 86
>>>3 13:32:45
>>> 0 0 0 630575 4046 0 47 0 0 0 0 322 1247 396 0 3 96 0
>>>13:32:50
>>> 1 0 0 630600 3994 4 7 0 0 0 0 294 3137 1028 11 1 88
>>>1 13:32:55
>>> 3 0 0 628444 6097 10 4 0 0 0 0 284 2145 695 2 1 95 2
>>>13:33:00
>>> 0 0 0 629526 5000 1 9 0 0 0 0 286 5779 578 2 2 96 0
>>>13:33:05
>>> 1 1 0 639068 126 114 118 9 193 1123 1915 601 37815 13174 19 10
>>>42 29 13:33:10
>>> 2 2 0 633948 126 976 2250 146 123 2109 6019 1374 12285 7024 22 11
>>>14 53 13:33:15
>>> 2 1 0 626833 120 750 3249 2 16 2550 15039 942 7091 2967 10 12
>>>33 45 13:33:20
>>> 2 0 0 595753 35013 1 6 3 0 769 1945 299 8583 3066 18 5 76
>>>0 13:33:25
>>> 0 0 0 595752 35005 0 12 1 0 0 0 288 8319 1959 3 3 92
>>>1 13:33:30
>>> 0 0 0 595752 34984 4 8 0 0 0 0 294 1418 458 1 1 98 0
>>>13:33:35
>>> 0 0 0 595925 34797 2 5 0 0 0 0 303 2337 644 1 1 98 1
>>>13:33:40
>>> 0 0 0 595943 34772 1 11 0 0 0 0 290 3241 948 2 1 95 1
>>>13:33:45
>>> 1 0 0 595806 34851 11 73 0 1 0 0 378 4743 739 2 7 87 4
>>>13:33:50
>>> 0 0 0 595806 34734 16 12 0 0 0 0 289 2368 1005 2 1 94
>>>3 13:33:55
>>> 0 0 0 595806 34734 0 4 0 0 0 0 274 1600 335 0 0 99 0
>>>13:34:00
>>>
>>>Any input would be appreciated.
>>
>>That does look to be quite a lot of paging and these days memory is so
>>cheap you should aim to page very litte. Nmon (not shipped as part of AIX
>>but downloadable from IBM) will tell you more, but you have to be root to
>>run it - you say you are not the admin, do you have root?
>>
>>What is your paging space occupancy - lsps -s? You don't have to be root
>>to run this.
>>
>>IMO given what you've said your admin is deficient in his duties, if
>>someone reports a potential paging/memory problem to me at least I
>>investigate it to see if a problem really exists.
>>
>>4GB doesn't sounsd much for an Oracle/ERP system to me, Oracle and every
>>ERP app I have seen are very memory hungry. But you give no indication of
>>the scale of the operation. How many users? How big is the database?
>>
>>Your wait time is getting high too and it doesn't always correalate with
>>the paging activity. Iostat will give an indication of which disks are
>>hot. It's possible that some Oracle tuning is needed. Do you have a decent
>>DBA?
>>
>>Is the application unacceptably slow?
>>
>>Regards, Ian
>
>

The "OutOfMemory" from a java process is not necessarily a system memory
problem. It could be an application threshold being reached, or the max
heap size of the originating process is set too low. Yes, your system
is low on memory, but one issue may not be related to the other here.
Possibly, but not definitely.

>> Our OS level is 5.2.0.0 or 5200-08, I'm not sure how it is typically

OK, then the vmo -p -o lru_file_repage=0 option is active in 5200-08,
so you should tune that.

The doc link I gave you explains all ..

HTH
Mark Taylor