SEO

Hi all ,

I have an AIX server with the following configuration

System Model: IBM,7026-M80
Machine Serial Number: 107296F
Processor Type: PowerPC_RS64-III
Number Of Processors: 8
Processor Clock Speed: 500 MHz
CPU Type: 64-bit

We now an performance issue related to cpu which i can see it from the
output of vmstat.


# vmstat 2
System Configuration: lcpu=8 mem=8192MB
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
5 1 1524920 2063 0 4 5 43 147 0 1248 32226 2329 28 12 59
1
4 0 1523596 3904 0 0 25 265 357 0 2455 36837 4830 38 25 36
0
7 0 1525917 1575 0 0 0 0 0 0 1540 56658 5288 26 41 33
0
7 0 1524655 2780 0 0 0 0 0 0 1305 57006 4944 21 41 37
1
9 2 1526162 997 0 0 0 0 0 0 1792 43574 3002 77 21 2
1
20 1 1527498 122 0 0 24 261 379 0 1572 15606 2167 97 3 0
0
18 0 1524855 3485 0 5 29 366 557 0 1387 18862 2337 94 6 0
0
15 0 1527585 767 0 0 0 0 0 0 1271 20509 1898 91 9 0
0
9 0 1528297 116 0 0 1 32 36 0 1906 11855 2279 94 3 3
0
10 0 1527763 1304 0 0 11 260 314 0 2089 32441 3290 69 19 11
0
3 0 1525732 3339 0 0 0 0 0 0 1233 13562 2311 29 6 64
1
5 2 1527906 1065 0 0 0 0 0 0 1215 28048 2691 50 16 31
3
5 0 1527310 1636 0 0 0 0 0 0 1660 15754 2824 57 7 35
0
7 0 1524965 3960 0 0 0 0 0 0 1314 28249 2848 54 17 29
0
2 1 1527469 1445 0 0 0 0 0 0 1329 10786 1852 38 6 56
0
2 0 1526365 2495 0 5 0 0 0 0 1879 33815 4166 25 20 54
1
2 0 1525050 3786 0 0 0 0 0 0 1727 17780 3244 17 10 72
0
7 0 1527304 1573 0 0 0 0 0 0 1308 24240 2627 33 15 50
1
5 0 1526761 2065 0 1 0 0 0 0 2704 23377 4970 34 12 53
0


also i can see form the sar command that the runq is 100% full.


# sar -q 2 10

AIX inwsp01 2 5 0007296F4C00 11/08/06

System Configuration: lcpu=8

11:11:24 runq-sz %runocc swpq-sz %swpocc
11:11:26 2.5 100
11:11:28 6.0 100 3.0 50
11:11:30 10.0 100
11:11:32 5.0 100
11:11:34 9.0 100
11:11:36 3.0 100
11:11:38 5.0 100
11:11:41 3.0 67
11:11:43 4.5 100
11:11:45 6.5 100 1.0 50

Pls advice me why this is happening or is that we need to tune
something.

I have 7 different application running and i could a process taking 21%
of cpu , the next process 10% ........ from the ps -ef output.

Also can you pls tell me in detail the C value in a ps -ef command as
the first process that takes the 21% of cpu has a C value of 127 , 57
.......

Regards
Krishna

Vmstat is not enough to precisely pinpoint the root cause.

You have 8 CPUs, and during perf problems, there are more that 20
threads running.
There are also very few cycles spent on IO operations. Little paging.

I'd identify processes, that are responsible for these 100% CPU peaks.
Use following commands:
ps -ef | sort +3 -r | head -n 5
ps auxwww | head -n 5

I'd also advise to measure application execution time during peaks (if
possible to do so):
time ./app_name;./pdt_report 3
This command will run app_name, and print stats about how long has it
taken and how much CPU has it used.

Next step would be to examine their configs, algorithms (if docs
available), looking for possible bottlenecks. Or call application
support.

I hope this will help, it's based on perf tuning redbook, and saved my
butt quite a few times. :-)