Here is the output from iostat -d 4
Disks: % tm_act Kbps tps
Kb_read Kb_wrtn

hdisk5 0.0 0.0 0.0
1784 1436

hdisk3 0.3 5.5 0.9
203802 603651

hdisk4 0.0 0.0 0.0
0 0

hdisk2 0.4 5.2 1.0
148531 603175

hdisk1 0.0 0.0 0.0
0 0

hdisk6 0.0 0.0 0.0
2296 1816

hdisk0 7.8 514.0 37.2
71408704 3603944


I think there is no extra ordinary disk activity. Here is the out put of
iostat.

"Patrick Begou" <Patrick.Begou@hmg.inpg.fr> wrote in message
news:ck06ch$84t$1@news.grenet.fr...
> Rizwan Abbasi wrote:
> > Dears, need the group wisdom to solve this issue I have an IBMp650
orignally
> > configured for two LPARs with two cpus each but with pathetic
performance
> > now a single lpar with all the 4 CPUs but performance still not good as
in
> > vmstat output. Pls note that the same application was running excelent
> > under IBM old S80 with 6CPUs of 450MHz. Any help/hints or tips will be
most
> > apprecaited.
> >
> >
> >
> > System Configuration: lcpu=4 mem=6144MB
> >
> > kthr memory page faults cpu
> >
> > ----- ----------- ------------------------ ------------ -----------
> >
> > r b avm fre re pi po fr sr cy in sy cs us sy id wa
> >
> > 21 1 431992 745 0 0 0 345 887 0 908 123949 917 10 38 49 3
> >
> > 40 0 431997 719 0 0 0 0 0 0 697 276607 708 16 84 0 0
> > ^ ^
> Your system shows that you have a lot of system calls (276607) and that
> your application only use 16% of cpu for calculations. I don't know what
> your system calls are. File access ?
> Can you check with the command:
>
> iostat 4
> or
> iostat -d 4 (just disks activity)
>
> the activity of your disks (every 4 seconds in this example)?
>
> You are also using most of the RAM (small free list: 719).
>
> May be your database is quite big and cannot be "cached" in RAM. So most
> of the cpu time is used for disk I/O. But I'm not working with database
> and I have not a large knowledge for optimization of such aplications.
>
> Patrick
>
>
>
> .
> .
> .
> >
> > Avg CPU load: 100% <<========Yes but 84% is system time
> >
> > Run que Avg: 28-34
> >
> > Block Que Avg: 1,2
> >
> > Paging: 1% <<===== your system do not swap, so you have enought memory
> for your application but may be you need more for file
> caching.
>
> >
> >

Riz wrote:
> Here is the output from iostat -d 4
> Disks: % tm_act Kbps tps
> Kb_read Kb_wrtn
>
> hdisk5 0.0 0.0 0.0
> 1784 1436
>
> hdisk3 0.3 5.5 0.9
> 203802 603651
>
> hdisk4 0.0 0.0 0.0
> 0 0
>
> hdisk2 0.4 5.2 1.0
> 148531 603175
>
> hdisk1 0.0 0.0 0.0
> 0 0
>
> hdisk6 0.0 0.0 0.0
> 2296 1816
>
> hdisk0 7.8 514.0 37.2
> 71408704 3603944
>
>
> I think there is no extra ordinary disk activity. Here is the out put of
> iostat.

May be you should wait additional time (iostat is sampling every 4s in
this case) to have more detailed informations. May be restrict the
information to the DD where is located your database while it is in use

iostat -d your_physical_volume 4

and look at the values for several minutes (check with "vmstat 4" your
system activity in parallel).


But may be your sys time is not related to I/Os. But you must identify
what is this sys time to solve your problem.

Rizwan Abbasi wrote:
> Hi,
> Many thanks for the groups valuable thoughts, The old s80 was running AIX
> 4.3.3 and memeory there was 6GB,
> here is the sar output
> System Configuration: lcpu=4
>
>
> 11:29:46 %usr %sys %wio %idle
>
> 11:29:56 13 87 0 0
>
> 11:30:06 12 88 0 0
>
> 11:30:16 13 87 0 0
>
>
> Average 12 88 0 0
>
>
> here is the data requested from the topas
>
> Kernel: 87.4%
> User:12.6%
> Idle:0%
>
> Name PID CPU% PgSp Owner % Used 0.7
>
> _progres 34166 10.8 3.9 tascsd1 NFS (calls/sec) % Free 99.2
>
> _progres 152118 10.5 3.3 exaud10 ServerV2 0
>
> _progres 123042 10.1 2.4 mahfcl3 ClientV2 0 Press:
>
> _progres 151626 9.8 3.1 afticf9 ServerV3 0 "h" for help
>
> _progres 129912 9.7 3.0 basktf6 ClientV3 0 "q" to quit
>
> ksh 53210 9.2 95.2 support
>
>
>
> Any help or hints or tips will be most apprecaited.
>
>

Hmmm. a couple of things:

1) Please, be a little kinder when pulling data from topas as the
formatting gets pretty ugly

2) To be more specific, note that pressing 'c' while looking at topas
will let you verify that all the cpus are nailed.

Now, from the part of the list you have posted, of the 100% cpu usage,
progress is taking 50%? Just a shot in the dark, but the question about
progress configuration wasn't addressed. Now, I have no experience with
progress, but with other databases, you generally spawn less working
processes than cpu - typically leave a processor for the OS and overhead
and the rest for the database. It looks like you still have 5 database
processes, which would be good for a 6 cpu system, but with 4 cpus
perhaps you have a bunch of extra overhead having those extra processes
fighting for cpu. Particularly if there is an attempt to set processor
affinity? Maybe?

So, how can I find out whats causing my sys time v.high?

"Patrick Begou" <Patrick.Begou@hmg.inpg.fr> wrote in message
news:ck0mh2$csu$1@news.grenet.fr...
> Riz wrote:
> > Here is the output from iostat -d 4
> > Disks: % tm_act Kbps tps
> > Kb_read Kb_wrtn
> >
> > hdisk5 0.0 0.0
0.0
> > 1784 1436
> >
> > hdisk3 0.3 5.5
0.9
> > 203802 603651
> >
> > hdisk4 0.0 0.0
0.0
> > 0 0
> >
> > hdisk2 0.4 5.2
1.0
> > 148531 603175
> >
> > hdisk1 0.0 0.0
0.0
> > 0 0
> >
> > hdisk6 0.0 0.0
0.0
> > 2296 1816
> >
> > hdisk0 7.8 514.0
37.2
> > 71408704 3603944
> >
> >
> > I think there is no extra ordinary disk activity. Here is the out put of
> > iostat.
>
> May be you should wait additional time (iostat is sampling every 4s in
> this case) to have more detailed informations. May be restrict the
> information to the DD where is located your database while it is in use
>
> iostat -d your_physical_volume 4
>
> and look at the values for several minutes (check with "vmstat 4" your
> system activity in parallel).
>
>
> But may be your sys time is not related to I/Os. But you must identify
> what is this sys time to solve your problem.

Rizwan Abbasi wrote:
> Dears, need the group wisdom to solve this issue I have an IBMp650 orignally
> configured for two LPARs with two cpus each but with pathetic performance
> now a single lpar with all the 4 CPUs but performance still not good as in
> vmstat output. Pls note that the same application was running excelent
> under IBM old S80 with 6CPUs of 450MHz. Any help/hints or tips will be most
> apprecaited.

A 4-way 650 should run circles around the older S80.

Curious though - your cpu time is in the sys column which means lots of
system calls are being executed rather than application code. I would
suggest using tprof to see a bit more about what's going on.

# mkdir /tmp/tprof
# cd /tmp/tprof
# tprof -x sleep 10
# tprof -k -s -e

Look at the __prof.all file. That's off the top of my head, so
hopefully that's right...you'll see some detail about what's being executed.

Greg.