SEO

Dears, need the group wisdom to solve this issue I have an IBMp650 orignally
configured for two LPARs with two cpus each but with pathetic performance
now a single lpar with all the 4 CPUs but performance still not good as in
vmstat output. Pls note that the same application was running excelent
under IBM old S80 with 6CPUs of 450MHz. Any help/hints or tips will be most
apprecaited.



System Configuration: lcpu=4 mem=6144MB

kthr memory page faults cpu

----- ----------- ------------------------ ------------ -----------

r b avm fre re pi po fr sr cy in sy cs us sy id wa

21 1 431992 745 0 0 0 345 887 0 908 123949 917 10 38 49 3

40 0 431997 719 0 0 0 0 0 0 697 276607 708 16 84 0 0

40 0 431997 706 0 0 0 0 0 0 693 277172 688 16 84 0 0

40 0 431997 696 0 0 0 0 0 0 696 276998 699 17 83 0 0

40 0 431997 672 0 0 0 0 0 0 692 277235 696 20 80 0 0

40 0 431997 648 0 0 0 0 0 0 694 292152 734 17 83 0 0

40 0 431997 624 0 0 0 0 0 0 690 277085 699 18 82 0 0

40 0 431997 610 0 0 0 0 0 0 695 277311 708 16 84 0 0

40 0 431997 570 0 0 0 0 0 0 695 277084 703 17 83 0 0

40 0 431997 554 0 0 0 0 0 0 692 277030 701 16 84 0 0



Here is the details for OS etc.

OSLEVE: 5.2 ML04

Machine: IBM p650 (4 CPU)

RAM: 6GB

Application: MFGPRO

Databse: Progress 9.1D07

No of users : 130 (concurrent approx 100 sessions)

Printers: 40+

Java version 32bit 1.3.1

OS Kernel : 32bit (64bit kernel not supported by progress)

Avg CPU load: 100%

Run que Avg: 28-34

Block Que Avg: 1,2

Paging: 1%

"Rizwan Abbasi" <abbasi2@attglobal.net> wrote in message
news:41627237_4@news2.prserv.net...
> Dears, need the group wisdom to solve this issue I have an IBMp650
orignally
> configured for two LPARs with two cpus each but with pathetic performance
> now a single lpar with all the 4 CPUs but performance still not good as in
> vmstat output. Pls note that the same application was running excelent
> under IBM old S80 with 6CPUs of 450MHz. Any help/hints or tips will be
most
> apprecaited.
>
>
>
> System Configuration: lcpu=4 mem=6144MB
>
> kthr memory page faults cpu
>
> ----- ----------- ------------------------ ------------ -----------
>
> r b avm fre re pi po fr sr cy in sy cs us sy id wa
>
> 21 1 431992 745 0 0 0 345 887 0 908 123949 917 10 38 49 3
>
> 40 0 431997 719 0 0 0 0 0 0 697 276607 708 16 84 0 0
>
> 40 0 431997 706 0 0 0 0 0 0 693 277172 688 16 84 0 0
>
> 40 0 431997 696 0 0 0 0 0 0 696 276998 699 17 83 0 0
>
> 40 0 431997 672 0 0 0 0 0 0 692 277235 696 20 80 0 0
>
> 40 0 431997 648 0 0 0 0 0 0 694 292152 734 17 83 0 0
>
> 40 0 431997 624 0 0 0 0 0 0 690 277085 699 18 82 0 0
>
> 40 0 431997 610 0 0 0 0 0 0 695 277311 708 16 84 0 0
>
> 40 0 431997 570 0 0 0 0 0 0 695 277084 703 17 83 0 0
>
> 40 0 431997 554 0 0 0 0 0 0 692 277030 701 16 84 0 0
>
>
>
> Here is the details for OS etc.
>
> OSLEVE: 5.2 ML04
>
> Machine: IBM p650 (4 CPU)
>
> RAM: 6GB
>
> Application: MFGPRO
>
> Databse: Progress 9.1D07
>
> No of users : 130 (concurrent approx 100 sessions)
>
> Printers: 40+
>
> Java version 32bit 1.3.1
>
> OS Kernel : 32bit (64bit kernel not supported by progress)
>
> Avg CPU load: 100%
>
> Run que Avg: 28-34
>
> Block Que Avg: 1,2
>
> Paging: 1%
>
>


Is your database the top process? I would download nmon and take a look at
the top process.
http://www-106.ibm.com/developerwork...s/analyze_aix/
Was the old box 5.2 ML4?
Is the application certified to run on 5.2?
Is the firmware on the 650 current?( it changes almost monthly)
lscfg -vp | grep -p Plat
Platform Firmware:
ROM Level.(alterable).......3K040323
This was current a few weeks ago, IBMs firmware site is down at the moment.

Did anything in your database change when you moved hardware? (binaries,
config, code)
We have a bunch of 650 and no performance issues. Of course we don't have
the same application.
One thing to note, the cpus are different. If the app was compiled with
flags for the old architecture this may be
a problem on the new cpus.
Keep digging you will find it!

Doug

hi
System seems CPU bound....upgrade CPUs
SASA

"Rizwan Abbasi" <abbasi2@attglobal.net> wrote in message news:<41627237_4@news2.prserv.net>...
> Dears, need the group wisdom to solve this issue I have an IBMp650 orignally
> configured for two LPARs with two cpus each but with pathetic performance
> now a single lpar with all the 4 CPUs but performance still not good as in
> vmstat output. Pls note that the same application was running excelent
> under IBM old S80 with 6CPUs of 450MHz. Any help/hints or tips will be most
> apprecaited.
>
>
>
> System Configuration: lcpu=4 mem=6144MB
>
> kthr memory page faults cpu
>
> ----- ----------- ------------------------ ------------ -----------
>
> r b avm fre re pi po fr sr cy in sy cs us sy id wa
>
> 21 1 431992 745 0 0 0 345 887 0 908 123949 917 10 38 49 3
>
> 40 0 431997 719 0 0 0 0 0 0 697 276607 708 16 84 0 0
>
> 40 0 431997 706 0 0 0 0 0 0 693 277172 688 16 84 0 0
>
> 40 0 431997 696 0 0 0 0 0 0 696 276998 699 17 83 0 0
>
> 40 0 431997 672 0 0 0 0 0 0 692 277235 696 20 80 0 0
>
> 40 0 431997 648 0 0 0 0 0 0 694 292152 734 17 83 0 0
>
> 40 0 431997 624 0 0 0 0 0 0 690 277085 699 18 82 0 0
>
> 40 0 431997 610 0 0 0 0 0 0 695 277311 708 16 84 0 0
>
> 40 0 431997 570 0 0 0 0 0 0 695 277084 703 17 83 0 0
>
> 40 0 431997 554 0 0 0 0 0 0 692 277030 701 16 84 0 0
>
>
>
> Here is the details for OS etc.
>
> OSLEVE: 5.2 ML04
>
> Machine: IBM p650 (4 CPU)
>
> RAM: 6GB
>
> Application: MFGPRO
>
> Databse: Progress 9.1D07
>
> No of users : 130 (concurrent approx 100 sessions)
>
> Printers: 40+
>
> Java version 32bit 1.3.1
>
> OS Kernel : 32bit (64bit kernel not supported by progress)
>
> Avg CPU load: 100%
>
> Run que Avg: 28-34
>
> Block Que Avg: 1,2
>
> Paging: 1%

"sasa queer" <sasa1973pk@yahoo.com> wrote in message
news:702930ee.0410050959.15a658df@posting.google.c om...
> hi
> System seems CPU bound....upgrade CPUs
> SASA

The box in question has 1.4 GHz cpus in it and you can't up grade them.
Doug

>
> "Rizwan Abbasi" <abbasi2@attglobal.net> wrote in message
news:<41627237_4@news2.prserv.net>...
> > Dears, need the group wisdom to solve this issue I have an IBMp650
orignally
> > configured for two LPARs with two cpus each but with pathetic
performance
> > now a single lpar with all the 4 CPUs but performance still not good as
in
> > vmstat output. Pls note that the same application was running excelent
> > under IBM old S80 with 6CPUs of 450MHz. Any help/hints or tips will be
most
> > apprecaited.
> >
> >
> >
> > System Configuration: lcpu=4 mem=6144MB
> >
> > kthr memory page faults cpu
> >
> > ----- ----------- ------------------------ ------------ -----------
> >
> > r b avm fre re pi po fr sr cy in sy cs us sy id wa
> >
> > 21 1 431992 745 0 0 0 345 887 0 908 123949 917 10 38 49 3
> >
> > 40 0 431997 719 0 0 0 0 0 0 697 276607 708 16 84 0 0
> >
> > 40 0 431997 706 0 0 0 0 0 0 693 277172 688 16 84 0 0
> >
> > 40 0 431997 696 0 0 0 0 0 0 696 276998 699 17 83 0 0
> >
> > 40 0 431997 672 0 0 0 0 0 0 692 277235 696 20 80 0 0
> >
> > 40 0 431997 648 0 0 0 0 0 0 694 292152 734 17 83 0 0
> >
> > 40 0 431997 624 0 0 0 0 0 0 690 277085 699 18 82 0 0
> >
> > 40 0 431997 610 0 0 0 0 0 0 695 277311 708 16 84 0 0
> >
> > 40 0 431997 570 0 0 0 0 0 0 695 277084 703 17 83 0 0
> >
> > 40 0 431997 554 0 0 0 0 0 0 692 277030 701 16 84 0 0
> >
> >
> >
> > Here is the details for OS etc.
> >
> > OSLEVE: 5.2 ML04
> >
> > Machine: IBM p650 (4 CPU)
> >
> > RAM: 6GB
> >
> > Application: MFGPRO
> >
> > Databse: Progress 9.1D07
> >
> > No of users : 130 (concurrent approx 100 sessions)
> >
> > Printers: 40+
> >
> > Java version 32bit 1.3.1
> >
> > OS Kernel : 32bit (64bit kernel not supported by progress)
> >
> > Avg CPU load: 100%
> >
> > Run que Avg: 28-34
> >
> > Block Que Avg: 1,2
> >
> > Paging: 1%

>> System seems CPU bound....upgrade CPUs
>> SASA

> The box in question has 1.4 GHz cpus in it and you can't up grade them.
> Doug

Hmmm, doesn't IBM offer an upgrade option for p650 owners --> p5-570 ??
As I remember from the announcement letter, you were not limited to
the original number of processors, either.

From the high runqueue numbers posted, the system seems to have
more dispatchable work than CPUs to execute. With the Power5 SMT
feature might provide some benefits to this workload, & the ability
to have more than 4 CPUs would also be helpful.

Eric

Eric.Jones wrote:
>
>>> System seems CPU bound....upgrade CPUs SASA
>
>
>> The box in question has 1.4 GHz cpus in it and you can't up grade
>> them. Doug
>
>
> Hmmm, doesn't IBM offer an upgrade option for p650 owners --> p5-570 ??
> As I remember from the announcement letter, you were not limited to
> the original number of processors, either.
>
> From the high runqueue numbers posted, the system seems to have
> more dispatchable work than CPUs to execute. With the Power5 SMT
> feature might provide some benefits to this workload, & the ability
> to have more than 4 CPUs would also be helpful.
>
> Eric

Umm, folks, the OP already told us that it ran acceptably on 6 450 MHz
previous generation processors - so it should be smokin on this machine.

Shouldn't we be asking for more info, like:

Was it running on 5.2000-04 on the other machine?

What differences in the progress configuration to tune it to less
processors?

Doesn't it seem a bit odd it is sys not usr that is hogging all the cpu?

Even if one doesn't want to get and set up nmon, even topas or sar will
give more information as to just what is hogging the system ....b

"Timothy J. Bogart" <tbogart@frii.net> wrote in message
news:416328d2$0$202$75868355@news.frii.net...
> Eric.Jones wrote:
> >
> >>> System seems CPU bound....upgrade CPUs SASA
> >
> >
> >> The box in question has 1.4 GHz cpus in it and you can't up grade
> >> them. Doug
> >
> >
> > Hmmm, doesn't IBM offer an upgrade option for p650 owners --> p5-570 ??
> > As I remember from the announcement letter, you were not limited to
> > the original number of processors, either.
> >
> > From the high runqueue numbers posted, the system seems to have
> > more dispatchable work than CPUs to execute. With the Power5 SMT
> > feature might provide some benefits to this workload, & the ability
> > to have more than 4 CPUs would also be helpful.
> >
> > Eric
>
> Umm, folks, the OP already told us that it ran acceptably on 6 450 MHz
> previous generation processors - so it should be smokin on this machine.
>
> Shouldn't we be asking for more info, like:

Yep, that is why I asked all those questions .
You have to go back a few since the questions were sniped buy the IBM
salesmen ;-)
Doug


>
> Was it running on 5.2000-04 on the other machine?
>
> What differences in the progress configuration to tune it to less
> processors?
>
> Doesn't it seem a bit odd it is sys not usr that is hogging all the cpu?
>
> Even if one doesn't want to get and set up nmon, even topas or sar will
> give more information as to just what is hogging the system ....b

Hi,
Many thanks for the groups valuable thoughts, The old s80 was running AIX
4.3.3 and memeory there was 6GB,
here is the sar output
System Configuration: lcpu=4


11:29:46 %usr %sys %wio %idle

11:29:56 13 87 0 0

11:30:06 12 88 0 0

11:30:16 13 87 0 0


Average 12 88 0 0


here is the data requested from the topas

Kernel: 87.4%
User:12.6%
Idle:0%

Name PID CPU% PgSp Owner % Used 0.7

_progres 34166 10.8 3.9 tascsd1 NFS (calls/sec) % Free 99.2

_progres 152118 10.5 3.3 exaud10 ServerV2 0

_progres 123042 10.1 2.4 mahfcl3 ClientV2 0 Press:

_progres 151626 9.8 3.1 afticf9 ServerV3 0 "h" for help

_progres 129912 9.7 3.0 basktf6 ClientV3 0 "q" to quit

ksh 53210 9.2 95.2 support



Any help or hints or tips will be most apprecaited.


"Douglas R. Probst" <doug@catseye.pha.pa.us> wrote in message
news:cjviud$h62$1@bob.news.rcn.net...
>
> "Timothy J. Bogart" <tbogart@frii.net> wrote in message
> news:416328d2$0$202$75868355@news.frii.net...
> > Eric.Jones wrote:
> > >
> > >>> System seems CPU bound....upgrade CPUs SASA
> > >
> > >
> > >> The box in question has 1.4 GHz cpus in it and you can't up grade
> > >> them. Doug
> > >
> > >
> > > Hmmm, doesn't IBM offer an upgrade option for p650 owners --> p5-570
??
> > > As I remember from the announcement letter, you were not limited to
> > > the original number of processors, either.
> > >
> > > From the high runqueue numbers posted, the system seems to have
> > > more dispatchable work than CPUs to execute. With the Power5 SMT
> > > feature might provide some benefits to this workload, & the ability
> > > to have more than 4 CPUs would also be helpful.
> > >
> > > Eric
> >
> > Umm, folks, the OP already told us that it ran acceptably on 6 450 MHz
> > previous generation processors - so it should be smokin on this machine.
> >
> > Shouldn't we be asking for more info, like:
>
> Yep, that is why I asked all those questions .
> You have to go back a few since the questions were sniped buy the IBM
> salesmen ;-)
> Doug
>
>
> >
> > Was it running on 5.2000-04 on the other machine?
> >
> > What differences in the progress configuration to tune it to less
> > processors?
> >
> > Doesn't it seem a bit odd it is sys not usr that is hogging all the cpu?
> >
> > Even if one doesn't want to get and set up nmon, even topas or sar will
> > give more information as to just what is hogging the system ....b
>
>

Hi,

Just to add the data just posted is of new p650.

Thanks
Riz

"Rizwan Abbasi" <abbasi2@attglobal.net> wrote in message
news:41639869_3@news2.prserv.net...
> Hi,
> Many thanks for the groups valuable thoughts, The old s80 was running AIX
> 4.3.3 and memeory there was 6GB,
> here is the sar output
> System Configuration: lcpu=4
>
>
> 11:29:46 %usr %sys %wio %idle
>
> 11:29:56 13 87 0 0
>
> 11:30:06 12 88 0 0
>
> 11:30:16 13 87 0 0
>
>
> Average 12 88 0 0
>
>
> here is the data requested from the topas
>
> Kernel: 87.4%
> User:12.6%
> Idle:0%
>
> Name PID CPU% PgSp Owner % Used 0.7
>
> _progres 34166 10.8 3.9 tascsd1 NFS (calls/sec) % Free
99.2
>
> _progres 152118 10.5 3.3 exaud10 ServerV2 0
>
> _progres 123042 10.1 2.4 mahfcl3 ClientV2 0 Press:
>
> _progres 151626 9.8 3.1 afticf9 ServerV3 0 "h" for
help
>
> _progres 129912 9.7 3.0 basktf6 ClientV3 0 "q" to quit
>
> ksh 53210 9.2 95.2 support
>
>
>
> Any help or hints or tips will be most apprecaited.
>
>
> "Douglas R. Probst" <doug@catseye.pha.pa.us> wrote in message
> news:cjviud$h62$1@bob.news.rcn.net...
> >
> > "Timothy J. Bogart" <tbogart@frii.net> wrote in message
> > news:416328d2$0$202$75868355@news.frii.net...
> > > Eric.Jones wrote:
> > > >
> > > >>> System seems CPU bound....upgrade CPUs SASA
> > > >
> > > >
> > > >> The box in question has 1.4 GHz cpus in it and you can't up grade
> > > >> them. Doug
> > > >
> > > >
> > > > Hmmm, doesn't IBM offer an upgrade option for p650 owners --> p5-570
> ??
> > > > As I remember from the announcement letter, you were not limited to
> > > > the original number of processors, either.
> > > >
> > > > From the high runqueue numbers posted, the system seems to have
> > > > more dispatchable work than CPUs to execute. With the Power5 SMT
> > > > feature might provide some benefits to this workload, & the ability
> > > > to have more than 4 CPUs would also be helpful.
> > > >
> > > > Eric
> > >
> > > Umm, folks, the OP already told us that it ran acceptably on 6 450 MHz
> > > previous generation processors - so it should be smokin on this
machine.
> > >
> > > Shouldn't we be asking for more info, like:
> >
> > Yep, that is why I asked all those questions .
> > You have to go back a few since the questions were sniped buy the IBM
> > salesmen ;-)
> > Doug
> >
> >
> > >
> > > Was it running on 5.2000-04 on the other machine?
> > >
> > > What differences in the progress configuration to tune it to less
> > > processors?
> > >
> > > Doesn't it seem a bit odd it is sys not usr that is hogging all the
cpu?
> > >
> > > Even if one doesn't want to get and set up nmon, even topas or sar
will
> > > give more information as to just what is hogging the system ....b
> >
> >
>
>

Rizwan Abbasi wrote:
> Dears, need the group wisdom to solve this issue I have an IBMp650 orignally
> configured for two LPARs with two cpus each but with pathetic performance
> now a single lpar with all the 4 CPUs but performance still not good as in
> vmstat output. Pls note that the same application was running excelent
> under IBM old S80 with 6CPUs of 450MHz. Any help/hints or tips will be most
> apprecaited.
>
>
>
> System Configuration: lcpu=4 mem=6144MB
>
> kthr memory page faults cpu
>
> ----- ----------- ------------------------ ------------ -----------
>
> r b avm fre re pi po fr sr cy in sy cs us sy id wa
>
> 21 1 431992 745 0 0 0 345 887 0 908 123949 917 10 38 49 3
>
> 40 0 431997 719 0 0 0 0 0 0 697 276607 708 16 84 0 0
> ^ ^
Your system shows that you have a lot of system calls (276607) and that
your application only use 16% of cpu for calculations. I don't know what
your system calls are. File access ?
Can you check with the command:

iostat 4
or
iostat -d 4 (just disks activity)

the activity of your disks (every 4 seconds in this example)?

You are also using most of the RAM (small free list: 719).

May be your database is quite big and cannot be "cached" in RAM. So most
of the cpu time is used for disk I/O. But I'm not working with database
and I have not a large knowledge for optimization of such aplications.

Patrick



..
..
..
>
> Avg CPU load: 100% <<========Yes but 84% is system time
>
> Run que Avg: 28-34
>
> Block Que Avg: 1,2
>
> Paging: 1% <<===== your system do not swap, so you have enought memory
for your application but may be you need more for file
caching.

>
>