Hello

We have a p570 box running AIX 5.2 and DB2 database (resides on SAN
disk) with 8GB of Memory and 12GB of paging space
Notice the following behaviour when DB2 backup was started @ 21:05
hrs.
Would like to find out how to improve the performance of the system.
Any advice is appreciated.
N.B - NMON is used to collect statistics from no. 1 to 4 below.

1. High disk utilization for hdisk0 and hdisk1 (mirror rootvg) only.
They become very busy reaching 100%.

Disk %Busy hdisk0 hdisk1
20:55:47 3.2 3.6
21:00:47 6 7.2
21:05:47 60.6 69.6
21:10:54 99.8 101
21:15:58 95.8 98.1
21:20:58 26.2 33.7
21:25:58 15.6 25.1

2. Filesystem Cache increased from 30% to 70% .

Memory Process%FScache%System% Free% Pinned% User%
20:55:47 25.8 30 5.1 39.1 8.1
52.2
21:00:47 25.9 31.1 5.1 37.9
8.1 53.4
21:05:47 23.8 71.2 4.9 0
8.1 91.4
21:10:54 19.5 75.7 4.8 0
8.1 91.5
21:15:58 19.3 75.9 4.8 0
8.1 91.5
21:20:58 19.4 75.7 4.8 0
8.1 91.5

3. Real Free Memory will dropped to zero

Memory Real Free Virtual free Real free(MB)
20:55:47 39.1 69.7 3201
21:00:47 37.9 69.7 3101.9
21:05:47 0 69.5 2
21:10:54 0 69.1 2.7
21:15:58 0 69.1 1
21:20:58 0.1 69.1 4.2
21:25:58 0 69.1 1.9

4. Sudden increase in Pagein and Pageout.

Paging db304b faults pgin pgout pgsin pgsout
20:55:47 49742 3532.7 61.9 1 0
21:00:47 50569.7 3016.5 67 2.1 0
21:05:47 31936.3 2587.4 6844.9 33.7 182.7
21:10:54 12810.4 2516.2 5617.8 37.6 326
21:15:58 35961.7 4789.2 8631.2 143.9 143.3
21:20:58 46539.3 2231.2 5133.8 46.5 28.3
21:25:58 48137.4 1624.5 4261.4 15.2 12.1

5. Paging space usage maintain at 30% before and after DB2 backup

Wed Jul 26 20:50:14 GMT 2006
Page Space Physical Volume Volume Group Size %Used Active
Auto Type
hd6 hdisk1 rootvg 12288MB 30 yes
yes lv

Wed Jul 26 21:00:15 GMT 2006
Page Space Physical Volume Volume Group Size %Used Active
Auto Type
hd6 hdisk1 rootvg 12288MB 30 yes
yes lv

Wed Jul 26 21:10:31 GMT 2006
Page Space Physical Volume Volume Group Size %Used Active
Auto Type
hd6 hdisk1 rootvg 12288MB 31 yes
yes lv

Wed Jul 26 21:20:15 GMT 2006
Page Space Physical Volume Volume Group Size %Used Active
Auto Type
hd6 hdisk1 rootvg 12288MB 31 yes
yes lv

6. "vmstat" and "vmstat -v" are attached below - collected during
DB2 backup.

System Configuration: lcpu=4 mem=8192MB
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
2 1 1210663 496 0 5 6 448 907 0 303 44536 9546 15 20 55
10
1 21 1211219 77 0 31 343 13285 17613 0 3688 26512 11213 8 33
0 59
2 21 1211458 7 0 21 319 10788 12657 0 3682 11238 10307 4 27
1 68
6 16 1211017 4 0 26 384 11701 14078 0 3472 18415 8883 5 31
0 63
0 26 1210627 158 0 21 403 6573 7642 0 3566 14818 12230 3 21 0
76
0 22 1210535 171 0 34 392 5682 6556 0 3535 6531 9613 2 10 0
88
0 23 1210544 0 0 22 384 2805 3366 0 3091 5787 9369 2 9 0
90
0 24 1211117 1 0 26 448 3814 4553 0 3746 4511 12266 1 10 0
89
0 25 1211370 0 0 27 478 6766 10125 0 3431 8833 12423 2 15 0
83
0 19 1211496 1 0 18 396 2521 3207 0 3566 4544 11354 1 9 0
90
1 21 1210222 528 0 44 432 14072 20908 0 3592 162999 10209 8
38 0 53
4 15 1209850 439 0 84 244 16775 22938 0 3384 29124 8698 12 47
0 41
4 21 1209729 0 0 25 362 7470 9051 0 3700 14568 12184 5 21 0
74
0 19 1210441 78 0 33 469 17568 24675 0 3641 5331 9208 2 27 0
71
0 22 1209695 592 0 46 489 10581 12828 0 3468 12020 7325 5 24
0 71

2097152 memory pages
1984927 lruable pages
419 free pages
2 memory pools
171158 pinned pages
80.1 maxpin percentage
20.0 minperm percentage
80.0 maxperm percentage
77.3 numperm percentage
1534625 file pages
0.0 compressed percentage
0 compressed pages
27.8 numclient percentage
80.0 maxclient percentage
552457 client pages
0 remote pageouts scheduled
40509 pending disk I/Os blocked with no pbuf
0 paging space I/Os blocked with no psbuf
3613045 filesystem I/Os blocked with no fsbuf
0 client filesystem I/Os blocked with no fsbuf
25777 external pager filesystem I/Os blocked with no
sbuf

Thanks

h2ooo2h@sg.yahoo.com.sg wrote:
> Hello
>
> We have a p570 box running AIX 5.2 and DB2 database (resides on SAN
> disk) with 8GB of Memory and 12GB of paging space
> Notice the following behaviour when DB2 backup was started @ 21:05
> hrs.
> Would like to find out how to improve the performance of the system.
> Any advice is appreciated.
> N.B - NMON is used to collect statistics from no. 1 to 4 below.
>
> 1. High disk utilization for hdisk0 and hdisk1 (mirror rootvg) only.
> They become very busy reaching 100%.
>
> Disk %Busy hdisk0 hdisk1
> 20:55:47 3.2 3.6
> 21:00:47 6 7.2
> 21:05:47 60.6 69.6
> 21:10:54 99.8 101
> 21:15:58 95.8 98.1
> 21:20:58 26.2 33.7
> 21:25:58 15.6 25.1
>
> 2. Filesystem Cache increased from 30% to 70% .
>
> Memory Process%FScache%System% Free% Pinned% User%
> 20:55:47 25.8 30 5.1 39.1 8.1
> 52.2
> 21:00:47 25.9 31.1 5.1 37.9
> 8.1 53.4
> 21:05:47 23.8 71.2 4.9 0
> 8.1 91.4
> 21:10:54 19.5 75.7 4.8 0
> 8.1 91.5
> 21:15:58 19.3 75.9 4.8 0
> 8.1 91.5
> 21:20:58 19.4 75.7 4.8 0
> 8.1 91.5
>
> 3. Real Free Memory will dropped to zero
>
> Memory Real Free Virtual free Real free(MB)
> 20:55:47 39.1 69.7 3201
> 21:00:47 37.9 69.7 3101.9
> 21:05:47 0 69.5 2
> 21:10:54 0 69.1 2.7
> 21:15:58 0 69.1 1
> 21:20:58 0.1 69.1 4.2
> 21:25:58 0 69.1 1.9
>
> 4. Sudden increase in Pagein and Pageout.
>
> Paging db304b faults pgin pgout pgsin pgsout
> 20:55:47 49742 3532.7 61.9 1 0
> 21:00:47 50569.7 3016.5 67 2.1 0
> 21:05:47 31936.3 2587.4 6844.9 33.7 182.7
> 21:10:54 12810.4 2516.2 5617.8 37.6 326
> 21:15:58 35961.7 4789.2 8631.2 143.9 143.3
> 21:20:58 46539.3 2231.2 5133.8 46.5 28.3
> 21:25:58 48137.4 1624.5 4261.4 15.2 12.1
>
> 5. Paging space usage maintain at 30% before and after DB2 backup
>
> Wed Jul 26 20:50:14 GMT 2006
> Page Space Physical Volume Volume Group Size %Used Active
> Auto Type
> hd6 hdisk1 rootvg 12288MB 30 yes
> yes lv
>
> Wed Jul 26 21:00:15 GMT 2006
> Page Space Physical Volume Volume Group Size %Used Active
> Auto Type
> hd6 hdisk1 rootvg 12288MB 30 yes
> yes lv
>
> Wed Jul 26 21:10:31 GMT 2006
> Page Space Physical Volume Volume Group Size %Used Active
> Auto Type
> hd6 hdisk1 rootvg 12288MB 31 yes
> yes lv
>
> Wed Jul 26 21:20:15 GMT 2006
> Page Space Physical Volume Volume Group Size %Used Active
> Auto Type
> hd6 hdisk1 rootvg 12288MB 31 yes
> yes lv
>
> 6. "vmstat" and "vmstat -v" are attached below - collected during
> DB2 backup.
>
> System Configuration: lcpu=4 mem=8192MB
> kthr memory page faults cpu
> ----- ----------- ------------------------ ------------ -----------
> r b avm fre re pi po fr sr cy in sy cs us sy id wa
> 2 1 1210663 496 0 5 6 448 907 0 303 44536 9546 15 20 55
> 10
> 1 21 1211219 77 0 31 343 13285 17613 0 3688 26512 11213 8 33
> 0 59
> 2 21 1211458 7 0 21 319 10788 12657 0 3682 11238 10307 4 27
> 1 68
> 6 16 1211017 4 0 26 384 11701 14078 0 3472 18415 8883 5 31
> 0 63
> 0 26 1210627 158 0 21 403 6573 7642 0 3566 14818 12230 3 21 0
> 76
> 0 22 1210535 171 0 34 392 5682 6556 0 3535 6531 9613 2 10 0
> 88
> 0 23 1210544 0 0 22 384 2805 3366 0 3091 5787 9369 2 9 0
> 90
> 0 24 1211117 1 0 26 448 3814 4553 0 3746 4511 12266 1 10 0
> 89
> 0 25 1211370 0 0 27 478 6766 10125 0 3431 8833 12423 2 15 0
> 83
> 0 19 1211496 1 0 18 396 2521 3207 0 3566 4544 11354 1 9 0
> 90
> 1 21 1210222 528 0 44 432 14072 20908 0 3592 162999 10209 8
> 38 0 53
> 4 15 1209850 439 0 84 244 16775 22938 0 3384 29124 8698 12 47
> 0 41
> 4 21 1209729 0 0 25 362 7470 9051 0 3700 14568 12184 5 21 0
> 74
> 0 19 1210441 78 0 33 469 17568 24675 0 3641 5331 9208 2 27 0
> 71
> 0 22 1209695 592 0 46 489 10581 12828 0 3468 12020 7325 5 24
> 0 71
>
> 2097152 memory pages
> 1984927 lruable pages
> 419 free pages
> 2 memory pools
> 171158 pinned pages
> 80.1 maxpin percentage
> 20.0 minperm percentage
> 80.0 maxperm percentage
> 77.3 numperm percentage
> 1534625 file pages
> 0.0 compressed percentage
> 0 compressed pages
> 27.8 numclient percentage
> 80.0 maxclient percentage
> 552457 client pages
> 0 remote pageouts scheduled
> 40509 pending disk I/Os blocked with no pbuf
> 0 paging space I/Os blocked with no psbuf
> 3613045 filesystem I/Os blocked with no fsbuf
> 0 client filesystem I/Os blocked with no fsbuf
> 25777 external pager filesystem I/Os blocked with no
> sbuf
>
> Thanks

reduce minperm and maxperm a lot from the default.
turn on dio on the filesystems (or cio if you can get to aix 5.3), and
/ or turn off db2 filesystem caching within db2.

also, how are you doing the db backup? to disk? to tape? ...?

On 28 Jul 2006 10:48:41 -0700, aix@mail.com wrote:

>
>h2ooo2h@sg.yahoo.com.sg wrote:
>> Hello
>>
>> We have a p570 box running AIX 5.2 and DB2 database (resides on SAN
>> disk) with 8GB of Memory and 12GB of paging space
>> Notice the following behaviour when DB2 backup was started @ 21:05
>> hrs.
>> Would like to find out how to improve the performance of the system.
>> Any advice is appreciated.
>> N.B - NMON is used to collect statistics from no. 1 to 4 below.
>>
>> 1. High disk utilization for hdisk0 and hdisk1 (mirror rootvg) only.
>> They become very busy reaching 100%.
>>
>> Disk %Busy hdisk0 hdisk1
>> 20:55:47 3.2 3.6
>> 21:00:47 6 7.2
>> 21:05:47 60.6 69.6
>> 21:10:54 99.8 101
>> 21:15:58 95.8 98.1
>> 21:20:58 26.2 33.7
>> 21:25:58 15.6 25.1
>>
>> 2. Filesystem Cache increased from 30% to 70% .
>>
>> Memory Process%FScache%System% Free% Pinned% User%
>> 20:55:47 25.8 30 5.1 39.1 8.1
>> 52.2
>> 21:00:47 25.9 31.1 5.1 37.9
>> 8.1 53.4
>> 21:05:47 23.8 71.2 4.9 0
>> 8.1 91.4
>> 21:10:54 19.5 75.7 4.8 0
>> 8.1 91.5
>> 21:15:58 19.3 75.9 4.8 0
>> 8.1 91.5
>> 21:20:58 19.4 75.7 4.8 0
>> 8.1 91.5
>>
>> 3. Real Free Memory will dropped to zero
>>
>> Memory Real Free Virtual free Real free(MB)
>> 20:55:47 39.1 69.7 3201
>> 21:00:47 37.9 69.7 3101.9
>> 21:05:47 0 69.5 2
>> 21:10:54 0 69.1 2.7
>> 21:15:58 0 69.1 1
>> 21:20:58 0.1 69.1 4.2
>> 21:25:58 0 69.1 1.9
>>
>> 4. Sudden increase in Pagein and Pageout.
>>
>> Paging db304b faults pgin pgout pgsin pgsout
>> 20:55:47 49742 3532.7 61.9 1 0
>> 21:00:47 50569.7 3016.5 67 2.1 0
>> 21:05:47 31936.3 2587.4 6844.9 33.7 182.7
>> 21:10:54 12810.4 2516.2 5617.8 37.6 326
>> 21:15:58 35961.7 4789.2 8631.2 143.9 143.3
>> 21:20:58 46539.3 2231.2 5133.8 46.5 28.3
>> 21:25:58 48137.4 1624.5 4261.4 15.2 12.1
>>
>> 5. Paging space usage maintain at 30% before and after DB2 backup
>>
>> Wed Jul 26 20:50:14 GMT 2006
>> Page Space Physical Volume Volume Group Size %Used Active
>> Auto Type
>> hd6 hdisk1 rootvg 12288MB 30 yes
>> yes lv
>>
>> Wed Jul 26 21:00:15 GMT 2006
>> Page Space Physical Volume Volume Group Size %Used Active
>> Auto Type
>> hd6 hdisk1 rootvg 12288MB 30 yes
>> yes lv
>>
>> Wed Jul 26 21:10:31 GMT 2006
>> Page Space Physical Volume Volume Group Size %Used Active
>> Auto Type
>> hd6 hdisk1 rootvg 12288MB 31 yes
>> yes lv
>>
>> Wed Jul 26 21:20:15 GMT 2006
>> Page Space Physical Volume Volume Group Size %Used Active
>> Auto Type
>> hd6 hdisk1 rootvg 12288MB 31 yes
>> yes lv
>>
>> 6. "vmstat" and "vmstat -v" are attached below - collected during
>> DB2 backup.
>>
>> System Configuration: lcpu=4 mem=8192MB
>> kthr memory page faults cpu
>> ----- ----------- ------------------------ ------------ -----------
>> r b avm fre re pi po fr sr cy in sy cs us sy id wa
>> 2 1 1210663 496 0 5 6 448 907 0 303 44536 9546 15 20 55
>> 10
>> 1 21 1211219 77 0 31 343 13285 17613 0 3688 26512 11213 8 33
>> 0 59
>> 2 21 1211458 7 0 21 319 10788 12657 0 3682 11238 10307 4 27
>> 1 68
>> 6 16 1211017 4 0 26 384 11701 14078 0 3472 18415 8883 5 31
>> 0 63
>> 0 26 1210627 158 0 21 403 6573 7642 0 3566 14818 12230 3 21 0
>> 76
>> 0 22 1210535 171 0 34 392 5682 6556 0 3535 6531 9613 2 10 0
>> 88
>> 0 23 1210544 0 0 22 384 2805 3366 0 3091 5787 9369 2 9 0
>> 90
>> 0 24 1211117 1 0 26 448 3814 4553 0 3746 4511 12266 1 10 0
>> 89
>> 0 25 1211370 0 0 27 478 6766 10125 0 3431 8833 12423 2 15 0
>> 83
>> 0 19 1211496 1 0 18 396 2521 3207 0 3566 4544 11354 1 9 0
>> 90
>> 1 21 1210222 528 0 44 432 14072 20908 0 3592 162999 10209 8
>> 38 0 53
>> 4 15 1209850 439 0 84 244 16775 22938 0 3384 29124 8698 12 47
>> 0 41
>> 4 21 1209729 0 0 25 362 7470 9051 0 3700 14568 12184 5 21 0
>> 74
>> 0 19 1210441 78 0 33 469 17568 24675 0 3641 5331 9208 2 27 0
>> 71
>> 0 22 1209695 592 0 46 489 10581 12828 0 3468 12020 7325 5 24
>> 0 71
>>
>> 2097152 memory pages
>> 1984927 lruable pages
>> 419 free pages
>> 2 memory pools
>> 171158 pinned pages
>> 80.1 maxpin percentage
>> 20.0 minperm percentage
>> 80.0 maxperm percentage
>> 77.3 numperm percentage
>> 1534625 file pages
>> 0.0 compressed percentage
>> 0 compressed pages
>> 27.8 numclient percentage
>> 80.0 maxclient percentage
>> 552457 client pages
>> 0 remote pageouts scheduled
>> 40509 pending disk I/Os blocked with no pbuf
>> 0 paging space I/Os blocked with no psbuf
>> 3613045 filesystem I/Os blocked with no fsbuf
>> 0 client filesystem I/Os blocked with no fsbuf
>> 25777 external pager filesystem I/Os blocked with no
>> sbuf
>>
>> Thanks
>
>reduce minperm and maxperm a lot from the default.
>turn on dio on the filesystems (or cio if you can get to aix 5.3), and
>/ or turn off db2 filesystem caching within db2.
>
>also, how are you doing the db backup? to disk? to tape? ...?


correction, at 21:05 hrs, we started off with backup of the OS
(rootvg) Filesystems to TAPE and not doing DB2 backup.

On Fri, 28 Jul 2006 23:32:12 +0800, h2ooo2h@sg.yahoo.com.sg wrote:

>We have a p570 box running AIX 5.2 and DB2 database (resides on SAN
>disk) with 8GB of Memory and 12GB of paging space
>Notice the following behaviour when DB2 backup was started @ 21:05
>hrs.

>1. High disk utilization for hdisk0 and hdisk1 (mirror rootvg) only.
>They become very busy reaching 100%.

Either this I/O is due to paging or it is due to the output of the DB2
backup being directed to rootvg filespaces.

>2. Filesystem Cache increased from 30% to 70% .

(Hopefully) you've set up DB2 to do CIO (non-cached) reads and writes.
You didn't mention what version of DB2. In any case, it's an attribute
of the tablespace creation. Your cache increase is because you're
either writing to a filesystem or you're doing a "file-based" backup
-- i.e. using TAR or something to save the underlying DB2 data files.

>3. Real Free Memory will dropped to zero

>4. Sudden increase in Pagein and Pageout.

>5. Paging space usage maintain at 30% before and after DB2 backup

All of these are consistent with the above hypothesis.

Recommendations:

1. Be sure DB2 tablespaces are set to bypass AIX cache. Talk to your
DBA.

2. Use the DB2 backup commands, not a utility that captures the
underlying files while DB2 is shut down.

3. If you are doing a backup to a file, be sure that the destination
filesystem is mounted with the "release behind on write" option.

-- David