I'm running an AIX 5.1 system with TSM. The memory always seems to be
100% utilized. When I run a large I/O (started using "dd" to test this
to bypass the file system) I see that the swap area start going nuts and
I/Os to disk crawl. If I cancel the "dd" then I notice that the I/Os
start hauling ass. Why is AIX sending disk I/Os to memory instead of
going directly to disk? The disk subsystem is attached to an IBM FC
card. The disk subsystem shows minimal I/Os until I cancel the "dd";
then is starts cranking as AIX commits the cache to the disk array. How
do I get AIX to write directly to disk?

Does anyone out there run TSM on AIX? Are you at 100% memory utilization?

Jim85CJ wrote:

> I'm running an AIX 5.1 system with TSM. The memory always seems to be
> 100% utilized. When I run a large I/O (started using "dd" to test this
> to bypass the file system) I see that the swap area start going nuts and
> I/Os to disk crawl. If I cancel the "dd" then I notice that the I/Os
> start hauling ass. Why is AIX sending disk I/Os to memory instead of
> going directly to disk? The disk subsystem is attached to an IBM FC
> card. The disk subsystem shows minimal I/Os until I cancel the "dd";
> then is starts cranking as AIX commits the cache to the disk array. How
> do I get AIX to write directly to disk?
>

Jim85CJ wrote:

> I'm running an AIX 5.1 system with TSM. The memory always seems to be
> 100% utilized.
Thats fine. (no matter what app you run)
unused memory is wasted money

> When I run a large I/O (started using "dd" to test this
> to bypass the file system)

Please tell us the exact command. do you
dd <somewhere >/filesystem/file
or
dd </dev/hdisk# >/dev/null


> I see that the swap area start going nuts and
> I/Os to disk crawl. If I cancel the "dd" then I notice that the I/Os
> start hauling ass. Why is AIX sending disk I/Os to memory instead of
> going directly to disk?

To minimize head-movement -> to gain performance.
AIX keeps the data in ram until sync-daemon runs or other requirements
are met.


> The disk subsystem is attached to an IBM FC
> card. The disk subsystem shows minimal I/Os until I cancel the "dd";
> then is starts cranking as AIX commits the cache to the disk array. How
> do I get AIX to write directly to disk?
I dont see a way to do this on system-side.
you could run sync-daemon every 5 seconds, f.e. but i doubt your
performance will improve - it will probably suffer
you could also decrease numperm with vmo - but that will have the same
effects

your application can tell aix to bypass memory - it has to open the file
with O_SYNC . this actually does not mean "bypass memory" but it comes close

this is normal, memory is made up of working pages ( computational
data ) and filesystem cache ( persistient pages )

AIX as default will act like and NFS server and allow up to 80%
filesystem cache on top of the computational meory it requires to run
the kernel / applications etc..

minfree is set by default to 120, so your memory usgae will grow until
you only have 120 free pages on the free list, then the scanner kicks
in to try and free up pages to fulfil the kernels requests for
memory.

if numperm ( number of persistient pages i.e. filesystem cache ) is
lower than maxperm, then the scanner is indescriminate in what pages
it marks as candidates to page out ... either computational --> paging
space or pesistient --> the file on the filesystem from whence it
came.

you can control this by limiting the size of the filesystem cache with
the maxperm setting, i.e lower it until you stop paging pages out.
you can also calculate how much computational pages you require for
you current load by running vmstat 1 10 and taking the highes value
for "avm" and multiplying that by 4096 (4k page), then you can work
out how much of your memory is required just to run the AIX kernel and
Applications, and you can set the rest for use as filesystem cache.

What you will find with TSM is that it will cause many many pages to
be cached as it reads the filesystem for a backup ... so you need to
limit this by lowering maxperm/minperm .... set to something like 30
and 15 to start with and see how you go.

ref: http://publibn.boulder.ibm.com/doc_l...gd/2365c75.htm


HTH
Mark Taylor

Isn't using system cache for I/Os risky? If the system crashes and a
couple of MB of I/Os are in memory (but ACKd back to the application)
that data is lost and the application now has corrupted data. I don't
think I've seen any other OS do it this way.

Holger van Koll wrote:

> Jim85CJ wrote:
>
>> I'm running an AIX 5.1 system with TSM. The memory always seems to be
>> 100% utilized.
>
> Thats fine. (no matter what app you run)
> unused memory is wasted money
>
>> When I run a large I/O (started using "dd" to test this to bypass the
>> file system)
>
>
> Please tell us the exact command. do you
> dd <somewhere >/filesystem/file
> or
> dd </dev/hdisk# >/dev/null
>
>
>> I see that the swap area start going nuts and I/Os to disk crawl. If
>> I cancel the "dd" then I notice that the I/Os start hauling ass. Why
>> is AIX sending disk I/Os to memory instead of going directly to disk?
>
>
> To minimize head-movement -> to gain performance.
> AIX keeps the data in ram until sync-daemon runs or other requirements
> are met.
>
>
>> The disk subsystem is attached to an IBM FC card. The disk subsystem
>> shows minimal I/Os until I cancel the "dd"; then is starts cranking as
>> AIX commits the cache to the disk array. How do I get AIX to write
>> directly to disk?
>
> I dont see a way to do this on system-side.
> you could run sync-daemon every 5 seconds, f.e. but i doubt your
> performance will improve - it will probably suffer
> you could also decrease numperm with vmo - but that will have the same
> effects
>
> your application can tell aix to bypass memory - it has to open the file
> with O_SYNC . this actually does not mean "bypass memory" but it comes
> close

Jim85CJ wrote:

> Isn't using system cache for I/Os risky? If the system crashes and a
> couple of MB of I/Os are in memory (but ACKd back to the application)
> that data is lost and the application now has corrupted data. I don't
> think I've seen any other OS do it this way.

Read the man pages for I/O calls. If you don't open a file O_SYNC (or
some such) then there is *no* guanrantee made by any *X-ish OS that your
I/O has actually gone to disk, except maybe an option on mounting the
filesystem which the developer can't control.

When I "dd" to a file (not a raw device) it uses RAM up to 90% used.
I've never seen "d" do this on any other "UNIX" system. Must be
something in AIX.

Simon Marchese wrote:

> Jim85CJ wrote:
>
>> Isn't using system cache for I/Os risky? If the system crashes and a
>> couple of MB of I/Os are in memory (but ACKd back to the application)
>> that data is lost and the application now has corrupted data. I don't
>> think I've seen any other OS do it this way.
>
>
> Read the man pages for I/O calls. If you don't open a file O_SYNC (or
> some such) then there is *no* guanrantee made by any *X-ish OS that your
> I/O has actually gone to disk, except maybe an option on mounting the
> filesystem which the developer can't control.
>

On 2004-06-10, Jim85CJ <jim_85cj@NOSPAMyahoo.com> wrote:
> Does anyone out there run TSM on AIX? Are you at 100% memory utilization?
>

Yes&Yes.

But I mount the TSM-volumes with '-o rbrw' so that AIX will drop pages
after they've been read from disk, or written to disk.

The database and log-volumes are mounted with '-o rbw' so that read pages
are cached, but written pages are dropped.


-jf

so would you say that being at 100% memory utilization is normal for a
TSM system? Seems odd to me. I also think it is weird that AIX uses
RAM as file system cache.

Jan-Frode Myklebust wrote:

> On 2004-06-10, Jim85CJ <jim_85cj@NOSPAMyahoo.com> wrote:
>
>>Does anyone out there run TSM on AIX? Are you at 100% memory utilization?
>>
>
>
> Yes&Yes.
>
> But I mount the TSM-volumes with '-o rbrw' so that AIX will drop pages
> after they've been read from disk, or written to disk.
>
> The database and log-volumes are mounted with '-o rbw' so that read pages
> are cached, but written pages are dropped.
>
>
> -jf

In article <7lEBc.8900$bs4.5535@newsread3.news.atl.earthlink. net>, Jim85CJ <jim_85cj@NOSPAMyahoo.com> wrote:
> I also think it is weird that AIX uses RAM as file system cache.

Why would it be weird? Most all modern UNIX and UNIX-like OSes uses some
RAM as the fs cache source because memory is about 10,000 times faster
than even the fastest hard drives.

Ones that I know which uses some memory as fs cache is Solaris, Linux,
and I think FreeBSD/OpenBSD? Quite a few others, too, I'm sure.

The size of computational (for apps) pages vs buffer pages can be
adjusted on AIX via a percentage-based setting through VM tuning options
(vmtune under v4 or vmo under v5) if you need to reduce the fs cache and
provide more memory for applications.

-Dan