SEO

I have several systems that run AIX 5L and Informix databases on them.
Most of these systems show values I would expect on a system. However,
one seems to give me some weird values in the lsps command.

Here are the details:

System Model: IBM,9131-52A
Number Of Processors: 4
Processor Clock Speed: 1648 MHz
CPU Type: 64-bit
Kernel Type: 64-bit
Memory Size: 7936 MB
Good Memory Size: 7936 MB

Informix Version 9.40 FC7

# oslevel -s
5200-10-00-0000

# lsps -a
Page Space Physical Volume Volume Group Size %Used Active
Auto Type
paging00 hdisk3 datavg 6400MB 37 yes
yes lv
hd6 hdisk5 rootvg 6656MB 100 yes
yes lv

# lsps -s
Total Paging Space Percent Used
13056MB 69%

# vmstat 10 10
System Configuration: lcpu=4 mem=7936MB
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
1 1 2715508 79451 0 10 10 335 776 0 786 16640 5220 7 2 90
2
1 0 2715519 79440 0 0 0 0 0 0 601 11137 3835 7 1 92
0
1 0 2715520 79411 0 0 0 0 0 0 641 34843 12786 24 2 74
0
1 0 2715529 79401 0 0 0 0 0 0 606 4249 1631 2 0 97 0
1 0 2715673 79242 0 0 0 0 0 0 648 17652 6941 11 1 88
0
1 0 2716015 78898 0 0 0 0 0 0 643 17268 6116 11 1 88
0
2 0 2716193 78742 0 0 0 0 0 0 697 22620 8057 14 1 85
0
2 0 2717113 77816 0 0 0 0 0 0 809 25793 9227 16 1 83
0
5 0 2716951 77872 0 1 0 0 0 0 777 26851 4535 12 3 84
0
0 0 2716921 77902 0 0 0 0 0 0 593 4828 1846 3 0 97 0

# vmstat -v
2031616 memory pages
1952077 lruable pages
73804 free pages
1 memory pools
333094 pinned pages
80.1 maxpin percentage
20.0 minperm percentage
80.0 maxperm percentage
57.1 numperm percentage
1114800 file pages
0.0 compressed percentage
0 compressed pages
57.7 numclient percentage
80.0 maxclient percentage
1127373 client pages
0 remote pageouts scheduled
5943302 pending disk I/Os blocked with no pbuf
25321 paging space I/Os blocked with no psbuf
2740 filesystem I/Os blocked with no fsbuf
0 client filesystem I/Os blocked with no fsbuf
1436111 external pager filesystem I/Os blocked with no
fsbuf

# lsattr -El aio0
autoconfig available STATE to be configured at system restart True
fastpath enable State of fast path True
kprocprio 39 Server PRIORITY True
maxreqs 4096 Maximum number of REQUESTS True
maxservers 50 MAXIMUM number of servers per cpu True
minservers 10 MINIMUM number of servers True


The Informix databases are operating on raw LVs and so there should
not be a double-caching issue that I have heard mentioned. I have
other databases on other servers with less memory and those do not
seem to be causing high paging activity.

The users are not complaining about slow performance or any noticeable
issues, so just wondering if anyone may have a suggestion as to
whether this is a problem, and if there are any settings I should be
looking at that could tell me why this system has decided to use a
significant portion of its paging space for no apparent reason.

Thanks in advance.

Steve

On Apr 23, 5:28 pm, "steven_nospam at Yahoo! Canada"
<steven_nos...@yahoo.ca> wrote:
> I have several systems that run AIX 5L and Informix databases on them.
> Most of these systems show values I would expect on a system. However,
> one seems to give me some weird values in the lsps command.
>
> Here are the details:
>
> System Model: IBM,9131-52A
> Number Of Processors: 4
> Processor Clock Speed: 1648 MHz
> CPU Type: 64-bit
> Kernel Type: 64-bit
> Memory Size: 7936 MB
> Good Memory Size: 7936 MB
>
> Informix Version 9.40 FC7
>
> # oslevel -s
> 5200-10-00-0000
>
> # lsps -a
> Page Space Physical Volume Volume Group Size %Used Active
> Auto Type
> paging00 hdisk3 datavg 6400MB 37 yes
> yes lv
> hd6 hdisk5 rootvg 6656MB 100 yes
> yes lv
>
> # lsps -s
> Total Paging Space Percent Used
> 13056MB 69%
>
> # vmstat 10 10
> System Configuration: lcpu=4 mem=7936MB
> kthr memory page faults cpu
> ----- ----------- ------------------------ ------------ -----------
> r b avm fre re pi po fr sr cy in sy cs us sy id wa
> 1 1 2715508 79451 0 10 10 335 776 0 786 16640 5220 7 2 90
> 2
> 1 0 2715519 79440 0 0 0 0 0 0 601 11137 3835 7 1 92
> 0
> 1 0 2715520 79411 0 0 0 0 0 0 641 34843 12786 24 2 74
> 0
> 1 0 2715529 79401 0 0 0 0 0 0 606 4249 1631 2 0 97 0
> 1 0 2715673 79242 0 0 0 0 0 0 648 17652 6941 11 1 88
> 0
> 1 0 2716015 78898 0 0 0 0 0 0 643 17268 6116 11 1 88
> 0
> 2 0 2716193 78742 0 0 0 0 0 0 697 22620 8057 14 1 85
> 0
> 2 0 2717113 77816 0 0 0 0 0 0 809 25793 9227 16 1 83
> 0
> 5 0 2716951 77872 0 1 0 0 0 0 777 26851 4535 12 3 84
> 0
> 0 0 2716921 77902 0 0 0 0 0 0 593 4828 1846 3 0 97 0
>
> # vmstat -v
> 2031616 memory pages
> 1952077 lruable pages
> 73804 free pages
> 1 memory pools
> 333094 pinned pages
> 80.1 maxpin percentage
> 20.0 minperm percentage
> 80.0 maxperm percentage
> 57.1 numperm percentage
> 1114800 file pages
> 0.0 compressed percentage
> 0 compressed pages
> 57.7 numclient percentage
> 80.0 maxclient percentage
> 1127373 client pages
> 0 remote pageouts scheduled
> 5943302 pending disk I/Os blocked with no pbuf
> 25321 paging space I/Os blocked with no psbuf
> 2740 filesystem I/Os blocked with no fsbuf
> 0 client filesystem I/Os blocked with no fsbuf
> 1436111 external pager filesystem I/Os blocked with no
> fsbuf
>
> # lsattr -El aio0
> autoconfig available STATE to be configured at system restart True
> fastpath enable State of fast path True
> kprocprio 39 Server PRIORITY True
> maxreqs 4096 Maximum number of REQUESTS True
> maxservers 50 MAXIMUM number of servers per cpu True
> minservers 10 MINIMUM number of servers True
>
> The Informix databases are operating on raw LVs and so there should
> not be a double-caching issue that I have heard mentioned. I have
> other databases on other servers with less memory and those do not
> seem to be causing high paging activity.
>
> The users are not complaining about slow performance or any noticeable
> issues, so just wondering if anyone may have a suggestion as to
> whether this is a problem, and if there are any settings I should be
> looking at that could tell me why this system has decided to use a
> significant portion of its paging space for no apparent reason.
>
> Thanks in advance.
>
> Steve

1)
From
http://www.ibm.com/developerworks/vi...ory+setti ngs

Part 3

Paging allocation type:
- deferred page space allocation
- late page space allocation
- early page space allocation.

The default policy of AIX is deferred page space allocation.

2)
For some reason your system is using pretty much file cache. Could be
NFS, log files and what the heck i know but it looks like the
computational pages get page out. So check for busy filesystem.

In case the paging rate is low the system is working as designed ;-)

hth
Hajo

On Apr 23, 10:52 pm, Hajo Ehlers <serv...@metamodul.com> wrote:
> On Apr 23, 5:28 pm, "steven_nospam at Yahoo! Canada"
>
>
>
> <steven_nos...@yahoo.ca> wrote:
> > I have several systems that run AIX 5L and Informix databases on them.
> > Most of these systems show values I would expect on a system. However,
> > one seems to give me some weird values in the lsps command.
>
> > Here are the details:
>
> > System Model: IBM,9131-52A
> > Number Of Processors: 4
> > Processor Clock Speed: 1648 MHz
> > CPU Type: 64-bit
> > Kernel Type: 64-bit
> > Memory Size: 7936 MB
> > Good Memory Size: 7936 MB
>
> > Informix Version 9.40 FC7
>
> > # oslevel -s
> > 5200-10-00-0000
>
> > # lsps -a
> > Page Space Physical Volume Volume Group Size %Used Active
> > Auto Type
> > paging00 hdisk3 datavg 6400MB 37 yes
> > yes lv
> > hd6 hdisk5 rootvg 6656MB 100 yes
> > yes lv
>
> > # lsps -s
> > Total Paging Space Percent Used
> > 13056MB 69%
>
> > # vmstat 10 10
> > System Configuration: lcpu=4 mem=7936MB
> > kthr memory page faults cpu
> > ----- ----------- ------------------------ ------------ -----------
> > r b avm fre re pi po fr sr cy in sy cs us sy id wa
> > 1 1 2715508 79451 0 10 10 335 776 0 786 16640 5220 7 2 90
> > 2
> > 1 0 2715519 79440 0 0 0 0 0 0 601 11137 3835 7 1 92
> > 0
> > 1 0 2715520 79411 0 0 0 0 0 0 641 34843 12786 24 2 74
> > 0
> > 1 0 2715529 79401 0 0 0 0 0 0 606 4249 1631 2 0 97 0
> > 1 0 2715673 79242 0 0 0 0 0 0 648 17652 6941 11 1 88
> > 0
> > 1 0 2716015 78898 0 0 0 0 0 0 643 17268 6116 11 1 88
> > 0
> > 2 0 2716193 78742 0 0 0 0 0 0 697 22620 8057 14 1 85
> > 0
> > 2 0 2717113 77816 0 0 0 0 0 0 809 25793 9227 16 1 83
> > 0
> > 5 0 2716951 77872 0 1 0 0 0 0 777 26851 4535 12 3 84
> > 0
> > 0 0 2716921 77902 0 0 0 0 0 0 593 4828 1846 3 0 97 0
>
> > # vmstat -v
> > 2031616 memory pages
> > 1952077 lruable pages
> > 73804 free pages
> > 1 memory pools
> > 333094 pinned pages
> > 80.1 maxpin percentage
> > 20.0 minperm percentage
> > 80.0 maxperm percentage
> > 57.1 numperm percentage
> > 1114800 file pages
> > 0.0 compressed percentage
> > 0 compressed pages
> > 57.7 numclient percentage
> > 80.0 maxclient percentage
> > 1127373 client pages
> > 0 remote pageouts scheduled
> > 5943302 pending disk I/Os blocked with no pbuf
> > 25321 paging space I/Os blocked with no psbuf
> > 2740 filesystem I/Os blocked with no fsbuf
> > 0 client filesystem I/Os blocked with no fsbuf
> > 1436111 external pager filesystem I/Os blocked with no
> > fsbuf
>
> > # lsattr -El aio0
> > autoconfig available STATE to be configured at system restart True
> > fastpath enable State of fast path True
> > kprocprio 39 Server PRIORITY True
> > maxreqs 4096 Maximum number of REQUESTS True
> > maxservers 50 MAXIMUM number of servers per cpu True
> > minservers 10 MINIMUM number of servers True
>
> > The Informix databases are operating on raw LVs and so there should
> > not be a double-caching issue that I have heard mentioned. I have
> > other databases on other servers with less memory and those do not
> > seem to be causing high paging activity.
>
> > The users are not complaining about slow performance or any noticeable
> > issues, so just wondering if anyone may have a suggestion as to
> > whether this is a problem, and if there are any settings I should be
> > looking at that could tell me why this system has decided to use a
> > significant portion of its paging space for no apparent reason.
>
> > Thanks in advance.
>
> > Steve
>
> 1)
> Fromhttp://www.ibm.com/developerworks/views/aix/libraryview.jsp?search_by...
>
> Part 3
>
> Paging allocation type:
> - deferred page space allocation
> - late page space allocation
> - early page space allocation.
>
> The default policy of AIX is deferred page space allocation.
>
> 2)
> For some reason your system is using pretty much file cache. Could be
> NFS, log files and what the heck i know but it looks like the
> computational pages get page out. So check for busy filesystem.
>
> In case the paging rate is low the system is working as designed ;-)
>
> hth
> Hajo

A high % size of paging space in lsps could be partly due to the fact
that at various points of time comp pages have been paged out to
paging space. This is because about 57% of the memory is filled with
file pages which have been protected due to their high repage rates
and numperm being between minperm and maxperm.
Now unless paging space garbage collection is enabled, space in the
paging space is reserved (and shows up in lsps as used) even if the
page is brought to RAM again unless the process stops and releases
memory. (as per perf management guide)
Secondly, does having almost double the paging space (of the amount of
RAM) really helps?
Also what is the lru_file_repage parameter set to?

-Ashok Sangra

Hi there,
To find out what is using such paging space us the command svmon -Pg .
It will show you a list of processes where you can find a column
called Pgsp that indicates you how many 4k pages the process is using.
I usually use the command on the following way:
svmon -Pg |grep -p Pid|more

Regards,

Jackson

> Secondly, does having almost double the paging space (of the amount of
> RAM) really helps?
> Also what is the lru_file_repage parameter set to?
>
> -Ashok Sangra-

Hi Ashok,

Thanks for the advice. I will review what you mentioned. As for the
paging space sizes, our policy is that if the system has less that 4GB
of memory, we provide double that size for paging space. Once you get
to 4GB or more, we normally reduce this calculation to be equal to
actual memory as the extra paging space is not of much benefit. These
are calculations based on IBM recommendations and the redbooks we have
reviewed for AIX 4 and AIX 5.

The system in question has 8GB of memory, so I would normally set it
to 8GB of paging space. However, one of our operators was nervous
about the paging space going to 100% full and so increased the size to
see if it would help.

Steve

On Apr 24, 8:25 am, Jack <jakrai...@yahoo.com> wrote:
> Hi there,
> To find out what is using such paging space us the command svmon -Pg .
> It will show you a list of processes where you can find a column
> called Pgsp that indicates you how many 4k pages the process is using.
> I usually use the command on the following way:
> svmon -Pg |grep -p Pid|more
>
> Regards,
>
> Jackson

Hi Jackson,

Thanks for the command. I tried it out and determined that the
processes using up the most 4k pages were the Informix "oninit" pids.
This is what I suspected, but at least now I can see which PIDs in
particular are using the most, so I can try to fine tune a bit more
and check for incorrect onconfig settings (incorrect as in different
than our other servers that don't have this problem.)

Here is what was displayed as the highest usage:

-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit
Mthrd LPage
975072 oninit 21579 5460 1359218 1369143 Y
N N
909510 oninit 21550 5460 1359205 1369105 Y
N N
925902 oninit 21472 5460 1359092 1369013 Y
N N
852150 oninit 21468 5460 1359088 1369015 Y
N N
856240 oninit 21455 5460 1359087 1369008 Y
N N
938044 oninit 21455 5460 1359085 1369005 Y
N N
700416 oninit 21455 5460 1359082 1369005 Y
N N
966882 oninit 21462 5460 1359075 1368996 Y
N N
1847378 oninit 134123 5460 496749 506238 Y
N N
901270 oninit 134074 5460 496694 506154 Y
N N
1728628 oninit 133980 5460 496582 506062 Y
N N
520402 oninit 133994 5460 496579 506066 Y
N N
1532078 oninit 133977 5460 496579 506058 Y
N N
860278 oninit 133977 5460 496578 506058 Y
N N
598160 oninit 133995 5460 496578 506065 Y
N N
548974 oninit 133977 5460 496578 506058 Y
N N
1327106 oninit 133995 5460 496578 506065 Y
N N
1822948 oninit 133978 5460 496576 506057 Y
N N
1544276 oninit 133978 5460 496575 506054 Y
N N
864346 oninit 133978 5460 496573 506054 Y
N N
676034 oninit 133978 5460 496572 506055 Y
N N
1237182 oninit 134010 5460 496532 506042 Y
N N
-------------------------------------------------------------------------------

Thx.

SteveN

Upon further review of the issue, we found that the issue with the
paging is because of what we believe is a bug in Informix 9.4 FC7. We
have only three systems still running that version, and the symptom we
see there is that the memory is not released after completing certain
procedures in the Informix database.

The paging space issue seems to be more noticeable on the one server,
as that one does a lot of information warehouse data analysis, which
use the "questionable" routines that were causing the problem.

Temporary solution: Run "onmode -F" on the instance to free up memory
segments that were reserved but not released properly.

Permanent solution: Upgrade to Informix 10 or higher. We tested the
same scenario on that platform and it worked fine. It does not chew up
memory and start swapping it after some time.

So it ends up it is not really an AIX issue, but hope this may help
someone who is running Informix on AIX, or the AIX admin can advise
their Infromix DBA that it is clearly a RDBMS issue causing the
problem.

SteveN