SEO

I'm new to this group, but a bit of googling suggests it is the
appropriate place to find people in the know about the internal
behaviour of shared libraries on AIX.

I'd very much appreciate any help I can get in trying to understand
(and overcome) an odd problem we are seeing on AIX 4.3.3:

We support an application written using a third party tool which
generates 'C' and builds shared libraries which are then pulled into
the application via dynamic loading (dlopen(), dlsym(), etc).

The third party tool being used runs on multiple versions of UNIX, and
when building shared libraries on AIX seems to attempt to follow the
naming conventions on other UNIX systems by naming the shared objects
as libn.so.n (eg lib8.so.1)

If a function in one of these shared libraries is updated, the tool
deletes the old library and creates a new one, but changes the name by
adding a new dot number (eg lib8.so.1.1) extension, ad infinitum.
Occasionally, one of the numbers gets incremented instead of having a
new one appended, but we cannot fathom the circumstances under which
this happens and the tool which does this is a black box to us. In
all circumstances the effect is that the old version of the library
ceases to exist and a new version with a modified name comes into
existence.

If we run genkld on the system, multiple entries are seen for the
libraries which have been modified, but only one of the referenced
library files actually exists.

Sometimes, when a change has been made to one of these libraries,
other processes which had previously called functions in the old
version of the library will fall over with a message referencing the
pathname of the library which has been changed. Unfortunately, this
message is reported to us in a manner that doesn't say exactly how it
arose (dlopen(), dlsym(), or otherwise), and somewhat truncated so we
can't tell if it is the old version or the new that is being
complained about. The fact that the application process concerned can
then be restarted without error makes me believe the pathname
referenced is the old one though.

What I'm wondering is whether there are circumstances or tunable
parameters which would cause AIX to page out shared objects which
haven't been referenced for a while in the expectation that they can
be re-read from disk if required. If so, perhaps this explains the
problem: the old library is located and dlopen()ed, the function
required is called and returns; time passes, pages of memory
associated with the shared object are stolen; the function address is
then re-referenced and AIX tries to bring back the pages from the
original library, but finds that the path it has stored is no longer
valid?

If this is a plausible explanation of the problem, the next step is to
work out how to avoid it. We have other AIX sites running 4.3.3 and
another running 5.1 where we have never seen this problem occur, but I
am having a lot of trouble trying to find significant environmental
differences between the sites. There are differences, of course, but
identifying which are significant is an issue.

I can't even see why this site should be paging more than the others -
whenever I look it seems to have plenty of free memory - lsattr
indicates it has 1GB to share between 25-30 users! Perhaps there is a
limit to the amount of memory that can be allocated to the global
shared library segment and this is tuned lower at this site than the
others, but I can't see any mention of such a limit in the fine
manual.

I realise this has been a long and involved first post, but I suppose
I'm really just at a loss as to how to progress from here and would
greatly appreciate any help others with more AIX expertise might be
able to offer. The obvious steps that we've already taken have been
to raise a support call with the third party whose tool we use to
generate the shared libraries and to stop the online generation of
changes to functions at this site for the time being.

Best Regards,

Ken Wallis