SEO

Hi all,

I've got an application built on a pseries machine under AIX 5.3.
Under certain circumstances (which we have yet to identify
precisely!), the application may hang, segv, throw a memory error, or
even a "catastrophe in malloc" error. The problem is, the error is
very sporadic: when it happens, it happens with a particular function
which retrieves data from a database. Once it start happening though,
it happens consistently, to all users of that particular app, at
exactly the same point in the program (and it isn't always the same
point ... each time it happens, for the user, it is somewhere
different, but it seems to be traceable to the same internal (shared)
library function ... a function that allocates memory for an array of
structs to receive database information). Here's the strange part:
once an error has started appearing, the only cure is a reboot of the
entire machine (!)

Since restarting just the application isn't enough to resolve the
problem (even rebuilding the app doesn't do it), it seems like we may
be looking at some kind of memory corruption issue. Unfortunately,
I've never been onsite when the problem crops up (fortunately it's not
*too* frequent!), so I don't know what was going on with the machine
(paging, filesystem full, etc.).

One thing I've noticed though, is this: the application uses
functions from the pthread library (libpthreads) and the reentrant C
library (libc_r). I've looked at the xlc_r stanza in /etc/vac.cfg and
have the following configuration:

* IBM XL C/C++ Enterprise Edition Version 7.0 for 53

* standard c compiler aliased as xlc_r (53 Threads)
xlc_r: use = DEFLT
crt = /lib/crt0.o
mcrt = /lib/mcrt0.o
gcrt = /lib/gcrt0.o
libraries = -L/usr/vac/lib,-lxlopt,-L/usr/lib/threads,-lpthreads,-lc
proflibs = -L/lib/profiled,-L/usr/lib/profiled
hdlibs = -L/usr/vac/lib,-lhmd_r
options = -qlanglvl=extc89,-qcpluscmt,-qkeyword=inline,-qalias=ansi,-
qthreaded,-D_THREAD_SAFE,-D__VACPP_MULTI__

Now what I am wondering is this: shouldn't we perhaps substitute this
line:
crt = /lib/crt0.o (and mcrt0 and gcrt0)
For the thread-safe versions of the reentrant C runtime start-up
routines ? crt0_r.o ,mcrt0_r.o, gcrt0_r.o ?

I've seen an application core dump when it needed to be linked to the
_r versions of these routines (back in AIX 4.x), but is there any
chance that this could explain the strange and spontaneous errors we
have been seeing?

Any helpful suggestions will be greatly appreciated!

Regards,

Jim
Rio de Janeiro

jimwgramling@gmail.com writes:

> I've got an application built on a pseries machine under AIX 5.3.
> Under certain circumstances (which we have yet to identify
> precisely!), the application may hang, segv, throw a memory error, or
> even a "catastrophe in malloc" error.

All typical symptoms of heap corruption...

> Since restarting just the application isn't enough to resolve the
> problem (even rebuilding the app doesn't do it), it seems like we may
> be looking at some kind of memory corruption issue.

It is extremely likely that you are looking at heap corruption.

> Unfortunately,
> I've never been onsite when the problem crops up (fortunately it's not
> *too* frequent!), so I don't know what was going on with the machine
> (paging, filesystem full, etc.).

It wouldn't have helped anyway -- by the time you observe the
problem, the part of code that actually did the damage is likely
nowhere to be seen (not currently on the stack).

> For the thread-safe versions of the reentrant C runtime start-up
> routines ? crt0_r.o ,mcrt0_r.o, gcrt0_r.o ?

On AIX 5.1, crt0_r.o is a symlink to crt0.o.
Most likely the same is true on AIX 5.3.

> Any helpful suggestions will be greatly appreciated!

You need a heap debugger.

Start with this one:
http://www.redbooks.ibm.com/redbooks...tml/wwhelp.htm
and go to 4.2.4 The debug malloc allocator.

Your other options are (commercial) Insure++, Purify and ZeroFault.

Cheers,
--
In order to understand recursion you must first understand recursion.
Remove /-nsp/ for email.

Thanks Paul! That was very helpful indeed!

You were quite correct: crt0_r.o is a symlink to crt0.o. And, I spent
most of today playing around with the malloc debug allocator ... and
actually managed to capture a core dump, although I haven't figured
out anything from it yet. It's been many years since I've worked with
AIX and debugging a core dump is somewhat intimidating!

Am I right in the following assumption: if I shut down every instance
of the particular application, shouldn't the allocated heap be
released, unless it is a shared memory segment or data segments
associated with a shared library held open by some other app? I ask
this, because I have tracked down the most likely culprit to a
particular shared library, which is linked both to the client app as
well as the server app. Even after shutting down all client apps, the
problem continued after restarting the executable. But once I shut
down the server, and released the shared library (as verified with
genkld), the problem disappeared.

I am somewhat at a loss at how to proceed. The shared library is part
of a proprietary commercial software package. Right now, I'm hoping
for some inspiration, or a lucky break!

Thanks again for your input.

Regards,

Jim

jimwgramling@gmail.com writes:

> Am I right in the following assumption: if I shut down every instance
> of the particular application, shouldn't the allocated heap be
> released, unless it is a shared memory segment or data segments
> associated with a shared library held open by some other app?

Heap is not shared between processes; shutting down one instance of
a particular application has no effect on any other instances of it.

But SYSV shared memory is another story -- shutting down *all*
instances of apps using it doesn't clean shared memory.

> I ask
> this, because I have tracked down the most likely culprit to a
> particular shared library, which is linked both to the client app as
> well as the server app. Even after shutting down all client apps, the
> problem continued after restarting the executable. But once I shut
> down the server, and released the shared library (as verified with
> genkld), the problem disappeared.

Perhaps the server "voluntarily" cleans up shared memory (if it
even uses shared memory).

You can run "ipcs" before and after starting the server -- any new
shared memory segments indicate that the server does use shared
memory.

> I am somewhat at a loss at how to proceed. The shared library is part
> of a proprietary commercial software package.

That's tough. You could plead with the vendor to provide you a
debugging version of that library.

> Right now, I'm hoping for some inspiration, or a lucky break!

Lucky break always helps :-)

Cheres,
--
In order to understand recursion you must first understand recursion.
Remove /-nsp/ for email.

Paul wrote:

> Heap is not shared between processes; shutting down one instance of
> a particular application has no effect on any other instances of it.
>
> But SYSV shared memory is another story -- shutting down *all*
> instances of apps using it doesn't clean shared memory.
>

But that is the whole point of this problem: the server (an Ingres
database server, fyi) does use shared memory. The client / fe
software absolutely does not. When this problem crops up *within the
front-end (user program)*, it appears in all instances of the
application (other users, other processes). For example, yesterday,
the app started throwing a memory allocation error whenever the user
chose a certain menu option, after returning from a system call which
spawned an external process. If another user were to run the same
program, the user would get the same error if he/she followed the same
sequence. In this case, the application trapped the allocation
failure, but in the past, there have been segv's and application
hangs.

I ran with MALLOCTYPE=debug and captured a core. We were absolutely
alone on the machine, which allowed us to test the application in a
controlled environment; there were no external factors that we could
perceive (system errors, swapping, etc.)

The problem persisted in all instances (processes) of the user app
until I shut down the server.

The only thing the app has in common with the server itself (other
than the data ... a database table, which is not written or otherwise
changed, and is accessable from other programs), is a set of shared
runtime libraries used by the front end program as well as the
server's management tools.

So what could be happening that affects all instances of a user
process??

Thanks again for your feedback, Paul!

Regards,

Jim



> > I ask
> > this, because I have tracked down the most likely culprit to a
> > particular shared library, which is linked both to the client app as
> > well as the server app. Even after shutting down all client apps, the
> > problem continued after restarting the executable. But once I shut
> > down the server, and released the shared library (as verified with
> > genkld), the problem disappeared.
>
> Perhaps the server "voluntarily" cleans up shared memory (if it
> even uses shared memory).
>
> You can run "ipcs" before and after starting the server -- any new
> shared memory segments indicate that the server does use shared
> memory.
>
> > I am somewhat at a loss at how to proceed. The shared library is part
> > of a proprietary commercial software package.
>
> That's tough. You could plead with the vendor to provide you a
> debugging version of that library.
>
> > Right now, I'm hoping for some inspiration, or a lucky break!
>
> Lucky break always helps :-)
>
> Cheres,
> --
> In order to understand recursion you must first understand recursion.
> Remove /-nsp/ for email.

jimwgramling@gmail.com writes:

>> But SYSV shared memory is another story -- shutting down *all*
>> instances of apps using it doesn't clean shared memory.
>>
>
> But that is the whole point of this problem: the server (an Ingres
> database server, fyi) does use shared memory. The client / fe
> software absolutely does not.

How do you know that the client doesn't attach to the Ingres shared
memory segment? Your own code may not use it, but that "client
shared library" that you link against may well do so.

Note that for the server to use shared memory only makes sense if
either
- the server itself is implemented as several (more than one)
communicating processes, or
- its clients *also* use shared memory.

> I ran with MALLOCTYPE=debug and captured a core.

It might help if you posted the actual crash stack trace from dbx.

> The problem persisted in all instances (processes) of the user app
> until I shut down the server.

This is consistent with the server getting into a state where
it provides "unexpected" answer to all the clients.

Whether this is due to a bug in the server (provides "wrong" answer),
or in the clients (which do not anticipate all possible answers),
is hard to tell without more details.

Cheers,
--
In order to understand recursion you must first understand recursion.
Remove /-nsp/ for email.

> Note that for the server to use shared memory only makes sense if
> either
> - the server itself is implemented as several (more than one)
> communicating processes, or
> - its clients *also* use shared memory.

The Ingres installation is a set of processes ... it uses shared
memory for the locking system and other internal resources, and also
the transaction logging system, as well as for an internal table cache
which can be quite large, if you use multiple dbms servers. The
Ingres client software is completely independent of the dbms server,
using tcp sockets for all "client/server" communication, even if the
client software is local. The dbms server can be remote, which of
course rules out any possibility of using the dbms shared memory.

> Whether this is due to a bug in the server (provides "wrong" answer),
> or in the clients (which do not anticipate all possible answers),
> is hard to tell without more details.

I understand how the possibility of corrupted data sent back from the
server would look like a likely possibility based on the symptoms,
but, although I don't completely rule it out, there are reasons to
believe it is unlikely.

Today I opened an issue with Ingres Corp tech support, since the user
app is based on Ingres development tools. They of course want a test
case, which is almost impossible based on the spurious nature of the
error, but hopefully we will be able to get them some diagnostics
which may aid in discovering what is going on.

Thanks for your help!

Regards,

Jim