SEO

I help to administer about 40 AIX boxen, about 5 of these are oracle
application servers (9i r1, aix 4.3.3 & 5.2r4). We are currently
rebooting these machines nightly because java does not let go of memory
correctly and will bog the box down(thats what I have been told
anyways.) Has anyone else experienced this type of problem and found a
work around to free up memory that has been snatched up by java/oracle
app server but is not being used.

This gets really silly when our testers ask to have the stress-test
server rebooted before they run a stress-test... The poor box (p650 aix
5.2r4) ends up being rebooted multiple times a day which just seems so
wrong with AIX!?!?

In article <1114005641.998658.162260@l41g2000cwc.googlegroups .com>, nate.tade@gmail.com wrote:
> I help to administer about 40 AIX boxen, about 5 of these are oracle
> application servers (9i r1, aix 4.3.3 & 5.2r4). We are currently
> rebooting these machines nightly because java does not let go of memory
> correctly and will bog the box down(thats what I have been told
> anyways.) Has anyone else experienced this type of problem and found a
> work around to free up memory that has been snatched up by java/oracle
> app server but is not being used.
>
> This gets really silly when our testers ask to have the stress-test
> server rebooted before they run a stress-test... The poor box (p650 aix
> 5.2r4) ends up being rebooted multiple times a day which just seems so
> wrong with AIX!?!?
>

You just said the problem. It's not with AIX, but that java is not
letting go of memory properly. What about just bouncing oracle (that
is if java is running internal to oracle). Once the process dies, then
the memory is returned back to the kernel.

You could also look at using 'vmo' if you are running into a lot of swap.

nate.tade@gmail.com wrote:
> I help to administer about 40 AIX boxen, about 5 of these are oracle
> application servers (9i r1, aix 4.3.3 & 5.2r4). We are currently
> rebooting these machines nightly because java does not let go of
memory
> correctly and will bog the box down(thats what I have been told
> anyways.) Has anyone else experienced this type of problem and found
a
> work around to free up memory that has been snatched up by
java/oracle
> app server but is not being used.
>
> This gets really silly when our testers ask to have the stress-test
> server rebooted before they run a stress-test... The poor box (p650
aix
> 5.2r4) ends up being rebooted multiple times a day which just seems
so
> wrong with AIX!?!?


I have a server just like the one you describe, running AIX 4.3.3ML11
with Oracle 9i, Oracle App sever and Java 1.1.8.8

We have other Power 4+ AIX boxes running Oracle 9i at AIX 5.2ML4, and
some p5 systems at AIX 5.3ML1 with Oracle 10g.

None of them have memory problems, or need re-booting. I suspect you
have a memory leak in one of your Java applications using the Oracle
App server.

I would install NMON on the servers, select the "T" option for "Top
Processes", then select the "4" option, which will sort processes in
decending order by size.

This will help identify the app with the memory leak.
Make sure you are current on Oracle service for your release, also.

We only boot AIX after ML updates... etc

nate.tade@gmail.com <nate.tade@gmail.com> wrote:
> I help to administer about 40 AIX boxen, about 5 of these are oracle
> application servers (9i r1, aix 4.3.3 & 5.2r4). We are currently
> rebooting these machines nightly because java does not let go of memory
> correctly and will bog the box down(thats what I have been told
> anyways.) Has anyone else experienced this type of problem and found a
> work around to free up memory that has been snatched up by java/oracle
> app server but is not being used.
>
> This gets really silly when our testers ask to have the stress-test
> server rebooted before they run a stress-test... The poor box (p650 aix
> 5.2r4) ends up being rebooted multiple times a day which just seems so
> wrong with AIX!?!?

The simple solution would be to kill and restart the Java program and
not the machine.
If the problem is within oracle, shut down the instance.
Kill all Java processes left alive, and restart the instance.

That should be faster as well.

Yours,
Laurenz Albe

I would agree with installing NMON and monitoring. However, I had two
H80's on 4.3.3 ml 09, oracle 8.1.7 and 6h1's with 5.2 and oracle 9.2.
They too loose memory and paging space, but I found with help from our
developers that it was not java but was the listener process to the
database. Try bouncing the listeners for the apps servers, when I've
done this for my db servers, the memory has returned.

HTH,
Pete's

nate.tade@gmail.com wrote:
> I help to administer about 40 AIX boxen, about 5 of these are oracle
> application servers (9i r1, aix 4.3.3 & 5.2r4). We are currently
> rebooting these machines nightly because java does not let go of memory
> correctly and will bog the box down(thats what I have been told
> anyways.) Has anyone else experienced this type of problem and found a
> work around to free up memory that has been snatched up by java/oracle
> app server but is not being used.
>
> This gets really silly when our testers ask to have the stress-test
> server rebooted before they run a stress-test... The poor box (p650 aix
> 5.2r4) ends up being rebooted multiple times a day which just seems so
> wrong with AIX!?!?
>


Have you ever look into the Performance Management Guide ?
There is a section on Java tuning. Check for the AIXTHREAD_SCOPE
parameter, maybe, playing around with this param solves your problem

Unfortunetly I am dealing with fragmented memory at that point and the
belief is that AIX cannot get fragmented memory back without a reboot.
Maybe I am wrong about that.

<nate.tade@gmail.com> wrote in message
news:1114005641.998658.162260@l41g2000cwc.googlegr oups.com...
> I help to administer about 40 AIX boxen, about 5 of these are oracle
> application servers (9i r1, aix 4.3.3 & 5.2r4). We are currently
> rebooting these machines nightly because java does not let go of memory
> correctly and will bog the box down(thats what I have been told
> anyways.) Has anyone else experienced this type of problem and found a
> work around to free up memory that has been snatched up by java/oracle
> app server but is not being used.
>
> This gets really silly when our testers ask to have the stress-test
> server rebooted before they run a stress-test... The poor box (p650 aix
> 5.2r4) ends up being rebooted multiple times a day which just seems so
> wrong with AIX!?!?
>
>
Try running Enovia LCA and CATIA V5 - its just as bad.

nate.tade@gmail.com wrote:
> Unfortunetly I am dealing with fragmented memory at that point and the
> belief is that AIX cannot get fragmented memory back without a reboot.
> Maybe I am wrong about that.
>

No offense, but our company does similar stuff with reboots and it is
just avoiding the problem.

1) Java is usually 32-bit, and the segmented architecture of the
environment can mean that some segments for some uses can fill up and
cause problems. A segment is 256MB so look for that number as a boundary.

2) Java itself uses garbage collection for tidying up after itself.
Often these sorts of issues appear when the garbage isn't collected due
to load.

With properly tested applications we just don't get this behaviour.
Check the applications first...... NMon is an execellent tool, and there
are Java tools (i.e. JProbe, Tivoli Performance Viewer with WebSphere,
etc).

Don't give in to rebooting a fairly stable OS regularly just for the
sake of avoiding fixing bad appps! Our management try to get us to do
the same but it is nonsense.......

One tip - Often when we get the same behaviouur, there is some hint of a
trigger in "errpt". For example, on some of our systems when the Java
processes seem to be playing up there has been a previous problem in MQ
or a core dump in some other system that has caused things to back up...

Cheers

Colin

Is your Oracle and Java running with an own user?
If yes try the following:

stop Oracle and java

#ipcs | grep <orauser>

gives you something like:
m 1048584 0xd6cd9648 --rw-r----- orap18 dba

remove the shared memory segment with:

# ipcrm -m 1048584

and then do the same with the java stuff...

HTH
Axel