SEO

Hi Guru's

I am facinga strange problem in AIX 5.3 (ML04), i have been fcing
issues with libraries.. Everything is working fine untill all of a
sudden, i start getting library loading error. i cannot execute any
command, every comman gives an output saying library cannot be loaded,
and no one can even telnet to the machine, when ever anyone tries to
telnet they get th emessage



telnet (cagney)

Could not load program login:
Dependent module libc.a(shr.o) could not be loaded.
Could not load module libc.a(shr.o).

Connect to host lost...

Is there any one who has come across this problem?? or know the reason
for this issue????

Only a reboot can fix this issue......

Regards,
Pattabhi Raman

Hi,
You are facing resource issues. Had the exact same problem a few weeks
ago when somebody changed an app users limits to unlimited. Try to find
the app consumimg the most memory etc. Stop that app and the problem
will go away.
You will probably need to kill semaphores as well. You will then need
to put in limits on the user to prevent losing your box.


Unix wrote:
> Hi Guru's
>
> I am facinga strange problem in AIX 5.3 (ML04), i have been fcing
> issues with libraries.. Everything is working fine untill all of a
> sudden, i start getting library loading error. i cannot execute any
> command, every comman gives an output saying library cannot be loaded,
> and no one can even telnet to the machine, when ever anyone tries to
> telnet they get th emessage
>
>
>
> telnet (cagney)
>
> Could not load program login:
> Dependent module libc.a(shr.o) could not be loaded.
> Could not load module libc.a(shr.o).
>
> Connect to host lost...
>
> Is there any one who has come across this problem?? or know the reason
> for this issue????
>
> Only a reboot can fix this issue......
>
> Regards,
> Pattabhi Raman

Unix wrote:
> Hi Guru's
>
> I am facinga strange problem in AIX 5.3 (ML04), i have been fcing
> issues with libraries.. Everything is working fine untill all of a
> sudden, i start getting library loading error. i cannot execute any
> command, every comman gives an output saying library cannot be loaded,
> and no one can even telnet to the machine, when ever anyone tries to
> telnet they get th emessage
>
> telnet (cagney)
>
> Could not load program login:
> Dependent module libc.a(shr.o) could not be loaded.
> Could not load module libc.a(shr.o).

I was having the same problem. IBM AIX support informed me that there
is a bug in the 64-bit kernel in AIX 5.3 ML4 which causes a memory
leak. APAR IY84780 will address the issue; unfortunately, it is not
available yet.

http://www-1.ibm.com/support/docview...id=isg1IY84780

IBM support has found that enabling MODS will normally prevent the
problem from occurring until the APAR is available. The following is
from a document that IBM sent to me.

MODS (Memory Overlay Detection System)


Overview
Some of the most difficult types of problems to debug are what are
generally called "memoryoverlays." Memory overlays include the
following:

· Writing to memory that is owned by another program or routine
· Writing past the end (or before the beginning) of declared
variables or arrays
· Writing past the end (or before the beginning) of
dynamically-allocated memory
· Writing to or reading from freed memory
· Freeing memory twice
· Calling memory allocation routines with incorrect parameters or
under incorrect conditions.

In the kernel environment (including the AIX kernel, kernel extensions,
and device drivers), memory overlay problems have been especially
difficult to debug because tools for finding them have not been
available.

Starting with AIX Version 4.2.1, however, the Memory Overlay Detection
System helps detect memory overlay problems in the AIX kernel, kernel
extensions, and device drivers.

How to Enable MODS
# bosdebug -M
# bosboot -a -d /dev/ipldevice
# reboot the system

How to Disaable MODS
# bosdebug -o
# bosboot -a -d /dev/ipldevice
# reboot the system

How to check the current status of MODS
# bosdebug -L
Memory debugger off <<<<< MODS is off
Memory sizes 0
Network memory sizes 0
Kernel debugger off
Real Time Kernel off
Backtracking fault log off

How MODS Works
The primary goal of the MODS feature is to produce a dump file that
accurately identifies the problem. MODS works by turning on additional
checking to help detect the conditions listed above. When any of these
conditions is detected, your system crashes immediately and produces a
dump file that points directly at the offending code. (Previously, a
system dump might point to unrelated code that happened to be running
later when the invalid situation was finally detected.)

Turning MODS on has a small impact on overall system performance and
causes somewhat more memory to be used in the kernel and the network
memory heaps. If your system is running at full CPU utilization or if
you are already near the maximums for kernel memory usage, turning on
the MODS may cause some performance degradation.

Wow, That was great piece of information.

Thanks a lot.

Regards,
Pattabhi Raman
dhoffman@cerner.com wrote:
> Unix wrote:
> > Hi Guru's
> >
> > I am facinga strange problem in AIX 5.3 (ML04), i have been fcing
> > issues with libraries.. Everything is working fine untill all of a
> > sudden, i start getting library loading error. i cannot execute any
> > command, every comman gives an output saying library cannot be loaded,
> > and no one can even telnet to the machine, when ever anyone tries to
> > telnet they get th emessage
> >
> > telnet (cagney)
> >
> > Could not load program login:
> > Dependent module libc.a(shr.o) could not be loaded.
> > Could not load module libc.a(shr.o).
>
> I was having the same problem. IBM AIX support informed me that there
> is a bug in the 64-bit kernel in AIX 5.3 ML4 which causes a memory
> leak. APAR IY84780 will address the issue; unfortunately, it is not
> available yet.
>
> http://www-1.ibm.com/support/docview...id=isg1IY84780
>
> IBM support has found that enabling MODS will normally prevent the
> problem from occurring until the APAR is available. The following is
> from a document that IBM sent to me.
>
> MODS (Memory Overlay Detection System)
>
>
> Overview
> Some of the most difficult types of problems to debug are what are
> generally called "memoryoverlays." Memory overlays include the
> following:
>
> · Writing to memory that is owned by another program or routine
> · Writing past the end (or before the beginning) of declared
> variables or arrays
> · Writing past the end (or before the beginning) of
> dynamically-allocated memory
> · Writing to or reading from freed memory
> · Freeing memory twice
> · Calling memory allocation routines with incorrect parameters or
> under incorrect conditions.
>
> In the kernel environment (including the AIX kernel, kernel extensions,
> and device drivers), memory overlay problems have been especially
> difficult to debug because tools for finding them have not been
> available.
>
> Starting with AIX Version 4.2.1, however, the Memory Overlay Detection
> System helps detect memory overlay problems in the AIX kernel, kernel
> extensions, and device drivers.
>
> How to Enable MODS
> # bosdebug -M
> # bosboot -a -d /dev/ipldevice
> # reboot the system
>
> How to Disaable MODS
> # bosdebug -o
> # bosboot -a -d /dev/ipldevice
> # reboot the system
>
> How to check the current status of MODS
> # bosdebug -L
> Memory debugger off <<<<< MODS is off
> Memory sizes 0
> Network memory sizes 0
> Kernel debugger off
> Real Time Kernel off
> Backtracking fault log off
>
> How MODS Works
> The primary goal of the MODS feature is to produce a dump file that
> accurately identifies the problem. MODS works by turning on additional
> checking to help detect the conditions listed above. When any of these
> conditions is detected, your system crashes immediately and produces a
> dump file that points directly at the offending code. (Previously, a
> system dump might point to unrelated code that happened to be running
> later when the invalid situation was finally detected.)
>
> Turning MODS on has a small impact on overall system performance and
> causes somewhat more memory to be used in the kernel and the network
> memory heaps. If your system is running at full CPU utilization or if
> you are already near the maximums for kernel memory usage, turning on
> the MODS may cause some performance degradation.

Hi All,

Now that i have applied the fix pack and updated my machine to ML5,
shoul i disable MODS??

Regards,
Pattabhi Raman
Unix wrote:
> Wow, That was great piece of information.
>
> Thanks a lot.
>
> Regards,
> Pattabhi Raman
> dhoffman@cerner.com wrote:
> > Unix wrote:
> > > Hi Guru's
> > >
> > > I am facinga strange problem in AIX 5.3 (ML04), i have been fcing
> > > issues with libraries.. Everything is working fine untill all of a
> > > sudden, i start getting library loading error. i cannot execute any
> > > command, every comman gives an output saying library cannot be loaded,
> > > and no one can even telnet to the machine, when ever anyone tries to
> > > telnet they get th emessage
> > >
> > > telnet (cagney)
> > >
> > > Could not load program login:
> > > Dependent module libc.a(shr.o) could not be loaded.
> > > Could not load module libc.a(shr.o).
> >
> > I was having the same problem. IBM AIX support informed me that there
> > is a bug in the 64-bit kernel in AIX 5.3 ML4 which causes a memory
> > leak. APAR IY84780 will address the issue; unfortunately, it is not
> > available yet.
> >
> > http://www-1.ibm.com/support/docview...id=isg1IY84780
> >
> > IBM support has found that enabling MODS will normally prevent the
> > problem from occurring until the APAR is available. The following is
> > from a document that IBM sent to me.
> >
> > MODS (Memory Overlay Detection System)
> >
> >
> > Overview
> > Some of the most difficult types of problems to debug are what are
> > generally called "memoryoverlays." Memory overlays include the
> > following:
> >
> > · Writing to memory that is owned by another program or routine
> > · Writing past the end (or before the beginning) of declared
> > variables or arrays
> > · Writing past the end (or before the beginning) of
> > dynamically-allocated memory
> > · Writing to or reading from freed memory
> > · Freeing memory twice
> > · Calling memory allocation routines with incorrect parameters or
> > under incorrect conditions.
> >
> > In the kernel environment (including the AIX kernel, kernel extensions,
> > and device drivers), memory overlay problems have been especially
> > difficult to debug because tools for finding them have not been
> > available.
> >
> > Starting with AIX Version 4.2.1, however, the Memory Overlay Detection
> > System helps detect memory overlay problems in the AIX kernel, kernel
> > extensions, and device drivers.
> >
> > How to Enable MODS
> > # bosdebug -M
> > # bosboot -a -d /dev/ipldevice
> > # reboot the system
> >
> > How to Disaable MODS
> > # bosdebug -o
> > # bosboot -a -d /dev/ipldevice
> > # reboot the system
> >
> > How to check the current status of MODS
> > # bosdebug -L
> > Memory debugger off <<<<< MODS is off
> > Memory sizes 0
> > Network memory sizes 0
> > Kernel debugger off
> > Real Time Kernel off
> > Backtracking fault log off
> >
> > How MODS Works
> > The primary goal of the MODS feature is to produce a dump file that
> > accurately identifies the problem. MODS works by turning on additional
> > checking to help detect the conditions listed above. When any of these
> > conditions is detected, your system crashes immediately and produces a
> > dump file that points directly at the offending code. (Previously, a
> > system dump might point to unrelated code that happened to be running
> > later when the invalid situation was finally detected.)
> >
> > Turning MODS on has a small impact on overall system performance and
> > causes somewhat more memory to be used in the kernel and the network
> > memory heaps. If your system is running at full CPU utilization or if
> > you are already near the maximums for kernel memory usage, turning on
> > the MODS may cause some performance degradation.