SEO

Did you do anything about the network messages?
Informix told you there were network problems at 8:27AM, and didn't crash until 10AM.
Did you try to fix or figure out the network problem in that 1.5 hours before informix gave up?
My guess is you have some important part of informix on a shared/network drive and informix could not get to it.



-----Original Message-----
From: informix-list-bounces@iiug.org [mailto:informix-list-bounces@iiug.org] On Behalf Of Omar "Muñoz
Sent: Sunday, February 17, 2008 11:35 AM
To: informix-list-bounces@iiug.org; informix-list@iiug.org
Subject: Help about server which hangs during restart

Hi.

Once - when we tried to bring a database online
after we had shut it down without problems as a part
of a weekly task- database hung out for about an hour.
Online.log looked like this:

08:19:13 On-Line Mode
08:19:13 Affinitied VP 1 to phys proc 4
08:27:11 VP Notify mechanism incomplete after 5
minutes. This can be due to slo
w network file access. Will try 12 more times
08:35:11 VP Notify mechanism incomplete after 5
minutes. This can be due to slo
w network file access. Will try 11 more times
08:43:09 VP Notify mechanism incomplete after 5
minutes. This can be due to slo
w network file access. Will try 10 more times
08:51:08 VP Notify mechanism incomplete after 5
minutes. This can be due to slo
w network file access. Will try 9 more times

After that, database crashed, and online.log showed:

09:54:56 notifyvp(): vp 3, pid 22717 of class 0
didn't rcv
09:54:56 notifyvp(): vp 4, pid 22718 of class 0
didn't rcv
09:54:56 notifyvp(): vp 5, pid 22719 of class 0
didn't rcv
09:54:56 notifyvp(): vp 6, pid 22720 of class 0
didn't rcv
09:54:56 notifyvp(): vp 7, pid 22721 of class 0
didn't rcv
09:54:56 notifyvp(): vp 8, pid 22722 of class 0
didn't rcv
09:54:56 notifyvp(): vp 9, pid 22723 of class 0
didn't rcv
09:54:56 Assert Failed: mt_notifyvp timed out
09:54:56 IBM Informix Dynamic Server Version 9.40.FC7
09:54:56 Who: Session(1, informix@orion, 0,
35930e028)
Thread(7, main_loop(), 3592cc028, 1)
File: mt.c Line: 11121
09:54:56 stack trace for pid 22637 written to
/respaldo2/informix/tmp/siisa/af.
3ef58cf
09:54:56 See Also:
/respaldo2/informix/tmp/siisa/af.3ef58cf,
shmem.3ef58cf.0
09:56:12 Error writing
'/respaldo2/informix/tmp/siisa/shmem.3ef58cf.0' errno
=
28
09:56:12 mt.c, line 11121, thread 7, proc id 22637,
mt_notifyvp timed out.
09:56:14 The Master Daemon Died
09:56:15 PANIC: Attempting to bring system down

We finally were able to bring informix online only by
restarting the entire server (a Sun server with
Solaris 9). Even so, I'd like to ask you which is the
reason why this kind of error happens. I'm talking
about an informix 9.40 FC7 database.

Thanks in advance

Omar Munoz



__________________________________________________ __________________________________
Looking for last minute shopping deals?
Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsea...egory=shopping
_______________________________________________
Informix-list mailing list
Informix-list@iiug.org
http://www.iiug.org/mailman/listinfo/informix-list
================================================== ==========
The information contained in this message may be privileged
and confidential and protected from disclosure. If the reader
of this message is not the intended recipient, or an employee
or agent responsible for delivering this message to the
intended recipient, you are hereby notified that any reproduction,
dissemination or distribution of this communication is strictly
prohibited. If you have received this communication in error,
please notify us immediately by replying to the message and
deleting it from your computer. Thank you. Tellabs
================================================== ==========

On 17 Feb, 17:42, "Sebastian, Norma J."
<NormaJean.Sebast...@tellabs.com> wrote:
> Did you do anything about the network messages?
> Informix told you there were network problems at 8:27AM, and didn't crash until 10AM.
> Did you try to fix or figure out the network problem in that 1.5 hours before informix gave up?
> My guess is you have some important part of informix on a shared/network drive and informix could not get to it.
>
>
>
> -----Original Message-----
> From: informix-list-boun...@iiug.org [mailto:informix-list-boun...@iiug.org] On Behalf Of Omar "Muñoz
> Sent: Sunday, February 17, 2008 11:35 AM
> To: informix-list-boun...@iiug.org; informix-l...@iiug.org
> Subject: Help about server which hangs during restart
>
> Hi.
>
> * *Once - when we tried to bring a database online
> after we had shut it down without problems as a part
> of a weekly task- database hung out for about an hour.
> Online.log looked like this:
>
> 08:19:13 *On-Line Mode
> 08:19:13 *Affinitied VP 1 to phys proc 4
> 08:27:11 *VP Notify mechanism incomplete after 5
> minutes. This can be due to slo
> w network file access. Will try 12 more times
> 08:35:11 *VP Notify mechanism incomplete after 5
> minutes. This can be due to slo
> w network file access. Will try 11 more times
> 08:43:09 *VP Notify mechanism incomplete after 5
> minutes. This can be due to slo
> w network file access. Will try 10 more times
> 08:51:08 *VP Notify mechanism incomplete after 5
> minutes. This can be due to slo
> w network file access. Will try 9 more times
>
> After that, database crashed, and online.log showed:
>
> 09:54:56 *notifyvp(): vp 3, pid 22717 of class 0
> didn't rcv
> 09:54:56 *notifyvp(): vp 4, pid 22718 of class 0
> didn't rcv
> 09:54:56 *notifyvp(): vp 5, pid 22719 of class 0
> didn't rcv
> 09:54:56 *notifyvp(): vp 6, pid 22720 of class 0
> didn't rcv
> 09:54:56 *notifyvp(): vp 7, pid 22721 of class 0
> didn't rcv
> 09:54:56 *notifyvp(): vp 8, pid 22722 of class 0
> didn't rcv
> 09:54:56 *notifyvp(): vp 9, pid 22723 of class 0
> didn't rcv
> 09:54:56 *Assert Failed: mt_notifyvp timed out
> 09:54:56 *IBM Informix Dynamic Server Version 9.40.FC7
> 09:54:56 * Who: Session(1, informix@orion, 0,
> 35930e028)
> * * * * * * * * Thread(7, main_loop(), 3592cc028, 1)
> * * * * * * * * File: mt.c Line: 11121
> 09:54:56 *stack trace for pid 22637 written to
> /respaldo2/informix/tmp/siisa/af.
> 3ef58cf
> 09:54:56 * See Also:
> /respaldo2/informix/tmp/siisa/af.3ef58cf,
> shmem.3ef58cf.0
> 09:56:12 *Error writing
> '/respaldo2/informix/tmp/siisa/shmem.3ef58cf.0' errno
> =
> 28
> 09:56:12 *mt.c, line 11121, thread 7, proc id 22637,
> mt_notifyvp timed out.
> 09:56:14 *The Master Daemon Died
> 09:56:15 *PANIC: Attempting to bring system down
>
> We finally were able to bring informix online only by
> restarting the entire server (a Sun server with
> Solaris 9). Even so, I'd like to ask you which is the
> reason why this kind of error happens. I'm talking
> about an informix 9.40 FC7 database.
>
> * *Thanks in advance
>
> * * * * * * * * * * * Omar Munoz
>
> * * * __________________________________________________ _________________________*_________
> Looking for last minute shopping deals? *
> Find them fast with Yahoo! Search. *http://tools.search.yahoo.com/newsea...egory=shopping
> _______________________________________________
> Informix-list mailing list
> Informix-l...@iiug.orghttp://www.iiug.org/mailman/listinfo/informix-list
> ================================================== ==========
> The information contained in this message may be privileged
> and confidential and protected from disclosure. If the reader
> of this message is not the intended recipient, or an employee
> or agent responsible for delivering this message to the
> intended recipient, you are hereby notified that any reproduction,
> dissemination or distribution of this communication is strictly
> prohibited. If you have received this communication in error,
> please notify us immediately by replying to the message and
> deleting it from your computer. Thank you. Tellabs
> ================================================== ==========- Hide quoted text -
>
> - Show quoted text -

No Norma, this does not have to be a network issue, you are wrong. All
it means if that the VP's timed out communicating.
Perhaps one of the threads was hung in an OS call or there is an
Informix bug? DO NOT MAKE ASSUMPTIONS.

There is a known vp notify issue in 9.40.UC7

http://www-1.ibm.com/support/docview...=utf-8&lang=en

Patch to 9.40.UC7W1X1 to fix. PS When will 9.40.xC8 be out?

Omar, there could be some other issue. Who knows, could be a hardware
issue an OS issue or an Informix issue.

You have DUMPSHMEM enabled but if you check under /usr/include/sys/
errno.h on your system you will find errno 28 is no more space.
There is either not enough space under "'/respaldo2/informix/tmp/siisa/
shmem" to dump shared memory or it is trying to write a
file >2Gb on a file system that is not largefile enabled.

What did onstat -g stk all give? Did you strace/truss the server
pids?
Where there any issues in the OS logs?

You need to do more investigation when the problem happens.