SEO

#1 (**permalink**) 04-19-2008, 08:37 PM

This message is in MIME format. Since your mail reader does not understand
this format, some or all of this message may not be legible.

------_=_NextPart_001_01C40068.E7163E90
Content-Type: text/plain;
charset="iso-8859-1"

Hi,

I apologize for the MIME earlier. I hope this email is ok now.

-----Original Message-----
From: Rajasekaran, Rajesh
Sent: Monday, March 01, 2004 2:35 PM
To: 'informix-list@iiug.org'
Cc: 'Andrew J. Dibbins'
Subject: RE: ONBAR-VERITAS DB restore performance issue... [293]

Hi,
I would like to post some updates on this performance issue topic to get
some more suggestions. From the bptm logs. it shows that the delay is on the
client side. But, we are yet to isolate the issue btw the DB server, Solaris
9 and the storage array Sun storedge A5200 on this host client. We put all
the latest patches for Solaris 9, Volume manager 3.5 mp2, Veritas Netbackup
4.5 MP5. In fact, we actually applied all the firmware revision for the
storedge array.

We defined our DB volumes layout on RAID-5 with 8 columns and 32 k stripe
size. I see the I/O is utilized to 35 % only at the max. We did perform the
alternate client restore with RAID 5 h/w on the Sun storedge T3 Array. But
this one is with the VERITAS volume manager 3.5 MP2 - RAID5 software level.
Iam not sure whether that is causing the performance issue. From the
Informix side, there is nothing much to tune for ONBAR except the
BAR_NB_XPORT_COUNT since iam doing the serial whole system restore. I even
bumped up that parameter to 10000 from 100. I found that there is a KAIO
bug fixed in Solaris 9 but thats for Oracle. So, we even tried disabling the
KAIO and ran the restore with AIO threads and still didnt improve the
performance at all.
We still didnt get the 10000 KB/SEC and we get only 500 KB/SEC.
Please post your comments/suggestions on this topic. I would like to clarify
one more thing here.
My question:
During the whole system restore, since i had the performance issues, Every
time i troubleshoot and fix something, i got to re-start the restore using
"onbar -RESTART" since i enabled the Restartable restore in the Onconfig
file. Since the troubleshooting has gone for few days and there are times
before i start the restartable restore, i changed the ixbar file to include
the latest incremental backups to restore.
Now today, it just completed the whole system restore and for some reasons,
it didnt restore the incremental backup at all. Hence I'm performing the log
restores for all logs since the last level-0 restore that is just completed.
In the past, i changed the ixbar files during the restartable restore to
include the latest logs from PRD. That used to restore upto the last logs
that i just included in the ixbar file. Iam not sure why it didnt restore
the level-1 the same way in this instance.
1) Can't the ixbar file be changed during the restartable restore to
purposely include the latest level-1's ?
Thanks
RAJ

-----Original Message-----
From: Andrew J. Dibbins [mailto:andy_d@rapier.demon.co.uk]
Sent: Friday, February 27, 2004 4:08 PM
To: Rajasekaran, Rajesh
Subject: RE: ONBAR-VERITAS DB restore performance issue... [293]

Hi Rajesh,

Just to confirm:

SIZE_DATA_BUFFERS = 16
NUMBER_DATA_BUFFERS = 1048576

Surely, you mean the other way round

SIZE_DATA_BUFFERS = 1048576
NUMBER = 16

Anyway, the size figure is way too high, it should be 65536 for dlt 7k, and
should never be bigger.

I don't think its a fix for the problems your experiencing, but it will
surely improve overal backup performance.

Andy
-----Original Message-----
From: Rajasekaran, Rajesh [mailto:RRajasekaran@forestpharm.com]
Sent: Friday, February 27, 2004 1:40 PM
To: 'Andrew J. Dibbins'; Rajasekaran, Rajesh
Subject: RE: ONBAR-VERITAS DB restore performance issue... [293]

Andy,
We have the SUN L40 DLT 7000 tape drives. There are 2 such drives in the
robotic library. Our SIZE_DATA_BUFFERS,NUMBER_DATA_BUFFERS are set to
16,1048576 respectively.
Thanks
RAJ
-----Original Message-----
From: Andrew J. Dibbins [mailto:andy_d@rapier.demon.co.uk]
Sent: Thursday, February 26, 2004 3:12 PM
To: Rajasekaran....
Subject: RE: ONBAR-VERITAS DB restore performance issue... [293]

Hi Rajash,
This probably won't matter, because of your bptm logging, but check
/usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS and NUMBER_DATA_BUFFERS
Your don't mention the tape drives in use, but consider SIZE_DATA_BUFFERS
set to:
64k for dlt
256k for lto's and stk9940B
and our best results for the number of buffers is around 16, but although
I've never used this there is a extra parameter RESTORE_DATA_BUFFERS (check
manual).

Also, make sure shared memory paramters are in /etc/system on the NBU
master, as its this
that gets the used when the above gets allocated.
Clearly the raid 5 isn't clever, you could consider changing the strip size
to 64K, some benefit, perhaps.
Keep us posted.
Andy
Home: andrew.dibbins@rapier.demon.co.uk
Work: andrew.dibbins@sun.com
-----Original Message-----
From: forum.subscriber@iiug.org [mailto:forum.subscriber@iiug.org]On
Behalf Of Rajasekaran....
Sent: Thursday, February 26, 2004 2:00 PM
To: sapmix@iiug.org
Subject: ONBAR-VERITAS DB restore performance issue... [293]

Hi,
We have a performance issue with ONBAR restore for our IDS 7.31UD2XG. We
use the Veritas Netbackup 4.5GA under Solaris 8. The backup (whole system
serial) that we are trying to restore is the whole system restore. The
restore throughput used to be 10 mb/sec and now its around 3 mb/sec. The
restore is going over the network.
The resore client has 100 MBPS network and the master server is 1000
Mbps/sec (gigabit network). I dont see much usage on the CPU,memory on both
the box. Our network ppl say that the load between the two host is normal.
We somehow identified that the slow down could be very specific to the
restore client and it may be due to disk writes. But, we also have the view

that the storage array is only 25% used during the restore. The storage
array is Sun storedge A5200 array on Sun Fire V880 host. The DB volume
layouts are RAID5 with 8 cols and 32k stripe size. I know RAID5 is going to
slow down writes due to parity writing but i dont think it should degrade to

the extent of more than 75%.
Our VERITAS bptm logs from the master server shows that the delay is after
it sends the data to the client. For an object, 6.5 hrs out of 10 hrs was
the delay for one object. This used to complete in 4.5 hrs. But on this
instance, the disk is not heavily loaded and hence more data can be pumped
up.
Considering the storage array is not utilized much, Should the
BAR_NB_XPORT_COUNT and the BAR_NB_XFER_SIZE be increased ? I just left the
values to 100 & 31 respectively. I am assuming that increasing this
parameter will only have effect on parallel backups/restores. Also,
BAR_NB_XFER_SIZE is set to 31 by default on 2k pagesize system. But it says

xbsa has 64kb limit. At the same time, it can have values upto 1
million.Will it help if i increase beyond 32 k ?
Will tuning both these parameters for a serial whole system restore help ?
Please pass on your suggestions/comments.
Regards
Rajesh Rajasekaran
Informix Database Administrator
Forest Pharmaceuticals Inc.
(314) 493-7073

__________________________________________________ __________________
This e-mail and its attachments may contain Forest Laboratories, Inc.
proprietary information that is privileged, confidential or subject to
copyright belonging to Forest Laboratories, Inc. This e-mail is intended
solely for the use of the individual or entity to which it is addressed. If
you are not the intended recipient of this e-mail, or the employee or agent
responsible for delivering this e-mail to the intended recipient, you are
hereby notified that any dissemination, distribution, copying or action
taken in relation to the contents of and attachments to this e-mail is
strictly prohibited and may be unlawful. If you have received this e-mail in

error, please notify the sender immediately and permanently delete the
original and any copy of this e-mail and any printout.

__________________________________________________ __________________
This e-mail and its attachments may contain Forest Laboratories, Inc.
proprietary information that is privileged, confidential or subject to
copyright belonging to Forest Laboratories, Inc. This e-mail is intended
solely for the use of the individual or entity to which it is addressed. If
you are not the intended recipient of this e-mail, or the employee or agent
responsible for delivering this e-mail to the intended recipient, you are
hereby notified that any dissemination, distribution, copying or action
taken in relation to the contents of and attachments to this e-mail is
strictly prohibited and may be unlawful. If you have received this e-mail in
error, please notify the sender immediately and permanently delete the
original and any copy of this e-mail and any printout.

__________________________________________________ __________________
This e-mail and its attachments may contain Forest Laboratories, Inc.
proprietary information that is privileged, confidential or subject to
copyright belonging to Forest Laboratories, Inc. This e-mail is intended
solely for the use of the individual or entity to which it is addressed. If
you are not the intended recipient of this e-mail, or the employee or agent
responsible for delivering this e-mail to the intended recipient, you are
hereby notified that any dissemination, distribution, copying or action
taken in relation to the contents of and attachments to this e-mail is
strictly prohibited and may be unlawful. If you have received this e-mail in
error, please notify the sender immediately and permanently delete the
original and any copy of this e-mail and any printout.

------_=_NextPart_001_01C40068.E7163E90
Content-Type: text/html;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Diso-8859-1">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
5.5.2653.12">
<TITLE>ONBAR-VERITAS DB restore performance issue...</TITLE>
</HEAD>
<BODY>
 

Hi, 


I apologize for the MIME earlier.  I hope this =
email is ok now.


  


-----Original Message-----
 From: Rajasekaran, Rajesh 
 Sent: Monday, March 01, 2004 2:35 PM
 To: 'informix-list@iiug.org'
 Cc: 'Andrew J. Dibbins'
 Subject: RE: ONBAR-VERITAS DB restore performance =
issue... [293] 

 

Hi,
 I would like to post some updates on this =
performance issue topic to get some more suggestions. From the bptm =
logs. it shows that the delay is on the client side. But, we are yet to =
isolate the issue btw the DB server, Solaris 9 and the storage array =
Sun storedge A5200 on this host client. We put all the latest patches =
for Solaris 9, Volume manager 3.5 mp2, Veritas Netbackup 4.5 MP5.  =
In fact, we actually applied all the firmware revision for the storedge =
array. 

   We defined our DB volumes layout on =
RAID-5 with 8 columns and 32 k stripe size. I see the I/O is utilized =
to 35 % only at the max. We did perform the alternate client restore =
with RAID 5 h/w on the Sun storedge T3 Array. But this one is with the =
VERITAS volume manager 3.5 MP2 - RAID5 software level. Iam not sure =
whether that is causing the performance issue. From the Informix side, =
there is nothing much to tune for ONBAR except the BAR_NB_XPORT_COUNT =
since iam doing the serial whole system restore. I even bumped up that =
parameter to 10000 from 100.  I found that there is a KAIO bug =
fixed in Solaris 9 but thats for Oracle. So, we even tried disabling =
the KAIO and ran the restore with AIO threads and still didnt improve =
the performance at all.  

We still didnt get the 10000 KB/SEC and we get only =
500 KB/SEC.
 Please post your comments/suggestions on this topic. =
I would like to clarify one more thing here.
 My question:
 During the whole system restore, since i had the =
performance issues, Every time i troubleshoot and fix something, i got =
to re-start the restore using "onbar -RESTART" since i =
enabled the Restartable restore in the Onconfig file. Since the =
troubleshooting has gone for few days and there are times before i =
start the restartable restore, i changed the ixbar file to include the =
latest incremental backups to restore.

Now today, it just completed the whole system restore =
and for some reasons, it didnt restore the incremental backup at all. =
Hence I'm performing the log restores for all logs since the last =
level-0 restore that is just completed. In the past, i changed the =
ixbar files during the restartable restore to include the latest logs =
from PRD. That used to restore upto the last logs that i just included =
in the ixbar file. Iam not sure why it didnt restore the level-1 the =
same way in this instance.

1) Can't the ixbar file be changed during the =
restartable restore to purposely include the latest level-1's ?
 Thanks
 RAJ


-----Original Message-----
 From: Andrew J. Dibbins [<A =
HREF=3D"mailto:andy_d@rapier.demon.co.uk">mailto:a ndy_d@rapier.demon.co.=
uk</A>]
 Sent: Friday, February 27, 2004 4:08 PM
 To: Rajasekaran, Rajesh
 Subject: RE: ONBAR-VERITAS DB restore performance =
issue... [293] 

 

Hi Rajesh,


Just to confirm:


SIZE_DATA_BUFFERS =3D 16
 NUMBER_DATA_BUFFERS =3D 1048576


Surely, you mean the other way round


SIZE_DATA_BUFFERS =3D 1048576
 NUMBER =3D 16


Anyway, the size figure is way too high, it should be =
65536 for dlt 7k, and should never be bigger.


I don't think its a fix for the problems your =
experiencing, but it will surely improve overal backup =
performance.


Andy
 -----Original Message-----
 From: Rajasekaran, Rajesh [<A =
HREF=3D"mailto:RRajasekaran@forestpharm.com">mailt o:RRajasekaran@forestp=
harm.com</A>]
 Sent: Friday, February 27, 2004 1:40 PM
 To: 'Andrew J. Dibbins'; Rajasekaran, Rajesh
 Subject: RE: ONBAR-VERITAS DB restore performance =
issue... [293] 

 
 
 

Andy, 
    We have the SUN L40 DLT 7000 tape =
drives. There are 2 such drives in the robotic library.  Our =
SIZE_DATA_BUFFERS,NUMBER_DATA_BUFFERS are set to 16,1048576 =
respectively.

Thanks 
 RAJ 
 -----Original Message----- 
 From: Andrew J. Dibbins [<A =
HREF=3D"mailto:andy_d@rapier.demon.co.uk">mailto:a ndy_d@rapier.demon.co.=
uk</A>] 
 Sent: Thursday, February 26, 2004 3:12 PM 
 To: Rajasekaran.... 
 Subject: RE: ONBAR-VERITAS DB restore performance =
issue... [293] 

 

Hi Rajash, 
 This probably won't matter, because of your bptm =
logging, but check 
 /usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS and =
NUMBER_DATA_BUFFERS 
 Your don't mention the tape drives in use, but =
consider SIZE_DATA_BUFFERS 
 set to: 
 64k for dlt 
 256k for lto's and stk9940B 
 and our best results for the number of buffers is =
around 16, but although 
 I've never used this there is a extra parameter =
RESTORE_DATA_BUFFERS (check 
 manual). 

 

Also, make sure shared memory paramters are in =
/etc/system on the NBU 
 master, as its this 
 that gets the used when the above gets allocated. =

 Clearly the raid 5 isn't clever, you could consider =
changing the strip size 
 to 64K, some benefit, perhaps. 
 Keep us posted. 
 Andy 
 Home: andrew.dibbins@rapier.demon.co.uk 
 Work: andrew.dibbins@sun.com 
 -----Original Message----- 
 From: forum.subscriber@iiug.org [<A =
HREF=3D"mailto:forum.subscriber@iiug.org">mailto:f orum.subscriber@iiug.o=
rg</A>]On 
 Behalf Of Rajasekaran.... 
 Sent: Thursday, February 26, 2004 2:00 PM 
 To: sapmix@iiug.org 
 Subject: ONBAR-VERITAS DB restore performance =
issue... [293] 

 

Hi, 
    We have a performance issue with ONBAR =
restore for our IDS 7.31UD2XG. We 
 use the Veritas Netbackup 4.5GA under Solaris 8. The =
backup (whole system 
 serial) that we are trying to restore is the whole =
system restore. The 
 restore throughput used to be 10 mb/sec and now its =
around 3 mb/sec. The 
 restore is going over the network. 
 The resore client has 100 MBPS network and the =
master server is 1000 
 Mbps/sec (gigabit network). I dont see much usage on =
the CPU,memory on both 
 the box. Our network ppl say that the load between =
the two host is normal. 
    We somehow identified that the slow =
down could be very specific to the 
 restore client  and it may be due to disk =
writes. But, we also have the view 
 that the storage array is only 25% used during the =
restore. The storage 
 array is Sun storedge A5200 array on Sun Fire V880 =
host. The DB volume 
 layouts are RAID5 with 8 cols and 32k stripe size. I =
know RAID5 is going to 
 slow down writes due to parity writing but i dont =
think it should degrade to 
 the extent of more than 75%. 
 Our VERITAS bptm logs from the master server shows =
that the delay is after 
 it sends the data to the client. For an object, 6.5 =
hrs out of 10 hrs was 
 the delay for one object. This used to complete in =
4.5 hrs. But on this 
 instance, the disk is not heavily loaded and hence =
more data can be pumped 
 up. 
 Considering the storage array is not utilized much, =
Should the 
 BAR_NB_XPORT_COUNT and the BAR_NB_XFER_SIZE be =
increased ? I just left the 
 values to 100 & 31 respectively. I am assuming =
that increasing this 
 parameter will only have effect on parallel =
backups/restores.  Also, 
 BAR_NB_XFER_SIZE  is set to 31 by default on 2k =
pagesize system. But it says 
 xbsa has 64kb limit. At the same time, it can have =
values upto 1 
 million.Will it help if i increase beyond 32 k ? =

 Will tuning both these parameters for a serial whole =
system restore help ? 
 Please pass on your suggestions/comments. 
 Regards 
 Rajesh Rajasekaran 
 Informix Database Administrator 
 Forest Pharmaceuticals Inc. 
 (314) 493-7073 

 
 
 
 

_________________________________________ ______________________=
_____ 
 This e-mail and its attachments may contain Forest =
Laboratories, Inc. 
 proprietary information that is privileged, =
confidential or subject to 
 copyright belonging to Forest Laboratories, Inc. =
This e-mail is intended 
 solely for the use of the individual or entity to =
which it is addressed. If 
 you are not the intended recipient of this e-mail, =
or the employee or agent 
 responsible for delivering this e-mail to the =
intended recipient, you are 
 hereby notified that any dissemination, =
distribution, copying or action 
 taken in relation to the contents of and attachments =
to this e-mail is 
 strictly prohibited and may be unlawful. If you have =
received this e-mail in 
 error, please notify the sender immediately and =
permanently delete the 
 original and any copy of this e-mail and any =
printout. 

 
 
 

_________________________________________ ______________________=
_____  
 This e-mail and its attachments may contain Forest =
Laboratories, Inc. proprietary information that is privileged, =
confidential or subject to copyright belonging to Forest Laboratories, =
Inc. This e-mail is intended solely for the use of the individual or =
entity to which it is addressed. If you are not the intended recipient =
of this e-mail, or the employee or agent responsible for delivering =
this e-mail to the intended recipient, you are hereby notified that any =
dissemination, distribution, copying or action taken in relation to the =
contents of and attachments to this e-mail is strictly prohibited and =
may be unlawful. If you have received this e-mail in error, please =
notify the sender immediately and permanently delete the original and =
any copy of this e-mail and any printout.
 
 

_______________________________________________ _____________________&=
nbsp;=20
 This e-mail and its attachments may contain Forest Laboratories, =
Inc. proprietary information that is privileged, confidential or =
subject to copyright belonging to Forest Laboratories, Inc. This e-mail =
is intended solely for the use of the individual or entity to which it =
is addressed. If you are not the intended recipient of this e-mail, or =
the employee or agent responsible for delivering this e-mail to the =
intended recipient, you are hereby notified that any dissemination, =
distribution, copying or action taken in relation to the contents of =
and attachments to this e-mail is strictly prohibited and may be =
unlawful. If you have received this e-mail in error, please notify the =
sender immediately and permanently delete the original and any copy of =
this e-mail and any printout.

</BODY>
</HTML>
------_=_NextPart_001_01C40068.E7163E90--
sending to informix-list

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode