RE: ONBAR-VERITAS DB restore performance issue... [293]

#1 (**permalink**) 04-19-2008, 09:36 PM

This message is in MIME format. Since your mail reader does not understand
this format, some or all of this message may not be legible.

------_=_NextPart_001_01C3FFCC.A5B6FCB0
Content-Type: text/plain;
charset="iso-8859-1"

Hi,

I would like to post some updates on this performance issue topic to get
some more suggestions. From the bptm logs. it shows that the delay is on the
client side. But, we are yet to isolate the issue btw the DB server, Solaris
9 and the storage array Sun storedge A5200 on this host client. We put all
the latest patches for Solaris 9, Volume manager 3.5 mp2, Veritas Netbackup
4.5 MP5. In fact, we actually applied all the firmware revision for the
storedge array.

We defined our DB volumes layout on RAID-5 with 8 columns and 32 k stripe
size. I see the I/O is utilized to 35 % only at the max. We did perform the
alternate client restore with RAID 5 h/w on the Sun storedge T3 Array. But
this one is with the VERITAS volume manager 3.5 MP2 - RAID5 software level.
Iam not sure whether that is causing the performance issue. From the
Informix side, there is nothing much to tune for ONBAR except the
BAR_NB_XPORT_COUNT since iam doing the serial whole system restore. I even
bumped up that parameter to 10000 from 100. I found that there is a KAIO
bug fixed in Solaris 9 but thats for Oracle. So, we even tried disabling the
KAIO and ran the restore with AIO threads and still didnt improve the
performance at all.

We still didnt get the 10000 KB/SEC and we get only 500 KB/SEC.

Please post your comments/suggestions on this topic. I would like to clarify
one more thing here.

My question:

During the whole system restore, since i had the performance issues, Every
time i troubleshoot and fix something, i got to re-start the restore using
"onbar -RESTART" since i enabled the Restartable restore in the Onconfig
file. Since the troubleshooting has gone for few days and there are times
before i start the restartable restore, i changed the ixbar file to include
the latest incremental backups to restore.

Now today, it just completed the whole system restore and for some reasons,
it didnt restore the incremental backup at all. Hence I'm performing the log
restores for all logs since the last level-0 restore that is just completed.
In the past, i changed the ixbar files during the restartable restore to
include the latest logs from PRD. That used to restore upto the last logs
that i just included in the ixbar file. Iam not sure why it didnt restore
the level-1 the same way in this instance.

1) Can't the ixbar file be changed during the restartable restore to
purposely include the latest level-1's ?

Thanks

RAJ

-----Original Message-----
From: Andrew J. Dibbins [mailto:andy_d@rapier.demon.co.uk]
Sent: Friday, February 27, 2004 4:08 PM
To: Rajasekaran, Rajesh
Subject: RE: ONBAR-VERITAS DB restore performance issue... [293]

Hi Rajesh,

Just to confirm:

SIZE_DATA_BUFFERS = 16
NUMBER_DATA_BUFFERS = 1048576

Surely, you mean the other way round

SIZE_DATA_BUFFERS = 1048576
NUMBER = 16

Anyway, the size figure is way too high, it should be 65536 for dlt 7k, and
should never be bigger.

I don't think its a fix for the problems your experiencing, but it will
surely improve overal backup performance.

Andy

-----Original Message-----
From: Rajasekaran, Rajesh [mailto:RRajasekaran@forestpharm.com]
Sent: Friday, February 27, 2004 1:40 PM
To: 'Andrew J. Dibbins'; Rajasekaran, Rajesh
Subject: RE: ONBAR-VERITAS DB restore performance issue... [293]

Andy,

We have the SUN L40 DLT 7000 tape drives. There are 2 such drives in the
robotic library. Our SIZE_DATA_BUFFERS,NUMBER_DATA_BUFFERS are set to
16,1048576 respectively.

Thanks
RAJ
-----Original Message-----
From: Andrew J. Dibbins [ mailto:andy_d@rapier.demon.co.uk
<mailto:andy_d@rapier.demon.co.uk> ]
Sent: Thursday, February 26, 2004 3:12 PM
To: Rajasekaran....
Subject: RE: ONBAR-VERITAS DB restore performance issue... [293]

Hi Rajash,

This probably won't matter, because of your bptm logging, but check
/usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS and NUMBER_DATA_BUFFERS

Your don't mention the tape drives in use, but consider SIZE_DATA_BUFFERS
set to:

64k for dlt
256k for lto's and stk9940B

and our best results for the number of buffers is around 16, but although
I've never used this there is a extra parameter RESTORE_DATA_BUFFERS (check
manual).

Also, make sure shared memory paramters are in /etc/system on the NBU
master, as its this
that gets the used when the above gets allocated.

Clearly the raid 5 isn't clever, you could consider changing the strip size
to 64K, some benefit, perhaps.

Keep us posted.

Andy

Home: andrew.dibbins@rapier.demon.co.uk
Work: andrew.dibbins@sun.com

-----Original Message-----
From: forum.subscriber@iiug.org [ mailto:forum.subscriber@iiug.org
<mailto:forum.subscriber@iiug.org> ]On
Behalf Of Rajasekaran....
Sent: Thursday, February 26, 2004 2:00 PM
To: sapmix@iiug.org
Subject: ONBAR-VERITAS DB restore performance issue... [293]

Hi,

We have a performance issue with ONBAR restore for our IDS 7.31UD2XG. We
use the Veritas Netbackup 4.5GA under Solaris 8. The backup (whole system
serial) that we are trying to restore is the whole system restore. The
restore throughput used to be 10 mb/sec and now its around 3 mb/sec. The
restore is going over the network.

The resore client has 100 MBPS network and the master server is 1000
Mbps/sec (gigabit network). I dont see much usage on the CPU,memory on both
the box. Our network ppl say that the load between the two host is normal.

We somehow identified that the slow down could be very specific to the
restore client and it may be due to disk writes. But, we also have the view

that the storage array is only 25% used during the restore. The storage
array is Sun storedge A5200 array on Sun Fire V880 host. The DB volume
layouts are RAID5 with 8 cols and 32k stripe size. I know RAID5 is going to
slow down writes due to parity writing but i dont think it should degrade to

the extent of more than 75%.

Our VERITAS bptm logs from the master server shows that the delay is after
it sends the data to the client. For an object, 6.5 hrs out of 10 hrs was
the delay for one object. This used to complete in 4.5 hrs. But on this
instance, the disk is not heavily loaded and hence more data can be pumped
up.

Considering the storage array is not utilized much, Should the
BAR_NB_XPORT_COUNT and the BAR_NB_XFER_SIZE be increased ? I just left the
values to 100 & 31 respectively. I am assuming that increasing this
parameter will only have effect on parallel backups/restores. Also,
BAR_NB_XFER_SIZE is set to 31 by default on 2k pagesize system. But it says

xbsa has 64kb limit. At the same time, it can have values upto 1
million.Will it help if i increase beyond 32 k ?

Will tuning both these parameters for a serial whole system restore help ?

Please pass on your suggestions/comments.

Regards
Rajesh Rajasekaran
Informix Database Administrator
Forest Pharmaceuticals Inc.
(314) 493-7073

__________________________________________________ __________________
This e-mail and its attachments may contain Forest Laboratories, Inc.
proprietary information that is privileged, confidential or subject to
copyright belonging to Forest Laboratories, Inc. This e-mail is intended
solely for the use of the individual or entity to which it is addressed. If
you are not the intended recipient of this e-mail, or the employee or agent
responsible for delivering this e-mail to the intended recipient, you are
hereby notified that any dissemination, distribution, copying or action
taken in relation to the contents of and attachments to this e-mail is
strictly prohibited and may be unlawful. If you have received this e-mail in

error, please notify the sender immediately and permanently delete the
original and any copy of this e-mail and any printout.

__________________________________________________ __________________
This e-mail and its attachments may contain Forest Laboratories, Inc.
proprietary information that is privileged, confidential or subject to
copyright belonging to Forest Laboratories, Inc. This e-mail is intended
solely for the use of the individual or entity to which it is addressed. If
you are not the intended recipient of this e-mail, or the employee or agent
responsible for delivering this e-mail to the intended recipient, you are
hereby notified that any dissemination, distribution, copying or action
taken in relation to the contents of and attachments to this e-mail is
strictly prohibited and may be unlawful. If you have received this e-mail in
error, please notify the sender immediately and permanently delete the
original and any copy of this e-mail and any printout.

__________________________________________________ __________________
This e-mail and its attachments may contain Forest Laboratories, Inc.
proprietary information that is privileged, confidential or subject to
copyright belonging to Forest Laboratories, Inc. This e-mail is intended
solely for the use of the individual or entity to which it is addressed. If
you are not the intended recipient of this e-mail, or the employee or agent
responsible for delivering this e-mail to the intended recipient, you are
hereby notified that any dissemination, distribution, copying or action
taken in relation to the contents of and attachments to this e-mail is
strictly prohibited and may be unlawful. If you have received this e-mail in
error, please notify the sender immediately and permanently delete the
original and any copy of this e-mail and any printout.

------_=_NextPart_001_01C3FFCC.A5B6FCB0
Content-Type: text/html;
charset="iso-8859-1"

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>RE: ONBAR-VERITAS DB restore performance issue... [293]</TITLE>

<META content="MSHTML 6.00.2800.1400" name=GENERATOR></HEAD>
<BODY>
<DIV>
Hi,

I would
like to post some updates on this performance issue topic to get some more
suggestions. From the bptm logs. it shows that the delay is on the client
side. But, we are yet to isolate the issue btw the DB server, Solaris 9 and the
storage array Sun storedge A5200 on this host client. We put all the latest
patches for Solaris 9, Volume manager 3.5 mp2, Veritas Netbackup 4.5 MP5. 
In fact, we actually applied all the firmware revision for the storedge array.

   We defined our DB volumes layout on RAID-5 with 8 columns
and 32 k stripe size. I see the I/O is utilized to 35 % only at the max. We did
perform the alternate client restore with RAID 5 h/w on the Sun storedge T3
Array. But this one is with the VERITAS volume manager 3.5 MP2 - RAID5 software
level. Iam not sure whether that is causing the performance issue.
From the Informix side, there is nothing much to tune for ONBAR except
the BAR_NB_XPORT_COUNT since iam doing the serial whole system restore. I even
bumped up that parameter to 10000 from 100.  I found that
there is a KAIO bug fixed in Solaris 9 but thats for Oracle. So, we even
tried disabling the KAIO and ran the restore with AIO threads and still didnt
improve the performance at all.  
We still
didnt get the 10000 KB/SEC and we get only 500 KB/SEC.
Please
post your comments/suggestions on this topic. I would like to clarify one more
thing here.
My question:
During the whole system restore, since i had the performance issues, Every time i troubleshoot and fix something, i
got to re-start the restore using "onbar -RESTART" since i enabled the
Restartable restore in the Onconfig file. Since the troubleshooting has gone for
few days and there are times before i start the restartable restore, i changed
the ixbar file to include the latest incremental backups to restore.
Now today, it just completed the whole system restore and for
some reasons, it didnt restore the incremental backup at all. Hence I'm
performing the log restores for all logs since the last level-0 restore that is
just completed. In the past, i changed the ixbar files during the restartable
restore to include the latest logs from PRD. That used to restore upto the last
logs that i just included in the ixbar file. Iam not sure why it didnt restore
the level-1 the same way in this instance.
1) Can't the ixbar file be changed during the restartable
restore to purposely include the latest level-1's ?
Thanks
RAJ</DIV>
<BLOCKQUOTE dir=ltr style="MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader dir=ltr align=left>-----Original Message----- From: Andrew J. Dibbins
[mailto:andy_d@rapier.demon.co.uk] Sent: Friday, February 27, 2004
4:08 PM To: Rajasekaran, Rajesh Subject: RE: ONBAR-VERITAS
DB restore performance issue... [293] </DIV>
<DIV>Hi
Rajesh,</DIV>
<DIV> </DIV>
<DIV>Just
to confirm:</DIV>
<DIV> </DIV>
<DIV>SIZE_DATA_BUFFERS = 16</DIV>
<DIV>NUMBER_DATA_BUFFERS = 1048576</DIV>
<DIV> </DIV>
<DIV>Surely, you mean the other way round</DIV>
<DIV> </DIV>
<DIV>SIZE_DATA_BUFFERS = 1048576</DIV>
<DIV>NUMBER = 16</DIV>
<DIV> </DIV>
<DIV>Anyway, the size figure is way too high, it should be 65536 for
dlt 7k, and should never be bigger.</DIV>
<DIV> </DIV>
<DIV>I
don't think its a fix for the problems your experiencing, but it will surely
improve overal backup performance.</DIV>
<DIV> </DIV>
<DIV>Andy</DIV>
<BLOCKQUOTE dir=ltr style="MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader dir=ltr align=left>-----Original Message----- From: Rajasekaran, Rajesh
[mailto:RRajasekaran@forestpharm.com] Sent: Friday, February 27,
2004 1:40 PM To: 'Andrew J. Dibbins'; Rajasekaran,
Rajesh Subject: RE: ONBAR-VERITAS DB restore performance issue...
[293] </DIV> 
Andy, 
   We have the SUN L40 DLT 7000 tape drives. There
are 2 such drives in the robotic library.  Our
SIZE_DATA_BUFFERS,NUMBER_DATA_BUFFERS are set to 16,1048576
respectively.
Thanks RAJ -----Original Message----- From: Andrew J.
Dibbins [<A
href="mailto:andy_d@rapier.demon.co.uk">mailto:and y_d@rapier.demon.co.uk</A>]
 Sent: Thursday, February 26, 2004 3:12 PM To: Rajasekaran.... Subject: RE:
ONBAR-VERITAS DB restore performance issue... [293] 
Hi Rajash, 
This probably won't matter, because of your bptm logging,
but check /usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS and
NUMBER_DATA_BUFFERS 
Your don't mention the tape drives in use, but consider
SIZE_DATA_BUFFERS set to: 
64k for dlt 256k for lto's and
stk9940B 
and our best results for the number of buffers is around 16,
but although I've never used this there is a extra
parameter RESTORE_DATA_BUFFERS (check manual). 
Also, make sure shared memory paramters are in /etc/system
on the NBU master, as its this that gets the used when the above gets allocated. 
Clearly the raid 5 isn't clever, you could consider changing
the strip size to 64K, some benefit, perhaps.

Keep us posted. 
Andy 
Home: andrew.dibbins@rapier.demon.co.uk Work: andrew.dibbins@sun.com 
-----Original Message----- From:
forum.subscriber@iiug.org [<A
href="mailto:forum.subscriber@iiug.org">mailto:for um.subscriber@iiug.org</A>]On
 Behalf Of Rajasekaran.... Sent:
Thursday, February 26, 2004 2:00 PM To:
sapmix@iiug.org Subject: ONBAR-VERITAS DB restore
performance issue... [293] 
Hi, 
   We have a performance issue with ONBAR restore
for our IDS 7.31UD2XG. We use the Veritas Netbackup
4.5GA under Solaris 8. The backup (whole system serial) that we are trying to restore is the whole system restore.
The restore throughput used to be 10 mb/sec and now
its around 3 mb/sec. The restore is going over the
network. 
The resore client has 100 MBPS network and the master server
is 1000 Mbps/sec (gigabit network). I dont see much
usage on the CPU,memory on both the box. Our network
ppl say that the load between the two host is normal. 
   We somehow identified that the slow down could
be very specific to the restore client  and it
may be due to disk writes. But, we also have the view that the storage array is only 25% used during the restore. The
storage array is Sun storedge A5200 array on Sun
Fire V880 host. The DB volume layouts are RAID5 with
8 cols and 32k stripe size. I know RAID5 is going to slow down writes due to parity writing but i dont think it should
degrade to the extent of more than 75%. 
Our VERITAS bptm logs from the master server shows that the
delay is after it sends the data to the client. For
an object, 6.5 hrs out of 10 hrs was the delay for
one object. This used to complete in 4.5 hrs. But on this instance, the disk is not heavily loaded and hence more data can be
pumped up. 
Considering the storage array is not utilized much, Should
the BAR_NB_XPORT_COUNT and the BAR_NB_XFER_SIZE be
increased ? I just left the values to 100 & 31
respectively. I am assuming that increasing this parameter will only have effect on parallel backups/restores. 
Also, BAR_NB_XFER_SIZE  is set to 31 by default
on 2k pagesize system. But it says xbsa has 64kb
limit. At the same time, it can have values upto 1 million.Will it help if i increase beyond 32 k ? 
Will tuning both these parameters for a serial whole system
restore help ? 
Please pass on your suggestions/comments. 
Regards Rajesh Rajasekaran
 Informix Database Administrator Forest Pharmaceuticals Inc. (314)
493-7073 
___________________________________________ _________________________
 This e-mail and its attachments may contain Forest
Laboratories, Inc. proprietary information that is
privileged, confidential or subject to copyright
belonging to Forest Laboratories, Inc. This e-mail is intended
 solely for the use of the individual or entity to which it
is addressed. If you are not the intended recipient
of this e-mail, or the employee or agent responsible
for delivering this e-mail to the intended recipient, you are
 hereby notified that any dissemination, distribution,
copying or action taken in relation to the contents
of and attachments to this e-mail is strictly
prohibited and may be unlawful. If you have received this e-mail in
 error, please notify the sender immediately and permanently
delete the original and any copy of this e-mail and
any printout. 
_______________________________________________ _____________________ 
 This e-mail and its attachments may contain Forest Laboratories, Inc.
proprietary information that is privileged, confidential or subject to
copyright belonging to Forest Laboratories, Inc. This e-mail is intended
solely for the use of the individual or entity to which it is addressed. If
you are not the intended recipient of this e-mail, or the employee or agent
responsible for delivering this e-mail to the intended recipient, you are
hereby notified that any dissemination, distribution, copying or action
taken in relation to the contents of and attachments to this e-mail is
strictly prohibited and may be unlawful. If you have received this e-mail in
error, please notify the sender immediately and permanently delete the
original and any copy of this e-mail and any
printout.</BLOCKQUOTE></BLOCKQUOTE></BODY></HTML>
 
 
 

_______________________________________________ _____________________ 

This e-mail and its attachments may contain Forest Laboratories, Inc. proprietary information that is privileged, confidential or subject to copyright belonging to Forest Laboratories, Inc. This e-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this e-mail, or the employee or agent responsible for delivering this e-mail to the intended recipient, you are hereby notified that any dissemination, distribution, copying or action taken in relation to the contents of and attachments to this e-mail is strictly prohibited and may be unlawful. If you have received this e-mail in error, please notify the sender immediately and permanently delete the original and any copy of this e-mail and any printout.

------_=_NextPart_001_01C3FFCC.A5B6FCB0--
sending to informix-list

#2 (**permalink**) 04-19-2008, 09:37 PM

Rajasekaran, Rajesh wrote:

> This message is in MIME format.

Don't post MIME. Or HTML - it's incredibly horrible.

> We defined our DB volumes layout on RAID-5

Careful - you'll set Art off.

Try searching for 'NO RAID 5' written by Art Kagel in
groups.google.com - read and learn.

--
Jonathan Leffler #include <disclaimer.h>
Email: jleffler@earthlink.net, jleffler@us.ibm.com
Guardian of DBD::Informix v2003.04 -- http://dbi.perl.org/

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode