vBulletin Search Engine Optimization
| |||||||
| Register | FAQ | Members List | Calendar | Search | Today's Posts | Mark Forums Read |
| ||||
| This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C41E34.A549DFD2 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Alexey, Our level-0 bkup on QAS is currently taking 10-11 hrs for the 400 = GB database with the serial backup and single stream. The level-1 bkup = used to take not more than 2 hrs to backup. Our PRD level-1 is still taking = 1-1.5 hrs and its storedge array is SUN 3510 FC array. Moreover on the QAS, = I dont think its the T3 array throughput bcos if thats the case, it would = have degraded the level-0 as well but its not. We dont take level-2 here. = Iam not suspecting the DLT drive too. Its somewhere btw the ONBAR & the IDS. Let me check what else is changed on the box such as O/S patches, T3 = disk array related patches in the recent past.=20 More suggestions are greatly appreciated. Thanks RAJ -----Original Message----- From: Alexey Sonkin [mailto:alexeis@grandvirtual.com] Sent: Thursday, April 08, 2004 5:19 PM To: 'Rajasekaran, Rajesh'; 'sapmix@iiug.org' Cc: 'informix-list@iiug.org' Subject: RE: Incremental Archive performance issue...... Rajesh, It's not clear to me from Your email how long it takes to make the level-0 backup of Your QAS. I suppose it is about the same as Your Level-1 backup. Is Your question why You see such a significant difference between=20 level-1 and level-0 on PRD and no difference on QA? During Level-1 backup, Informix is reading from the database EXACTLY THE SAME amount of data as during level-0 backup. For each page IDS makes timestamp comparison to decide, whether to put that page into L1 backup or not. The only difference between L1 and L2 is how much data the engine is sending to tape. It means, that in Your case, on QAS backup performance is limited by the T3 array throughput rather then by DLT limitations. More powerful (I suppose) production disk array can ship data much faster to IDS for analysis, and backup performance is constrained by the DLT limitations... ------------------------------------------ Alexey Sonkin =A0 -----Original Message----- From: Rajasekaran, Rajesh [mailto:RRajasekaran@forestpharm.com]=20 Sent: Thursday, April 08, 2004 4:56 PM To: 'sapmix@iiug.org' Cc: 'informix-list@iiug.org' Subject: Incremental Archive performance issue...... Hi,=20 =A0=A0=A0 Our QAS system is a Sun E450 box with a sun storedge T/3 = array attached. Its a solaris o/s with IDS 7.31 database. The DB size is around 400 GB. = Its backup is directed over the network to a tape library attached to a = master server which is another host. Our Master server is also the=A0 PRD box. = Its a Sun Fire V880 with Sun L40 tape libary attached. We have the Veritas Netbackup Data center 4.5 for our backups & restores. The QAS DB backup is going over the network to the master server's DLT = tape library that gives 35 GB/HR performance. We got that throughput for our = QAS level-0 bkup even though its over the T3 network. But the Level-1 = backup has severe performance degradation and it takes almost 5-7 hrs to complete. = Our PRD level-1's are taking anywhere btw 1.5 - 2 hrs. We are only running the onbar whole system serial bkup. It should not = put much I/O bottlenecks since only one dbspace bkup is running at one time = and not parallel.=20 =A0I can see the iostat shows 10000-25000 Kb/sec for the T3 storedge = array continously during the level-1 bkups. The informix "onstat -g stq" = shows Full queue 0, Empty queue with Cnt 100 all the time. I dont see the I/O bottleneck here and i dont see the problem with the Veritas Netbackup = side since our level-0 works as expected. I dont see any = "arc_very_old_pages()" running during the level-1 bkups. Questions=20 1) What could be the reason for level-1 backup degradation in = performance when level-0 is performing well ?=20 2) What better way to monitor btw the onbar and the engine ? Will the bar_dbug =3D 9 lvl trace shows=A0 more detail towards my problem ? Any suggestions/comments is greatly appreciated.=20 =A0=20 Regards=20 Rajesh Rajasekaran=20 Informix Database Administrator=20 Forest Pharmaceuticals Inc.=20 (314) 493-7073=20 __________________________________________________ __________________ =20 This e-mail and its attachments may contain Forest Laboratories, Inc. proprietary information that is privileged, confidential or subject to copyright belonging to Forest Laboratories, Inc. This e-mail is = intended solely for the use of the individual or entity to which it is = addressed. If you are not the intended recipient of this e-mail, or the employee or = agent responsible for delivering this e-mail to the intended recipient, you = are hereby notified that any dissemination, distribution, copying or action taken in relation to the contents of and attachments to this e-mail is strictly prohibited and may be unlawful. If you have received this = e-mail in error, please notify the sender immediately and permanently delete the original and any copy of this e-mail and any printout. ------_=_NextPart_001_01C41E34.A549DFD2 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN"> <HTML> <HEAD> <META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; = charset=3Diso-8859-1"> <META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version = 5.5.2657.73"> <TITLE>RE: Incremental Archive performance issue......</TITLE> </HEAD> <BODY> <P><FONT SIZE=3D2>Alexey,</FONT> </P> <P><FONT SIZE=3D2> Our level-0 bkup on = QAS is currently taking 10-11 hrs for the 400 GB database with the = serial backup and single stream. The level-1 bkup used to take not more = than 2 hrs to backup. Our PRD level-1 is still taking 1-1.5 hrs and its = storedge array is SUN 3510 FC array. Moreover on the QAS, I dont = think its the T3 array throughput bcos if thats the case, it would have = degraded the level-0 as well but its not. We dont take level-2 here. = Iam not suspecting the DLT drive too. Its somewhere btw the ONBAR & = the IDS.</FONT></P> <P><FONT SIZE=3D2> Let me check what else is changed on the = box such as O/S patches, T3 disk array related patches in the = recent past. </FONT> </P> <P><FONT SIZE=3D2>More suggestions are greatly appreciated.</FONT> </P> <P><FONT SIZE=3D2>Thanks</FONT> <BR><FONT SIZE=3D2>RAJ</FONT> </P> <P><FONT SIZE=3D2>-----Original Message-----</FONT> <BR><FONT SIZE=3D2>From: Alexey Sonkin [<A = HREF=3D"mailto:alexeis@grandvirtual.com">mailto:al exeis@grandvirtual.com= </A>]</FONT> <BR><FONT SIZE=3D2>Sent: Thursday, April 08, 2004 5:19 PM</FONT> <BR><FONT SIZE=3D2>To: 'Rajasekaran, Rajesh'; 'sapmix@iiug.org'</FONT> <BR><FONT SIZE=3D2>Cc: 'informix-list@iiug.org'</FONT> <BR><FONT SIZE=3D2>Subject: RE: Incremental Archive performance = issue......</FONT> </P> <BR> <BR> <P><FONT SIZE=3D2>Rajesh,</FONT> </P> <P><FONT SIZE=3D2>It's not clear to me from Your email how long it = takes</FONT> <BR><FONT SIZE=3D2>to make the level-0 backup of Your QAS.</FONT> <BR><FONT SIZE=3D2>I suppose it is about the same as Your Level-1 = backup.</FONT> </P> <P><FONT SIZE=3D2>Is Your question why You see such a significant = difference between </FONT> <BR><FONT SIZE=3D2>level-1 and level-0 on PRD and no difference on = QA?</FONT> </P> <P><FONT SIZE=3D2>During Level-1 backup, Informix is reading from the = database</FONT> <BR><FONT SIZE=3D2>EXACTLY THE SAME amount of data as during level-0 = backup.</FONT> <BR><FONT SIZE=3D2>For each page IDS makes timestamp comparison to = decide, whether</FONT> <BR><FONT SIZE=3D2>to put that page into L1 backup or not. The only = difference</FONT> <BR><FONT SIZE=3D2>between L1 and L2 is how much data the engine is = sending to tape.</FONT> </P> <P><FONT SIZE=3D2>It means, that in Your case, on QAS backup = performance</FONT> <BR><FONT SIZE=3D2>is limited by the T3 array throughput rather then by = DLT limitations.</FONT> <BR><FONT SIZE=3D2>More powerful (I suppose) production disk array can = ship data</FONT> <BR><FONT SIZE=3D2>much faster to IDS for analysis, and backup = performance is constrained</FONT> <BR><FONT SIZE=3D2>by the DLT limitations...</FONT> </P> <BR> <P><FONT SIZE=3D2>------------------------------------------</FONT> <BR><FONT SIZE=3D2>Alexey Sonkin</FONT> </P> <P><FONT SIZE=3D2>=A0</FONT> <BR><FONT SIZE=3D2>-----Original Message-----</FONT> <BR><FONT SIZE=3D2>From: Rajasekaran, Rajesh [<A = HREF=3D"mailto:RRajasekaran@forestpharm.com">mailt o:RRajasekaran@forestp= harm.com</A>] </FONT> <BR><FONT SIZE=3D2>Sent: Thursday, April 08, 2004 4:56 PM</FONT> <BR><FONT SIZE=3D2>To: 'sapmix@iiug.org'</FONT> <BR><FONT SIZE=3D2>Cc: 'informix-list@iiug.org'</FONT> <BR><FONT SIZE=3D2>Subject: Incremental Archive performance = issue......</FONT> </P> <P><FONT SIZE=3D2>Hi, </FONT> <BR><FONT SIZE=3D2>=A0=A0=A0 Our QAS system is a Sun E450 box with a = sun storedge T/3 array attached.</FONT> <BR><FONT SIZE=3D2>Its a solaris o/s with IDS 7.31 database. The DB = size is around 400 GB. Its</FONT> <BR><FONT SIZE=3D2>backup is directed over the network to a tape = library attached to a master</FONT> <BR><FONT SIZE=3D2>server which is another host. Our Master server is = also the=A0 PRD box. Its a</FONT> <BR><FONT SIZE=3D2>Sun Fire V880 with Sun L40 tape libary attached. We = have the Veritas</FONT> <BR><FONT SIZE=3D2>Netbackup Data center 4.5 for our backups & = restores.</FONT> <BR><FONT SIZE=3D2>The QAS DB backup is going over the network to the = master server's DLT tape</FONT> <BR><FONT SIZE=3D2>library that gives 35 GB/HR performance. We got that = throughput for our QAS</FONT> <BR><FONT SIZE=3D2>level-0 bkup even though its over the T3 network. = But the Level-1 backup has</FONT> <BR><FONT SIZE=3D2>severe performance degradation and it takes almost = 5-7 hrs to complete. Our</FONT> <BR><FONT SIZE=3D2>PRD level-1's are taking anywhere btw 1.5 - 2 = hrs.</FONT> <BR><FONT SIZE=3D2>We are only running the onbar whole system serial = bkup. It should not put</FONT> <BR><FONT SIZE=3D2>much I/O bottlenecks since only one dbspace bkup is = running at one time and</FONT> <BR><FONT SIZE=3D2>not parallel. </FONT> <BR><FONT SIZE=3D2>=A0I can see the iostat shows 10000-25000 Kb/sec for = the T3 storedge array</FONT> <BR><FONT SIZE=3D2>continously during the level-1 bkups. The informix = "onstat -g stq" shows</FONT> <BR><FONT SIZE=3D2>Full queue 0, Empty queue with Cnt 100 all the time. = I dont see the I/O</FONT> <BR><FONT SIZE=3D2>bottleneck here and i dont see the problem with the = Veritas Netbackup side</FONT> <BR><FONT SIZE=3D2>since our level-0 works as expected. I dont see any = "arc_very_old_pages()"</FONT> <BR><FONT SIZE=3D2>running during the level-1 bkups.</FONT> <BR><FONT SIZE=3D2>Questions </FONT> <BR><FONT SIZE=3D2>1) What could be the reason for level-1 backup = degradation in performance</FONT> <BR><FONT SIZE=3D2>when level-0 is performing well ? </FONT> <BR><FONT SIZE=3D2>2) What better way to monitor btw the onbar and the = engine ? Will the</FONT> <BR><FONT SIZE=3D2>bar_dbug =3D 9 lvl trace shows=A0 more detail = towards my problem ?</FONT> <BR><FONT SIZE=3D2>Any suggestions/comments is greatly appreciated. = </FONT> <BR><FONT SIZE=3D2>=A0 </FONT> <BR><FONT SIZE=3D2>Regards </FONT> <BR><FONT SIZE=3D2>Rajesh Rajasekaran </FONT> <BR><FONT SIZE=3D2>Informix Database Administrator </FONT> <BR><FONT SIZE=3D2>Forest Pharmaceuticals Inc. </FONT> <BR><FONT SIZE=3D2>(314) 493-7073 </FONT> </P> <BR> <BR> <BR> <P>_______________________________________________ _____________________&= nbsp;=20 <BR>This e-mail and its attachments may contain Forest Laboratories, = Inc. proprietary information that is privileged, confidential or = subject to copyright belonging to Forest Laboratories, Inc. This e-mail = is intended solely for the use of the individual or entity to which it = is addressed. If you are not the intended recipient of this e-mail, or = the employee or agent responsible for delivering this e-mail to the = intended recipient, you are hereby notified that any dissemination, = distribution, copying or action taken in relation to the contents of = and attachments to this e-mail is strictly prohibited and may be = unlawful. If you have received this e-mail in error, please notify the = sender immediately and permanently delete the original and any copy of = this e-mail and any printout.</P> </BODY> </HTML> ------_=_NextPart_001_01C41E34.A549DFD2-- sending to informix-list |