Rajesh,

As You mentioned, Your DLT speed is 35GB/hour.
This is why it takes 10-11 hours to make the L0 backup
of Your 400GB database. Your L0 backup speed is limited
by the DLT device.

For L1 archive, the backup speed is mostly limited by the
disk array throughput: as I've already mentioned, Informix
must read the entire database (like in L0 archive)

You can also check whether any significant database writes
occur during the backup. If this is the case, You might experience
'very old page' problem


------------------------------------------
Alexey Sonkin
*
-----Original Message-----
From: Rajasekaran, Rajesh [mailto:RRajasekaran@forestpharm.com]
Sent: Friday, April 09, 2004 9:15 AM
To: 'Alexey Sonkin'
Cc: 'sapmix@iiug.org'; 'informix-list@iiug.org'
Subject: RE: Incremental Archive performance issue......

Alexey,
***** Our level-0 bkup on QAS is currently taking 10-11 hrs for the 400 GB
database with the serial backup and single stream. The level-1 bkup used
to take not more than 2 hrs to backup. Our PRD level-1 is still taking
1-1.5 hrs and its storedge array is SUN 3510 FC array.* Moreover on the QAS,

I dont think its the T3 array throughput bcos if thats the case,
it would have degraded the level-0 as well but its not. We dont take
level-2 here. Iam not suspecting the DLT drive too. Its somewhere btw the
ONBAR & the IDS.
*Let me check what else is changed on the box* such as O/S patches,
T3 disk array related patches in the recent past.
More suggestions are greatly appreciated.
Thanks
RAJ

-----Original Message-----
From: Alexey Sonkin [mailto:alexeis@grandvirtual.com]
Sent: Thursday, April 08, 2004 5:19 PM
To: 'Rajasekaran, Rajesh'; 'sapmix@iiug.org'
Cc: 'informix-list@iiug.org'
Subject: RE: Incremental Archive performance issue......

Rajesh,
It's not clear to me from Your email how long it takes
to make the level-0 backup of Your QAS.
I suppose it is about the same as Your Level-1 backup.
Is Your question why You see such a significant difference between
level-1 and level-0 on PRD and no difference on QA?
During Level-1 backup, Informix is reading from the database
EXACTLY THE SAME amount of data as during level-0 backup.
For each page IDS makes timestamp comparison to decide, whether
to put that page into L1 backup or not. The only difference
between L1 and L2 is how much data the engine is sending to tape.
It means, that in Your case, on QAS backup performance
is limited by the T3 array throughput rather then by DLT limitations.
More powerful (I suppose) production disk array can ship data
much faster to IDS for analysis, and backup performance is constrained
by the DLT limitations...

------------------------------------------
Alexey Sonkin
*
-----Original Message-----
From: Rajasekaran, Rajesh [mailto:RRajasekaran@forestpharm.com]
Sent: Thursday, April 08, 2004 4:56 PM
To: 'sapmix@iiug.org'
Cc: 'informix-list@iiug.org'
Subject: Incremental Archive performance issue......
Hi,
*** Our QAS system is a Sun E450 box with a sun storedge T/3 array attached.

Its a solaris o/s with IDS 7.31 database. The DB size is around 400 GB. Its
backup is directed over the network to a tape library attached to a master
server which is another host. Our Master server is also the* PRD box. Its a
Sun Fire V880 with Sun L40 tape libary attached. We have the Veritas
Netbackup Data center 4.5 for our backups & restores.
The QAS DB backup is going over the network to the master server's DLT tape
library that gives 35 GB/HR performance. We got that throughput for our QAS
level-0 bkup even though its over the T3 network. But the Level-1 backup has

severe performance degradation and it takes almost 5-7 hrs to complete. Our
PRD level-1's are taking anywhere btw 1.5 - 2 hrs.
We are only running the onbar whole system serial bkup. It should not put
much I/O bottlenecks since only one dbspace bkup is running at one time and
not parallel.
*I can see the iostat shows 10000-25000 Kb/sec for the T3 storedge array
continously during the level-1 bkups. The informix "onstat -g stq" shows
Full queue 0, Empty queue with Cnt 100 all the time. I dont see the I/O
bottleneck here and i dont see the problem with the Veritas Netbackup side
since our level-0 works as expected. I dont see any "arc_very_old_pages()"
running during the level-1 bkups.
Questions
1) What could be the reason for level-1 backup degradation in performance
when level-0 is performing well ?
2) What better way to monitor btw the onbar and the engine ? Will the
bar_dbug = 9 lvl trace shows* more detail towards my problem ?
Any suggestions/comments is greatly appreciated.
*
Regards
Rajesh Rajasekaran
Informix Database Administrator
Forest Pharmaceuticals Inc.
(314) 493-7073


__________________________________________________ __________________*
This e-mail and its attachments may contain Forest Laboratories, Inc.
proprietary information that is privileged, confidential or subject to
copyright belonging to Forest Laboratories, Inc. This e-mail is intended
solely for the use of the individual or entity to which it is addressed. If
you are not the intended recipient of this e-mail, or the employee or agent
responsible for delivering this e-mail to the intended recipient, you are
hereby notified that any dissemination, distribution, copying or action
taken in relation to the contents of and attachments to this e-mail is
strictly prohibited and may be unlawful. If you have received this e-mail in
error, please notify the sender immediately and permanently delete the
original and any copy of this e-mail and any printout.

sending to informix-list

In message <c56p0h$cfc$1@terabinaries.xmission.com>, Alexey Sonkin
<alexeis@grandvirtual.com> writes
>
>
>Rajesh,
>
>As You mentioned, Your DLT speed is 35GB/hour.
>This is why it takes 10-11 hours to make the L0 backup
>of Your 400GB database. Your L0 backup speed is limited
>by the DLT device.
>
>For L1 archive, the backup speed is mostly limited by the
>disk array throughput: as I've already mentioned, Informix
>must read the entire database (like in L0 archive)

Surely this depends on some extent as to how much of the database has
actually changed since the last L0 archive?

>
>You can also check whether any significant database writes
>occur during the backup. If this is the case, You might experience
>'very old page' problem
>
>
>------------------------------------------
>Alexey Sonkin
>*
>-----Original Message-----
>From: Rajasekaran, Rajesh [mailto:RRajasekaran@forestpharm.com]
>Sent: Friday, April 09, 2004 9:15 AM
>To: 'Alexey Sonkin'
>Cc: 'sapmix@iiug.org'; 'informix-list@iiug.org'
>Subject: RE: Incremental Archive performance issue......
>
>Alexey,
>***** Our level-0 bkup on QAS is currently taking 10-11 hrs for the 400 GB
>database with the serial backup and single stream. The level-1 bkup used
>to take not more than 2 hrs to backup. Our PRD level-1 is still taking
>1-1.5 hrs and its storedge array is SUN 3510 FC array.* Moreover on the QAS,
>
>I dont think its the T3 array throughput bcos if thats the case,
>it would have degraded the level-0 as well but its not. We dont take
>level-2 here. Iam not suspecting the DLT drive too. Its somewhere btw the
>ONBAR & the IDS.
>*Let me check what else is changed on the box* such as O/S patches,
>T3 disk array related patches in the recent past.
>More suggestions are greatly appreciated.
>Thanks
>RAJ
>
>-----Original Message-----
>From: Alexey Sonkin [mailto:alexeis@grandvirtual.com]
>Sent: Thursday, April 08, 2004 5:19 PM
>To: 'Rajasekaran, Rajesh'; 'sapmix@iiug.org'
>Cc: 'informix-list@iiug.org'
>Subject: RE: Incremental Archive performance issue......
>
>Rajesh,
>It's not clear to me from Your email how long it takes
>to make the level-0 backup of Your QAS.
>I suppose it is about the same as Your Level-1 backup.
>Is Your question why You see such a significant difference between
>level-1 and level-0 on PRD and no difference on QA?
>During Level-1 backup, Informix is reading from the database
>EXACTLY THE SAME amount of data as during level-0 backup.
>For each page IDS makes timestamp comparison to decide, whether
>to put that page into L1 backup or not. The only difference
>between L1 and L2 is how much data the engine is sending to tape.
>It means, that in Your case, on QAS backup performance
>is limited by the T3 array throughput rather then by DLT limitations.
>More powerful (I suppose) production disk array can ship data
>much faster to IDS for analysis, and backup performance is constrained
>by the DLT limitations...
>
>------------------------------------------
>Alexey Sonkin
>*
>-----Original Message-----
>From: Rajasekaran, Rajesh [mailto:RRajasekaran@forestpharm.com]
>Sent: Thursday, April 08, 2004 4:56 PM
>To: 'sapmix@iiug.org'
>Cc: 'informix-list@iiug.org'
>Subject: Incremental Archive performance issue......
>Hi,
>*** Our QAS system is a Sun E450 box with a sun storedge T/3 array attached.
>
>Its a solaris o/s with IDS 7.31 database. The DB size is around 400 GB. Its
>backup is directed over the network to a tape library attached to a master
>server which is another host. Our Master server is also the* PRD box. Its a
>Sun Fire V880 with Sun L40 tape libary attached. We have the Veritas
>Netbackup Data center 4.5 for our backups & restores.
>The QAS DB backup is going over the network to the master server's DLT tape
>library that gives 35 GB/HR performance. We got that throughput for our QAS
>level-0 bkup even though its over the T3 network. But the Level-1 backup has
>
>severe performance degradation and it takes almost 5-7 hrs to complete. Our
>PRD level-1's are taking anywhere btw 1.5 - 2 hrs.
>We are only running the onbar whole system serial bkup. It should not put
>much I/O bottlenecks since only one dbspace bkup is running at one time and
>not parallel.
>*I can see the iostat shows 10000-25000 Kb/sec for the T3 storedge array
>continously during the level-1 bkups. The informix "onstat -g stq" shows
>Full queue 0, Empty queue with Cnt 100 all the time. I dont see the I/O
>bottleneck here and i dont see the problem with the Veritas Netbackup side
>since our level-0 works as expected. I dont see any "arc_very_old_pages()"
>running during the level-1 bkups.
>Questions
>1) What could be the reason for level-1 backup degradation in performance
>when level-0 is performing well ?
>2) What better way to monitor btw the onbar and the engine ? Will the
>bar_dbug = 9 lvl trace shows* more detail towards my problem ?
>Any suggestions/comments is greatly appreciated.

--
Five Cats
Email to: cats_spam at uk2 dot net

"Five Cats" <cats_spam@[127.0.0.1]> wrote in message
news:Hc7W+WA3NmeAFwKT@[127.0.0.1]...
> In message <c56p0h$cfc$1@terabinaries.xmission.com>, Alexey Sonkin
> <alexeis@grandvirtual.com> writes
> >

> >For L1 archive, the backup speed is mostly limited by the
> >disk array throughput: as I've already mentioned, Informix
> >must read the entire database (like in L0 archive)
>
> Surely this depends on some extent as to how much of the database has
> actually changed since the last L0 archive?

And how, pray, is the Informix engine to know if a page has changed since
the last level 0 archive????

In message <c5ju7r$2jjcs$1@ID-162943.news.uni-berlin.de>, Neil Truby
<neil.truby@ardenta.com> writes
>"Five Cats" <cats_spam@[127.0.0.1]> wrote in message
>news:Hc7W+WA3NmeAFwKT@[127.0.0.1]...
>> In message <c56p0h$cfc$1@terabinaries.xmission.com>, Alexey Sonkin
>> <alexeis@grandvirtual.com> writes
>> >
>
>> >For L1 archive, the backup speed is mostly limited by the
>> >disk array throughput: as I've already mentioned, Informix
>> >must read the entire database (like in L0 archive)
>>
>> Surely this depends on some extent as to how much of the database has
>> actually changed since the last L0 archive?
>
>And how, pray, is the Informix engine to know if a page has changed since
>the last level 0 archive????

The date stamp on the page.


--
Five Cats
Email to: cats_spam at uk2 dot net

Neil Truby wrote:
>
> "Five Cats" <cats_spam@[127.0.0.1]> wrote in message
> news:Hc7W+WA3NmeAFwKT@[127.0.0.1]...
> > In message <c56p0h$cfc$1@terabinaries.xmission.com>, Alexey Sonkin
> > <alexeis@grandvirtual.com> writes
> > >
>
> > >For L1 archive, the backup speed is mostly limited by the
> > >disk array throughput: as I've already mentioned, Informix
> > >must read the entire database (like in L0 archive)
> >
> > Surely this depends on some extent as to how much of the database has
> > actually changed since the last L0 archive?
>
> And how, pray, is the Informix engine to know if a page has changed since
> the last level 0 archive????

onstat -g arc
oncheck -pr

are your friends.

The rest is to compare page timestamps with these entries
in the reserved pages.

dic_k
--
Richard Kofler
SOLID STATE EDV
Dienstleistungen GmbH
Vienna/Austria/Europe

"Five Cats" <cats_spam@[127.0.0.1]> wrote in message
news:OhnEx9ADObfAFwwd@[127.0.0.1]...
> In message <c5ju7r$2jjcs$1@ID-162943.news.uni-berlin.de>, Neil Truby
> <neil.truby@ardenta.com> writes
> >"Five Cats" <cats_spam@[127.0.0.1]> wrote in message
> >news:Hc7W+WA3NmeAFwKT@[127.0.0.1]...
> >> In message <c56p0h$cfc$1@terabinaries.xmission.com>, Alexey Sonkin
> >> <alexeis@grandvirtual.com> writes
> >> >
> >
> >> >For L1 archive, the backup speed is mostly limited by the
> >> >disk array throughput: as I've already mentioned, Informix
> >> >must read the entire database (like in L0 archive)
> >>
> >> Surely this depends on some extent as to how much of the database has
> >> actually changed since the last L0 archive?
> >
> >And how, pray, is the Informix engine to know if a page has changed since
> >the last level 0 archive????
>
> The date stamp on the page.

OK, but it has to read the page to know this. So Informix has to read all
pages (at least, all those with at least some data in them), even for an
incremental backup.

Neil Truby wrote:
>
> "Five Cats" <cats_spam@[127.0.0.1]> wrote in message
> news:OhnEx9ADObfAFwwd@[127.0.0.1]...
> > In message <c5ju7r$2jjcs$1@ID-162943.news.uni-berlin.de>, Neil Truby
> > <neil.truby@ardenta.com> writes
> > >"Five Cats" <cats_spam@[127.0.0.1]> wrote in message
> > >news:Hc7W+WA3NmeAFwKT@[127.0.0.1]...
> > >> In message <c56p0h$cfc$1@terabinaries.xmission.com>, Alexey Sonkin
> > >> <alexeis@grandvirtual.com> writes
> > >> >
> > >
> > >> >For L1 archive, the backup speed is mostly limited by the
> > >> >disk array throughput: as I've already mentioned, Informix
> > >> >must read the entire database (like in L0 archive)
> > >>
> > >> Surely this depends on some extent as to how much of the database has
> > >> actually changed since the last L0 archive?
> > >
> > >And how, pray, is the Informix engine to know if a page has changed since
> > >the last level 0 archive????
> >
> > The date stamp on the page.
>
> OK, but it has to read the page to know this. So Informix has to read all
> pages (at least, all those with at least some data in them), even for an
> incremental backup.

exactly this!
The engine must read all pages (even those, which have no more
any data in them - as this might be the change), but Level 1 in
contrary to a full system backup (Level 0) in general terms brings
less
pages onto the XBSA pipe or onto the respective SW interface of
ontape, and therefore is rather less dependant on the tape speed
(or output speed to disk if using disk pools or ontaping to disk)
And YES - as J. Tong pointed out many times: ontape to disk never
was unsupported! )))

dic_k
--
Richard Kofler
SOLID STATE EDV
Dienstleistungen GmbH
Vienna/Austria/Europe

On Wed, 14 Apr 2004 23:13:29 GMT, Richard Kofler
<richard.kofler@chello.at> wrote:

>Neil Truby wrote:
>>
>> "Five Cats" wrote:
>> > Neil Truby writes:
>> > >"Five Cats" wrote:
>> > >> Alexey Sonkin writes:
>> > >> >
>> > >
>> > >> >For L1 archive, the backup speed is mostly limited by the
>> > >> >disk array throughput: as I've already mentioned, Informix
>> > >> >must read the entire database (like in L0 archive)
>> > >>
>> > >> Surely this depends on some extent as to how much of the database has
>> > >> actually changed since the last L0 archive?
>> > >
>> > >And how, pray, is the Informix engine to know if a page has changed since
>> > >the last level 0 archive????
>> >
>> > The date stamp on the page.
>>
>> OK, but it has to read the page to know this. So Informix has to read all
>> pages (at least, all those with at least some data in them), even for an
>> incremental backup.
>
>exactly this!
>The engine must read all pages (even those, which have no more
>any data in them - as this might be the change), but Level 1 in
>contrary to a full system backup (Level 0) in general terms brings
>less
>pages onto the XBSA pipe or onto the respective SW interface of
>ontape, and therefore is rather less dependant on the tape speed
>(or output speed to disk if using disk pools or ontaping to disk)
>And YES - as J. Tong pointed out many times: ontape to disk never
>was unsupported! )))
>
>dic_k

Fortunately it is smart enough to not read *all* pages - it will
consult the chunk free list and only examine those pages actually
allocated to a tablespace or system object (i.e. logical log, etc.).
Then again, it does this for all levels of archive. Timestamps need
to be read for any archive level - even a full, as you have to deal
with pages updated after the archive start time - so the "scan time"
for archives of any level will be pretty much the same. It is the
time spent writing to tape that will differ for a full vs.
incremental. Another possible factor is the amount of update activity
during the archive, as the physical log needs to be checked regularly
to grab pages that have been updated after the archive start (though
the amount of impact depends on the version you are using - older
stuff, pre-7.2 I think, writes the pages from the plog to the backup
during checkpoints; the later versions collect them in a temp table
and add them to the archive at the end.)

The tape hardware can also be a factor. Newer, high-speed tape drives
see maximum i/o performance when a constant stream of data is
supplied. Writing in smaller "chunks" sporadically results in much
poorer performance on such drives. As an incremental archive is
likely to produce the latter behavior, you can expect to see reduced
performance on that end.

Dave Kosenko