SEO

This is a question that I also posted on Dell hardware forums, and I realize it
probably belongs there more than here. But I am thinking someone might have
some anecdotal information that could help me and this post may help someone
else down the road.

My PowerEdge 1800 (dual 3ghz Xeon, 3GB ram) came with a single SATA drive. I
installed and developed a modest database application (Postgresql, Windows 2003
Server, Python) on it and was pretty impressed with the performance.

I wanted to add some more space, speed and reliability so I bought a used 3ware
9500s SATA raid card. I bought three more of the same drives (Seagate
ST3808110as) and configured them in RAID 5 (3 in RAID one as hot spare). I
reinstalled the OS and software (my efforts to ghost were not
fruitful...another story), and the first thing I did was run a sql script to
make tables, indexes, sequences etc for my app and import about 20MB of data.

When I had this installed on a single SATA drive running from the PE1800's
on-board SATA interface, this operation took anywhere from 65-80 seconds.

With my new RAID card and drives, this operation took 272 seconds!?

I did a quick search and found a few posts about how RAID 5 with Databases is a
poor choice and one in particular about Postgres and how you could expect
performance to be halved with RAID 5 over a single drive or RAID 1 (until you
get above 6 disks in your RAID 5 array). So, a poorly planned configuration on
my part.

I scrubbed the RAID config, made two RAID 1 containers (also read about how
moving database logs to a different partition than the database data is optimal
for speed and reliability). I installed the OS on the first RAID 1 volume, the
Postgresql apps and data on the other, and used Junction from sysinternals to
put the pg_xlogs back on the OS partition (does Postgresql have an easier way
to do this on Windows?).

Well, things didn't improve noticeably - 265 seconds.

Next step, turn on the 3ware RAID card's write cache (even though I have no
Battery Backup Unit on the RAID card and am warned about possible data loss in
the event of power loss).

This helped - down to 172 seconds.

Is this loss in performance just the normal overhead involved when adding a
raid card - writes now having to go to two drives instead of one? Or maybe is
the SATA interface on the Dell 1800s motherboard faster than the interface on
the 3ware raid card (SATA II ?).

Thanks for any help you guidance you can provide.

---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

On Apr 28, 2006, at 11:37 AM, Erik Myllymaki wrote:

> When I had this installed on a single SATA drive running from the
> PE1800's on-board SATA interface, this operation took anywhere from
> 65-80 seconds.
>
> With my new RAID card and drives, this operation took 272 seconds!?

switch it to RAID10 and re-try your experiment. if that is fast,
then you know your raid controller does bad RAID5.

anyhow, I have in one server (our office mail server and part-time
development testing box) an adaptec SATA RAID from dell. it is
configured for RAID5 and does well for normal office stuff, but when
we do postgres tests on it, it just is plain old awful.

but I have some LSI based cards on which RAID5 is plenty fast and
suitable for the DB, but those are SCSI.

For what it is worth, the Dell PE1850 internal PERC4/Si card is
wicked fast when hooked up with a pair of U320 SCSI drives.



---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

It's also possible that the single SATA drive you were testing (or the
controller it was attached to) is lying about fsync and performing write
caching behind your back, whereas your new controller and drives are
not.

You'll find a lot more info on the archives of this list about it, but
basically if your application is committing a whole lot of small
transactions, then it will run fast (but not safely) on a drive which
lies about fsync, but slower on a better disk subsystem which doesn't
lie about fsync.

Try running a test with fsync=off with your new equipment and if it
suddenly starts running faster, then you know that's the problem.
You'll either have a choice of losing all of your data the next time the
system shuts down uncleanly but being fast, or of running slow, or of
fixing the applications to use chunkier transactions.

-- Mark

On Fri, 2006-04-28 at 13:36 -0400, Vivek Khera wrote:
> On Apr 28, 2006, at 11:37 AM, Erik Myllymaki wrote:
>
> > When I had this installed on a single SATA drive running from the
> > PE1800's on-board SATA interface, this operation took anywhere from
> > 65-80 seconds.
> >
> > With my new RAID card and drives, this operation took 272 seconds!?
>
> switch it to RAID10 and re-try your experiment. if that is fast,
> then you know your raid controller does bad RAID5.
>
> anyhow, I have in one server (our office mail server and part-time
> development testing box) an adaptec SATA RAID from dell. it is
> configured for RAID5 and does well for normal office stuff, but when
> we do postgres tests on it, it just is plain old awful.
>
> but I have some LSI based cards on which RAID5 is plenty fast and
> suitable for the DB, but those are SCSI.
>
> For what it is worth, the Dell PE1850 internal PERC4/Si card is
> wicked fast when hooked up with a pair of U320 SCSI drives.
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faq

---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match

Erik,

I think you have a mismatch in your Linux driver and firmware for your 3Ware
card. Download a matched Linux driver and firmware from www.3ware.com and
your problems should disappear.

- Luke


On 4/28/06 8:37 AM, "Erik Myllymaki" <erik.myllymaki@aviawest.com> wrote:

> This is a question that I also posted on Dell hardware forums, and I realize
> it
> probably belongs there more than here. But I am thinking someone might have
> some anecdotal information that could help me and this post may help someone
> else down the road.
>
> My PowerEdge 1800 (dual 3ghz Xeon, 3GB ram) came with a single SATA drive. I
> installed and developed a modest database application (Postgresql, Windows
> 2003
> Server, Python) on it and was pretty impressed with the performance.
>
> I wanted to add some more space, speed and reliability so I bought a used
> 3ware
> 9500s SATA raid card. I bought three more of the same drives (Seagate
> ST3808110as) and configured them in RAID 5 (3 in RAID one as hot spare). I
> reinstalled the OS and software (my efforts to ghost were not
> fruitful...another story), and the first thing I did was run a sql script to
> make tables, indexes, sequences etc for my app and import about 20MB of data.
>
> When I had this installed on a single SATA drive running from the PE1800's
> on-board SATA interface, this operation took anywhere from 65-80 seconds.
>
> With my new RAID card and drives, this operation took 272 seconds!?
>
> I did a quick search and found a few posts about how RAID 5 with Databases is
> a
> poor choice and one in particular about Postgres and how you could expect
> performance to be halved with RAID 5 over a single drive or RAID 1 (until you
> get above 6 disks in your RAID 5 array). So, a poorly planned configuration on
> my part.
>
> I scrubbed the RAID config, made two RAID 1 containers (also read about how
> moving database logs to a different partition than the database data is
> optimal
> for speed and reliability). I installed the OS on the first RAID 1 volume, the
> Postgresql apps and data on the other, and used Junction from sysinternals to
> put the pg_xlogs back on the OS partition (does Postgresql have an easier way
> to do this on Windows?).
>
> Well, things didn't improve noticeably - 265 seconds.
>
> Next step, turn on the 3ware RAID card's write cache (even though I have no
> Battery Backup Unit on the RAID card and am warned about possible data loss in
> the event of power loss).
>
> This helped - down to 172 seconds.
>
> Is this loss in performance just the normal overhead involved when adding a
> raid card - writes now having to go to two drives instead of one? Or maybe is
> the SATA interface on the Dell 1800s motherboard faster than the interface on
> the 3ware raid card (SATA II ?).
>
> Thanks for any help you guidance you can provide.
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faq
>
>

I have been in discussion with 3ware support and after adjusting some settings,
the 3ware card in RAID 1 gets better performance than the single drive. I guess
this had everything to do with the write (and maybe read?) cache.

Of course now i am in a dangerous situation - using volatile write cache
without a BBU.

If I were to use a UPS to ensure a soft shutdown in the event of power loss, am
I somewhat as safe as if I were to purchase a BBU for this RAID card?



Thanks.

Mark Lewis wrote:
> It's also possible that the single SATA drive you were testing (or the
> controller it was attached to) is lying about fsync and performing write
> caching behind your back, whereas your new controller and drives are
> not.
>
> You'll find a lot more info on the archives of this list about it, but
> basically if your application is committing a whole lot of small
> transactions, then it will run fast (but not safely) on a drive which
> lies about fsync, but slower on a better disk subsystem which doesn't
> lie about fsync.
>
> Try running a test with fsync=off with your new equipment and if it
> suddenly starts running faster, then you know that's the problem.
> You'll either have a choice of losing all of your data the next time the
> system shuts down uncleanly but being fast, or of running slow, or of
> fixing the applications to use chunkier transactions.
>
> -- Mark
>
> On Fri, 2006-04-28 at 13:36 -0400, Vivek Khera wrote:
>> On Apr 28, 2006, at 11:37 AM, Erik Myllymaki wrote:
>>
>>> When I had this installed on a single SATA drive running from the
>>> PE1800's on-board SATA interface, this operation took anywhere from
>>> 65-80 seconds.
>>>
>>> With my new RAID card and drives, this operation took 272 seconds!?
>> switch it to RAID10 and re-try your experiment. if that is fast,
>> then you know your raid controller does bad RAID5.
>>
>> anyhow, I have in one server (our office mail server and part-time
>> development testing box) an adaptec SATA RAID from dell. it is
>> configured for RAID5 and does well for normal office stuff, but when
>> we do postgres tests on it, it just is plain old awful.
>>
>> but I have some LSI based cards on which RAID5 is plenty fast and
>> suitable for the DB, but those are SCSI.
>>
>> For what it is worth, the Dell PE1850 internal PERC4/Si card is
>> wicked fast when hooked up with a pair of U320 SCSI drives.
>>
>>
>>
>> ---------------------------(end of broadcast)---------------------------
>> TIP 3: Have you checked our extensive FAQ?
>>
>> http://www.postgresql.org/docs/faq
>
> ---------------------------(end of broadcast)---------------------------
> TIP 9: In versions below 8.0, the planner will ignore your desire to
> choose an index scan if your joining column's datatypes do not
> match

---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend

A UPS will make it less likely that the system will reboot and destroy
your database due to a power failure, but there are other causes for a
system reboot.

With a BBU, the only component that can fail and cause catastrophic data
loss is the RAID itself.

With a UPS, you are additionally vulnerable to OS crashes, failures in
non-RAID hardware, UPS failures, or anything else that would necessitate
a hard reboot.

So a UPS is a decent replacement for a BBU only if you trust your app
server/OS more than you value your data.

-- Mark Lewis


On Mon, 2006-05-01 at 10:58 -0700, Erik Myllymaki wrote:
> I have been in discussion with 3ware support and after adjusting some settings,
> the 3ware card in RAID 1 gets better performance than the single drive. I guess
> this had everything to do with the write (and maybe read?) cache.
>
> Of course now i am in a dangerous situation - using volatile write cache
> without a BBU.
>
> If I were to use a UPS to ensure a soft shutdown in the event of power loss, am
> I somewhat as safe as if I were to purchase a BBU for this RAID card?
>
>
>
> Thanks.
>
> Mark Lewis wrote:
> > It's also possible that the single SATA drive you were testing (or the
> > controller it was attached to) is lying about fsync and performing write
> > caching behind your back, whereas your new controller and drives are
> > not.
> >
> > You'll find a lot more info on the archives of this list about it, but
> > basically if your application is committing a whole lot of small
> > transactions, then it will run fast (but not safely) on a drive which
> > lies about fsync, but slower on a better disk subsystem which doesn't
> > lie about fsync.
> >
> > Try running a test with fsync=off with your new equipment and if it
> > suddenly starts running faster, then you know that's the problem.
> > You'll either have a choice of losing all of your data the next time the
> > system shuts down uncleanly but being fast, or of running slow, or of
> > fixing the applications to use chunkier transactions.
> >
> > -- Mark
> >
> > On Fri, 2006-04-28 at 13:36 -0400, Vivek Khera wrote:
> >> On Apr 28, 2006, at 11:37 AM, Erik Myllymaki wrote:
> >>
> >>> When I had this installed on a single SATA drive running from the
> >>> PE1800's on-board SATA interface, this operation took anywhere from
> >>> 65-80 seconds.
> >>>
> >>> With my new RAID card and drives, this operation took 272 seconds!?
> >> switch it to RAID10 and re-try your experiment. if that is fast,
> >> then you know your raid controller does bad RAID5.
> >>
> >> anyhow, I have in one server (our office mail server and part-time
> >> development testing box) an adaptec SATA RAID from dell. it is
> >> configured for RAID5 and does well for normal office stuff, but when
> >> we do postgres tests on it, it just is plain old awful.
> >>
> >> but I have some LSI based cards on which RAID5 is plenty fast and
> >> suitable for the DB, but those are SCSI.
> >>
> >> For what it is worth, the Dell PE1850 internal PERC4/Si card is
> >> wicked fast when hooked up with a pair of U320 SCSI drives.
> >>
> >>
> >>
> >> ---------------------------(end of broadcast)---------------------------
> >> TIP 3: Have you checked our extensive FAQ?
> >>
> >> http://www.postgresql.org/docs/faq
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 9: In versions below 8.0, the planner will ignore your desire to
> > choose an index scan if your joining column's datatypes do not
> > match

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

On May 1, 2006, at 1:58 PM, Erik Myllymaki wrote:

> Of course now i am in a dangerous situation - using volatile write
> cache without a BBU.
>

It should be against the law to make RAID cards with caches that are
not battery backed.

> If I were to use a UPS to ensure a soft shutdown in the event of
> power loss, am I somewhat as safe as if I were to purchase a BBU
> for this RAID card?

no. not at all.


---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

good points, thanks.

Tom Arthurs wrote:
> UPS does not protect against the tech behind the rack unplugging the
> power cable, or an accidental power cycle from exercising the wrong
> switch. Both are probably more common causes of failure than a total
> power outage.
>
> Erik Myllymaki wrote:
>> I have been in discussion with 3ware support and after adjusting some
>> settings, the 3ware card in RAID 1 gets better performance than the
>> single drive. I guess this had everything to do with the write (and
>> maybe read?) cache.
>>
>> Of course now i am in a dangerous situation - using volatile write
>> cache without a BBU.
>>
>> If I were to use a UPS to ensure a soft shutdown in the event of power
>> loss, am I somewhat as safe as if I were to purchase a BBU for this
>> RAID card?
>>
>>
>>
>> Thanks.
>>
>> Mark Lewis wrote:
>>> It's also possible that the single SATA drive you were testing (or the
>>> controller it was attached to) is lying about fsync and performing write
>>> caching behind your back, whereas your new controller and drives are
>>> not.
>>>
>>> You'll find a lot more info on the archives of this list about it, but
>>> basically if your application is committing a whole lot of small
>>> transactions, then it will run fast (but not safely) on a drive which
>>> lies about fsync, but slower on a better disk subsystem which doesn't
>>> lie about fsync.
>>>
>>> Try running a test with fsync=off with your new equipment and if it
>>> suddenly starts running faster, then you know that's the problem.
>>> You'll either have a choice of losing all of your data the next time the
>>> system shuts down uncleanly but being fast, or of running slow, or of
>>> fixing the applications to use chunkier transactions.
>>>
>>> -- Mark
>>>
>>> On Fri, 2006-04-28 at 13:36 -0400, Vivek Khera wrote:
>>>> On Apr 28, 2006, at 11:37 AM, Erik Myllymaki wrote:
>>>>
>>>>> When I had this installed on a single SATA drive running from the
>>>>> PE1800's on-board SATA interface, this operation took anywhere
>>>>> from 65-80 seconds.
>>>>>
>>>>> With my new RAID card and drives, this operation took 272 seconds!?
>>>> switch it to RAID10 and re-try your experiment. if that is fast,
>>>> then you know your raid controller does bad RAID5.
>>>>
>>>> anyhow, I have in one server (our office mail server and part-time
>>>> development testing box) an adaptec SATA RAID from dell. it is
>>>> configured for RAID5 and does well for normal office stuff, but
>>>> when we do postgres tests on it, it just is plain old awful.
>>>>
>>>> but I have some LSI based cards on which RAID5 is plenty fast and
>>>> suitable for the DB, but those are SCSI.
>>>>
>>>> For what it is worth, the Dell PE1850 internal PERC4/Si card is
>>>> wicked fast when hooked up with a pair of U320 SCSI drives.
>>>>
>>>>
>>>>
>>>> ---------------------------(end of
>>>> broadcast)---------------------------
>>>> TIP 3: Have you checked our extensive FAQ?
>>>>
>>>> http://www.postgresql.org/docs/faq
>>>
>>> ---------------------------(end of broadcast)---------------------------
>>> TIP 9: In versions below 8.0, the planner will ignore your desire to
>>> choose an index scan if your joining column's datatypes do not
>>> match
>>
>> ---------------------------(end of broadcast)---------------------------
>> TIP 6: explain analyze is your friend

---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend

UPS does not protect against the tech behind the rack unplugging the
power cable, or an accidental power cycle from exercising the wrong
switch. Both are probably more common causes of failure than a total
power outage.

Erik Myllymaki wrote:
> I have been in discussion with 3ware support and after adjusting some
> settings, the 3ware card in RAID 1 gets better performance than the
> single drive. I guess this had everything to do with the write (and
> maybe read?) cache.
>
> Of course now i am in a dangerous situation - using volatile write
> cache without a BBU.
>
> If I were to use a UPS to ensure a soft shutdown in the event of power
> loss, am I somewhat as safe as if I were to purchase a BBU for this
> RAID card?
>
>
>
> Thanks.
>
> Mark Lewis wrote:
>> It's also possible that the single SATA drive you were testing (or the
>> controller it was attached to) is lying about fsync and performing write
>> caching behind your back, whereas your new controller and drives are
>> not.
>>
>> You'll find a lot more info on the archives of this list about it, but
>> basically if your application is committing a whole lot of small
>> transactions, then it will run fast (but not safely) on a drive which
>> lies about fsync, but slower on a better disk subsystem which doesn't
>> lie about fsync.
>>
>> Try running a test with fsync=off with your new equipment and if it
>> suddenly starts running faster, then you know that's the problem.
>> You'll either have a choice of losing all of your data the next time the
>> system shuts down uncleanly but being fast, or of running slow, or of
>> fixing the applications to use chunkier transactions.
>>
>> -- Mark
>>
>> On Fri, 2006-04-28 at 13:36 -0400, Vivek Khera wrote:
>>> On Apr 28, 2006, at 11:37 AM, Erik Myllymaki wrote:
>>>
>>>> When I had this installed on a single SATA drive running from the
>>>> PE1800's on-board SATA interface, this operation took anywhere
>>>> from 65-80 seconds.
>>>>
>>>> With my new RAID card and drives, this operation took 272 seconds!?
>>> switch it to RAID10 and re-try your experiment. if that is fast,
>>> then you know your raid controller does bad RAID5.
>>>
>>> anyhow, I have in one server (our office mail server and part-time
>>> development testing box) an adaptec SATA RAID from dell. it is
>>> configured for RAID5 and does well for normal office stuff, but
>>> when we do postgres tests on it, it just is plain old awful.
>>>
>>> but I have some LSI based cards on which RAID5 is plenty fast and
>>> suitable for the DB, but those are SCSI.
>>>
>>> For what it is worth, the Dell PE1850 internal PERC4/Si card is
>>> wicked fast when hooked up with a pair of U320 SCSI drives.
>>>
>>>
>>>
>>> ---------------------------(end of
>>> broadcast)---------------------------
>>> TIP 3: Have you checked our extensive FAQ?
>>>
>>> http://www.postgresql.org/docs/faq
>>
>> ---------------------------(end of broadcast)---------------------------
>> TIP 9: In versions below 8.0, the planner will ignore your desire to
>> choose an index scan if your joining column's datatypes do not
>> match
>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: explain analyze is your friend

---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?

http://archives.postgresql.org

We use the 3Ware BBUs and they¹re very nice, they self monitor and let you
know about their capacity if it¹s a problem.

- Luke


On 5/1/06 11:43 AM, "Erik Myllymaki" <erik.myllymaki@aviawest.com> wrote:

> good points, thanks.
>
> Tom Arthurs wrote:
>> > UPS does not protect against the tech behind the rack unplugging the
>> > power cable, or an accidental power cycle from exercising the wrong
>> > switch. Both are probably more common causes of failure than a total
>> > power outage.
>> >
>> > Erik Myllymaki wrote:
>>> >> I have been in discussion with 3ware support and after adjusting some
>>> >> settings, the 3ware card in RAID 1 gets better performance than the
>>> >> single drive. I guess this had everything to do with the write (and
>>> >> maybe read?) cache.
>>> >>
>>> >> Of course now i am in a dangerous situation - using volatile write
>>> >> cache without a BBU.
>>> >>
>>> >> If I were to use a UPS to ensure a soft shutdown in the event of power
>>> >> loss, am I somewhat as safe as if I were to purchase a BBU for this
>>> >> RAID card?
>>> >>
>>> >>
>>> >>
>>> >> Thanks.
>>> >>
>>> >> Mark Lewis wrote:
>>>> >>> It's also possible that the single SATA drive you were testing (orthe
>>>> >>> controller it was attached to) is lying about fsync and performing>>>>
write
>>>> >>> caching behind your back, whereas your new controller and drives are
>>>> >>> not.
>>>> >>>
>>>> >>> You'll find a lot more info on the archives of this list about it,but
>>>> >>> basically if your application is committing a whole lot of small
>>>> >>> transactions, then it will run fast (but not safely) on a drive which
>>>> >>> lies about fsync, but slower on a better disk subsystem which doesn't
>>>> >>> lie about fsync.
>>>> >>>
>>>> >>> Try running a test with fsync=off with your new equipment and if it
>>>> >>> suddenly starts running faster, then you know that's the problem.
>>>> >>> You'll either have a choice of losing all of your data the next time
the
>>>> >>> system shuts down uncleanly but being fast, or of running slow, orof
>>>> >>> fixing the applications to use chunkier transactions.
>>>> >>>
>>>> >>> -- Mark
>>>> >>>
>>>> >>> On Fri, 2006-04-28 at 13:36 -0400, Vivek Khera wrote:
>>>>> >>>> On Apr 28, 2006, at 11:37 AM, Erik Myllymaki wrote:
>>>>> >>>>
>>>>>> >>>>> When I had this installed on a single SATA drive running from the
>>>>>> >>>>> PE1800's on-board SATA interface, this operation took anywhere
>>>>>> >>>>> from 65-80 seconds.
>>>>>> >>>>>
>>>>>> >>>>> With my new RAID card and drives, this operation took 272 seconds!?
>>>>> >>>> switch it to RAID10 and re-try your experiment. if that is fast,
>>>>> >>>> then you know your raid controller does bad RAID5.
>>>>> >>>>
>>>>> >>>> anyhow, I have in one server (our office mail server and part-time
>>>>> >>>> development testing box) an adaptec SATA RAID from dell. it is
>>>>> >>>> configured for RAID5 and does well for normal office stuff, but
>>>>> >>>> when we do postgres tests on it, it just is plain old awful.
>>>>> >>>>
>>>>> >>>> but I have some LSI based cards on which RAID5 is plenty fast and
>>>>> >>>> suitable for the DB, but those are SCSI.
>>>>> >>>>
>>>>> >>>> For what it is worth, the Dell PE1850 internal PERC4/Si card is
>>>>> >>>> wicked fast when hooked up with a pair of U320 SCSI drives.
>>>>> >>>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> ---------------------------(end of
>>>>> >>>> broadcast)---------------------------
>>>>> >>>> TIP 3: Have you checked our extensive FAQ?
>>>>> >>>>
>>>>> >>>> http://www.postgresql.org/docs/faq
>>>> >>>
>>>> >>> ---------------------------(end of
>>>> broadcast)---------------------------
>>>> >>> TIP 9: In versions below 8.0, the planner will ignore your desire to
>>>> >>> choose an index scan if your joining column's datatypes do not
>>>> >>> match
>>> >>
>>> >> ---------------------------(end of broadcast)---------------------------
>>> >> TIP 6: explain analyze is your friend
>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: explain analyze is your friend
>
>