On Fri, Sep 07, 2007 at 12:33:41PM +0200, Tobias Brox wrote:
>Advantages:
>
> 1. Higher I/O (at least the salesman claims so)

Benchmark it. It is extremely unlikely that you'll get I/O *as good as*
DAS at a similar price point.

> 2. Easier to upgrade the disk capacity

Is this an issue? You may find that you can simply get dramatically more
space for the money with DAS and not have to worry about an upgrade.
Also, you can use the postgres tablespace functionality to migrate data
to a new partition fairly transparently.

> 3. Easy to set up "warm standby" functionality. (Then again, if the
> postgres server fails miserably, it's likely to be due to a disk
> crash).

You may find that using db replication will gain you even more
reliability for less money.

>Also, my boss states that "all big enterprises uses SAN nowadays".

Use SAN *for what*?

Mike Stone

---------------------------(end of broadcast)---------------------------
TIP 7: You can help support the PostgreSQL project by donating at

http://www.postgresql.org/about/donate

I'm getting a san together to consolidate my disk space usage for my
servers. It's iscsi based and I'll be pxe booting my servers from it.
The idea is to keep spares on hand for one system (the san) and not have
to worry about spares for each specific storage system on each server.
This also makes growing filesystems and such pretty simple. Redundancy
is also good since I'll have two iscsi switches plugged into two cisco
ethernet switches and two different raid controllers on the jbod. I'll
start plugging my servers into each switch for further redundancy. In
the end I could loose disks, ethernet switches, cables, iscsi switches,
raid controller, whatever, and it keeps on moving.

That said, I'm not putting my postgres data on the san. The DB server
will boot from the san and use it for is OS, but there are 6 15k SAS
disks in it setup with raid-10 that will be used for the postgres data
mount. The machine is a dell 2950 and uses an LSI raid card.

The end result is a balance of cost, performance, and reliability. I'm
using iscsi for the cost, reliability, and ease of use, but where I need
performance I'm sticking to local disks.

schu

---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly

We are currently running our database against on SAN share. It looks like this:

2 x RAID 10 (4 disk SATA 7200 each)

Raid Group 0 contains the tables + indexes
Raid Group 1 contains the log files + backups (pg_dump)

Our database server connects to the san via iSCSI over Gig/E using
jumbo frames. File system is XFS (noatime).

I believe our raid controller is an ARECA. Whatever it is, it has the
option of adding a battery to it but I have not yet been able to
convince my boss that we need it.

Maintenance is nice, we can easily mess around with the drive shares,
expand and contract them, snapshot them, yadda yadda yadda. All
things which we NEVER do to our database anyway.

Performance, however, is a mixed bag. It handles concurrency very
well. We have a number of shares (production shares, data shares, log
file shares, backup shares, etc. etc.) spread across the two raid
groups and it handles them with aplomb.

Throughput, however, kinda sucks. I just can't get the kind of
throughput to it I was hoping to get. When our memory cache is blown,
the database can be downright painful for the next few minutes as
everything gets paged back into the cache.

I'd love to try a single 8 disk RAID 10 with battery wired up directly
to our database, but given the size of our company and limited funds,
it won't be feasible any time soon.

Bryan

On 9/7/07, Matthew Schumacher <matt.s@aptalaska.net> wrote:
> I'm getting a san together to consolidate my disk space usage for my
> servers. It's iscsi based and I'll be pxe booting my servers from it.
> The idea is to keep spares on hand for one system (the san) and not have
> to worry about spares for each specific storage system on each server.
> This also makes growing filesystems and such pretty simple. Redundancy
> is also good since I'll have two iscsi switches plugged into two cisco
> ethernet switches and two different raid controllers on the jbod. I'll
> start plugging my servers into each switch for further redundancy. In
> the end I could loose disks, ethernet switches, cables, iscsi switches,
> raid controller, whatever, and it keeps on moving.
>
> That said, I'm not putting my postgres data on the san. The DB server
> will boot from the san and use it for is OS, but there are 6 15k SAS
> disks in it setup with raid-10 that will be used for the postgres data
> mount. The machine is a dell 2950 and uses an LSI raid card.
>
> The end result is a balance of cost, performance, and reliability. I'm
> using iscsi for the cost, reliability, and ease of use, but where I need
> performance I'm sticking to local disks.
>
> schu
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly
>

---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

Bryan Murphy wrote:

>Our database server connects to the san via iSCSI over Gig/E using
>jumbo frames. File system is XFS (noatime).
>
>
>
>
....

>Throughput, however, kinda sucks. I just can't get the kind of
>throughput to it I was hoping to get. When our memory cache is blown,
>the database can be downright painful for the next few minutes as
>everything gets paged back into the cache.
>
>
>

Remember that Gig/E is bandwidth limited to about 100 Mbyte/sec. Maybe
a little faster than that downhill with a tailwind, but not much.
You're going to get much better bandwidth connecting to a local raid
card talking to local disks simply due to not having the ethernet as a
bottleneck. iSCSI is easy to set up and manage, but it's slow. This is
the big advantage Fibre Channel has- serious performance. You can have
multiple channels on a single fibre channel card- IIRC, QLogic's cards
have a default of 4 channels- each pumping 400 Mbyte/sec. At which
point the local bus rapidly becomes the bottleneck. Of course, this
comes at the cost of a signifigant increase in complexity.

Brian


---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Alan Hodgson wrote:
> On Friday 07 September 2007 10:56, "Bryan Murphy"
> <bryan.murphy@gmail.com> wrote:
>> Our database server connects to the san via iSCSI over Gig/E using
>> jumbo frames. File system is XFS (noatime).
>>
>> Throughput, however, kinda sucks. I just can't get the kind of
>> throughput to it I was hoping to get.
>
> A single Gig/E couldn't even theoretically do better than 125MB/sec, so
> yeah I would expect throughput sucks pretty bad.

We have a customer that has a iSCSI SAN that can bond multiple Gig/E
connections that provides them with reasonable performance. You should
see if yours allows it.

Joshua D. Drake


>


- --

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997 http://www.commandprompt.com/
UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFG4ZbGATb/zqfZUUQRAhtmAKCh/PsmkL/JOPq4++Aci2/XwDDJ7wCfbwJs
5vBg+TG5xQFKoJMdybpjDWo=
=up8R
-----END PGP SIGNATURE-----

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

On Friday 07 September 2007 10:56, "Bryan Murphy"
<bryan.murphy@gmail.com> wrote:
> Our database server connects to the san via iSCSI over Gig/E using
> jumbo frames. File system is XFS (noatime).
>
> Throughput, however, kinda sucks. I just can't get the kind of
> throughput to it I was hoping to get.

A single Gig/E couldn't even theoretically do better than 125MB/sec, so
yeah I would expect throughput sucks pretty bad.

--
"A democracy is a sheep and two wolves deciding on what to have for
lunch. Freedom is a well armed sheep contesting the results of the
decision." -- Benjamin Franklin


---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly

On Fri, 7 Sep 2007, Tobias Brox wrote:

> We're also considering to install postgres on SAN - that is, my boss is
> convinced this is the right way to go.
> Advantages:
> 1. Higher I/O (at least the salesman claims so)

Shockingly, the salesman is probably lying to you. The very concept of
SAN says that you're putting something in between your system and the
disks, and that something therefore must slow things down compared to
connecting directly. iSCSI, FC, whatever you're using as the
communications channel can't be as fast as a controller card with a good
interface straight into the motherboard. For example, a PCI-E x16 disk
controller card maxes out at 4GB/s in each direction; good luck bonding
enough iSCSI or FC channels together to reach that transfer rate and
getting something even remotely cost-competative with an internal card.

The cases where a SAN can improve upon performance over direct discs are
when the comparison isn't quite fair; for example:

1) The SAN allows spreading the load over more disks than you can fit
internally in the system
2) The SAN provides a larger memory cache than the internal cards you're
comparing against

If you're in one of those situations, then perhaps the salesman's claim
could have some merit. There are lots of reasons one might want to use a
SAN, but a higher I/O rate when fairly comparing to connecting disks
directly is unlikely to be on that list.

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD

---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly

On Sep 6, 2007, at 2:42 PM, Scott Marlowe wrote:

> I'd recommend against Dell unless you're at a company that orders
> computers by the hundred lot. My experience with Dell has been that
> unless you are a big customer you're just another number (a small one
> at that) on a spreadsheet.

I order maybe 5-6 servers per year from dell, and the sales rep knows
me when I call him. Just set up a business account.

That said, lately I've been buying Sun X4100's for my DB servers.
These machines are built like tanks and extremely fast. The only
difficulty is hooking up disks to them. The only sane choice is to
use a fibre channel card to an external array. The only dual-channel
internal SCSI RAID controller that fits is an Adaptec model, and it
is to be avoided.

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

On Fri, 7 Sep 2007, Tobias Brox wrote:

> We're also considering to install postgres on SAN - that is, my boss is
> convinced this is the right way to go.
>
> Advantages:
>
> 1. Higher I/O (at least the salesman claims so)

only if you buy better disks for the SAN then for the local system (note
that this includes battery backed ram for write caching. the SAN will
include a bunch becouse it's performance would _suck_ otherwise. if you
don't put any on your stand-alone system you are comparing apples to
oranges)

> 2. Easier to upgrade the disk capacity

only if you buy a SAN with a lot of empty drive slots, but wouldn't buy a
system with empty drive slots.

> 3. Easy to set up "warm standby" functionality. (Then again, if the
> postgres server fails miserably, it's likely to be due to a disk
> crash).

and if postgres dies for some other reason the image on disk needs repair,
unless you script stopping postgres when the SAN does it's snapshots,
those snapshots are not going to be that good. the problems are useually
repairable, but that makes starting your warm spare harder.

> Also, my boss states that "all big enterprises uses SAN nowadays".

your bos would be very surprised at what the really big shops are doing
(and not doing). yes they have a SAN, they have many SANs, from many
different vendors, and they have many systems that don't use the SAN and
use local disks instead. when you get really large you can find just about
anything _somewhere_ in the company.

> Disadvantages:
>
> 1. Risky? One gets the impression that there are frequent problems
> with data integrity when reading some of the posts in this thread.

SAN's add more parts and more potential points of failure, then when you
add the SAN replication to the mix things get even more 'interesting'.
doing SAN replication across a significant distance to your DR facility
can be a LOT harder to get working right then the salesman makes it sound.
it's not uncommon to see a san replication decide that it's going to take
a week to catch up after doing a DR test for example.

> 2. Expensive

no, _extremely expensive. price one and then look at how much hardware you
could buy instead. you can probably buy much mroe storage, and a couple
complete spare systems (do replication to a local spare as well as your
remote system) and end up with even more reliability.

> 3. "Single point of failure" ... but that you have either it's a SAN or
> a local disk, one will anyway need good backup systems (and eventually
> "warm standby"-servers running from physically separated disks).

no, with local disks you can afford to have multiple systems so that you
don't have a SPOF

> 4. More complex setup?
>
> 5. If there are several hosts with write permission towards the same
> disk, I can imagine the risks being higher for data integrity
> breakages. Particularly, I can imagine that if two postgres instances
> is started up towards the same disk (due to some sysadmin mistake), it
> could be disasterous.

when you are useing a SAN for a database the SAN vendor will have you
allocate complete disks to each box, so you don't have multiple boxes
hitting the same drive, but you also don't get a lot of the anvantages the
salesman talks about.

David Lang

---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

david@lang.hm wrote:
> On Fri, 7 Sep 2007, Tobias Brox wrote:
>
>> We're also considering to install postgres on SAN - that is, my boss is
>> convinced this is the right way to go.
>>
>> Advantages:
>>
>> 1. Higher I/O (at least the salesman claims so)
>

In general a SAN does not provide more I/O than direct attached storage.
It is all about the BUS, Controller and drive types.

> only if you buy better disks for the SAN then for the local system (note
> that this includes battery backed ram for write caching. the SAN will
> include a bunch becouse it's performance would _suck_ otherwise. if you
> don't put any on your stand-alone system you are comparing apples to
> oranges)
>
>> 2. Easier to upgrade the disk capacity
>
> only if you buy a SAN with a lot of empty drive slots, but wouldn't buy
> a system with empty drive slots.

Well there are SANs that have trays that can be stacked, but then again
you can get the same thing with DAS too.

>
>> 3. Easy to set up "warm standby" functionality. (Then again, if the
>> postgres server fails miserably, it's likely to be due to a disk
>> crash).
>
>> Also, my boss states that "all big enterprises uses SAN nowadays".
>

Uhmm as someone who consults with many of the big enterprises that are
running PostgreSQL, that is *not* true.

>> 2. Expensive
>
> no, _extremely expensive. price one and then look at how much hardware

Let me just +1 this. The amount of DAS storage you can get for 30k is
amazing compared to the amount of SAN you can get for 30k.

Joshua D. Drake

> you could buy instead. you can probably buy much mroe storage, and a
> couple complete spare systems (do replication to a local spare as well
> as your remote system) and end up with even more reliability.
>
>> 3. "Single point of failure" ... but that you have either it's a SAN or
>> a local disk, one will anyway need good backup systems (and eventually
>> "warm standby"-servers running from physically separated disks).
>
> no, with local disks you can afford to have multiple systems so that you
> don't have a SPOF
>
>> 4. More complex setup?
>>
>> 5. If there are several hosts with write permission towards the same
>> disk, I can imagine the risks being higher for data integrity
>> breakages. Particularly, I can imagine that if two postgres instances
>> is started up towards the same disk (due to some sysadmin mistake), it
>> could be disasterous.
>
> when you are useing a SAN for a database the SAN vendor will have you
> allocate complete disks to each box, so you don't have multiple boxes
> hitting the same drive, but you also don't get a lot of the anvantages
> the salesman talks about.
>
> David Lang
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: don't forget to increase your free space map settings
>


- --

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997 http://www.commandprompt.com/
UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFG4b9/ATb/zqfZUUQRAnBiAJ4kdOicN3If4scLAVdaU4nS+srGHQCgnkR2
C6RvSyLcAtgQ1bJJEau8s00=
=lqbw
-----END PGP SIGNATURE-----

---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings