I'm building an e-mail service that has two requirements: It should
index messages on the fly to have lightening search results, and it
should be able to handle large amounts of space. The server is going
to be dedicated only for e-mail with 250GB of storage in Raid-5. I'd
like to know how PostgreSQL could handle such a large amount of data.
How much RAM would I need? I expect my users to have a 10GB quota per
e-mail account.
Thanks for your advice,

--
Charles A. Landemaine.

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

I'm building an e-mail service that has two requirements: It should
index messages on the fly to have lightening search results, and it
should be able to handle large amounts of space. The server is going
to be dedicated only for e-mail with 250GB of storage in Raid-5. I'd
like to know how PostgreSQL could handle such a large amount of data.
How much RAM would I need? I expect my users to have a 10GB quota per
e-mail account.
Thanks for your advice,

--
Charles A. Landemaine.

---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

On Thu, 2007-01-04 at 15:00 -0300, Charles A. Landemaine wrote:
> I'm building an e-mail service that has two requirements: It should
> index messages on the fly to have lightening search results, and it
> should be able to handle large amounts of space. The server is going
> to be dedicated only for e-mail with 250GB of storage in Raid-5.

Well Raid 5 is likely a mistake. Consider RAID 10.


> I'd
> like to know how PostgreSQL could handle such a large amount of data.

250GB is not really that much data for PostgreSQL I have customers with
much larger data sets.


> How much RAM would I need?

Lots... which is about all I can tell you without more information. How
many customers? Are you using table partitioning? How will you be
searching? Full text or regex?

Joshua D. Drake


> I expect my users to have a 10GB quota per
> e-mail account.
> Thanks for your advice,
>
--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate




---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?

http://archives.postgresql.org

On Thu, 4 Jan 2007 15:00:05 -0300
"Charles A. Landemaine" <landemaine@gmail.com> wrote:

> I'm building an e-mail service that has two requirements: It should
> index messages on the fly to have lightening search results, and it
> should be able to handle large amounts of space. The server is going
> to be dedicated only for e-mail with 250GB of storage in Raid-5. I'd
> like to know how PostgreSQL could handle such a large amount of data.
> How much RAM would I need? I expect my users to have a 10GB quota per
> e-mail account.

Well this is a bit like asking "what's the top speed for a van that
can carry 8 people", it really isn't enough information to be able
to give you a good answer.

It depends on everything from the data model you use to represent
your E-mail messages, to configuration, to hardware speeds, etc.

In general, you want as much RAM as you can afford for the project,
the more the better. I'd say 2-4GB is the minimum. And RAID-5 isn't
very good for database work in general, you'll get better performance
from RAID 1+0.

---------------------------------
Frank Wiles <frank@wiles.org>
http://www.wiles.org
---------------------------------


---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings

Frank Wiles wrote:
> On Thu, 4 Jan 2007 15:00:05 -0300
> "Charles A. Landemaine" <landemaine@gmail.com> wrote:
>
>> I'm building an e-mail service that has two requirements: It should
>> index messages on the fly to have lightening search results, and it
>> should be able to handle large amounts of space. The server is going
>> to be dedicated only for e-mail with 250GB of storage in Raid-5. I'd
>> like to know how PostgreSQL could handle such a large amount of data.
>> How much RAM would I need? I expect my users to have a 10GB quota per
>> e-mail account.
>

I wouldn't do it this way, I would use cyrus. It stores the messages in
plain text each in it's own file (like maildir) but then it also indexes
the headers in bdb format and also can index the search database in bdb
format. The result is a very simple mail store that can perform
searches very fast. The only problem is that the search index in only
updated periodically.

If you need more information then look at the cyrus-users list as this
is WAY off topic.

schu

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

Charles A. Landemaine wrote:
| I'm building an e-mail service that has two requirements: It should
| index messages on the fly to have lightening search results, and it
| should be able to handle large amounts of space. The server is going
| to be dedicated only for e-mail with 250GB of storage in Raid-5. I'd
| like to know how PostgreSQL could handle such a large amount of data.
| How much RAM would I need? I expect my users to have a 10GB quota per
| e-mail account.
| Thanks for your advice,
|

Hello, Charles.

I'll second people's suggestions to stay away from RAID5; the kind of
workload a mail storage will have is one that is approximately an even mix
of writes (in database terms, INSERTs, UPDATEs and DELETEs) and reads, and
we all know RAID5 is a loser when it comes to writing a lot, at least when
you're building arrays with less than 10-15 drives. I'd suggest you go for
RAID10 for the database cluster and an extra drive for WAL.

Another point of interest I'd like to mention is one particular aspect of
the workflow of an e-mail user: we will typically touch the main inbox a lot
and leave most of the other folders pretty much intact for most of the time.
This suggests per-inbox quota might be useful, maybe in addition to the
overall quota, because then you can calculate your database working set more
easily, based on usage statistics for a typical account. Namely, if the
maximum size of an inbox is x MB, with y% average utilization, and you plan
for z users, of which w% will be typically active in one day, your database
working set will be somewhere in the general area of (x * y%) * (z * w%) MB.
Add to that the size of the indexes you create, and you have a very
approximate idea of the amount of RAM you need to place in your machines to
keep your performance from becoming I/O-bound.

The main reason I'm writing this mail though, is to suggest you take a look
at Oryx, http://www.oryx.com/; They used to have this product called
Mailstore, which was designed to be a mail store using PostgreSQL as a
backend, and has since evolved to a bit more than just that, it seems.
Perhaps it could be of help to you while building your system, and I'm sure
the people at Oryx will be glad to hear from you while, and after you've
built your system.

Kind regards,
- --
~ Grega Bremec
~ gregab at p0f dot net
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)

iD8DBQFFncGcfu4IwuB3+XoRA9Y9AJ0WA+0aooVvGMOpQXGStz kRNVDCjwCeNdfs
CArTFwo6geR1oRBFDzFRY/U=
=Y1Lf
-----END PGP SIGNATURE-----

---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings

On Fri, 2007-01-05 at 04:10 +0100, Grega Bremec wrote:
> he main reason I'm writing this mail though, is to suggest you take a
> look
> at Oryx, http://www.oryx.com/; They used to have this product called
> Mailstore, which was designed to be a mail store using PostgreSQL as a
> backend, and has since evolved to a bit more than just that, it seems.
> Perhaps it could be of help to you while building your system, and I'm
> sure
> the people at Oryx will be glad to hear from you while, and after
> you've
> built your system.
>
> Kind regards,
> --
> ~ Grega Bremec
re above...
http://www.archiveopteryx.org/1.10.html

---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend

On Fri, Jan 05, 2007 at 01:15:44PM -0500, Reid Thompson wrote:
> On Fri, 2007-01-05 at 04:10 +0100, Grega Bremec wrote:
> > he main reason I'm writing this mail though, is to suggest you take a
> > look
> > at Oryx, http://www.oryx.com/; They used to have this product called
> > Mailstore, which was designed to be a mail store using PostgreSQL as a
> > backend, and has since evolved to a bit more than just that, it seems.
> > Perhaps it could be of help to you while building your system, and I'm
> > sure
> > the people at Oryx will be glad to hear from you while, and after
> > you've
> > built your system.
> >
> > Kind regards,
> > --
> > ~ Grega Bremec
> re above...
> http://www.archiveopteryx.org/1.10.html

You should also look at http://dbmail.org/ , which runs on several
databases (PostgreSQL included).
--
Jim Nasby jim@nasby.net
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster