I need to prepare a file system for many small files. The will be
MailDir/ type individual mail message files store by postfix. What must
I do to prepare for this type of file handling for a mail server storing
individual message files in th MailDir/ format?

Thanks

"BN" <BN@BN.COMO> wrote in message
news:11lr0mvfa18at1f@news.supernews.com...
>I need to prepare a file system for many small files. The will be
> MailDir/ type individual mail message files store by postfix. What must
> I do to prepare for this type of file handling for a mail server storing
> individual message files in th MailDir/ format?
>
> Thanks

Just make sure your not using directio, it will kill your performance with
many small files :-)

--
Rodrick R. Brown
Unix Systems Admin
http://www.rodrickbrown.com
rodrick.brown[@]gmail.com

When in 1986 Apple bought a Cray X-MP and announced that they would use it
to design the next Apple Macintosh, Seymour Cray replied, "This is very
interesting because I am using an Apple Macintosh to design the Cray-2
supercomputer."

In article <Esh7f.616$u43.549@twister.nyc.rr.com>,
"Rodrick Brown" <rodrick.brown@gmail.com> wrote:

> "BN" <BN@BN.COMO> wrote in message
> news:11lr0mvfa18at1f@news.supernews.com...
> >I need to prepare a file system for many small files. The will be
> > MailDir/ type individual mail message files store by postfix. What must
> > I do to prepare for this type of file handling for a mail server storing
> > individual message files in th MailDir/ format?
> >
> > Thanks
>
> Just make sure your not using directio, it will kill your performance with
> many small files :-)

Don't know how postfix does this, but regardless of the filesystem, many
files in a single directory is going to kill performance, although I
have no feeling for how many would cause the problem. Maybe someone
could provide metrics for this? I know I've had problems with dirty
programming practices of evil developers who've created lots of /tmp/
files (100,000+) and never clean them up. Then /tmp runs out of space
and it takes me a long time to clean them up, even with find.

If files are stored in a tree directory like the netscape cache files,
that's faster and less of a performance issue. It would be better to
redesign the application not to put all files in a single directory.
Netscape's enterprise mail program did this. Don't know if other mail
programs do. sendmail does not.

--
DeeDee, don't press that button! DeeDee! NO! Dee...

>>
>>>I need to prepare a file system for many small files. The will be
>>>MailDir/ type individual mail message files store by postfix. What must
>>>I do to prepare for this type of file handling for a mail server storing
>>>individual message files in th MailDir/ format?
>>>
>>>Thanks
When you create your FS, it might be a good idea to make more inodes
than the default. Having a file system full when you still have space
free (all the inodes consumed) is more frequent in proxy caches (really
small files) than in email systems but, ...


>>
>>Just make sure your not using directio, it will kill your performance with
>>many small files :-)

True. In fact, bumping up the inode cache in the kernel would be a good
idea.
>
>
> Don't know how postfix does this, but regardless of the filesystem, many
> files in a single directory is going to kill performance, although I
> have no feeling for how many would cause the problem. Maybe someone
> could provide metrics for this?

The ideal number is, I believe, 255 entries per directory.
Openwave, for example, creates a matrix of buckets (directories) in a
255 folders, each with 255 folders, each sized not to have more than 255
files (more levels can be added as needed).

In Tiered Storage solutions I came around that problem and, I was amazed
how well solaris worked with LOTS of files in one directory. Of course
making an ls -l would give you 45 minutes until the output started
pouring but, it still worked.
I'd say that a few thousand (2, 3, ... 10) would be the maximum workable
number.

> If files are stored in a tree directory like the netscape cache files,
> that's faster and less of a performance issue. It would be better to
> redesign the application not to put all files in a single directory.
> Netscape's enterprise mail program did this. Don't know if other mail
> programs do. sendmail does not.

The first time I came across this was, in fact, with Netscape Messaging
server (2.x I believe). The mailboxes were in buckets but, the queues
weren't and, when a spam attack came up, the mail server would slow down
until it was stopped for all practical effects

To sum it up, Solaris works pretty well no matter what you throw at it
and, I'm sure even if you don't take any precautions on bumping up the
inode number and bumping up the inode cache, it would still work.
If you're planning in putting lots of files IN ONE DIR, *then* you will
not like the performance (but I don't know any O.S. that overcomes this
problem, Unix, OS/400 (or whatever they call it now), zOS.
windows was the only O.S. that could "overcome" that problem but, then
Microsoft realised that their choices were corrupting data when the
system crashed so, now they also have that issue (and it's bound to get
worst with Vista, he he he).

In article <43600abd$0$26321$a729d347@news.telepac.pt>,
Jaime Cardoso <jaimec@solsuni.pt> wrote:
>
<SNIP>
>
>The ideal number is, I believe, 255 entries per directory.
>Openwave, for example, creates a matrix of buckets (directories) in a
>255 folders, each with 255 folders, each sized not to have more than 255
>files (more levels can be added as needed).
>
>In Tiered Storage solutions I came around that problem and, I was amazed
>how well solaris worked with LOTS of files in one directory. Of course
>making an ls -l would give you 45 minutes until the output started
>pouring but, it still worked.
>I'd say that a few thousand (2, 3, ... 10) would be the maximum workable
> number.
>
>> If files are stored in a tree directory like the netscape cache files,
>> that's faster and less of a performance issue. It would be better to
>> redesign the application not to put all files in a single directory.
>> Netscape's enterprise mail program did this. Don't know if other mail
>> programs do. sendmail does not.
>
I myself like mostly-flat directories (I grep the saved result
of a "ls -lsAt >! ls.dn" to find things, already date-sorted for me),
ditto via dired (nifty tool!) in emacs).
The two dirs I spend most time working with have 3,000 and 6,000
files (and dirs). Works for me. Perhaps wouldn't back in
sun3 days, but man this blade100 is FAST!

David