SEO

Our system security requirements are such that we have to execute
entirely from RAM, so when the system is powered off, we return to a
known state. The question is - Is running Solaris from a tmpfs mounted
file system a reasonable thing to do, and if so, what sort of
performance should we expect - better or worse than a spinning disk.
If it is not reasonable, can anyone suggest an alternative.

Currently we are seeing very poor nfsd performance in particular. This
system acts primarily as a file server to a number of other VME
chassis', and the transfer of files to other chassis is no where near
meeting our performance requirements.

Our configuration is:

Hardware: VME based Themis UltraSparc IIe with 4GB RAM
Software: Solaris 9 4/03
Disk: 9GB M-Systems 3.5" Ultra Wide SCSI Flash - Write Protected

Basically we install and configure the OS as required then write
protect the flash disk. In addition to jumpers on the disk, the
following line from /etc/rcS.d/S40standardmounts.sh is commented out:
# /sbin/mount -m -o $mntops $mountp

Next we create the script /etc/rcS.d/S00WRITEprotected.sh so at boot
time we copy what we think we need from / to tmpfs then mount using
lofs onto /. The rest of / is mounted read-only. S40standardmounts.sh
still takes care of /proc, /etc/mnttab, and /dev/fd.

The S00WRITEprotected.sh script is as follows:

# Begin
mount -n /tmp

tar cpf - var | (cd /tmp; tar xpf -)
mount -n -F lofs /tmp/var /var

tar cpf - etc | (cd /tmp; tar xpf -)
mount -n -F lofs /tmp/etc /etc

tar cpf - dev | (cd /tmp; tar xpf -)
mount -n -F lofs /tmp/dev /dev

tar cpf - devices | (cd /tmp; tar xpf -) mknod
/tmp/devices/pci@1f,0/pci@2/scsi@4/sd@0,0:a b 32 24 mount -n -F lofs
/tmp/devices /devices

# Our directory for application data which NFS clients download from
tar cpf - foo | (cd /tmp; tar xpf -) mount -n -F lofs /tmp/foo /foo

cp /.rhosts /tmp/.rhosts
mount -n -F lofs /tmp/.rhosts /.rhosts # End

The /etc/vfstab file has entries for: /dev/fd, /proc, /tmp on tmpfs,
and /.

When fully populated with our operational software, we have approx 2GB
RAM free.

Some general findings in regard to performance:

- The less available memory on tmpfs, the poorer the performance.
- During NFS client access to data on tmpfs, such as performing an ls
-lRt, almost all the CPU time is spent on the system kernel;
specifically, the nfsd process.

We are in the process of applying all the recommended Solaris 9 patches
as of 08/01 to determine if any NFS bug fixes improve performance.

Does anyone have any suggestions as to how to improve performance for
this configuration, or alternate configurations, given we must execute
entirely from RAM.

Thanks in advance!

thomas.wessell@ngc.com writes:

>Our system security requirements are such that we have to execute
>entirely from RAM, so when the system is powered off, we return to a
>known state. The question is - Is running Solaris from a tmpfs mounted
>file system a reasonable thing to do, and if so, what sort of
>performance should we expect - better or worse than a spinning disk.
>If it is not reasonable, can anyone suggest an alternative.

How does executing from tmpfs help with this?

Doesn't executing from read-only disks work just as well?

>Currently we are seeing very poor nfsd performance in particular. This
>system acts primarily as a file server to a number of other VME
>chassis', and the transfer of files to other chassis is no where near
>meeting our performance requirements.

>Our configuration is:

>Hardware: VME based Themis UltraSparc IIe with 4GB RAM
>Software: Solaris 9 4/03
>Disk: 9GB M-Systems 3.5" Ultra Wide SCSI Flash - Write Protected

So what do you copy to tmpfs and how much of tmpfs is then used?

>Basically we install and configure the OS as required then write
>protect the flash disk. In addition to jumpers on the disk, the
>following line from /etc/rcS.d/S40standardmounts.sh is commented out:
> # /sbin/mount -m -o $mntops $mountp

>Next we create the script /etc/rcS.d/S00WRITEprotected.sh so at boot
>time we copy what we think we need from / to tmpfs then mount using
>lofs onto /. The rest of / is mounted read-only. S40standardmounts.sh
>still takes care of /proc, /etc/mnttab, and /dev/fd.

Ah, I see.

>When fully populated with our operational software, we have approx 2GB
>RAM free.

>Some general findings in regard to performance:

>- The less available memory on tmpfs, the poorer the performance.

No surprising. It sounds like you do not have enough memory.

Have you through about adding a swap device? This may sound strange
but since swap data is not examined on reboot a swap device is
similar to "throw away" memory. Swap will also catch the slack
of "reserved" swap which now means that you have physical memory which
is reserved but cannot be used.

> - During NFS client access to data on tmpfs, such as performing an ls
>-lRt, almost all the CPU time is spent on the system kernel;
>specifically, the nfsd process.

Probably in tmpfs code? Have you used lockstat to profile where the kernel
spends its time?

>Does anyone have any suggestions as to how to improve performance for
>this configuration, or alternate configurations, given we must execute
>entirely from RAM.

I find the "must execute from RAM" requirement rather strange;
it seems that your actual requirement is "must execute the same
thing always" and read-only media seems to serve that purpose
just as well, does it not?

Casper
--
Expressed in this posting are my opinions. They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

Thanks for the reply.

A little more background on our system requirements. This is a system
being developed for the government. The read-only flash disk contains
only unclassisifed software, like Solaris, Mercury OS, and some
commercial tools. Some of our operational software is NOT
unclassified; our software is encrypted onto a separate disk through an
encryption device prior to system startup. Once the system is powered
on, our software is decrypted (through an encryption device), and
written to tmpfs on this Solaris board. The driving requirement for
this whole configuration is that when power is turned off, the system
returns to an unclassified state. This means there can be no
non-volatile memory device used during system operation - which seems
would rule out a separate swap device. The board we're using is maxed
out at 4GB of RAM.

We have been using vmstat and prstat so far, and plan to use truss on
the nfsd process to try to determine where it is spending its time. We
will look into using lockstat as well. I don't know if there are
kernel, nfs, or tmpfs tuning parameters that we could tweak to improve
performance.

As soon as we enable writing to the disk, and have a swap partition
defined, the performance is pretty normal - which just can't operate in
this mode.

Thanks again.

On 2006-08-03, thomas.wessell@ngc.com <thomas.wessell@ngc.com> wrote:

[snip creative solution]

A few manufacturers produce storage that is based on RAM and behaves
like a spindle, with a backup battery to retain data when power goes
out. If one could manage this backup battery (remove it?), you'd end
up with a read-write device which could be mounted as normal UFS which
would clear out like ordinary RAM when de-powered.

I read an article on this about two years ago. Apparently the NSA was
using it as an ultra-fast SAN storage solution. It was expensive, but
when you're working on classified projects I guess budget limits are
a bit more flexible.

Just an idea - you could maybe get a module with 9Gb of storage and
used that as a swap device if you're looking to run with your current
solution and perhaps work with the vendor to ensure there is no power
when you turn it off.

--
Andre.