Aragorn wrote:

> again, that system has 32 GB of RAM in it, so your mileage may vary. ;-)

And I thought my 8GB were a lot! :-)

Aragorn wrote:
>>> again, that system has 32 GB of RAM in it, so your mileage may vary. ;-)
>> And I thought my 8GB were a lot! :-)
> That machine is destined to be used with multiple Gentoo instances running
> atop of the Xen hypervisor. My goal is to set up four Gentoo instances -
> the privileged virtual machine included - of which one will be a GUI
> workstation installation with KDE, OpenOffice and various other graphical
> applications, and the others being basically headless virtual servers. ;-)

I have been thinking of my own setup, not as expensive as yours and I was
starting to think about OpenVZ, as I will anyhow just be using Linux so not
needing everything that Xen has to offer. It seems that OpenVZ is kinder to
the CPU ( http://www.hpl.hp.com/techreports/2007/HPL-2007-59.pdf ).

But there is still one trouble I have, dedicating graphics cards to different
virtual environments and USB-ports. At the moment what I have thought about is
"Firewall/Gateway" which has more or less just a read only environment,
File-server to share files, Media-box (dedicated graphics, remote control),
Desktop (dedicated graphics, keyboard and mouse), Server running mail, web...

This would be more or less, everything I have today in one box, still quite
normal off the shelf hardware, but still a bit over 1000 Euro, but still
cheaper than those Indian 5000 Euro cars.


--

//Aho

J.O. Aho wrote:
> Aragorn wrote:
>>>> again, that system has 32 GB of RAM in it, so your mileage may vary.
>>>> ;-)
>>> And I thought my 8GB were a lot! :-)
>> That machine is destined to be used with multiple Gentoo instances
>> running
>> atop of the Xen hypervisor. My goal is to set up four Gentoo instances -
>> the privileged virtual machine included - of which one will be a GUI
>> workstation installation with KDE, OpenOffice and various other graphical
>> applications, and the others being basically headless virtual servers.
>> ;-)
>
> I have been thinking of my own setup, not as expensive as yours and I
> was starting to think about OpenVZ, as I will anyhow just be using Linux
> so not needing everything that Xen has to offer. It seems that OpenVZ is
> kinder to the CPU (
> http://www.hpl.hp.com/techreports/2007/HPL-2007-59.pdf ).
>
> But there is still one trouble I have, dedicating graphics cards to
> different virtual environments and USB-ports. At the moment what I have
> thought about is "Firewall/Gateway" which has more or less just a read
> only environment, File-server to share files, Media-box (dedicated
> graphics, remote control), Desktop (dedicated graphics, keyboard and
> mouse), Server running mail, web...
>
> This would be more or less, everything I have today in one box, still
> quite normal off the shelf hardware, but still a bit over 1000 Euro, but
> still cheaper than those Indian 5000 Euro cars.

Reading the man pages for the openvz control tool, it seems not be that easy
to dedicate anything else than network cards for a VE, so it seems that Xen
may be the way to go anyway...



--

//Aho

J.O. Aho wrote:

> Aragorn wrote:
>>>> again, that system has 32 GB of RAM in it, so your mileage may vary.
>>>> ;-)
>>> And I thought my 8GB were a lot! :-)
>> That machine is destined to be used with multiple Gentoo instances
>> running
>> atop of the Xen hypervisor. My goal is to set up four Gentoo instances -
>> the privileged virtual machine included - of which one will be a GUI
>> workstation installation with KDE, OpenOffice and various other graphical
>> applications, and the others being basically headless virtual servers.
>> ;-)
>
> I have been thinking of my own setup, not as expensive as yours and I was
> starting to think about OpenVZ, as I will anyhow just be using Linux so
> not needing everything that Xen has to offer. It seems that OpenVZ is
> kinder to the CPU ( http://www.hpl.hp.com/techreports/2007/HPL-2007-59.pdf
> ).

I know OpenVZ, yes, and for a while I have also contemplated using it, along
with checking out the KVM approach. :-) Gentoo has packages for OpenVZ in
Portage, by the way, although they /may/ still be masked - this I don't
know.

> But there is still one trouble I have, dedicating graphics cards to
> different virtual environments and USB-ports.

Yes, and I can help you out there - I _ think_- when it comes to *Xen,* as I
have been playing with an idea, but I have not been able to test its
validity yet.

When building the XenLinux-patched /dom0/ kernel, you have the option to use
either a virtual framebuffer in /domU/ or not, and only upon selecting that
option do you get a second option, namely to select a virtual keyboard
for /domU./

In other words, the kernel configurator is set up to have the virtual
keyboard depend on the virtual framebuffer. The idea here is to be able to
switch the console from /dom0/ to /domU/ via the /xm/ commandline utility;
you can switch back to the privileged virtual machine via the key
combination /Ctrl+]/ - you may need to press /AltGr/ for the "]" character,
depending on your locale.

Now, if you want to use graphics inside /domU/ and you want to be actually
only run X11 there - and not in X11, with a remote connection to an
application running inside /domU,/ which is what most people do but which
is not the right way as it introduces a stability hazard to your management
virtual machine by running X there - then you need to have a dedicated
video adapter for the /domU/ machine.

So far so good, but the problem there is that by default, the Linux kernel
will still be using whatever it considers to be the primary video adapter -
the one found at /dom0/ boot time - for non-graphical output. So if you
want to use the dedicated /domU/ video adapter, you are stuck to only being
able to use it in X11, which you _can_ easily set up by editing
*/etc/xorg.conf* and manually specifying the PCI address for that adapter.

This approach would then of course also mean that you'd have to have that
operating system session configured to boot to runlevel 5, with a display
manager, and that you wouldn't get any output during the character mode
phase of the /domU/ boot and shutdown, unless you're willing to switch
between monitors or monitor channels every time you switch back and forth
between character mode and X11.

Now, my as yet untested solution is the following. First make sure you have
the necessary peripherals installed. Keyboard, mouse, two videocards, each
of which is connected to a channel on your monitor(s). Next, configure
your */dom0/* XenLinux kernel with PCI passthrough, and with support for
both the virtual framebuffer and the virtual keyboard. Finish off your
configuration and save the /.config/ file.

Now, open the /.config/ file again with an editor, and comment out the
virtual framebuffer part, but leave the virtual keyboard enabled. Then
compile your /dom0/ kernel - this is the untested part, so if you break it,
you get to keep both pieces - and configure and compile your /domU/
kernel. Also, find out the PCI address of the videocard you intend to
dedicate to /domU/ and hide that device from Xen via its boot parameters,
using for instance...

pciback.hide=(1:00.06)

.... or whatever the PCI address for that card is.

If the above works as I think it would, then you should be able to switch
the console from /dom0/ to /domU/ in the usual manner, upon which you would
flick the switch - or push the button, depending on what monitor(s) you are
using - on your monitor(s) to switch channels. You should then have a
fully working console in /domU./

When compiling the evil proprietary nVidia driver in /domU/ however, you
must first export /IGNORE_XEN_PRESENCE/ as a variable inside the shell in
which you are building the driver. It will then be exported to all
subsequent shells created by the build process. Without that, the nVidia
driver will not build.

Eventhough I haven't gotten around to it yet, I believe it would also be
wisest to hide your soundcard - if any - from Xen and /dom0/ in the same
manner, via a Xen boot parameter. You would then of course not be able to
use any sound inside /dom0,/ but that is not what /dom0/ is for anyway. ;-)

Now, the above is of course for a Xen set-up. OpenVZ is very different
there in that you're not running multiple operating system kernels on top
of a hypervisor, but running multiple userspace instances on top of a
common kernel. OpenVZ does (to my knowledge) also not support X11, and
even if it did, it wouldn't be safe to do so, as you would be risking
kernel and hypervisor instability if you were to use the evil proprietary
driver.

With the Xen approach as I have laid out above, if the evil driver hangs the
kernel or even messes up the hardware framebuffer in such a way that you
need a reboot to recover from it, then the damage would still be limited to
that particular /domU/ virtual machine anyway, and your /dom0/ and
hypervisor would be safe, and as such, so would your other /domU/ virtual
machines be. So all of your individual server set-ups would continue to be
available while you are rebooting your GUI workstation virtual machine.

This is why it is wrong to run X11 in /dom0,/ unless /dom0/ is your
workstation installation and is not to be considered mission-critical.
Such a set-up would be good for using /domU/ as a testbed or sandbox only.
My intent however is to have all /domU/ virtual machines be
mission-critical, with the graphical virtual machine preferably being as
stable as possible but acknowledging that it holds a risk due to having to
use the evil proprietary nVidia driver for 3D graphics. If you don't need
3D acceleration, then you can use the more reliable (and less cumbersome to
set up) FOSS "nv" driver.

> At the moment what I have thought about is "Firewall/Gateway" which has
> more or less just a read only environment, File-server to share files,
> Media-box (dedicated graphics, remote control), Desktop (dedicated
> graphics, keyboard and mouse), Server running mail, web...

If you need dedicated graphics in more than one unprivileged virtual
machine, then you'll need to add an additional video adapter card for each
of them. You can't share one videocard among multiple virtual machines,
except via a virtual framebuffer. Likewise, your remote control apparatus
will have to be hidden from /dom0/ and must only be accessed by one
unprivileged virtual machine.

But then again still, this would not be feasible with OpenVZ. You'd need to
use Xen for such a set-up. I would also set up the firewalling, NAT and
gateway thing in /dom0,/ since that is the virtual machine that has access
to the physical NICs in your machine. You should then also set up the Xen
networking configuration to make use of routing instead of the default
choice of bridging.

If you want to use ethernet bridging set-up that Xen defaults to, then each
of your virtual machines must have a dedicated IP address of its own, and
this might be tricky with most ISPs. Mine for instance - Telenet here in
Belgium - gives me only two public IP addresses (via semi-static DHCP) for
my current contract, and when my machine will be fully set up, I intend to
switch to a contract that gives me a guaranteed static IP address, but then
I'll only have one. If I want anything more, then the next step is 8
static IP addresses over ADSL, but that would also require me to rent a
landline or a raw copper from Belgacom, as I'm on cable internet right now
and this 8 IP addresses deal is not possible over cable.

So if you do want bridging and your ISP only gives you one public IP
address, then you should get a router and set that one up as the default
gateway and firewall, and then you can have the router assign as many IP
addresses to your Xen box as you like.

> This would be more or less, everything I have today in one box, still
> quite normal off the shelf hardware, but still a bit over 1000 Euro, but
> still cheaper than those Indian 5000 Euro cars.

You mean that backpack on wheels from Tata Motors? Wasn't that 2500
Euro, by the way? ;-)

--
Aragorn
(registered GNU/Linux user #223157)

Aragorn wrote:

> [...]
>
> Now, if you want to use graphics inside /domU/ and you want to be actually
> only run X11 there - and not in X11, with a remote connection to an
^^^
That should read "...and not in /dom0..."/.

> application running inside /domU,/ which is what most people do but which
> is not the right way as it introduces a stability hazard to your
> management virtual machine by running X there - then you need to have a
> dedicated video adapter for the /domU/ machine.
>
> [...]
>
> When compiling the evil proprietary nVidia driver in /domU/ however, you
> must first export /IGNORE_XEN_PRESENCE/ as a variable inside the shell in
> which you are building the driver. It will then be exported to all
> subsequent shells created by the build process. Without that, the nVidia
> driver will not build.

The /IGNORE_XEN_PRESENCE/ variable should have a value other than "0".
So, ...

export IGNORE_XEN_PRESENCE=1

.... should work. If it's set to "0", then the driver won't compile. I got
that off the nVidia forums. ;-)

--
Aragorn
(registered GNU/Linux user #223157)

Aragorn wrote:
> [...]
> The hardware for that machine - Tyan twin-socket ccNUMA board, twin dualcore
> HE Opterons (2.6 GHz, 68 Watt), 32 GB ECC registered RAM (ATP pc5300), four
> Hitachi 15k SAS 147 GB disks in RAID 5 on an Adaptec 32105 PCIe adapter,
> Zippy 800 Watt EPS12V power supply and a CoolerMaster CM-832 chassis with
> four 12cm fans - has taken a serious bite out of my piggybank as well...

If it doesn't run Quake 4 with 200FPS it's not worth the money :P

Nikos Chantziaras wrote:

> Aragorn wrote:
>> [...]
>> The hardware for that machine - Tyan twin-socket ccNUMA board, twin
>> dualcore HE Opterons (2.6 GHz, 68 Watt), 32 GB ECC registered RAM (ATP
>> pc5300), four Hitachi 15k SAS 147 GB disks in RAID 5 on an Adaptec 32105
>> PCIe adapter, Zippy 800 Watt EPS12V power supply and a CoolerMaster
>> CM-832 chassis with four 12cm fans - has taken a serious bite out of my
>> piggybank as well...
>
> If it doesn't run Quake 4 with 200FPS it's not worth the money :P

That would largely depend on the video adapter and on the proprietary, evil
nVidia driver. ;-)

On the other hand, I do think that something that compiles a complete Linux
kernel in only just over one minute is worth its money, though...

--
Aragorn
(registered GNU/Linux user #223157)

Aragorn <aragorn@chatfactory.invalid> wrote:
> Darin McBride wrote:
>>
>> While it's safe, I wouldn't clear out distfiles. /var/tmp/portage,
>> however, is safe as long as you don't have a current emerge running
>> ;-)
>
> If you (and/or the OP) have a sufficient amount of RAM in your
> system, you could then make */var/tmp/portage* a /tmpfs./ It's what
> I do, but then again, that system has 32 GB of RAM in it, so your
> mileage may vary. ;-)

A minor problem with that is that /var/tmp/portage is also the home
directory for the portage user account, and some other persistent stuff
might be saved in it (like a .forward for mail or .distcc for distributed
compiling). Having user accounts pointing to temporary directories is
usually a bad idea, and I wish gentoo had used a different directory for
this.

Regards,
--
*Art

Aragorn wrote:

> I know OpenVZ, yes, and for a while I have also contemplated using it, along
> with checking out the KVM approach. :-) Gentoo has packages for OpenVZ in
> Portage, by the way, although they /may/ still be masked - this I don't
> know.

I haven't looked that far, but it seems to be unmasked according the Gentoo
Howto at the wiki, but this far dedicating hardware don't work more than for
NICs as far as I managed to figure things out, which means that OpenVZ won't
be an option for me.


> Now, if you want to use graphics inside /domU/ and you want to be actually
> only run X11 there - and not in X11, with a remote connection to an
> application running inside /domU,/ which is what most people do but which
> is not the right way as it introduces a stability hazard to your management
> virtual machine by running X there - then you need to have a dedicated
> video adapter for the /domU/ machine.
> So far so good, but the problem there is that by default, the Linux kernel
> will still be using whatever it considers to be the primary video adapter -
> the one found at /dom0/ boot time - for non-graphical output. So if you
> want to use the dedicated /domU/ video adapter, you are stuck to only being
> able to use it in X11, which you _can_ easily set up by editing
> */etc/xorg.conf* and manually specifying the PCI address for that adapter.

I have been thinking of using 3 graphics cards, two PCIe and one PCI, where I
was hoping to be able to use the PCI card as the main graphics adapter, as
console output don't require any monster graphics cards.
While the desktop and the media-box instances I was hoping to give them a PCIe
graphics card each, both should be able to run at the same time.


> This approach would then of course also mean that you'd have to have that
> operating system session configured to boot to runlevel 5, with a display
> manager, and that you wouldn't get any output during the character mode
> phase of the /domU/ boot and shutdown, unless you're willing to switch
> between monitors or monitor channels every time you switch back and forth
> between character mode and X11.

Yes, but in gentoo you default have initlevel 3 for graphical environment, as
people seems to only use "default", I do want to have things setup like in
RedHat where level 3 is multiuser mode with VT and level 5 is Xorg. Myself I
added graphical, no network as level 4 (reserved for future use in RedHat), so
I have an equivalent to level 2.


> With the Xen approach as I have laid out above, if the evil driver hangs the
> kernel or even messes up the hardware framebuffer in such a way that you
> need a reboot to recover from it, then the damage would still be limited to
> that particular /domU/ virtual machine anyway, and your /dom0/ and
> hypervisor would be safe, and as such, so would your other /domU/ virtual
> machines be. So all of your individual server set-ups would continue to be
> available while you are rebooting your GUI workstation virtual machine.

*nods*
That is one of the strong benefits with Xen over the others, but OpenVZ has
easier resource management system, where you can make changes on the fly. As I
have understood Xen, you have to restart the instance before it gets the new
resources.


> This is why it is wrong to run X11 in /dom0,/ unless /dom0/ is your
> workstation installation and is not to be considered mission-critical.
> Such a set-up would be good for using /domU/ as a testbed or sandbox only.
> My intent however is to have all /domU/ virtual machines be
> mission-critical, with the graphical virtual machine preferably being as
> stable as possible but acknowledging that it holds a risk due to having to
> use the evil proprietary nVidia driver for 3D graphics. If you don't need
> 3D acceleration, then you can use the more reliable (and less cumbersome to
> set up) FOSS "nv" driver.

No, I don't want a graphical environment for dom0, I want it to be as clean as
possible, the only thing I will run on it would be a NFS share, as I'm not
that happy about the file I/O in a domU and thinking about some kind of tftp
boots for the domUs.


>> At the moment what I have thought about is "Firewall/Gateway" which has
>> more or less just a read only environment, File-server to share files,
>> Media-box (dedicated graphics, remote control), Desktop (dedicated
>> graphics, keyboard and mouse), Server running mail, web...
>
> If you need dedicated graphics in more than one unprivileged virtual
> machine, then you'll need to add an additional video adapter card for each
> of them. You can't share one videocard among multiple virtual machines,
> except via a virtual framebuffer. Likewise, your remote control apparatus
> will have to be hidden from /dom0/ and must only be accessed by one
> unprivileged virtual machine.

It had been nice if Xen had the same way to deal with hardware as OpenVZ have
for their network devices and you can do the change in realtime with no need
to reboot.


> But then again still, this would not be feasible with OpenVZ. You'd need to
> use Xen for such a set-up. I would also set up the firewalling, NAT and
> gateway thing in /dom0,/ since that is the virtual machine that has access
> to the physical NICs in your machine. You should then also set up the Xen
> networking configuration to make use of routing instead of the default
> choice of bridging.

Was thinking about if it would be possible to dedicate a nic in a similar way
to a domU as a graphics card. IMHO it feels better if someone "hack" a domU
with only read only access to files, than getting access to dom0, and if
something would get messy then just restart the domU instead of a reboot of
the whole system.


> If you want to use ethernet bridging set-up that Xen defaults to, then each
> of your virtual machines must have a dedicated IP address of its own, and
> this might be tricky with most ISPs. Mine for instance - Telenet here in
> Belgium - gives me only two public IP addresses (via semi-static DHCP) for
> my current contract, and when my machine will be fully set up, I intend to
> switch to a contract that gives me a guaranteed static IP address, but then
> I'll only have one.

My ISP allows just one public IP (thanks to someone smart person at that
company, it's a static one), but the majority os ISPs over here offers only
DHCP and with one dynamic ip-address.
I don't want any direct access to the other domUs from the outside, so
bridging ain't the option for me.


> So if you do want bridging and your ISP only gives you one public IP
> address, then you should get a router and set that one up as the default
> gateway and firewall, and then you can have the router assign as many IP
> addresses to your Xen box as you like.

I have a route from my ISP, but I have disabled the routing and firewall, as I
feel I have more control over thing when I run it on my computer, which won't
loose logfiles just doe a blackout or restart.


>> This would be more or less, everything I have today in one box, still
>> quite normal off the shelf hardware, but still a bit over 1000 Euro, but
>> still cheaper than those Indian 5000 Euro cars.
> You mean that backpack on wheels from Tata Motors? Wasn't that 2500
> Euro, by the way? ;-)

Yes, that one. It may have dropped in price, but still the hardware solution I
have looked at is cheaper than the car.


--

//Aho

Aragorn wrote:
> Nikos Chantziaras wrote:
>
>> Aragorn wrote:
>>> [...]
>>> The hardware for that machine - Tyan twin-socket ccNUMA board, twin
>>> dualcore HE Opterons (2.6 GHz, 68 Watt), 32 GB ECC registered RAM (ATP
>>> pc5300), four Hitachi 15k SAS 147 GB disks in RAID 5 on an Adaptec 32105
>>> PCIe adapter, Zippy 800 Watt EPS12V power supply and a CoolerMaster
>>> CM-832 chassis with four 12cm fans - has taken a serious bite out of my
>>> piggybank as well...
>> If it doesn't run Quake 4 with 200FPS it's not worth the money :P
>
> That would largely depend on the video adapter and on the proprietary, evil
> nVidia driver. ;-)
>
> On the other hand, I do think that something that compiles a complete Linux
> kernel in only just over one minute is worth its money, though...
>

I would say it's cool to have, but not worth all the money, I can wait 5
minutes and afford to go on vacation.

--

//Aho