Hi,

I am trying to design a Linux Ha (high availability) cluster for my lamp
platform. Does anyone have any tips?

I have the following workload.
>One commercial website doing a pretty hefty 12 million heavy duty
counted banner views. Nxt to this it handles up to 20000 page views on
medium sized html/php pages.

>A new website which is even heavier, i think it will do a few million
hits each day, with a far heavier script.

I have the following hardware:
>Hp lpr450, 2u dual p2 450, 512 MB , 2x 4.1GB scsi 7.2k rpm, Win2k
server
>>This runs the first site (web and db) and has a cpu load on both
cpu's of around 40%.

>Compaq dl360, 1u dual p3 1266, 512 MB, 2x 9.1 GB scsi 10k rpm, Gentoo
(2.4 kernel)
>>This doesn't do anything yet, i think of making it my mysql server,
relaxing the above LPR.
>>However this sever crashed within the first week having it under a
bit of stress, might be any on the following, apache/php or mysql.
>>Compiling never gave me any problems, so the cpus are fine, as you
can see im hesitating to create a dedicated mysql server out of a server
which crsahed on me, while the current setup is working rock solid
without this machine.

>Home build server, 2u Dual p2 400, 512MB, 4x 18gb scsi 10k rpm, gentoo
(2.6 kernel).
>>This will become the file server, the dual p2 400 should be fast
enough to pump out the requested files. and the disks have more than
enough space to hold each of the other servers /var/www folder.
>>This server could also pull in backups from the log files of each
server and a backup of database from the mysql server.
>>How should i do the backups from mysql? should i run a script locking
the tables and copying the files, or should i make this server a mysql
slave, which automatically makes sure its up to date allll the time.
(but may heavier on the cpu's)

All servers have ecc reg mem.

Since i cannot afford new hardware yet, the new project will need it,
but i can't afford a nice new sun dual opteron yet, im stuck with the
current hardware.

Im thinking of making a dedicated mysql server from the dl360, since it
has fast disks, can be upgraden with extra MEM, and has the cpu power to
deal with loads of queries (up to 4 times as much as the dual 450, since
that also does webserving. bechmarks show me that there's almost no
delay in the speed php gets its results from mysql when i run them over
the network.

The network is build up out of two Baystack 450-24t switches, both have
2 vlans, one for internal communication, and one for internet traffic,
the seccond switch is a failover switch for the first, using spanning tree.

Does anyone have any idea's on how to do this the best way?
I am thinking of using sort of the same setup as wikipedia uses,

One or maybe two mysql servers for queries, (and maybe a server which
does bacups trough mysql slave replication)
A larger amount of apache/lighttpd webservers which connect to the mysql
server trough the 100mb lan.
Maybe 2 squid proxy's load balancing the apache server.
A file server. which gathers all of the backups, and hold all of the
php/html/image files for the webservers.

Since i do not yet have all of the required machines i might be
combining some functions on a single machine.

The squid proxy's should be configured in a failover config, making sure
each squid can take over the workload of the other if the primary goes down.
If a webserver crashes it will simply be removed from the squid list, so
it won't get any request anymore. The squids won't do a lot of caching
since i need to write almost all hits to the dbase.

The db servers would be configged as a master/slave pair, making sure
there's allways at least one server to query to. Php could check this
and make sure the apache's connect to right correct db server, load
balancing is not needed yet, its just done for failover.
Maybe there's a problem when the master mysql goes down, and the clients
start writing to the slave, that the master needs to get updated from
the slave server when it gets back online.

Any tips? :P

Il try again, but now with a bit of markup.
Hi,

I am trying to design a Linux Ha (high availability) cluster for my lamp
platform. Does anyone have any tips?

I have the following workload:

One commercial website.
12 million heavy duty counted banner views per month.
20000 page views on medium sized html/php pages per day.

A new website which is even heavier,
1/2 million hits each day, with a far heavier script.

I have the following hardware:
Hp lpr450,
2u dual p2 450, 512 MB ,
2x 4.1GB scsi 7.2k rpm,
Win2k server
This runs the first site (web and db)
Has a cpu load on both cpu's of around 40%.

Compaq dl360,
1u dual p3 1266, 512 MB,
2x 9.1 GB scsi 10k rpm,
Gentoo (2.4 kernel)

This doesn't do anything yet,

i think of making it my mysql server, relaxing the lpr.
However this sever crashed within the first week having it under a bit
of stress,
Might be any on the following, apache/php or mysql.
Compiling never gave me any problems,
So the cpus are fine,
As you can see im hesitating to create a dedicated mysql server out of
a server which crashed on me, while the current setup is working rock
solid without this machine.

Home build server,
2u Dual p2 400, 512MB,
4x 18gb scsi 10k rpm,
gentoo (2.6 kernel).

This will become the file server,
Should be fast enough to pump out the requested files.
and the disks have more than enough space
Should hold each of the other servers /var/www folder.
This server could also pull in backups
From the log files of each server
And a backup of database from the mysql server.

How should i do the backups from mysql?
Should i run a script locking the tables and copying the files,
Or should i make this server a mysql slave,
Which automatically makes sure its up to date all the time. (but may
heavier on the cpu's)

All servers have ecc reg mem.

Since i cannot afford new hardware yet im stuck with the current
hardware. The new project will need it, but i can't afford a nice new
sun dual opteron yet, I might be combining some functions on a single
machine.


Im thinking of making a dedicated mysql server from the dl360, since it
has fast disks, can be upgraden with extra MEM, and has the cpu power to
deal with loads of queries (up to 4 times as much as the dual 450, since
that also does webserving. bechmarks show me that there's almost no
delay in the speed php gets its results from mysql when i run them over
the network.

Does anyone have any idea's on how to do this the best way?
I am thinking of using sort of the same setup as wikipedia uses,

One or maybe two mysql servers for queries, (and maybe a server which
does bacups trough mysql slave replication)
A larger amount of apache/lighttpd webservers which connect to the mysql
server trough the 100mb lan.
Maybe 2 squid proxy's load balancing the apache server.
A file server. which gathers all of the backups, and hold all of the
php/html/image files for the webservers.


The squid proxy's should be configured in a failover config, making sure
each squid can take over the workload of the other if the primary goes down.
If a webserver crashes it will simply be removed from the squid list, so
it won't get any request anymore. The squids won't do a lot of caching
since i need to write almost all hits to the dbase.

The db servers would be configged as a master/slave pair, making sure
there's allways at least one server to query to. Php could check this
and make sure the apache's connect to right correct db server, load
balancing is not needed yet, its just done for failover.
Maybe there's a problem when the master mysql goes down, and the clients
start writing to the slave, that the master needs to get updated from
the slave server when it gets back online.

Any tips? :P

Jan Klopper enlightened us with:
> I have the following hardware:
> Hp lpr450,
> 2u dual p2 450, 512 MB ,
> 2x 4.1GB scsi 7.2k rpm,
> Win2k server

Drop it, replace it with Gentoo.

> Compaq dl360,
> 1u dual p3 1266, 512 MB,
> 2x 9.1 GB scsi 10k rpm,
> Gentoo (2.4 kernel)

I suggest a 2.6 kernel instead.

> Compiling never gave me any problems, So the cpus are fine,

Never assume something is fine solely based on the fact that it does
things the right way. Make sure you also test the memory - memtest86
is a good one.

> How should i do the backups from mysql?

Plenty of documentation on the MySQL site. I suggest you use a binary
log file and replicate that to a slave MySQL server. Then you can take
the slave down and make a proper backup, without affecting the live
server.

> Maybe 2 squid proxy's load balancing the apache server.

That sounds like a plan. Proxying static data is a lot easier on the
CPU than parsing & running complex PHP scripts.

> The squids won't do a lot of caching since i need to write almost
> all hits to the dbase.

See if you can change that. Why do you need to write hits to a
database? If it's for traffic logging, you can also use Squid's log
file and analyze that.

Sybren
--
The problem with the world is stupidity. Not saying there should be a
capital punishment for stupidity, but why don't we just take the
safety labels off of everything and let the problem solve itself?

Sybren Stuvel wrote:
> Jan Klopper enlightened us with:
>
>>I have the following hardware:
>> Hp lpr450,
>> 2u dual p2 450, 512 MB ,
>> 2x 4.1GB scsi 7.2k rpm,
>> Win2k server
>
>
> Drop it, replace it with Gentoo.
>

Was planning on doing so, but i can't just keep it offlie for a couple
of hours (and maybe more if the gentoo fails), i was planning on
building the cluster, and then migrating the services.
As soon as everything is switched over reinstall the machine with gentoo
and add it to the cluster.
>
>> Compaq dl360,
>> 1u dual p3 1266, 512 MB,
>> 2x 9.1 GB scsi 10k rpm,
>> Gentoo (2.4 kernel)
>
>
> I suggest a 2.6 kernel instead.
>
I know, but compiling a kernel and lilo/grub is a complete nightmnare on
these machines.
>
>>Compiling never gave me any problems, So the cpus are fine,
>
>
> Never assume something is fine solely based on the fact that it does
> things the right way. Make sure you also test the memory - memtest86
> is a good one.
>
Hmz,, il do so. (back to the datacenter :P )
>
>>How should i do the backups from mysql?
>
>
> Plenty of documentation on the MySQL site. I suggest you use a binary
> log file and replicate that to a slave MySQL server. Then you can take
> the slave down and make a proper backup, without affecting the live
> server.
>
Read about this, i will probably let the file server do this.
>
>>Maybe 2 squid proxy's load balancing the apache server.
>
>
> That sounds like a plan. Proxying static data is a lot easier on the
> CPU than parsing & running complex PHP scripts.
>
Static data will indeed be cached by the squids.
>
>>The squids won't do a lot of caching since i need to write almost
>>all hits to the dbase.
>
>
> See if you can change that. Why do you need to write hits to a
> database? If it's for traffic logging, you can also use Squid's log
> file and analyze that.
>
Nope, i don't think i have a lot of progress with changing the logging
to the squid login and parsing them, i do actually need al that
processing, and the scripts are pretty optimized and do cache internally.

The squids will just be there to allow loadbalancing, if one of the
squids dies, the other should take over the work from other.

> Sybren

in Jan Klopper's honest opinion:

> Sybren Stuvel wrote:
>> Jan Klopper enlightened us with:
>>
>>>I have the following hardware:
>>>Hp lpr450,
>>>2u dual p2 450, 512 MB ,
>>>2x 4.1GB scsi 7.2k rpm,
>>>Win2k server
>>
>>
>> Drop it, replace it with Gentoo.
>>
>
> Was planning on doing so, but i can't just keep it offlie for a couple
> of hours (and maybe more if the gentoo fails), i was planning on
> building the cluster, and then migrating the services.
> As soon as everything is switched over reinstall the machine with gentoo
> and add it to the cluster.

yes, good decision... but sincerely i'm not sure on how you'd be able to
create a cluster with LAMP and Wintel+IIS... I'd rather consider migrating
services one by one to the new server without worrying about clustering

>>
>>>Compaq dl360,
>>>1u dual p3 1266, 512 MB,
>>>2x 9.1 GB scsi 10k rpm,
>>>Gentoo (2.4 kernel)
>>
>>
>> I suggest a 2.6 kernel instead.
>>
> I know, but compiling a kernel and lilo/grub is a complete nightmnare on
> these machines.

it shouldn't be a great problem: the last week i had to install a RHEL21
with 2.4.9-e3 kernel to a HP DL580 wich is just a natural evolution of
yours DL360. Anyway i verified that every component is perfectly supported
by new kernel so you shouldn't be harmed by <=2.6.9 kernel.

>>
>>>Compiling never gave me any problems, So the cpus are fine,
>>
>>
>> Never assume something is fine solely based on the fact that it does
>> things the right way. Make sure you also test the memory - memtest86
>> is a good one.
>>
> Hmz,, il do so. (back to the datacenter :P )

Don't you have ECC memory? so probably you could skip memtest86

Good luck and enjoy Gentoo.
--
Hekaton Keires,
En to phronein maedhen aedistos bios.

Hekaton Keires wrote:
> in Jan Klopper's honest opinion:
>
>
>>Sybren Stuvel wrote:
>>
>>>Jan Klopper enlightened us with:
>>>
>>>
>>>>I have the following hardware:
>>>>Hp lpr450,
>>>>2u dual p2 450, 512 MB ,
>>>>2x 4.1GB scsi 7.2k rpm,
>>>>Win2k server
>>>
>>>
>>>Drop it, replace it with Gentoo.
>>>
>>
>>Was planning on doing so, but i can't just keep it offlie for a couple
>>of hours (and maybe more if the gentoo fails), i was planning on
>>building the cluster, and then migrating the services.
>>As soon as everything is switched over reinstall the machine with gentoo
>>and add it to the cluster.
>
>
> yes, good decision... but sincerely i'm not sure on how you'd be able to
> create a cluster with LAMP and Wintel+IIS... I'd rather consider migrating
> services one by one to the new server without worrying about clustering
>
I can't stop the machine, but was planning on building the cluster next
to it (altough it would not be very large), and then migrate the
services to the cluster, so i can reinstall that server and integrate
into the cluster. But things hav changed, Read on.

>
>>>>Compaq dl360,
>>>>1u dual p3 1266, 512 MB,
>>>>2x 9.1 GB scsi 10k rpm,
>>>>Gentoo (2.4 kernel)
>>>
>>>
>>>I suggest a 2.6 kernel instead.
>>>
>>
>>I know, but compiling a kernel and lilo/grub is a complete nightmnare on
>>these machines.
>
Its the raid array giving me the problems, its gives out different /dev/
locations for the disks, so the kernel has problems finding the boot
partition, but if i change it to enable the kernel to find its
partitions, lilo won;t do anything since the live-cd is giving a
completely different location, and grub/lilo won't continue.
>
> it shouldn't be a great problem: the last week i had to install a RHEL21
> with 2.4.9-e3 kernel to a HP DL580 wich is just a natural evolution of
> yours DL360. Anyway i verified that every component is perfectly supported
> by new kernel so you shouldn't be harmed by <=2.6.9 kernel.
>
Since i just bought 5 more dl360's i'd really like to have some help
installing them correctly, could i email you for help?
>
>>>>Compiling never gave me any problems, So the cpus are fine,
>>>
>>>
>>>Never assume something is fine solely based on the fact that it does
>>>things the right way. Make sure you also test the memory - memtest86
>>>is a good one.
>>>
>>
>>Hmz,, il do so. (back to the datacenter :P )
>
>
> Don't you have ECC memory? so probably you could skip memtest86
>
Yup i do, Registerd ECC mem., 512MB Kingston. (so i figured it would be
stable), The cpu's are both new, and so is the rest of the machine, the
disks are running in raid, so it can't be the disks.

> Good luck and enjoy Gentoo.

Gentoo's my friend, it runs on all the other machines, the last windows
box is that server. :P

in Jan Klopper's honest opinion:

> Its the raid array giving me the problems, its gives out different /dev/
> locations for the disks, so the kernel has problems finding the boot
> partition, but if i change it to enable the kernel to find its
> partitions, lilo won;t do anything since the live-cd is giving a
> completely different location, and grub/lilo won't continue.

what kind of prob? the module you need is probably cciss (Compaq Smart
Array) which in my case correctly managed RAID5 configuration.

probably your problem is live cd which does not contain cciss module in
initrd file - instead of them it has some scsi low level driver and so you
can see disks, but not the logical array constructed upon them...

the thing should be inverse: you shouldn't use scsi drivers but cciss
module. please remind me about the scsi controller - isn't it a LSI Logic
53c1030 U2 320 controller? that one gave me some probs - in fact i had to
download the drivers from the site (rpm was available) and that was the
only that worked... but as i told you - you shouldn't use these during
installation so just skip them and use cciss

> Since i just bought 5 more dl360's i'd really like to have some help
> installing them correctly, could i email you for help?

well probably i am not fast to reply - but I'll try to give you all the
support i can provide

>
> Gentoo's my friend, it runs on all the other machines, the last windows
> box is that server. :P

imho when something works, and does it well, leave it alone :P
--
Hekaton Keires,
En to phronein maedhen aedistos bios.

Hekaton Keires wrote:
> in Jan Klopper's honest opinion:
>
>
>>Its the raid array giving me the problems, its gives out different /dev/
>>locations for the disks, so the kernel has problems finding the boot
>>partition, but if i change it to enable the kernel to find its
>>partitions, lilo won;t do anything since the live-cd is giving a
>>completely different location, and grub/lilo won't continue.
>
>
> what kind of prob? the module you need is probably cciss (Compaq Smart
> Array) which in my case correctly managed RAID5 configuration.
>
> probably your problem is live cd which does not contain cciss module in
> initrd file - instead of them it has some scsi low level driver and so you
> can see disks, but not the logical array constructed upon them...
>
> the thing should be inverse: you shouldn't use scsi drivers but cciss
> module. please remind me about the scsi controller - isn't it a LSI Logic
> 53c1030 U2 320 controller? that one gave me some probs - in fact i had to
> download the drivers from the site (rpm was available) and that was the
> only that worked... but as i told you - you shouldn't use these during
> installation so just skip them and use cciss
>

Okay, that should clear things up, so if i load up a live-cd with the
correct scsi module it should just give me the same /dev locations and
thus give me a good enviroment to install grub/lilo what ever.

I think the module im running right now is:
compaq 5000 or something like that.

>>Since i just bought 5 more dl360's i'd really like to have some help
>>installing them correctly, could i email you for help?
>
>
> well probably i am not fast to reply - but I'll try to give you all the
> support i can provide
>

That'a okay, il try setting up the scsi setup first.
>
>>Gentoo's my friend, it runs on all the other machines, the last windows
>>box is that server. :P
>
>
> imho when something works, and does it well, leave it alone :P
Il want to embed the last windows box into the cluster when its al
running nice, (keeping the windows machine up and running is loads
harder, instead of just having it run inside the cluster)