A web farm is essential to a high traffic site. Redundancy, intelligent traffic flow, collaboration and monitoring are keys to maintaining uptime and availability. The correct infrastructure will mean a solid, consistent presence for a site with an industry standard 99.9999+% uptime .
At the core of any web farm is redundancy. Any single point of failure is a vulnerability. Here is sample topography of a basic implementation.
As you can, see several are potential failure points. To minimize these ricks, the following is a possible solution:
You can see that every major area is covered with redundant systems. This layout will provide a stable, constant environment for the site. Redundant systems obviously cost twice as much, but it’s the only way to assure availability and presence.
Navigating your site traffic is another essential part to a solid web farm. Typically a company will use a 3rd party appliance (NetScaler, BigIP, etc) to route traffic to the internal web servers. This allows the actual web servers to be “hidden” from the outside world, which is always a plus when fending off DOS attacks and such. The appliance will receive every inbound request and send it to the appropriate web servers. These appliances can be configured in a number of ways to direct traffic to specific servers under specific circumstances. (Round robin, highest load, etc)
A more cost effective way of load balancing is to use Microsoft NLB (network Load Balancing). This free tool comes with Windows Server and will do in a pinch. It is not as configurable as a 3rd party solution and will only allow a “Round Robin” technique for traffic flow. If 1 server is loaded up, it will still receive requests if it’s next in line.
The next area to address is collaboration of data across the web farm. Having 100 web servers for your new music download program is great, until you have an update to your software to apply to them. There are several 3rd party solutions for maintaining multiple file systems, but Microsoft does provide a free solution. Microsoft DFS (Distributed File System) allows updates to 1 server to be propagated to a number of other servers automatically. This greatly diminished the update process and allows for several systems to be maintained with ease.
Note: Microsoft DFS has a number of configuration options and your implementation may require exclusions or other settings.
All the redundancy and collaboration in the world won’t help you if your site goes down and no one knows about it. The most embarrassing thing for an admin is to get a call from your customer asking why the site isn’t up. To prevent such a herendous faux pa all you your systems should be monitored.
You can use several 3rd party applications to monitor your systems, which can be installed on your web server. These applications will alert you if anything goes awry.
An easier solution is a web based company that will constantly monitor you site by pinging a particular IP and running a script against the site. A good one I've used is AlertSite.
Hope this helps and happy farming!