the google file system

Filed in archive Technology

As Google heads for IPO the discussion is all around. One of the biggest achievements by Google engineers besides the parallel computing platform of estimated 100.000 Pc-like servers is the Google file systems. This PDF from the Google engineers Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung says how it was done.

Meanwhile Business 2.0 looks for a private cluster network.

"Lustre is one that's been developed by a Boston startup called, straightforwardly enough, Cluster File Systems. Six national laboratories are using Lustre, including Sandia, which plans to adopt it for nuclear-weapon simulations on its ASCI Red Storm supercomputer. Cluster CEO Phil Schwan says he also has some customers in the R&D departments of energy and auto companies who are using Lustre to work on very large data sets for oil exploration and vehicle design."

"What's in the labs now is on the cusp of hitting corporate data centers. Merrill Lynch (MER), for example, recently completed a project to install a cluster-like file system throughout its network, so analysts can collaborate on documents anywhere in the world. A cluster file system eliminates the need for backup -- a huge IT headache that consumes a lot of staff time -- and synchronization, which can frustrate road warriors trying to juggle files on their laptops. The biggest improvement comes in manageability, though: A cluster file system, once installed, looks like a single entity to a systems administrator, which vastly simplifies its operation. The biggest maintenance chore is sending someone through the data center to pull failed drives out of the rack and replace them."

I feel it's an area that could make good use of some well funded start-ups...