We have designed and implemented the Google File System, a scalable distributed file system for large distributed data-intensive applications. It provides fault tolerance while running on inexpensive commodity hardware, and it delivers high aggregate performance to a large number of clients.
The developers have released version 1.0 of the XtreemFS distributed file system. The result of a research project funded by the EU, XtreemFS is designed for distributed data management in grids, but is also meant to simplify accessing local files over the internet. It's easy to set up, supports RAID-0 (striping) over different machines and file replication over the network, encrypts data, and provides security mechanisms to ensure only authorised users can access shared files. Unlike sharing via NFS or SMB/CIFS, the file system can extend over several servers, provides security mechanisms, and is optimised for use over slow and not totally reliable internet connections. The file system runs in user space (via FUSE under Linux, via Dokan under Windows), so it can do without drivers or a kernel module. Linux and Windows are supported on the client side, and a Mac OS X client is under construction. The server runs under Linux and Solaris, and the software is licensed under the GPLv2.
Google File System, a scalable distributed file system for large distributed data-intensive applications. It provides fault tolerance while running on inexpensive commodity hardware, and it delivers high aggregate performance to a large number of clients.
S. Ghemawat, H. Gobioff, and S. Leung. Proceedings of the nineteenth ACM symposium on Operating systems principles, page 29--43. New York, NY, USA, ACM, (2003)