An Introduction to Distributed Systems
Serge Abiteboul, Ioana Manolescu, Philippe Rigaux, Marie-Christine Rousset, Pierre Senellart · Cambridge University Press eBooks · 2011
This chapter is an introduction to very large data management in distributed systems. Here, “very large” means a context where gigabytes (1,000 MB = 10 9 bytes) constitute the unit size for measuring data volumes. Terabytes (10 12 bytes) are commonly encountered, and many Web companies and scientific or financial institutions must deal with petabytes (10 15 bytes). In a near future, we can expect exabytes (10 18 bytes) data sets, with the world-wide digital universe roughly estimated (in 2010) as about 1 zetabytes (10 21 bytes). Distribution is the key for handling very large data sets. Distribution is necessary (but not sufficient) to bring scalability (i.e., the means of maintaining stable performance for steadily growing data collections by adding new resources to the system). However, distribution brings a number of technical problems that make the design and implementation of distributed storage, indexing, and computing a delicate issue. A prominent concern is the risk of failure . In an environment that consists of hundreds or thousands of computers (a common setting for large Web companies), it becomes very common to face the failure of components (hardware, network, local systems, disks), and the system must be ready to cope with it at any moment. Our presentation covers principles and techniques that recently emerged to handle Web-scale data sets. We examine the extension of traditional storage and indexing methods to large-scale distributed settings. We describe techniques to efficiently process point queries that aim at retrieving a particular object.