HTTP redirection for replica catalogue lookups in data grids
Heinz Stockinger, Andrew Bohdan Hanushevsky · 2002
Data distribution and replication in distributed systems require special purpose middleware tools for accessing replicated data. Data Grids, special forms of systems distributed over wide-area networks, need to handle data management issues like distribution and replication of large amounts of data in the Tera- and Petabyte scale. Replica catalogues are used for cataloguing and locating replicated files in distributed sites all around the globe. We present a novel and administratively scalable approach for distributing a replica catalogue and resolving file location information by using HTTP redirection. HTTP redirection servers managing local file catalogues allow for greater flexibility and local file management autonomy whereas a global replica catalogue provides the necessary mapping of logical files to individual sites. By distributing the catalogues a site can autonomously move files for load balancing within a site without notifying a global replica catalogue. Our approach scales well in terms of catalogue administration to a large number of sites and file entries and thus establishes a powerful middleware service. We present the design and implementation of our catalogue redirection servers and report on promising experimental results.