Kelp : an architecture for understanding global system behavior in massively scalable distributed systems
Yung, Arthur · cIRcle (University of British Columbia) · 2009
Current techniques do not scale distributed systems to millions of nodes because they cannot handle global behavior description and global coordination to such massive sizes. KELP addresses these problems with a loose, decentralized system of nodes that are connected together with a small-world network. Viewed as a network of randomly connected clusters, a small-world network supports massive scalability with its random connectivity while still supporting locality within its clusters. KELP uses two key properties of randomness to scale. First, there is a short typical distance of separation between any two nodes in the system. This is used to quickly infer global behavior. Second, nodes have relatively little knowledge of the overall system, which helps provide looser semantics for global coordination. Finally, on top of the small-world infrastructure, KELP provides massively scalable data structures to make building massive scale systems less ad-hoc.