Efficient local algorithms for distributed data mining in large scale peer to peer environments: a deterministic approach
Hillol Kargupta, Kanishka Bhaduri · 2008
Peer-to-peer (P2P) systems such as Gnutella, Napster, e-Mule, Kazaa, and Freenet are increasingly becoming popular for many applications that go beyond downloading music files without paying for it. Examples include P2P systems for network storage, web caching, searching and indexing of relevant documents and distributed network-threat analysis. These environments are rich in data and this data, if mined, can provide valuable source of information. Mining the web cache of users, for example, may often give information about their browsing patterns leading to efficient searching, resource utilization, query routing and more. However, most of the off-the-shelf data analysis techniques are designed for centralized applications where the entire data is stored in a single location. These techniques do not work in a highly decentralized, distributed environment such as a P2P network. We need distributed data mining algorithms that are fundamentally local, scalable, decentralized, asynchronous and anytime to solve this problem. This research proposes DeFraLC: a Determinsitic Framework for Local Computation of functions defined on data distributed in large scale (peer to peer) systems. Computing global data models in such environments can be very expensive. Moving all or some of the data to a central location does not work because of the high cost involved in centralization. The cost increases even more under a dynamic scenario where the peers' data and the network topology change arbitrarily. In this dissertation we have focused on developing algorithms for deterministic function-computation in large scale P2P environments. Our algorithmic framework is local which means that a peer can compute a function based on the information of only a handful of nearby neighbors and the communication overhead of the algorithm is upper bounded by some constant, independent of the size of the system. As a consequence, several messages can be pruned, leading to excellent scalability of our algorithms. The first algorithm that we have developed—PeGMA, Peer-to-Peer Generic Monitoring Algorithm—is capable of computing complex functions defined on the average of the horizontally distributed data. This generic algorithm is extremely accurate, highly scalable and can seamlessly adapt to changes in the data or the network. Following PeGMA, several interesting algorithms can be developed such as the L2 norm monitoring of distributed data which is a very powerful primitive. Using a two step feedback loop, a number of data mining algorithms have been proposed. The first step uses the local algorithm to raise a flag whenever the current data does not fit the function. The second step uses a feedback loop to sample data from the network and build a new function. The correctness of the local algorithm guarantees that once the computation terminates each peer has the same result compared to a centralized scenario. We propose solutions for P2P k-means monitoring, eigen monitoring and multivariate regression in P2P environments. Furthermore, we have shown how a complex data mining algorithm such as decision tree induction can be developed for P2P environments. Finally we have implemented all of the algorithms in a Distributed Data Mining Toolkit (DDMT) developed at the DIADIC research lab at UMBC. Our extensive experimental results show that the proposed algorithms are accurate, efficient and highly scalable.