Balancing cost and accuracy in distributed data mining
Robert L. Grossman, Andrei L. Turinsky · 2002
This dissertation identifies the fundamental trade-off between cost and accuracy in distributed data mining and explores the resulting issues. It suggests that an optimal distributed strategy must balance cost and accuracy by combining local data processing with data transfer across the network. A formal framework is developed for finding a proper data transfer strategy as a solution to a mathematical optimization problem. Furthermore, a natural connection to the mixture of models problem and the Expectation-Maximization approach is demonstrated and described in quantitative terms. This leads to the development of a class of algorithms that guide the network transfer. Finally, it is shown that a hierarchical predictive system may often be more appropriate than some popular methods widely used in distributed data mining nowadays. A new greedy algorithm is presented that allows one to build such a system by utilizing a properly chosen strategy for data transfer over the network.