Exploring Mathematical Strategies for Finding Hidden Features in Multi-Dimensional Big Datasets
Tri Dung Duong, Fang Ren, Apurva Mehta · DigitalCommons - CalPoly (California State Polytechnic University) · 2016
With advances in technology in brighter sources and larger and faster detectors, the amount of data generated at national user facilities such as SLAC is increasing exponentially. Humans have a superb ability to recognize patterns in complex and noisy data and therefore, data is still curated and analyzed by humans. However, a human brain is unable to keep up with the accelerated pace of data generation, and as a consequence, the rate of new discoveries hasn't kept pace with the rate of data creation. Therefore, new procedures to quickly assess and analyze the data are needed. Machine learning approaches are effective in reducing the complexity of data and finding hidden trends and contrasts in large datasets. The primary goal of this project is to develop a new algorithm using recent advances in image processing, machine learning techniques, and employing different types of distance metrics such as Euclidian, Manhattan, and Cosine to a large amount of diffraction data collected at a synchrotron beamline in high-throughput experimentation. The new algorithm enables analysis and extraction of hidden features from a large multi-dimensional dataset on-the-fly and near real-time with minimal computational cost and human intervention. When the algorithm is performed on a large number of x-ray diffraction patterns, the algorithm can be used to find the structural phase boundaries leading to the discovery of the composition-structure relationship, which is often an end goal of many materials science experiments.