Deep learning under data scarcity in science and engineering applications
Joshua R. Waite · 2024
This dissertation investigates challenges and approaches in leveraging techniques to overcome data scarcity in deep learning, with a focus on Atomic Force Microscopy (AFM) force curve characterization, shooter detection and tracking systems, and spatial reasoning in vision-language models (VLMs). In the AFM domain, we develop a few-shot learning framework to efficiently characterize force-distance curves, surpassing manual analysis speed and matching human accuracy. Additionally, we utilize data processing techniques to improve AFM data quality and reliability. Transitioning to shooter detection, we propose a holistic approach to detect entire shooters, enhancing tracking robustness and addressing obscured firearms. Leveraging domain randomization and transfer learning, our system effectively trains with synthetic data and operates seamlessly on edge hardware. To enhance spatial reasoning in VLMs, we propose a reinforcement learning (RL)-based sample selection framework. In this approach, the RL agent leverages feedback from a simulation environment and the VLM in order to generate feasible and informative samples targeted to challenge the weaknesses of the VLM. Throughout the dissertation, we address key research questions regarding learning under data scarcity and propose methodologies, including novel deep learning frameworks and innovative data synthesis strategies, to tackle these challenges, thereby advancing scientific research and public safety.