Multimodal Co-orchestration for Exploring Structure-Property Relationships in Combinatorial Libraries via Multi-Task Bayesian Optimization
Sergei V. Kalinin, Boris N. Slautin, Utkarsh Pratiush, Ilia N. Ivanov, Yongtao Liu, Rohit Kumar Pant, Xiaohang Zhang, Ichiro Takeuchi, Maxim A Ziatdinov · Microscopy and Microanalysis · 2024
The rapid evolution of automated synthesis approaches has significantly improved cost-effectiveness and noticeably accelerated the pace of designing novel materials [1,2]. However, a bottleneck arises in the characterization, often necessitating a comprehensive study of various electric, mechanical, chemical, and structural properties. This complexity makes the characterization process time-consuming, creating a significant gap between the rates of synthesis and exploration. Among the pioneering examples of such high-throughput synthesis are combinatorial libraries. The characterization of combinatorial libraries involves the exploration of various aspects of their structures and functionalities by a broad spectrum of local investigative methods, including SPM, electron microscopy, Raman microscopy, and more. The recent revolution in autonomous instrumentations brings forth an opportunity for the co-orchestration of multimodal tools, equipped with multiple sequential detection methods, or several characterization tools to explore identical samples. This enables exploration of combinatorial libraries in multiple locations by multiple tools simultaneously, or downstream characterization in automated synthesis systems. Here we introduce the multimodal co-orchestration framework, representing a significant advancement in optimizing material exploration through the simultaneous orchestration of multiple methodologies (Fig. 1A). The idea of a multimodal co-orchestration approach lies in leveraging information uncovered for one property (modality) to expedite the exploration of another property measured by a different method. This accelerates the overall characterization process. The combinatorial library characterization comprises sequential steps. At each step, the orchestrating agent is employed to determine the measurement modality and location at the following step based on the anticipated knowledge gain and measurement cost. The low-dimensional compositional space within a combinatorial library makes the implementation of Bayesian optimization (BO) a highly robust solution for governing the orchestrating agent [3]. However, raw measured data are often represented in high-dimensional datasets, such as spectra and images. We propose to employ variational autoencoders (VAE) to reduce the dimensionality of raw data. As a result, multimodal co-orchestration occurs by optimizing the exploration trajectory within the low-dimensional space, encompassing composition and VAE latent variables from different modalities. Typically, a single latent variable of VAE representation is chosen from each modality to form a learning dataset for multi-task Gaussian Processes (MTGP). The objective of MTGP is to predict both the mean values and uncertainties associated with the selected latent variables (modalities) within the compositional latent space. The multimodal acquisition function is constructed based on MTGP outcomes. At the beginning of the exploration, introducing new data at each exploration step can noticeably alter the VAE distributions, due to the limitless gathered knowledge about the system. This is associated with the initial co-orchestration stage, where the VAE and MTGP are retrained at each step from scratch. As information accumulates, injecting new data doesn't drastically influent the VAE distribution. The stability of the VAE distribution enables a transition to the steady co-orchestration stage with the application of incremental training (Fig. 1B). The effectiveness of the suggested framework was assessed by applying it to the co-orchestration of piezoresponse force microscopy hysteresis loops (BEPS) and micro-Raman spectra to explore the Sm-BiFeO3 combinatorial library (Sm-BFO). This system possesses a phase transition from the ferroelectric phase of pure rhombohedral BiFeO3 to a non-ferroelectric phase of orthorhombic 20% Sm-doped BiFeO3, through the morphotropic phase boundary [4]. For automated experiment simulation, we employ the pre-acquired BEPS and Raman datasets with spectra gained in equidistant locations within the library. Each dataset consists of 94 spectra. The measurement locations correspond to compositions ranging from 20% Sm-BFO (location 0) to pure BFO (location 93). The performed automated experiment simulation consists of the 30 subsequent exploration steps governed by the Maximum uncertainty acquisition function (Fig. 2A, C, E). The gradual decrease in the GP uncertainties mirrors the ongoing exploration (Fig. 2F). During the main part of the exploration, the model endeavors to capture the general trends in compositional profiles of modalities and correlations between them. This is reflected in the gradual adjustment and stabilization of both the multi-task coregularization matrix coefficients B10 and the kernel lengths of the MTGP latent processes (Fig. 2B, D). In the final steps, the algorithm explores noise, resulting in the reduction of kernel lengths and B10 coefficients. The workflow confirms its effectiveness in optimizing the exploration trajectory. In summary, we have introduced a co-orchestration workflow to guide the exploration of combinatorial libraries by the simultaneous application of multiple methods. This approach expedites combinatorial library exploration through the real-time utilization of acquired knowledge about one property to accelerate the exploration of other properties measured by different methods. The capabilities of multimodal co-orchestration were validated by the autonomous experiment simulations in the Sm-BFO library. However, the proposed framework is general and can be extended to multiple measurement modalities and arbitrary dimensionality of measured signals. We do believe that the co-orchestration workflow may significantly enhance the efficiency of combinatorial library exploration, narrowing the gap between synthesis and characterization rates [5]. Co-orchestration workflow. (A) General principle and (B) working schematics of multimodal co-orchestration. Co-orchestration experiment simulation. (A, C, E) MTGP reconstruction at the various exploration steps, (B) MTGP kernel length, (D) B10 coregularization coefficient between modalities, and (F) GP uncertainty. Modality 0 corresponds to the Raman spectra, while Modality 1 represents the BEPS hysteresis loops. The solid lines in (A, C, E) correspond to the MTGP predictions, rounds and stars represent the experimental points, ground truth compositional profiles illustrated by dashed lines. The background color strips indicate the selected modality at each exploration step.