Contributions à l'apprentissage local, asynchrone et décentralisé, et à l'apprentissage profond sur variété
Edouard Oyallon · HAL (Le Centre pour la Communication Scientifique Directe) · 2023
This document is a summary of some of the research I conducted from 2018 to 2023 to obtain the Habilitation à Diriger des Recherches. All the results mentioned are discussed in greater depth in the corresponding publications.The research topics I will discuss focus on the concepts of distributed learn- ing, local learning, and deep representation learning on manifolds. A central point I aim to develop is the creation of efficient distributed algorithms for train- ing deep neural networks, both in terms of computation time and the number of operations required for training. The first chapter provides an introduction to these fields and my research activities. The second chapter discusses the interplay between computer hardware and existing training paradigms. These interactions are crucial as they promise or achieve significant reductions in the training costs of deep networks. The third chapter outlines initial elements of re- search to create deep representations on manifolds and graphs. In particular, we have sought principles for modeling such architectures. The fourth chapter dis- cusses the local learning results I have obtained, especially layer-by-layer greedy learning. This corresponds to the idea of training networks by proposing weight updates for a layer based solely on local information to that layer, using as little global information related to all layers as possible. The fifth chapter summarizes results obtained in decentralized asynchronous learning, within frameworks of convex optimization and optimization of deep neural network weights. One of the interests of such approaches is their potential superiority, both from a prac- tical implementation standpoint and from an algorithmic perspective. Finally, the last chapter provides future perspectives on my research.