Challenges of Self-Supervised Learning for Unified, Multi-Modal, Multi-Task Transformer Models
Graham Annett, Tim Andersen, Robert D. Annett · 2022
The recent success of multi-modal multi-task transformer models combined with their ability to learn in a scalable self-supervised fashion has presented evidence that omnipotent models trained with heterogeneous data and tasks are within the realms of possibility. This paper presents several research questions and impediments related towards the training of generalized transformer architectures.