Effects of Architecture and Training on Embedding Geometry and Feature Discriminability in BERT
Maksim Podkorytov, Daniel Biś, Jinglun Cai, Kobra Amirizirtol, Xiuwen Liu · 2020
Natural language processing has improved substantially in the last few years due to the increased computational power and availability of text data. Bidirectional Encoder Representations from Transformers (BERT) have further improved the performance by using an auto-encoding model that incorporates larger bidirectional contexts. However, the underlying mechanisms of BERT for its effectiveness are not well understood. In the paper we investigate how the BERT architecture and its pretraining protocol affect the geometry of its embeddings and the effectiveness of its features for classification tasks. As an autoencoding model, during pre-training, it produces representations that are context dependent and at the same time must be able to "reconstruct" the original input sentences. The complex interactions of the two via transformers lead to interesting geometric properties of the embeddings and subsequently affect the inherent discriminability of the resulting representations. Our experimental results illustrate that the BERT models do not produce "effective" contextualized representations for words and their improved performance may mainly be due to fine-tuning or classifiers that model the dependencies explicitly by encoding syntactic patterns in the training data.