Transformers, Tables and Frame Semantics
Mario Ramirez, Alex Bogatu, Norman W. Paton, André Freitas · 2023
Transformer-based language models are able to capture linguistic patterns at scale by encoding both syntactic and semantic dimensions of natural language representations with the aim of achieving language understanding. While Transformers have been adapted for generating table embeddings, less research effort has been dedicated to investigating the extent to which these models can encode table semantics. To address this limitation, we propose a method to transfer knowledge from pre-trained natural language models to encode schema-level relationships and analyze the resulting model with respect to two schema-related tasks in different data scenarios.