Shifted PAUQ: Distribution shift in text-to-SQL
Oleg Somov, Elena Viktorovna Tutubalina · 2023
Semantic parsing plays a pivotal role in advancing the accessibility of human-computer interaction on a large scale.Spider, a widely recognized dataset for text-to-SQL, contains a wide range of natural language (NL) questions in English and corresponding SQL queries.Original splits of Spider and its adapted to Russian language and improved version, PAUQ, assume independence and identical distribution of training and testing data (i.i.d split).In this work, we propose a target length split and multilingual i.i.d split to measure compositionality and cross-language generalization.We present experimental results of popular text-to-SQL models on original, multilingual, and target length splits.We also construct a context-free grammar for the evaluation of compositionality in text-to-SQL in an out-of-distribution setting.We make the splits publicly available on Hug-gingFace hub via https://huggingface.co/ datasets/composite/pauq.