RobustQA: Benchmarking the Robustness of Domain Adaptation for Open-Domain Question Answering
Rujun Han, Peng Qi, Yuhao Zhang, Lan Liu, Juliette Burger, William Yang Wang, Zhiheng Huang, Bing Xiang, Dan Roth · 2023
Open-domain question answering (ODQA) is a crucial task in natural language processing.A typical ODQA system relies on a retriever module to select relevant contexts from a large corpus for a downstream reading comprehension model.Existing ODQA datasets consist mainly of Wikipedia corpus, and are insufficient to study models' generalizability across diverse domains, as models are trained and evaluated on the same genre of data.We propose RobustQA 1 , a novel benchmark consisting of datasets from 8 different domains, which facilitates the evaluation of ODQA's domain robustness.To build RobustQA, we annotate QA pairs in retrieval datasets with rigorous quality control.We further examine improving QA performances by incorporating unsupervised learning methods with target-domain corpus and adopting large generative language models.These methods can effectively improve model performances on RobustQA.However, experimental results demonstrate a significant gap from in-domain training, suggesting that Ro-bustQA is a challenging benchmark to evaluate ODQA domain robustness.1 Datasets and their processing code can be found here: https://github.com/rujunhan/RobustQA-dataFiQA-Finance Question: Why do investors buy stock that had appreciated?Document: Imagine how foolish the people that bought Apple at $100 must have felt.It was up tenfold for the $10 it traded at just years prior, how could it go any higher?Stocks have no memory.A stock's earnings may grow and justify the new higher price people are willing to pay.When FB came public, I remarked how I'd analyze the price and felt it was overvalued until its earnings came up.Just because it's gone down ever since, doesn't make it a buy, yet. LoTTE-LifestyleQuestion: What techniques, tricks or otherwise have you used to get upgrades on flights?Document: ... but I think the best way to get upgraded is to fly a lot with the airline.Generally when the flight's overbooked in one class, and they're trying to pick which person to upgrade, frequent flyer status is the first metric they use.The higher your status, the higher up the list you go! Having a high status with a partner airline can work too, high tiers with a partner airline usually comes below the airline's own frequent flyers, but above everyone else.Otherwise, if you're flying on your own that'll help in the event that there aren't enough frequent flyer to upgrade to free the required number of seats!Offering to pay may be an option too -if they're pretty full they may offer you a low price to upgrade.