Hurdles to Progress in Long-form Question Answering
Kalpesh Krishna, Aurko Roy, Mohit Iyyer · 2021
The task of long-form question answering (LFQA) involves retrieving documents relevant to a given question and using them to generate a paragraph-length answer.While many models have recently been proposed for LFQA, we show in this paper that the task formulation raises fundamental challenges regarding evaluation and dataset creation that currently preclude meaningful modeling progress.To demonstrate these challenges, we first design a new system that relies on sparse attention and contrastive retriever learning to achieve state-of-the-art performance on the ELI5 LFQA dataset.While our system tops the public leaderboard, a detailed analysis reveals several troubling trends: (1) our system's generated answers are not actually grounded in the documents that it retrieves; (2) ELI5 contains significant train / validation overlap, as at least 81% of ELI5 validation questions occur in paraphrased form in the training set; (3) ROUGE-L is not an informative metric of generated answer quality and can be easily gamed; and (4) human evaluations used for other text generation tasks are unreliable for LFQA.We offer suggestions to mitigate each of these issues, which we hope will lead to more rigorous LFQA research and meaningful progress in the future. 1 * Work done during an internship at Google Research. 1 Resources accompanying our paper can be found in https://github.com/martiansideofthemoon/ hurdles-longform-qa Val Q: Can you protect electronics from EMPs/solar flares?If so, how?Generation using random retrievals, 24.8 ROUGE-L Yes, you absolutely can, in fact you can build a Faraday cage around your electronics, and protect them from solar flares...This is what is done with the Faraday cage around your electronics, which is the problem.The reason it is expensive is because it requires a huge amount of power and is expensive to replace... designed to shield your electronics from solar flares, you will have to pay for the protection.This is because you have to buy a piece of equipment that is designed to shield your electronics from solar flares, and that is expensive.... This is also expensive, but not as expensive as the protection you need to shield your electronics from solar flares... designed to be as cheap as possible... Generation using predicted retrievals, 19.0 ROUGE-L Yes, you can shield them.But it's a slow process... Also, the equipment that's powered by them is a lot more expensive than you'd think, so it's hard to make sure that you're not just shielding them from your remote control.Train Q1: How does an EMP ruin electronics?What does it do?How would they be fixed?Can It be protected against?How?Train Q2: If Earth were hit with a massive EMP, would all of our currently technology be completely unusable permanently?Train Q3: Whenever a electromagnetic pulse (EMP) is released what does it do to electronics to disable them?Train Q4: If earth was hit with an EMP, could we ever restore electricity?If not, why?Train Q5: What are solar flares and why does it impact our electronics?Train Q6.When an EMP goes off, can the electronics affected be replaced?Gold Answer, 18.6 ROUGE-L I'll start with the grounding question, because that's the easiest to answer: Doesn't help a bit.All that matters is that the metal container is conductive and doesn't have gaps...completely seal your Faraday cage.Consider soldering the lid on to that paint can... look at little baggie it comes in.Sealed mylar.That protected that chip from air travel at 35,000 feet, land travel through rural, urban, and suburban areas, and all the electromagnetic radiation that the trip entails...No lead shielding.No safes.... Random Train Ans, 19.4 ROUGE-LThe fast lane/slow lane is a bit of a misnomer.It gives the impression that new, faster lanes are being built.In reality, normal speed will be...