Yes-Yes-Yes: Proactive Data Collection for ACL Rolling Review and Beyond
Nils Dycke, Ilia Kuznetsov, Iryna Gurevych · 2022
The shift towards publicly available text sources has enabled language processing at unprecedented scale, yet leaves under-serviced the domains where public and openly licensed data is scarce.Proactively collecting text data for research is a viable strategy to address this scarcity, but lacks systematic methodology taking into account the many ethical, legal and confidentiality-related aspects of data collection.Our work presents a case study on proactive data collection in peer reviewa challenging and under-resourced NLP domain.We outline ethical and legal desiderata for proactive data collection and introduce "Yes-Yes-Yes", the first donation-based peer reviewing data collection workflow that meets these requirements.We report on the implementation of Yes-Yes-Yes at ACL Rolling Review 1 and empirically study the implications of proactive data collection for the dataset size and the biases induced by the donation behavior on the peer reviewing platform.