Publication Details

NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned

MIN Sewon, FAJČÍK Martin, DOČEKAL Martin, ONDŘEJ Karel and SMRŽ Pavel et al. NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned. In: Proceedings of the NeurIPS 2020 Competition and Demonstration Track. Proceedings of Machine Learning Research, vol. 133. online: Proceedings of Machine Learning Research, 2021, pp. 86-111. ISSN 2640-3498. Available from: http://proceedings.mlr.press/v133/min21a/min21a.pdf
Czech title
Soutěž NeurIPS 2020 EfficientQA: Systémy, analýzy a získané lekce
Type
conference paper
Language
english
Authors
Min Sewon (UWASH)
Fajčík Martin, Ing., Ph.D. (DCGM FIT BUT)
Dočekal Martin, Ing. (DCGM FIT BUT)
Ondřej Karel, Ing. (DCGM FIT BUT)
and others
URL
Keywords

question answering, QA, ODQA, efficientQA, memory, disk memory, budget, efficient parameter, retrieval corpora

Abstract

We review the EfficientQA competition from NeurIPS 2020. The competition focused on open-domain question answering (QA), where systems take natural language questions as input and return natural language answers. The aim of the competition was to build systems that can predict correct answers while also satisfying strict on-disk memory budgets. These memory budgets were designed to encourage contestants to explore the trade-off between storing retrieval corpora or the parameters of learned models. In this report, we describe the motivation and organization of the competition, review the best submissions, and analyze system predictions to inform a discussion of evaluation for open-domain QA.

Published
2021
Pages
86-111
Journal
Proceedings of Machine Learning Research, vol. 133, no. 133, ISSN 2640-3498
Proceedings
Proceedings of the NeurIPS 2020 Competition and Demonstration Track
Series
Proceedings of Machine Learning Research
Conference
Thirty-fourth Conference on Neural Information Processing Systems, online, US
Publisher
Proceedings of Machine Learning Research
Place
online, US
BibTeX 
@INPROCEEDINGS{FITPUB12572,
   author = "Sewon Min and Martin Faj\v{c}\'{i}k and Martin Do\v{c}ekal and Karel Ond\v{r}ej and Pavel Smr\v{z} and et al.",
   title = "NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned",
   pages = "86--111",
   booktitle = "Proceedings of the NeurIPS 2020 Competition and Demonstration Track",
   series = "Proceedings of Machine Learning Research",
   journal = "Proceedings of Machine Learning Research",
   volume = 133,
   number = 133,
   year = 2021,
   location = "online, US",
   publisher = "Proceedings of Machine Learning Research",
   ISSN = "2640-3498",
   language = "english",
   url = "https://www.fit.vut.cz/research/publication/12572"
}
Back to top