Publication Details

Rethinking the Objectives of Extractive Question Answering

FAJČÍK Martin, JON Josef and SMRŽ Pavel. Rethinking the Objectives of Extractive Question Answering. In: Proceedings of the 3rd Workshop on Machine Reading for Question Answering. Proceedings of the 3rd Workshop on Machine Reading for Question Answering. Punta Cana: Association for Computational Linguistics, 2021, pp. 14-27. ISBN 978-1-954085-95-4. Available from: https://aclanthology.org/2021.mrqa-1.2/
Czech title
Přehodnocení objektivních funkcí systémov pre extrakční odpovídaní na otázky
Type
conference paper
Language
english
Authors
Fajčík Martin, Ing., Ph.D. (DCGM FIT BUT)
Jon Josef, Ing. (DCGM FIT BUT)
Smrž Pavel, doc. RNDr., Ph.D. (DCGM FIT BUT)
URL
Keywords

QA, extractive QA, independent objective, joint objective, compound objective

Abstract

This work demonstrates that using the objective with independence assumption for modelling the span probability P (a_s , a_e ) = P (a_s )P (a_e) of span starting at position a_s and ending at position a_e has adverse effects. Therefore we propose multiple approaches to modelling joint probability P (a_s , a_e) directly. Among those, we propose a compound objective, composed from the joint probability while still keeping the objective with independence assumption as an auxiliary objective. We find that the compound objective is consistently superior or equal to other assumptions in exact match. Additionally, we identified common errors caused by the assumption of independence and manually checked the counterpart predictions, demonstrating the impact of the compound objective on the real examples. Our findings are supported via experiments with three extractive QA models (BIDAF, BERT, ALBERT) over six datasets and our code, individual results and manual analysis are available online.

Published
2021
Pages
14-27
Proceedings
Proceedings of the 3rd Workshop on Machine Reading for Question Answering
Series
Proceedings of the 3rd Workshop on Machine Reading for Question Answering
Conference
Conference on Empirical Methods in Natural Language Processing, Punta Cana, DO
ISBN
978-1-954085-95-4
Publisher
Association for Computational Linguistics
Place
Punta Cana, DO
EID Scopus
2-s2.0-85138379950
BibTeX 
@INPROCEEDINGS{FITPUB12639,
   author = "Martin Faj\v{c}\'{i}k and Josef Jon and Pavel Smr\v{z}",
   title = "Rethinking the Objectives of Extractive Question Answering",
   pages = "14--27",
   booktitle = "Proceedings of the 3rd Workshop on Machine Reading for Question Answering",
   series = "Proceedings of the 3rd Workshop on Machine Reading for Question Answering",
   year = 2021,
   location = "Punta Cana, DO",
   publisher = "Association for Computational Linguistics",
   ISBN = "978-1-954085-95-4",
   language = "english",
   url = "https://www.fit.vut.cz/research/publication/12639"
}
Back to top