Publication Details

BUT-FIT at SemEval-2020 Task 4: Multilingual commonsense

JON Josef, FAJČÍK Martin, DOČEKAL Martin and SMRŽ Pavel. BUT-FIT at SemEval-2020 Task 4: Multilingual commonsense. In: Proceedings of the Fourteenth Workshop on Semantic Evaluation. Barcelona: Association for Computational Linguistics, 2020, pp. 374-390. ISBN 978-1-952148-31-6. Available from: https://www.aclweb.org/anthology/2020.semeval-1.46/
Czech title
BUT-FIT na SemEval-2020 Úkol 4: Vícejazyčný zdravý rozum
Type
conference paper
Language
english
Authors
Jon Josef, Ing. (DCGM FIT BUT)
Fajčík Martin, Ing., Ph.D. (DCGM FIT BUT)
Dočekal Martin, Ing. (DCGM FIT BUT)
Smrž Pavel, doc. RNDr., Ph.D. (DCGM FIT BUT)
URL
Keywords

NLP, commonsense, pretrained language models, multilingual, machine translation

Abstract



We participated in all three subtasks. In subtasks A and B, our submissions are based on pretrained language representation models (namely ALBERT) and data augmentation. We experimented with solving the task for another language, Czech, by means of multilingual models and machine translated dataset, or translated model inputs. We show that with a strong machine translation system, our system can be used in another language with a small accuracy loss. In subtask C, our submission, which is based on pretrained sequence-to-sequence model (BART), ranked 1st in BLEU score ranking, however, we show that the correlation between BLEU and human evaluation, in which our submission ended up 4th, is low. We analyse the metrics used in the evaluation and we propose an additional score based on model from subtask B, which correlates well with our manual ranking, as well as reranking method based on the same principle. We performed an error and dataset analysis for all subtasks and we present our findings.

Published
2020
Pages
374-390
Proceedings
Proceedings of the Fourteenth Workshop on Semantic Evaluation
Conference
The 28th International Conference on Computational Linguistics, Barcelona (online), ES
ISBN
978-1-952148-31-6
Publisher
Association for Computational Linguistics
Place
Barcelona, ES
EID Scopus
2-s2.0-85123684230
BibTeX 
@INPROCEEDINGS{FITPUB12424,
   author = "Josef Jon and Martin Faj\v{c}\'{i}k and Martin Do\v{c}ekal and Pavel Smr\v{z}",
   title = "BUT-FIT at SemEval-2020 Task 4: Multilingual commonsense",
   pages = "374--390",
   booktitle = "Proceedings of the Fourteenth Workshop on Semantic Evaluation",
   year = 2020,
   location = "Barcelona, ES",
   publisher = "Association for Computational Linguistics",
   ISBN = "978-1-952148-31-6",
   language = "english",
   url = "https://www.fit.vut.cz/research/publication/12424"
}
Back to top