Minatak QA dataset contains 20K English question-answer pairs linked to Wikidata, and additional translated 160K questions in eight different languages. Mintaka contains only question and answer pairs, so that the authors used models that can be trained end-to- end. The table given below shows the evaluation results on 3 language models (rows # 1-3), 3 knowledge graph-based models (rows # 4-6), and 2 retriever-reader models(rows # 7 & 8). Train, dev, and test set can be found here.
| Model | Year | hits@1 | Language | Reported By |
|---|---|---|---|---|
| XL-T5 (fine-tuned) | 2022 | 0.38 | English | Sen et al., 2022 |
| DPR (trained) | 2022 | 0.31 | English | Sen et al., 2022 |
| T5 | 2022 | 0.28 | English | Sen et al., 2022 |
| T5 for CBQA (zero-shot) | 2022 | 0.20 | English | Sen et al., 2022 |
| Rigel | 2022 | 0.20 | English | Sen et al., 2022 |
| EmbedKGQA | 2022 | 0.18 | English | Sen et al., 2022 |
| DPR (zero-shot) | 2022 | 0.15 | English | Sen et al., 2022 |
| KVMemNet | 2022 | 0.12 | English | Sen et al., 2022 |
| T5 for CBQA (translated) | 2022 | 0.31 | Multilingually | Sen et al., 2022 |
| Rigel | 2022 | 0.19 | Multilingually | Sen et al., 2022 |
| MT5 | 2022 | 0.16 | Multilingually | Sen et al., 2022 |
Sen, P., Aji, A. F., & Saffari, A. (2022). Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answering. Proceedings of the 29th International Conference on Computational Linguistics, 1604–1619. https://aclanthology.org/2022.coling-1.138