AfrIFact: Cultural Information Retrieval, Evidence Extraction and Fact Checking for African Languages

arXiv:2604.00706 · cs.CL · Submitted 2026-04-01 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "AfrIFact: Cultural Information Retrieval, Evidence Extraction and Fact Checking for African Languages".

Jane: Here's the gist of the paper. The gist: AfrIFact introduces a benchmark dataset covering information retrieval, evidence extraction, and fact-checking across ten African languages and English to assess the capability of embedding models and LLMs in low-resource settings.

Tom: Dataset Introduction AfrIFact is a multilingual benchmark containing more than 18,000 claims in 10 African languages and English to assess the capability of embedding models and LLMs to answer specific research questions First, to investigate the capability in extracting relevant documents from large-scale resources, we ask What is the effectiveness of embedding models in performing information retrieval for underresourced African languages?

Jane: Secondly, to identify and select the specific sentences or passages that support or refute a claim, assessing the quality and reliability of their evidence extraction, we explore How precisely can embedding models identify and extract relevant evidence from single document for claim verification?

Tom: Finally, we ask To what extent can current LLMs correctly verify the factuality of claims?

Title and authors: Tom: We started by looking at the title and authors of this paper, "AfrIFact: Cultural Information Retrieval, Evidence Extraction And Fact Checking for African Languages." It’s a very descriptive title that tells you exactly what the project is about—it's not just one thing; it’s retrieval, extraction, and fact-checking all bundled together.

Jane: The authors are a big international group here across Germany, Canada, Japan, England, Mexico, Nigeria and Senegal. That cross-continental collaboration suggests they were aiming for a really broad multilingual scope from the start.

Lu: What's interesting is that they explicitly state their focus on African languages and culturally grounded domains which sets them apart from datasets that might be more general or only cover English.

Meng: I wonder how much of this data actually comes from real, varied sources versus curated Wikipedia content or other public materials.

Lalam: The authors recruited native-speaking annotators through an African AI community, which suggests a focus on building the infrastructure for true multilingual support rather than just scraping existing text.

The paper's summary: Tom: So, to summarize what AfrIFact does, it’s creating a benchmark of over eighteen thousand claims in ten African languages and English. The main goal is to assess the capability of embedding models and LLMs in these specific low-resource environments.

Jane: It sets up three core research questions: first, how effective are embedding models at performing information retrieval for underresourced African languages? Second, how precisely can they identify and extract relevant evidence from a single document to verify a claim? And third, to what extent can current LLMs correctly verify the factuality of claims by classifying them as supported, refuted, or not enough information.

Lu: They are specifically testing their ability to do information retrieval—finding the right documents—evidence extraction—pulling out the supporting sentences—and then classification for fact-checking. That's a comprehensive test suite.

Meng: From an operational standpoint, this is impressive because it forces models to deal with the noise and linguistic diversity inherent in these languages at scale, which is where many production systems fail.

Lalam: The paper highlights that most existing datasets focus primarily on English, and AfrIFact focuses explicitly on African languages and supports culturally grounded domains enabling multilingual fact verification with evidence extraction and retrieval.

The paper's improvements: Tom: The authors point out several areas where their work contributes to the field. They highlight that existing efforts in African language IR datasets are often limited in scale or domain coverage, and AfrIFact aims to fix that by being more comprehensive.

Jane: They also noted a gap between English and African languages, especially in the health domain, which suggests they’re setting a baseline for where we need to improve accuracy across different linguistic groups.

Lu: They are pushing for improvements in cross-lingual retrieval capabilities, acknowledging that even the best embedding models lack these skills natively and suggesting specialized strategies are needed.

Meng: And on the LLM side, they show that while zero-shot fact verification is extremely challenging across languages, few-shot prompting can actually improve performance by up to forty-three percent in models like AfriqueQwen-14B.

Lalam: The authors conclude that a small scale dataset coupled with parameter efficient finetuning can help improve the performance of models, and they stress the need for a substantially larger test set for robust, high-quality evaluation.

Conclusion: Tom: So to wrap up on AfrIFact, it’s established as a benchmark for low-resource information retrieval and fact checking specifically tailored for African languages across health and culture news domains. The results show systematic gaps in culture-specific information retrieval, evidence retrieval, and fact verification that researchers need to address.

Jane: The implication is that we need to move toward creating systems that are not just good at English but robust enough to handle the linguistic nuances and domain specifics of African contexts when we talk about verifying claims.

Lu: They’ve laid out the systematic challenges, which points toward the next steps: expanding dataset coverage and building cross-lingual evaluation methods that are truly robust.

Meng: From a deployment view, it confirms that you can't just take a big model and expect it to work reliably in these specific niches without targeted fine-tuning based on data like this.

Lalam: AfrIFact is introduced as a crucial resource for pushing the boundaries of what AI can do when applied to real-world, underrepresented languages and contexts.

Israel Abebe Azime, Jesujoba O. Alabi, Crystina Zhang, Iffat Maab, Atnafu Lambebo Tonja, Tadesse Destaw Belay, Folasade Alabi, Salomey osei, Saminu Muhammad Aliyu, Nkechinyere Faith Aguobi, Bontu Fufa Balcha, Blessing Sibanda, Davis David, Mouhamadane Mboup, Daud Abolade, Neo Putini

Masakhane NLP, Saarland University, University of Waterloo, National Institute of Informatics, University College London, Instituto Politécnico Nacional Nacional de México, University of Ilorin Nigeria, Universidad de Deusto Spain, Bayero University Nigeria · Black Swan Addis Ababa University · Universite Alioune Diop Senegal · McGill University Mila-Quebec AI Institute Canada CIFAR AI Chair

cs.CL

Submitted: 2026-04-01

Updated: 2026-10-21

Comments: Accepted at 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP 2026)

Code: https://github.com/IsraelAbebe/AfriFact

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 77/100

The gist: The gist: AfrIFact introduces a benchmark dataset covering information retrieval, evidence extraction, and fact-checking across ten African languages and English to assess the capability of embedding

Key concepts

Information Retrieval (IR) Effectiveness: This tests how well embedding models can find relevant documents from large resources when dealing with underresourced African languages. The study uses metrics like nDCG@10 to measure this capability, revealing that embedding models struggle significantly with cross-lingual retrieval.
Evidence Extraction Precision: This measures the accuracy of models in identifying and pulling specific sentences or passages from a document that support or refute a given claim. The results show a larger gap between English and African languages in extracting evidence, especially within the health domain.
LLM Fact-Checking Accuracy: This evaluates how well Large Language Models can correctly classify claims as supported, refuted, or not enough information across different African languages. Zero-shot verification is challenging, but few-shot prompting and fine-tuning significantly boost accuracy.
Domain Specialization (Health vs. Culture/News): The dataset is split into health and culture/news domains to see how performance varies. Evidence extraction gaps between English and African languages are more pronounced in the health domain compared to cultural news documents.

Terminology

Accuracy — 1× in this paper · explained in 1 episode
Fine-Tuning — 1× in this paper · explained in 5 episodes
Low-Resource Languages — 1× in this paper · explained in 4 episodes
Verification — 1× in this paper · explained in 2 episodes

Summary

The gist: AfrIFact introduces a benchmark dataset covering information retrieval, evidence extraction, and fact-checking across ten African languages and English to assess the capability of embedding models and LLMs in low-resource settings.

Dataset Introduction

AfrIFact is a multilingual benchmark containing more than 18,000 claims in 10 African languages and English to assess the capability of embedding models and LLMs to answer specific research questions First, to investigate the capability in extracting relevant documents from large-scale resources, we ask What is the effectiveness of embedding models in performing information retrieval for underresourced African languages? Secondly, to identify and select the specific sentences or passages that support or refute a claim, assessing the quality and reliability of their evidence extraction, we explore How precisely can embedding models identify and extract relevant evidence from single document for claim verification? Finally, we ask To what extent can current LLMs correctly verify the factuality of claims? by asking LLMs to classify claims as SUPPORTED, REFUTED, or NOT ENOUGH INFORMATION. The dataset covers two domains: health and culturenews.

Data Curation and Structure

The dataset is structured around two main splits: AfrIFact-Health, which focuses on the factuality of healthcare information, particularly common diseases, recurrent outbreaks, and public health concerns, and AfrIFact-Culture-News, which focuses on cultural knowledge and news. The language coverage includes Amharic, Hausa, Igbo, Oromo, Shona, Swahili, Twi, Wolof, Yoruba, and isiZulu. For each selected language native-speaking annotators were recruited through an African AI community. The data construction pipeline for AfrIFact-Health involves steps such as Manual Filtering, Claim Generation and Evidence Extraction, and Claim Labeling.

Evaluation Methodology

The evaluation addresses three core research questions: information retrieval effectiveness (RQ1), document-level evidence extraction precision (RQ2), and LLM fact-checking accuracy (RQ3). For information retrieval, the study uses nDCG@10 as the primary metric and also reports Recall@100 in the Appendix. For evidence extraction, nDCG@3 is reported as the primary metric, with Recall@3 in the Appendix given the reduced search space for this task. Claim labeling involves asking LLMs to classify claims as SUPPORTED, REFUTED, or NOT ENOUGH INFORMATION.

Model Evaluation and Findings

The evaluation compares diverse models including embedding-based and lexical retrieval models. Analysis of Information Retrieval Capability shows that embedding models lack cross-lingual retrieval capabilities, and that cultural and news documents are easier to retrieve than healthcaredomain documents, both in large corpora and in single documents. Regarding LLMs, the findings indicate that "LLMs lack robust multilingual fact-verification capabilities in African languages, while few-shot prompting improves performance by up to 43% in AfriqueQwen-14B, and task-specific finetuning further improves fact-checking accuracy by up to 26%".

Evidence Extraction and Fact Checking Analysis

In Evidence Extraction Capability, the study observed a large gap between English and African languages, except in the case of Shona, for the health domain. The Evidence extraction gap between English and African languages is particularly pronounced in the Health domain compared to the Culture and News domains. For Fact Checking Capability, Zero-shot fact verification remains extremely challenging across languages, while Few-shot examples often yield dramatic improvements of up to 43 points for models such as AfriqueQwen-14B. The study concludes that A small scale dataset, coupled with parameter efficient finetuning, can help improve the performance of models.

Conclusion and Future Directions

The work introduces AfrIFact as a benchmark for low-resource information retrieval, evidence retrieval, and fact checking. The results reveal systematic gaps and challenges in culture-specific information retrieval, evidence retrieval, and fact verification, encouraging researchers to address these problems by expanding dataset coverage and creating robust systems that aim to eliminate misinformation >. The focus of this work is on creating a substantially larger test set to support robust, high-quality, human-generated, and cross-lingual evaluation.

Limitations

The limitations include the fact that Language inclusion in this dataset was guided by several factors, and the focus on health and culture-news domains might not cover all areas of fact-checking. Furthermore, the scope is limited to introducing dataset creation and benchmarking, overlooking crucial steps such as hyperparameter optimization, data augmentation, and in-depth analysis of trained models. The design choice to create a substantially larger test set forced the creation of a smaller training and validation set due to resource constraints.

Ethical Considerations

All annotators, language coordinators, and meta-annotators were compensated fairly in accordance with the agreement. The source data for AfrIFact is collected from publically available resources: AfriDocMT (health documents), Wikipedia (cultural content), and XLSum (news articles). No private or personally identifiable information has been collected. The claims were generated by annotators from public data. All contributors involved in the work received training in fact-checking and verification, along with video and textual guidelines for using the tools at each stage of the data annotation process. The annotation team consists of 30 annotators, 10 language coordinators, 3 meta-annotators, and 2 administrative assistants.

Acknowledgments

This dataset was created with support from Lacuna Fund and Google.org. Israel Abebe Azime is funded by the German Federal Ministry of Education and Research and the German federal states (http://www.nhr-verein.de/en/our-partners) as part of the National High-Performance Computing (NHR) joint funding program. The views expressed herein do not necessarily represent those of Lacuna Fund, its Steering Committee, its funders, Meridian Institute or CENIA.

References

The references listed include work on Automated Fact Checking (AFC) by Vlachos and Riedel, Information Retrieval (IR) classical evidence retrieval in AFC courses by Zheng et al., and the creation of datasets like X-FACT, XFEVER, and MultiClaim. The work also references models such as Qwen3-Embedding and various fine-tuning techniques like LoRA and QLoRA, as well as foundational papers on LLM dangers by Bender et al. The paper also cites work on cross-lingual information retrieval for African languages such as AfriCLIRMatrix and MultiFC. The dataset construction is inspired by AVeriTeC, and the paper also references work on sentence embeddings like SentenceBERT. The paper also cites work on fact-checking in general, such as FEVER and Llamafactory for efficient fine-tuning.

References (Continued)

The references include papers on the specific languages included in the dataset, such as those related to Amharic, Hausa, Igbo, Oromo, Shona, Swahili, Wolof, Yoruba, and isiZulu <ref:2604.

Improvements for AI systems

Cross-lingual Information Retrieval Improvement: Implement a mechanism that explicitly addresses embedding model limitations in cross-lingual retrieval by developing specialized retrieval strategies for underresourced African languages, as the paper notes that even the best embedding models lack cross-lingual retrieval capabilities.
Domain-Specific Evidence Extraction Refinement: Develop domain adaptation techniques to improve evidence extraction accuracy, specifically targeting the observed performance gap where English African Language evidence retrieval gap is larger on health compared to culturenews.
LLM Fact Verification Enhancement via Prompt Engineering: Utilize few-shot prompting strategies as shown in the results, which improves performance by up to 43% in AfriqueQwen-14B, to enhance the zero-shot fact verification capabilities of large language models across African languages.
Model Fine-tuning Strategy Optimization: Employ parameter efficient fine-tuning methods like QLoRA 8-bit quantization, which showed the biggest improvement for the AfrIFact-Health domain, with a 26% average accuracy improvement across African languages, to adapt base models effectively to the specific nuances of cultural and health domains.
Evidence Reliance Assessment: Design a verification layer that analyzes model behavior regarding evidence reliance, recognizing that Providing evidence does not consistently improve fact-checking accuracy in some cases, suggesting models often rely more on their internal knowledge than on explicitly provided evidence.

Abstract

Assessing the veracity of a claim made online is a complex and important task with real-world implications. When these claims are directed at communities with limited access to information and the content concerns issues such as healthcare and culture, the consequences intensify, especially in low-resource languages. In this work, we introduce AfrIFact, a dataset that covers the necessary steps for automatic fact-checking (i.e., information retrieval, evidence extraction, and fact checking), in ten African languages and English. Our evaluation results show that even the best embedding models lack cross-lingual retrieval capabilities, and that cultural and news documents are easier to retrieve than healthcare-domain documents, both in large corpora and in single documents. We show that LLMs lack robust multilingual fact-verification capabilities in African languages, while few-shot prompting improves performance by up to 43% in AfriqueQwen-14B, and task-specific fine-tuning further improves fact-checking accuracy by up to 26%. These findings, along with our release of the AfrIFact dataset, encourage work on low-resource information retrieval, evidence retrieval, and fact checking.