TukaBench: A Culturally Grounded Jailbreak Benchmark for African Languages
cs.CL, cs.AI
Submitted: 2026-05-31
Updated: 2026-09-22
Comments: Accept to EMNLP 2026 (main conference)
License: http://creativecommons.org/licenses/by/4.0/
The gist: Safety evaluation of Large Language Models (LLMs) remains heavily English-centric, leaving Low-Resource Languages (LRLs), particularly African ones, critically underexplored.
Terminology
Abstract
Safety evaluation of Large Language Models (LLMs) remains heavily English-centric, leaving Low-Resource Languages (LRLs), particularly African ones, critically underexplored. We introduce TUKABENCH, a jailbreak benchmark for seven African languages that extends JailbreakBench (JBB) beyond direct translation through four settings: human translation of JBB prompts, English adaptation to African contexts followed by human translation, human-curated prompts validated through interactions with GPT-5.2, and code-switched prompts combining English and African languages, isolating the effect of language, cultural grounding, and prompt evasiveness on model safety. Across closed and open models, prompting in African languages reduces refusal relative to English, with culturally adapted prompts leading to least refusal. The evaluation also surfaces two structural limitations: model comprehension failures and reduced LLM-as-a-judge reliability in LRLs. To capture the first, we introduce Deflection alongside Refused and Jailbroken; to assess the second, we validate outputs with human annotations, showing that judge-human agreement drops in lower-resource languages and less commonly supported scripts.
Sources
- UbuntuGuard: A Culturally-Grounded Policy Benchmark for Equitable AI Safety in African Languages
- One Word at a Time: Incremental Completion Decomposition Breaks LLM Safety
- FinSafetyBench: Evaluating LLM Safety in Real-World Financial Scenarios
- Do Methods to Jailbreak and Defend LLMs Generalize Across Languages?
- Break Me If You Can: Self-Jailbreaking of Aligned LLMs via Lexical Insertion Prompting
- International AI Safety Report
- A Cross-Language Investigation into Jailbreak Attacks in Large Language Models
- Boundary Point Jailbreaking of Black-Box LLMs
- BabelSafe: A Policy-Grounded Multilingual Safety Benchmark for LLMs
- Universal and Transferable Adversarial Attacks on Aligned Language Models
- Structured Semantic Cloaking for Jailbreak Attacks on Large Language Models
- ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming
- Low-Resource Languages Jailbreak GPT-4
- AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering