MasalBench: A Benchmark for Contextual and Cross-Cultural Understanding of Persian Proverbs in LLMs

arXiv:2601.22050 · cs.CL · Submitted 2026-01-29 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "MasalBench: A Benchmark for Contextual and Cross-Cultural Understanding of Persian Proverbs in LLMs".

Jane: The gist The study introduces MasalBench, a comprehensive benchmark designed to probe the contextual and cross-cultural understanding of Persian proverbs by LLMs.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: : So we’re wrapping up this look at MasalBench, which is this new benchmark for testing how well large language models understand Persian proverbs in context and across cultures.

Jane: : Right, so what’s the main thing we should know about the paper itself? What is its final message?

Tom: : The authors are focused on showing that while models can handle Persian proverbs within a local conversation, they struggle when asked to map those concepts onto an equivalent proverb in English.

Lu: : They created this benchmark to test if these models can manage that kind of cross-cultural abstraction, moving past simple memorization of definitions <ref:2601.22050#pg3>.

Meng: : From a deployment angle, it seems like we need better methods for teaching cultural analogy rather than just more data in the same language.

Lalam: : The paper shows that they perform well at Persian proverbs when they are in context within a Persian conversation, achieving accuracies above zero point nine zero <ref:2601.22050#pg1>.

Jane: : But when we test them on finding the equivalent English proverb, the performance drops to about zero point seven nine accuracy for the best model <ref:2601.22050#pg1>.

Tom: : That difference shows a real weakness in their cross-cultural reasoning capabilities when it comes to these specific cultural knowledge domains <ref:2601.22050#pg3>.

Lu: : It really shows that there’s a gap between understanding the language structure itself and applying that knowledge across different cultural frameworks.

Meng: : So for us, it means focusing on how we train models to reason across cultures, not just how well they predict the next word in Persian <ref:2601.22050#pg3>.

Jane: : It suggests that current models are good at local context but need significant work on deep, abstract cross-cultural reasoning for real utility.

Tom: : Exactly, and MasalBench gives us a concrete framework for assessing that specific kind of cultural understanding in other low-resource languages we might be looking at <ref:2601.22050#pg1>.

Conclusion: Tom: So we’re wrapping up our look at MasalBench, which is this new benchmark for testing how well large language models understand Persian proverbs in context and across cultures.

Jane: Right, so what’s the main thing we should know about the paper itself?

Tom: The authors are looking at a specific problem because we already know these models struggle with cultural knowledge and analogy when they move between languages.

Lu: They created this benchmark to see if the models can handle that kind of cross-cultural abstraction, not just memorizing definitions.

Meng: I’m curious about what it actually means for deployment, though I’m skeptical about their testing environment.

Lalam: Basically, the results show that the models are pretty good at Persian proverbs when they are in context within a Persian conversation, hitting above zero point nine zero accuracy in those tasks.

Jane: But what happens when you ask them to translate that same idea into an equivalent English proverb?

Tom: That’s where things get interesting, because the performance drops significantly there; the best model only hit about zero point seven nine accuracy for that task.

Lu: It really shows a gap between how well they understand the language structure itself and their ability to apply that knowledge across different cultural frameworks.

Meng: So what does this tell us about where we need to focus our efforts when we build these next generation models?

Jane: It suggests that current models are strong at understanding the local context but weak when it comes to deep, abstract cross-cultural reasoning.

Tom: Exactly, and MasalBench gives us a framework for assessing that specific kind of cross-cultural understanding in other low-resource languages.

Lalam: It’s a tool for figuring out if an AI truly gets the cultural layer of something beyond just pattern matching.

Jane: And it opens up a lot of avenues for research into building models with better cultural awareness, which is what we need to talk about next.

Ghazal Kalhor, Behnam Bahrak

School of Electrical and Computer Engineering, University of Tehran · Tehran Institute for Advanced Studies, Khatam University

cs.CL

Submitted: 2026-01-29

Updated: 2026-10-14

Comments: Accepted and forthcoming at the First Workshop on Pluralistic Value Alignment of LLMs (PlurVA-LLM 2026) at AACL-IJCNLP 2026

Code: https://github.com/kalhorghazal/MasalBench

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 73/100

The gist: The gist The study introduces MasalBench, a comprehensive benchmark designed to probe the contextual and cross-cultural understanding of Persian proverbs by LLMs.

Key concepts

MasalBench: A comprehensive benchmark designed to assess the contextual and cross-cultural understanding of Persian proverbs by LLMs. It involves two main tasks: understanding proverbs within a dialogue context and identifying their equivalent meaning in English.
Contextual Understanding: This task tests an LLM's ability to correctly interpret a Persian proverb within a given conversation. The benchmark uses multiple-choice questions where distractors are designed to be literal traps or plausible but incorrect meanings, measuring how well the model grasps the intended meaning in that specific situation.
Cross-Cultural Understanding: This task evaluates an LLM's ability to identify the English equivalent of a Persian proverb. The benchmark uses binary choices where one option is the correct equivalent, and distractors are crafted to sound similar in wording or tone, testing the model's grasp of cultural analogy.

Terminology

Benchmark — 2× in this paper · explained in 2 episodes
Low-Resource Languages — 2× in this paper · explained in 4 episodes
Accuracy — 1× in this paper · explained in 1 episode
Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes

Summary

The gist The study introduces MasalBench, a comprehensive benchmark designed to probe the contextual and cross-cultural understanding of Persian proverbs by LLMs.

Introduction and Motivation

Multilingual Large Language Models (LLMs) require mastery of conversational language rules to communicate effectively with users, and Persian proverbs are a key component of conversation in this low-resource language In contrast, the scarcity of such data in low-resource languages may limit LLM proficiency with their proverbs Persian is one of the low-resource languages for multilingual LLMs, yet it is exceptionally rich in the use of proverbs, both historically and in contemporary contexts To address this gap, MasalBench is introduced as a comprehensive benchmark for evaluating LLMs’ understanding of Persian proverbs at both contextual and cross-cultural levels.

Benchmark Construction

The benchmark construction involved two main tasks: contextual understanding and cross-cultural understanding.

For the contextual level, a set of 1,000 multiple-choice questions in dialogue form was proposed, where each distractor option served a specific purpose: a literal trap, a plausible but incorrect meaning, and an irrelevant meaning. This dataset was created by manually designing short dialogues for 100 proverbs and automating the design of distractors for the remaining proverbs.
For the cross-cultural level, 700 binary-choice questions were proposed to assess whether LLMs can identify the English equivalent of a given Persian proverb. The equivalent proverb was crafted to be most similar in meaning and usage, while the irrelevant proverb shared similar wording, tone, or imagery to serve as a convincing distractor.

Experimental Setup

The evaluation involved eight prominent multilingual LLMs of varying sizes with proficiency in Persian. The models evaluated included Llama 4 Scout (AI, 2025), Llama 3.3 70B Instruct (Meta AI, 2024), Qwen 2.5 72B Instruct (Qwen Team, 2025a), Qwen QwQ 32B (Qwen Team, 2025b), DeepSeek V3.1 (DeepSeek, 2025), DeepSeek R1 (DeepSeek-AI, 2025), GPT4.1 mini (OpenAI, 2025), and GPT-4o mini (OpenAI, 2024). The prompting setup employed zero-shot prompting with a temperature set to 0 for deterministic responses and top-p set to 1 so that all possibilities were considered.

Main Findings

The results indicated that multilingual LLMs perform strongly in understanding Persian proverbs in context, with model size and instruction tuning being key contributors. When errors occurred, they predominantly involved plausible distractors, reflecting the models’ tendency to select semantically coherent interpretations. In contrast, identifying the correct English equivalent of a Persian proverb proved more challenging, and although instruction tuning provided some benefit, its impact was limited due to the need for cultural knowledge and analogical reasoning.

Conclusion

MasalBench demonstrates that while LLMs are relatively strong at intra-language contextual reasoning, they are weaker at cross-cultural abstraction. The findings shed light on current limitations and offer a foundation for advancing research on cross-lingual and culturally informed language understanding. The study has several limitations, including relying solely on Gemini 2.5 Pro for benchmark construction and the inherent challenge of finding equivalent English proverbs.

References

Foote Koozegari: Persian Proverbs and Their Stories. Entesharat e Madreseh, Tehran, Iran

Gemini 2.5 Pro (Comanici et al., 2025)

DeepSeek V3.1 release notes

DeepSeek R1: Incentivizing reasoning capability in llms via reinforcement learning

Sundesh Donthi, Maximilian Spencer, Om Patel, Joon Yong Doh, Eid Rodan, Kevin Zhu, and Sean O'Brien. 2025. Improving llm abilities in idiomatic translation

Farhan Farsi et al., 2025. Melac: Massive evaluation of large language models with alignment of culture in persian language

Korosh Hadissi. 2010. A socio-historical approach to poetic origins of persian proverbs

Paria Khoshtab et al., 2024. Comparative study of multilingual idioms and similes in large language models

Jisu Kim et al., 2025. Memorization or reasoning? exploring the idiom understanding of llms

Chen Cecilia Liu et al., 2024. Are multilingual llms culturally-diverse reasoners? an investigation into multicultural proverbs and sayings

Meta AI. 2025. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation

OpenAI. 2024. Gpt4o mini: Advancing cost-efficient intelligence

OpenAI. 2025. Introducing gpt-4.1 in the api

Qwen Team. 2025a. Qwen2.5 technical report

Qwen Team. 2025b. Qwq-32b: Embracing the power of reinforcement learning

Mostafa Rahmandoost et al., 2011. Foote Koozegari: Persian Proverbs and Their Stories

Sara Rezaeimanesh et al., 2025. Large language models for persian-english idiom translation

Surendrabikram Thapa et al., 2025. Probing the limits of multilingual language understanding: Low-resource language proverbs as llm benchmark for ai wisdom

A Example Prompts

شما یک مدل زبانی هستید که برای ساخت مکالمه با کاربرد ضربالمثلهای فارسی طراحی شدهاید. به شما یک ضربالمثل فارسی و کاربرد آن داده میشود. وظیفه شما این است که یک مکالمهی کوتاه، طبیعی و محاورهای در دو بخش بین دو نفر بنویسید. نام شخصیتها باید ایرانی باشند. همچنین الزم است هدف نفر دوم از بهکار بردن ضربالمثل را توضیح دهید. خروجی را دقیقاً در قالب زیر ارائه کنید:

proverb>شب شراب نیرزد به بامداد خمار.>proverb/ >

usage>بسیاری از چیزهایی که در ابتدا خوب و لذتبخش و وسوسهانگیز به نظر میرسند، درنهایت پایانی ناخوش و تلخ دارند.>usage/>

English Translation You are a language model designed to generate conversations that make use of Persian proverbs. You are given a Persian proverb along with its usage description. Your task is to write a short, natural, and conversational dialogue in two turns between two speakers. In this dialogue, the second speaker must use the given proverb in response to the first speaker. The characters' names should be Iranian. Additionally, you must explain the intention behind the second speaker’s use of the proverb. Provide the output exactly in the following format:

first person": Javad: امیر واقعا دزدی کرده بود؟: person first

second person": سیروان: آش نخورده و دهن سوخته.: person second

third question": سیروان با این ضربالمثل چه منظوری داشت؟: "question

answer": ""

Improvements for AI systems

A robust evaluation framework for low-resource language proverbs should be developed by systematically comparing LLM performance across different model sizes and instruction tuning levels on both contextual and cross-cultural tasks, addressing the finding that model size and instruction tuning being key contributors to context understanding.
LLMs can be improved to better handle semantic coherence when making errors by training them on datasets that explicitly highlight the preference for plausible distractors, as this pattern is observed when models are uncertain about a proverb's meaning.
Cross-cultural understanding capabilities in LLMs can be enhanced by focusing training on cultural knowledge and analogical reasoning, as the paper notes that success in this task depends more on training exposure to cultural data than on improved instruction following.

Abstract

In recent years, multilingual Large Language Models (LLMs) have become an inseparable part of daily life, making it crucial for them to master the rules of conversational language in order to communicate effectively with users. While previous work has evaluated LLMs' understanding of figurative language in high-resource languages, their performance in low-resource languages remains underexplored. In this paper, we introduce MasalBench, a comprehensive benchmark for assessing LLMs' contextual and cross-cultural understanding of Persian proverbs, which are a key component of conversation in this low-resource language. We evaluate eight state-of-the-art LLMs on MasalBench and find that they perform well in identifying Persian proverbs in context, achieving accuracies above 0.90. However, their performance drops considerably when tasked with identifying equivalent English proverbs, with the best model achieving 0.79 accuracy. Our findings highlight the limitations of current LLMs in cultural knowledge and analogical reasoning, and they provide a framework for assessing cross-cultural understanding in other low-resource languages. MasalBench is available at https://github.com/kalhorghazal/MasalBench.