When Models Misread Ink: Auditing VLMs for Arabic Calligraphy Recognition

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "When Models Misread Ink".

Jane: The gist The main finding is that current VLMs exhibit culturally structured errors in Arabic calligraphy recognition, specifically defaulting to Thuluth and failing to recover Diwani,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper called "When Models Misread Ink: Auditing VLMs for Arabic Calligraphy Recognition". It’s by Niaz Mohaiman Abtahi, Evan Maksud, and Reyana Shahid. The main point here is that while these VLMs are supposed to be smart, they struggle with the fine-grained details of Arabic calligraphy because they don't really test those script traditions enough.

Jane: Exactly. What the authors claim is that their audit on twelve different Vision-Language Models shows that these systems aren't just making random mistakes; their errors are actually structured in a cultural way, specifically by defaulting to certain styles and failing to recognize others.

Lu: That’s right. The core finding is this default-label collapse, where every single numeric prompt run gets dominated by one label taking thirty-three to sixty-three percent of the predictions instead of sticking close to the actual numbers.

Meng: So if we're talking about what that actually means for deployment, it sounds like these models are just leaning on the most common style, which is Thuluth, way more than they should be.

Lalam: And they’re failing to recover Diwani when prompted for it because of this bias.

Tom: That’s the gist of it. The paper uses a cleaned one thousand one hundred ninety-five image six-way benchmark with numeric, visual cue, and Diwani-first prompts to test twelve different VLMs. And their best numeric prompt accuracy for closed and API models is sixty point nine percent, but for open weight models it drops to forty seven point two percent.

Jane: That drop in accuracy is significant when you think about how crucial these styles are culturally, as the abstract points out. It’s not just a technical error; it’s a cultural one.

Lu: The dataset itself is interesting too. They took DuwatBench and added images from RuFa for Ruq'ah and Nasta'liq, plus HICMA images for Kufic and Naskh to balance things out because the original set was too heavy on Thuluth.

Meng: So they tried to fix the imbalance before testing the models, which is a good first step, but it still seems like Thuluth is winning way too often.

Lalam: The final counts after curation show Kufic with one hundred six images and Naskh with two hundred eighteen, while Diwani has two hundred seventeen images.

Tom: And the main failure mode they found is that Thuluth is over-predicted in eleven of twelve numeric prompt runs. For closed and API models, they see Thuluth chosen two point zero five to two point nine three times its true frequency.

Jane: That multiplier suggests a pretty strong default bias, which is what the authors call default-label collapse. It’s not just a slight blip; it's a systematic over-prediction of one style.

Lu: They found that false Thuluth predictions most often come from true Diwani images, which makes sense because they are visual neighbors, but it also points to an uncertainty default setting in the model.

Paper summary: Meng: If I were deploying this, I’d be concerned about that uncertainty default; we need models that say "I don't know" instead of just guessing the most frequent thing.

Lalam: The paper shows that for eight of those twelve models, the default label is Thuluth. Even GPT-4o labels fifty-two point nine percent of all images as Thuluth, even though Thuluth only makes up eighteen point one percent of the set <ref:2610.22018#pg1,of all images as Thuluth>.

Tom: That gap between the default prediction and reality is what’s so telling. It shows that prompt descriptions alone aren't enough for these systems to handle this level of fine-grained distinction.

Jane: And when they tried prompting interventions, things got a bit messy. Visual cues actually moved accuracy by more than ten points for some models, like GPT-4o, but they hurt others, like GPT-five point six Terra which dropped accuracy by seven point three <ref:2610.22018#pg3>.

Lu: It suggests that the visual context matters immensely, but it also shows that what helps one model might actively confuse another model's internal representation.

Meng: If we’re trying to build a robust system, we can't just throw in prompts and hope for the best; we need consistency across different architectures.

Lalam: And foregrounding Diwani often didn't fix the problem; instead, it actually relocated the default label, meaning it might have pushed Thuluth even higher in some cases.

Tom: So what this means for us is that Arabic calligraphy recognition is a compact stress test for VLM robustness. It’s a way to measure which visual traditions these automated systems can actually make legible.

Jane: Right, and the implications are that we need to move beyond just measuring overall accuracy and start validating performance per style. Systems should report their uncertainty when they struggle with these specific cultural nuances instead of just giving a single label.

Lu: The authors are pointing toward the fact that these script styles encode cultural, historical, and religious contexts, so collapsing less visible traditions into more familiar labels is a problem we need to solve in the future.

Meng: From an engineering standpoint, if a model can’t reliably distinguish between Diwani and Thuluth at scale, it won't be ready for anything important. It's about reliability in production environments.

Lalam: So the final conclusion of "When Models Misread Ink: Auditing VLMs for Arabic Calligraphy Recognition" is that this task offers a compact, culturally grounded benchmark for measuring not just model accuracy but how errors redistribute visibility across significant visual traditions.

Tom: That’s the main point. It gives us a specific metric to see exactly where the blind spots are in current AI models when they encounter complex cultural imagery like Arabic calligraphy.

Jane: It shifts the focus from just getting a high overall score to understanding *why* a model fails, which is much more useful for improving these systems for real-world use.

Conclusion: Tom: So we're wrapping up this deep dive into "When Models Misread Ink: Auditing VLMs for Arabic Calligraphy Recognition". The paper looks at how current vision-language models handle Arabic calligraphy, and the main thing they found is that these models have these structured errors based on the cultural styles of the script.

Jane: Right, so it's not just about a few wrong guesses; it's about a pattern. It’s showing that when you test these models on different Arabic scripts—like Diwani or Thuluth—they default to one style way too often and can’t recover the others when you ask them to.

Lu: Exactly, it’s a default-label collapse issue. The authors audited twelve different vision-language models on a dataset of over a thousand images, and they found that for most of them, Thuluth is being picked as the answer way too much.

Meng: From an engineering standpoint, that's worrying because if we deploy these models, we can't rely on them just picking the most common style without understanding why it's defaulting.

Lalam: I agree with Meng; the system needs to report its uncertainty instead of just giving a single label when it’s struggling with a fine-grained cultural task like this.

Tom: So what does this mean for Arabic calligraphy in general? The paper suggests that these script styles carry cultural, historical, and religious context.

Jane: That context is huge, Tom. When an AI collapses the Diwani or Naskh into a default Thuluth label, it’s not just a bad prediction; it’s flattening a visual tradition.

Lu: It shows that for these systems to be useful in the real world, they need to learn to respect those less visible traditions instead of just sticking to the most common one because that's what the data says.

Meng: And if we can't get them right on something this compact and culturally grounded, then we have a bigger problem when they try to recognize more complex visual information in production.

Lalam: The conclusion is that Arabic calligraphy serves as a useful stress test for these vision-language models, showing us exactly where their legibility breaks down across different visual traditions.

Tom: It’s definitely a compact benchmark because it forces the model to distinguish between very similar scripts, which is tougher than just recognizing objects in general.

Jane: And it shifts the focus away from just chasing a high overall accuracy score toward actually validating how well the AI understands these specific cultural nuances.

Lu: So what comes next for this area? The authors hint that future work needs to move beyond simple prompting and into more robust ways for models to represent these distinct visual styles internally.