From prompting to evidence-based translation: A RAG+prompt system for Japanese-Chinese translation and its pedagogical potential

arXiv:2605.03387 · cs.CL · Submitted 2026-05-05 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "From prompting to evidence-based translation".

Jane: Here's the gist of the paper. The gist The RAG+Prompt translation system improves Ja→Zh translation of sentences containing NMCCs in an interpretable and auditable manner.

Tom: How it works The study evaluates a retrieval-augmented generation (RAG)+Prompt translation system that integrates linguistic analysis, embedding-based retrieval, prompt construction, and LLM generation without modifying the base model.

Jane: The analysis module outputs A1 (inner vs.

Tom: outer NMCC) and A2 (risk predictions: lexical choice/NMCC handling/word order/style/register); top-k = 5 similar Ja→Zh examples (L2 distance) and A1/A2 are inserted into an enhanced prompt.

Paper summary: Tom: So we’re looking at a paper called "From prompting to evidence-based translation: A RAG+prompt system for Japanese–Chinese translation and its pedagogical potential". This study looks at how to make large language models better when translating sentences that have those tricky noun-modifying clause constructions, or NMCCs.

Jane: That sounds really specific. So what’s the main idea here? What problem are they trying to solve with this RAG plus prompt system?

Tom: Basically, they found that large language models do pretty well on easy pairs, but they get unreliable when translating Japanese to Chinese sentences that have those noun-modifying clause constructions. They developed a system that doesn't change the base model itself but adds steps like linguistic analysis and retrieval to guide the AI better.

Lu: The core thesis is using this retrieval augmented generation RAG plus prompt translation system to improve Ja→Zh translation of sentences containing NMCCs in an interpretable and auditable manner, which is something they felt was missing in current methods.

Meng: Interpretable and auditable—that’s good for production, I guess. So how does this system actually work? What are the main components they built?

Tom: It integrates linguistic analysis, embedding-based retrieval, prompt construction, and then the LLM generation itself. The analysis module spits out two things: A1 which tells you if it’s an inner or outer NMCC, and A2 which predicts risks like word order issues or style problems.

Jane: So they aren't just asking the AI to translate; they are feeding it explicit evidence about the structure of the sentence first? That sounds like a smart way to handle those tricky grammatical dependencies.

Lalam: I see. The analysis module outputs A1 and A2, and then they grab top-k equals five similar Ja→Zh examples based on L2 distance from a knowledge base <ref:2605.03387#pg1,The analysis module outputs A1>. They then insert that A1 and A2 information into an enhanced prompt for the final generation step.

Tom: Exactly. The retrieval part uses embeddings generated by OpenAI’s text-embedding-ada-two to build an index, and they do a top-k nearest neighbor search using L2 distance, which is this similarity score calculation: one over one plus the distance squared > <ref:2605.03387#pg2>

Lu: And that means they are making sure the context window has both structural analysis and real examples of how similar things have been handled before. That should give the model much more grounding than just a bare prompt.

Jane: What’s interesting is that they test this using GPT-4o on a test set of sixty-six sentences, and they compare six different knowledge base sizes, ranging from zero up to two thousand examples <ref:2605.03387#pg1>.

Meng: Six knowledge base sizes? That implies there's an effect there. So what was the main result when they ran this experiment? What did the numbers show?

Paper summary: Tom: The sole metric they used is macro-averaged sentence-level BLEU, which is a one to four gram with brevity penalty, cased, and Chinese at the character level. The mean BLEU went from twenty-four point two eight at zero when RAG was disabled up to twenty-nine point nine six when they used two thousand examples, that’s a gain of five point six eight points or about twenty-three point four percent improvement > <ref:2605.03387#pg1>

Jane: So the upward trend held across all those sizes, meaning bigger knowledge bases actually led to higher translation scores. That suggests that having more context material really helps in this domain.

Lu: It supports the idea that for structurally complex sentences, simply adding more relevant external materials into the context window provides a tangible lift in both lexical precision and structural adaptation >

Meng: So if we think about deployment, what does that mean for running this system? Does it get heavy at scale? What’s the real cost of setting up this whole pipeline?

Tom: The paper mentions relying on GPT-4o as the base model, but they are testing it across six sizes <ref:2605.03387#pg1>. The engineering challenge is building this entire modular system—the analysis, retrieval, prompt construction—to be integrated without modifying the base model architecture itself >

Jane: It sounds like it’s moving from just asking a question to building a whole traceable workflow for translation. What does that imply for someone who only listens to the show?

Lu: It implies that students or users can move away from just submitting bare translation prompts and start building a personalized system with evidence-based prompting skills >

Meng: That’s an interesting shift in skill development, moving toward prompt literacy instead of just translation output. So what are the authors saying about where this is useful?

Tom: They emphasize that their system improves Ja→Zh translation of sentences containing NMCCs in an interpretable and auditable manner, which is the main claim they want to make >

Jane: So we’ve heard about how it works, and we’ve seen the results showing a measurable improvement tied to more knowledge. What do you think this means for future work with these types of complex language pairs?

Lu: Future work will focus on broadening genres, exploring different language pairs, and developing better evaluation metrics beyond just BLEU >

Meng: And they are acknowledging the limitations too. They explicitly state that they are relying on one base model, GPT-4o, one metric—BLEU—and knowledge bases compiled from published Ja→Zh materials > <ref:2605.03387#pg1>

Tom: So the caution is that this isn't a universal solution; it’s tied to those specific resources and models they used >

Jane: It’s a solid result for Ja→Zh translation of these hard structures, but we need more testing across different contexts to see how robust this method really is >

Conclusion: Tom: So we’ve been looking at this paper, "From prompting to evidence-based translation: A RAG+prompt system for Japanese–Chinese translation and its pedagogical potential."

Jane: It really boils down to taking that complex problem of translating Japanese sentences with noun-modifying clause constructions into something more structured and traceable.

Tom: Exactly. The authors are presenting this RAG plus prompt system, which basically layers linguistic analysis on top of a retrieval system before the AI even generates the translation.

Lu: What’s compelling is how they build that evidence chain. They aren't just guessing; they are explicitly telling the LLM what kind of grammatical relationship it’s dealing with, like whether it's an inner or outer noun-modifying clause.

Meng: From a deployment standpoint, I like that the system doesn't require retraining the base model itself. It’s an augmentation layer rather than a full model overhaul.

Lalam: I think what they show is that by providing those A1 and A2 risk predictions in the prompt, we get a much more reliable output because we’ve given it explicit instructions on potential lexical choice or word order problems beforehand.

Jane: That makes sense. So, when you look at the overall results, they show that this approach improves the quality of Japanese to Chinese translation on these tricky sentences.

Tom: The numbers support that too. They compared six different knowledge base sizes, and as we increased the size of those examples from zero up to two thousand the mean BLEU score went up significantly from twenty-four point two eight at zero to nearly thirty at two thousand examples.

Lu: That non-linear size effect is a key finding; larger knowledge bases really help with both lexical precision and adapting to the right sentence structure.

Jane: So for someone just listening, this means that when you’re using these advanced models for language tasks, you can start thinking about building that scaffolding around the AI instead of just asking one big question.

Tom: That’s the pedagogical side they focus on too. They see this as a way to teach students how to build evidence-based prompting skills rather than just submitting raw text into a prompt box.

Lu: It shifts the focus from simple translation tasks to understanding how to engineer prompts that are actually robust and auditable for complex grammar.

Meng: I guess for production, if we’re dealing with high-stakes content, having that audit trail of why the AI made a certain choice—based on those A1 and A2 outputs—is something I could actually use.

Jane: It sounds like this work lays a foundation for making LLM usage in specialized translation fields much more transparent and controllable.

Tom: It’s about moving from "get me a translation" to "here is the data, here is the structure, and here is my prediction of where things might go wrong."

Jane: And that's exactly what they achieved with this RAG plus prompt system for handling those noun-modifying clause constructions.

Wenshi Gu

Beihang University

cs.CL

Submitted: 2026-05-05

Updated: 2026-10-21

Comments: The authors have identified errors in the experimental data and have decided to withdraw this version of the manuscript

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 89/100

The gist: The gist The RAG+Prompt translation system improves Ja→Zh translation of sentences containing NMCCs in an interpretable and auditable manner.

Key concepts

NMCC: Noun-Modifying Clause Construction (NMCC) refers to a specific grammatical structure in Japanese sentences where a clause modifies a noun. This construction is often challenging for AI models because it requires deep understanding of how the modifying clause relates to the main noun, leading to potential translation errors.
RAG+Prompt System: This system combines linguistic analysis, retrieving similar examples from a database, and constructing detailed instructions (prompts) before feeding them to an LLM. It acts as an intelligent assistant that provides explicit evidence and task-specific guidance to guide the translation process accurately.
A1/A2 Analysis: The linguistic analysis module categorizes the NMCC type (A1: inner vs. outer relation) and predicts specific translation risks (A2: lexical choice, word order, style). These predictions are inserted into the prompt to help the LLM make more informed decisions during generation.
Knowledge Base Size Effect: The study found that performance improves non-linearly as the size of the retrieval knowledge base increases. Larger databases of Japanese-Chinese examples led to significantly higher BLEU scores, demonstrating that providing more relevant context substantially enhances translation accuracy for complex structures.

Terminology

RAG — 4× in this paper · explained in 3 episodes

Summary

The gist The RAG+Prompt translation system improves Ja→Zh translation of sentences containing NMCCs in an interpretable and auditable manner.

How it works

The study evaluates a retrieval-augmented generation (RAG)+Prompt translation system that integrates linguistic analysis, embedding-based retrieval, prompt construction, and LLM generation without modifying the base model. The analysis module outputs A1 (inner vs. outer NMCC) and A2 (risk predictions: lexical choice/NMCC handling/word order/style/register); top-k = 5 similar Ja→Zh examples (L2 distance) and A1/A2 are inserted into an enhanced prompt. This system is designed to improve translation quality by providing explicit evidence and task-specific instructions to the LLM.

The system follows a modular design with four core components: (1) linguistic analysis module; (2) embedding-based retrieval module; (3) prompt-construction module; and (4) translation generation module. The linguistic analysis module performs interpretable analysis of the NMCC in the source language to identify its type, and predicts potential translation errors in the target language. Specifically, it determines if an NMCC is an inner relation or outer relation (A1) and predicts risks such as lexical choice issues, NMCC handling issues, word-order errors, and style/register issues (A2).

Knowledge Base Construction and Retrieval

The embedding-based retrieval module is a key component that retrieves the Japanese example sentences most relevant to the current source language input together with their corresponding Chinese translations from a knowledge base. Sentence embeddings are generated using OpenAI’s text-embedding-ada-002 as the sole encoder, and an L2 (Euclidean) index is built over these embeddings to enable reproducible similarity search. At query time, the source language input is embedded to obtain a query vector in the same semantic space. The system performs top-k nearest-neighbor search with L2 distance (k=5) to identify relevant examples. A similarity score is calculated via the transformation similarity = 1 / 1 + d (1), which preserves strict monotonicity without altering the relative ranking of candidates.

Experimental Validation and Results

The empirical validation compares translation quality with and without RAG in Ja→Zh NMCC (Japanese→Chinese translation involving noun-modifying clause constructions). The sole evaluation metric used is macro-averaged sentence-level BLEU (1–4-gram with brevity penalty; cased; Chinese at the character level). The results show that mean BLEU increases from 24.28 at 0 (RAG disabled) to 29.96 at 2,000 (+5.68; +23.4%). The upward trend holds across sizes, with larger knowledge bases yielding higher scores.

Pedagogical Application

The system serves as a teaching aid that shifts students from submitting direct translation prompts to building a personalized RAG+Prompt translation system. This fosters prompt literacy and awareness of human–AI collaboration. The five-step process implemented in the classroom includes analysis, retrieval, construction, generation, and review. This involves students creating analysis worksheets and retrieval logs with justifications alongside successive prompt versions and review records to cultivate evidence-based prompting skills.

Discussion and Conclusion

The system’s advantages include the linguistic analysis module providing explicit judgments (A1 and A2) embedded into prompts for traceability, the embedding-based retrieval module creating a traceable evidence chain, and the prompt-construction module organizing inputs in a fixed order to stabilize outputs. Limitations include reliance on a single base model (GPT-4o), one metric (BLEU), and commercial APIs. The pedagogical value lies in creating an interpretable, auditable practice cycle where students develop translation skills and data governance through authentic tasks. Future work aims to broaden genres, language pairs, and evaluation metrics. The overall conclusion is that the RAG+Prompt translation system improves translation quality in an interpretable and auditable manner.

The paper is 2605.03387. The study was supported by the Educational Reform Project of Beihang University under the project Research on the Development of College Japanese Courses Oriented towards 'Engineering + Japanese'. The research was conducted by Wenshi Gu (guwenshi@buaa.edu.cn). It is important because it provides an interpretable and auditable method to improve the translation of structurally complex Japanese sentences containing noun-modifying clause constructions in the Ja→Zh direction. The findings demonstrate a non-linear size effect where larger knowledge bases substantially improve both lexical precision and structural adaptation in complex sentence translation. This research is significant because it operationalizes linguistic theory and translation knowledge into executable prompts, offering a transferable framework for cultivating evidence-based prompting skills in foreign language pedagogy. The methodology provides a unified technical foundation for assessing the empirical benefits of RAG+Prompt systems in this specific translation domain. The system's workflow coordinates analysis, retrieval, prompting, and generation to enhance accuracy and reproducibility. This work addresses the gap in existing research by focusing on Ja→Zh translation of sentences containing NMCCs. The results empirically support the system’s capacity to improve semantic fidelity and readability in Ja→Zh translation. This research is bounded by reliance on a single base model (GPT-4o), one metric (BLEU), and knowledge bases primarily compiled from published Ja→Zh materials. Future work will focus on replicating across genres and language pairs and improving knowledge-base governance. The research is a contribution to translation pedagogy by shifting the focus toward evidence-based prompting skills. It is a study on retrieval-augmented generation (RAG) and its application in translation and education. The paper is an empirical validation of the RAG+Prompt system’s contribution to improving translation quality. It is a study on Japanese–Chinese (Ja→Zh) translation involving noun-modifying clause constructions, i.e., sentences in Japanese containing relative clauses. The paper is a study on the effect of RAG knowledge base size on Ja→Zh translation quality. The paper is a study on how the size of the RAG knowledge base influences performance.

Improvements for AI systems

Retrieval-augmented generation (RAG) improves factual accuracy by incorporating task-relevant external materials into generation, which reduces hallucination in open-domain dialogue and translation tasks RAG incorporates task-relevant external materials into the generation process by concatenating evidence and prompts in the context window.
The system can perform linguistic feature analysis to output A1 (inner vs. outer NMCC) and A2 (risk predictions: lexical choice/NMCC handling/word order/style/register), providing an interpretable and auditable manner for translation, as stated in the abstract.
The system can improve translation quality by using a prompt-construction module that integrates linguistic analysis results, retrieved examples, and task instructions into a single enhanced prompt, reducing ambiguity and improving translation quality.
The AI system can be used as a teaching aid to foster prompt literacy by shifting students from submitting direct translation prompts to building a personalized RAG+Prompt translation system, which cultivates evidence-based prompting skills.
The system can dynamically scale performance based on knowledge base size, showing that larger knowledge bases yielding higher scores leads to substantial gains in macro-averaged sentence-level BLEU, peaking at 2,000 pairs.