Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context

arXiv:2605.13831 · cs.CV · Submitted 2026-05-13 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context".

Jane: Here's the gist of the paper. The gist: This work presents a systematic study of long-context continued pre-training for vision-language models, introducing MMProLong, which improves long-document VQA performance by 7.1% and maintains strong performance at contexts beyond 128K without additional training.

Tom: Systematic Study of LongPT The paper presents a systematic study of long-context continued pre-training (LongPT) for LVLMs, extending a 7B model from 32K to 128K context with extensive ablations on long-document data.

Jane: The authors first show that "long-document VQA is substantially more effective than OCR transcription".

Tom: Building on this observation, the ablations yield three key findings regarding long-context ability.

Title and authors: Jane: Now, let's talk about the title and who wrote this: "Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context <ref:2605.13831#pg1,Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context>." Who are the authors, Tom?

Tom: The team is a group of researchers from places like HKUST and ByteDance Seed. They are focused on how to make these models work well across much longer sequences than what we usually test them on.

Lu: It’s interesting because they aren't just looking at one small context length; they are testing the generalization ability beyond 128K tokens <ref:2605.13831#pg1>.

Jane: So, what does that mean for us right now? It suggests that a training recipe isn't fixed; it needs to be systematic enough to push these models past those standard benchmarks.

Meng: If you can train something on 128K and it still performs well at 256K or even 512K without retraining, that makes scaling much more predictable for production systems.

Lalam: The authors are trying to fill a gap in the literature where practical recipes for developing long-context vision-language models are still quite unexplored.

The paper's summary: Tom: So, the summary of this work is that they conduct a systematic study of LongPT, extending a 7B model from 32K to 128K context using long-document data and doing extensive ablations to find out what works <ref:2605.13831#pg1,extending a 7B model from 32K to 128K context>.

Jane: What’s the key finding they point out right away? They show that long-document VQA is substantially more effective than just using OCR transcription for training purposes.

Lu: That comparison is important because it suggests that having a high-quality, instruction-formatted supervision signal from the document itself is much better for learning long context.

Meng: I need to know what they are actually doing with the data synthesis part then, beyond just picking documents. Are they just feeding raw text in?

Tom: No, they construct five specific training tasks using a large pool of over one point five million PDF-formatted documents, selecting documents that are between thirty-two and fifty pages long.

Lalam: For the long-document VQA part, they use a segment-level pipeline to generate QA pairs from short, coherent page segments and then place that back into the full document context for the model to learn from.

The paper's improvements: Jane: So, what are the specific improvements or findings they highlight from their ablations? What did they discover about how to structure this training?

Tom: They found three key things. First, for the sequence length distribution, balanced data outperforms data that is strictly focused on a target length like 128K <ref:2605.13831#pg1>.

Lu: That’s a crucial point because it means we don't need to force every single training sample to be exactly 128K long; mixing different lengths seems better <ref:2605.13831#pg1>.

Meng: How does that translate to deployment? Does it mean our pipeline needs to generate samples of varying lengths, or is there a simpler way for us to curate our data?

Tom: They also found an eight:two mixture between extraction tasks and reasoning tasks yields the best overall scores, which they say preserves task diversity while maximizing performance <ref:2605.13831#pg3>.

Jane: And what about how far this works? The most impressive part is that MMProLong improves long-document VQA scores by seven point one percent at both the 64K and 128K contexts.

Conclusion: Tom: So, to wrap up, the core takeaway from "Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context" is that we have a practical LongPT recipe called MMProLong <ref:2605.13831#pg1,Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context>.

Jane: This recipe takes Qwen2 point 5-VL-7B and extends its context window to 128K, achieving a seven point one percent lift in long-document VQA performance compared to previous setups without needing extra training for contexts beyond that point <ref:2605.13831#pg1>.

Lu: The implication here is that we have an empirical foundation for advancing long-context vision-language models with data-efficient and transferable recipes based on the LongPT study.

Meng: From a deployment standpoint, this suggests a very scalable path toward training these models without needing massive amounts of task-specific supervision for every new long context scenario.

Lalam: The study also shows that this recipe generalizes to broader multimodal tasks, like webpage-based needle-in-a-haystack evaluation and long-context vision-text compression, even without task supervision.

Tom: We’ve covered the title, the summary, the specific recipe improvements like MMProLong, and what this means for pushing context lengths beyond 128K <ref:2605.13831#pg1>. That’s what we had on this paper.

Zhaowei Wang, Lishu Luo, Haodong Duan, Weiwei Liu, Sijin Wu, Ji Luo, Shen Yan, Shuai Peng, Sihang Yuan, Chaoyi Huang

Department of Computer Science, Hong Kong University of Science and Technology Department of ByteDance Seed

cs.CV

Submitted: 2026-05-13

Updated: 2026-10-18

Comments: Accepted at NeurIPS 2026. Code: https://github.com/zhaowei-wang-nlp/MMProLong

Code: https://github.com/zhaowei-wang-nlp/MMProLong

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 82/100

The gist: The gist: This work presents a systematic study of long-context continued pre-training for vision-language models, introducing MMProLong, which improves long-document VQA performance by 7.1% and

Key concepts

LongPT
Long-context continued pre-training is a technique used to train vision-language models on very long sequences of text and images. It involves continuing the model's training process beyond standard short context limits to enable it to understand and reason over massive documents.
MMProLong
MMProLong is the specific LongPT recipe developed for this study. It uses a combination of synthesizing data from long-document VQA tasks and natural sampling from a large document pool, specifically optimized for achieving high performance on long contexts up to 128K tokens.
Long-Document VQA
This task involves asking questions about entire documents rather than single images or short passages. The training data is synthesized by taking small, coherent page segments and generating QA pairs from them, placing those questions within the context of the full document for the model to answer.
Generalization Beyond 128K
This refers to the ability of a trained model to perform well on contexts significantly longer than what it was explicitly trained on. MMProLong demonstrates that this training method allows models to maintain strong performance at extended lengths like 512K tokens without needing additional, task-specific fine-tuning.

Terminology

Summary

The gist: This work presents a systematic study of long-context continued pre-training for vision-language models, introducing MMProLong, which improves long-document VQA performance by 7.1% and maintains strong performance at contexts beyond 128K without additional training.

Systematic Study of LongPT

The paper presents a systematic study of long-context continued pre-training (LongPT) for LVLMs, extending a 7B model from 32K to 128K context with extensive ablations on long-document data. The authors first show that long-document VQA is substantially more effective than OCR transcription. Building on this observation, the ablations yield three key findings regarding long-context ability.

Data Curation and Task Synthesis

The study constructs training data using a large document pool comprising over 1.5 million PDF-formatted documents, selecting documents with 32 to 50 pages to yield multimodal sequences ranging from 32K to 128K tokens. The five training tasks constructed are grouped into two categories: long-document VQA and OCR transcription. For long-document VQA data synthesis, a segment-level pipeline is used to generate a QA pair from a short, semantically coherent page segment and place it back into the full-document context. The three tasks of long-document VQA cover increasing evidence complexity: (i) single-page extraction (extract-single), (ii) multi-page extraction (extract-multi), and (iii) reasoning >.

LongPT Recipe Design

The final LongPT recipe, MMProLong, is obtained by longcontext continued pre-training from Qwen2.5-VL-7B with only a 5B-token budget. The recipe utilizes the following configuration: long-document VQA as the primary source of synthesizing data, natural sampling from the document pool over the target length range [32K, 128K], and an 8: 2 extraction-to-reasoning mixture.

Empirical Results and Generalization

MMProLong improves long-document VQA scores by 7.1% at 64K and 128K contexts and maintains strong performance at 256K and 512K contexts without additional training or adaptation. Furthermore, the recipe generalizes to broader multimodal long-context tasks, including webpage-based needle-in-a-haystack evaluation, long-context vision-text compression, and long-video understanding without task-specific supervision. The results show that MMProLong improves the overall average score on Qwen3-VL-8B from 50.03 to 61.75 on MM-NIAH.

Conclusion

The study establishes a practical LongPT recipe and an empirical foundation for advancing long-context vision-language models. This suggests a practical path toward training long-context vision-language models with data‑efficient, transferable LongPT recipes.

[Scoring Rationale]:

In summary, the student’s answer provides a comprehensive overview of the paper's core contribution and findings in structured sections. The student successfully identified the main finding (MMProLong's performance gain) and detailed the methodology (data synthesis tasks, recipe design) and generalization capabilities (extrapolation to longer contexts and other tasks). All required elements are present, covering the systematic study, key findings from ablations, the final recipe components, and empirical validation.

[Score]: 1 points

[JSON]:

"""

The gist: This work presents a systematic study of long-context continued pre-training for vision-language models, introducing MMProLong, which improves long-document VQA performance by 7.1% and maintains strong performance at contexts beyond 128K without additional training.

Systematic Study of LongPT

The paper presents a systematic study of long-context continued pre-training (LongPT) for LVLMs, extending a 7B model from 32K to 128K context with extensive ablations on long-document data. The authors first show that long-document VQA is substantially more effective than OCR transcription. Building on this observation, the ablations yield three key findings regarding long-context ability.

Data Curation and Task Synthesis

The study constructs training data using a large document pool comprising over 1.5 million PDF-formatted documents, selecting documents with 32 to 50 pages to yield multimodal sequences ranging from 32K to 128K tokens. The five training tasks constructed are grouped into two categories: long-document VQA and OCR transcription. For long-document VQA data synthesis, a segment-level pipeline is used to generate a QA pair from a short, semantically coherent page segment and place it back into the full-document context.

Improvements for AI systems

  1. LongPT Recipe Implementation: Adopt MMProLong to improve performance by 7.1% on long-document VQA while maintaining strong performance at 256K and 512K contexts without additional training.

  2. Data Mixture Strategy: Utilize an 8:2 extraction-to-reasoning mixture for long-context data, as this configuration achieves the best overall scores and preserves task diversity by retaining a small amount of reasoning data.

  3. Context Generalization: The resulting system can generalize to longer contexts up to 512K without further training or adaptation, and transfer capability to diverse tasks like webpage-based multimodal needle retrieval, long-context vision-text compression, and long-video understanding.

Abstract

Long-context modeling is becoming a core capability of modern large vision-language models (LVLMs), enabling sustained context management across long-document understanding, video analysis, and multi-turn tool use in long-horizon agents. Yet practical training recipes remain insufficiently explored, particularly for designing and balancing long-context data mixtures. In this work, we present a systematic study of long-context continued pre-training (LongPT) for LVLMs, extending a 7B model from 32K to 128K context with extensive ablations on long-document data. We first show that long-document VQA is substantially more effective than OCR transcription. Building on this observation, our ablations further yield three key findings: i) for sequence-length distribution, balanced data outperforms target-length-focused data (e.g., 128K), suggesting that long-context ability requires generalizable key-information retrieval across various lengths and positions; ii) retrieval remains the primary bottleneck, favoring retrieval-heavy mixtures with modest reasoning data for task diversity; iii) pure long-document VQA largely preserves short-context capabilities, suggesting that instruction-formatted long data reduces the need for short-data mixing. Based on these findings, we introduce MMProLong, obtained by long-context continued pre-training from Qwen2.5-VL-7B with only a 5B-token budget. MMProLong improves long-document VQA scores by 7.11% and maintains strong performance at 256K and 512K contexts beyond its 128K training window, without additional training. It further generalizes to webpage-based multimodal needle retrieval, long-context vision-text compression, and long-video understanding without task-specific supervision. Overall, our study establishes a practical LongPT recipe and an empirical foundation for advancing long-context vision-language models.

Sources

Related papers