Beyond Source Prompts: Towards Visually Grounded AI-Generated Video Quality Assessment

arXiv:2610.21758 · cs.CV · Submitted 2026-10-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Beyond Source Prompts".

Jane: The gist The proposed VGCap framework is an AIGV quality assessment framework guided by visually grounded captions that align better with generated content.

Tom: First, who's behind it and why it matters.

Paper summary: Jane: So wrapping up the discussion on "Beyond Source Prompts: Towards Visually Grounded AI-Generated Video Quality Assessment," it seems the authors are arguing that relying solely on source prompts for video quality assessment is insufficient because they don't reflect reality >

Tom: Right. The paper proposes VGCap, which uses visually grounded captions—a global one and segment-level ones—to bridge that gap between the intended text and the actual generated video >

Lu: The implication is that for AI video quality assessment to be useful, we need to move away from using just the initial source prompt as our primary textual reference >

Meng: So, what does this mean for us in production? Are we supposed to start building these captioning systems right now? >

Tom: Not necessarily overnight, but it shows that the next step in quality assessment needs to integrate visual evidence more deeply into the text used for judging >

Jane: It suggests that by incorporating features like those segment-level motion features and prompt-independent captions, we can get a much richer picture of spatial and temporal issues in AI videos >

Lu: The work provides a framework where you can assess quality across three distinct dimensions: spatial quality, temporal quality, and text-video alignment >

Meng: So it’s about getting better scores in all those areas instead of just focusing on one metric >

Tom: It's about making sure the assessment system understands both *what* is in the frame and *how* things move across time, which is what this paper does with VGCap >

Conclusion: Tom: So we’ve been looking at VGCap, which is this new framework for checking how good AI-generated videos are, and now we’re coming to the end of this segment with Tom and Jane wrapping up what it all means.

Jane: Right. It seems like the authors are basically saying that just using the initial text prompt to judge a video isn't enough anymore, because that text might not actually match what’s on screen.

Tom: Exactly. The paper proposes VGCap, which uses captions grounded in visual evidence—a global one and some segment-level ones—to check things more thoroughly.

Lu: What they did is create these two types of captions. A global caption that refines the original prompt using a few frames to see what the video is actually doing, and then segment captions that describe what’s happening in small chunks of time.

Meng: It sounds complicated from a training standpoint, but if it works at scale, it means we can get a better idea of where the AI is failing in the first place.

Jane: So for someone listening right now, this means we can move beyond just checking if the text matches; we can finally assess how well the AI captured both the look of things and how they moved over time.

Tom: It’s about getting a better picture of spatial quality, temporal quality, and whether the video actually looks like what it was supposed to be.

Lu: The experimental results on datasets like LGVQ and FETV show that this VGCap approach beats the other methods across all those three dimensions we are looking at.

Jane: So what does that actually change for us in production? It means if we want to build better video quality checks, we need to start thinking about using visual context to inform the text used for assessment.

Tom: Yeah, it pushes us toward systems where the text isn't just a starting point but something that has been visually verified against the output.

Lu: And their ablation studies on the components confirm that both types of captions—the whole video one and those local segment ones—are actually useful for predicting temporal quality.

Jane: That’s a big piece of information. It shows that focusing on localized motion features, described by those segment captions, is really helpful for understanding how time is handled in AI video generation.

Tom: So the core idea here is that we need more context than just the original source prompt to judge the quality of these videos.

Lu: The authors are proving that visually grounded captions give us better evidence for assessing spatial quality, temporal quality, and text-video alignment simultaneously.

Jane: It’s a solid framework built on refining the text using visual cues rather than relying solely on what was written at the very beginning.

Tom: So for anyone building these AI video tools, this paper points toward a method where we bake in that visual verification step right into the quality assessment pipeline.

Lu: We’ll be looking at how they plan to scale this caption construction process for real-time deployment next.

Kehao Zhou, Yixuan Li, Yuxuan Jiang, Fan Zhang

Visual Information Lab, University of Bristol

cs.CV

Submitted: 2026-10-15

Updated: 2026-10-15

Code: https://github.com/lolo123-123/VGCap

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 82/100

The gist: The gist The proposed VGCap framework is an AIGV quality assessment framework guided by visually grounded captions that align better with generated content.

Key concepts

Visually Grounded Captions: These are textual descriptions generated by analyzing the actual frames of an AI-generated video rather than just using the original text prompt. They serve as a bridge between the intended meaning of the source prompt and what is actually seen in the video, providing evidence for quality assessment.
Global Caption: This caption improves upon the original source prompt by minimally revising it based on visual evidence across multiple frames. It verifies and replaces parts of the prompt that contradict what is visually present in the video, creating a more accurate textual summary.
Segment Captions: These captions focus on describing localized temporal content within specific segments of the video. They are generated independently of the source prompt and verify against frames to ensure accuracy, offering detailed descriptions for short bursts of action.
Adaptive Global Blend Gate (AGBG): The AGBG is a mechanism used to intelligently fuse video features and textual caption features. It learns how much weight to give to the visual information versus the textual description at different points, allowing the framework to dynamically combine these inputs for better quality prediction.

Summary

The gist The proposed VGCap framework is an AIGV quality assessment framework guided by visually grounded captions that align better with generated content.

Introduction and Motivation

Recent advances in generative artificial intelligence have accelerated video generation technologies, but AI-generated videos (AIGVs) remain susceptible to various distortions, highlighting the need for dedicated video quality assessment methods Content semantics play an important role in AIGV quality assessment, with source prompts typically used as textual references However, these prompts may not accurately reflect the generated video content, resulting in a discrepancy between the intended textual description and the generated result This motivates the use of visually grounded captions that directly describe the generated content while preserving the source prompt as a reference for the intended semantics.

VGCap Framework Components

VGCap constructs two forms of captions to complement the intended semantics of the source prompt with visually grounded evidence of the realized video content. Specifically, it introduces two forms of captions: a global caption obtained by minimally revising the source prompt based on multi-frame visual evidence, and prompt-independent segment captions that describe localized temporal content. Together, these captions provide complementary textual evidence for assessing spatial quality, temporal quality, and text-video alignment.

Offline Caption Construction

The caption construction is performed offline, decoupling the captioning models from predictor training and inference. The construction involves two main types of captions:

Global Caption: This caption refines the source prompt to better describe the realized video content. For videos with at most 32 frames, all frames are retained; otherwise, 32 frames are uniformly sampled. A prompt-verification model decomposes the source prompt into six components: quantity, subject attributes, subject, action or state, object, and scene. Using only the sampled frames, a visual-description model independently generates descriptions for the same components. The prompt-verification model then verifies each prompt component as supported, contradicted, or uncertain based on frame content. Supported and uncertain components are retained, while contradicted components are replaced with the corresponding verified visual descriptions. The components are recomposed into a global caption while preserving the source-prompt structure.
Segment Captions: Segment captions describe localized temporal content. Videos are divided into segments based on frame counts: videos with fewer than 16, 16–23, 24–31, and at least 32 frames are evenly divided into one, two, three, and four non-overlapping segments. Up to eight frames are uniformly sampled from each segment. For each segment, the visual-description model generates an initial caption describing the visible subjects, actions or states, objects, and scene. The prompt-verification model verifies the caption against the corresponding frames. If an inconsistency is detected, the visual-description model revises the caption using the verification feedback, followed by a second verification. This frame-only process keeps the segment captions independent of the source prompt and other captions.

Feature Fusion and Assessment Branches

VGCap utilizes an Adaptive Global Blend Gate (AGBG) to fuse video and textual features, learning their relative contributions during fusion. The AGBG computes g = σ(γ), h = (1 − g)x + gy, where γ is a learnable scalar, σ(·) is the sigmoid function, g ∈ [0, 1] controls the textual contribution, and h is the fused feature. This formulation is applied across three quality branches:

** Spatial Quality Assessment:**

The spatial branch combines spatial video features with the global caption. It uses a pretrained SigLIP2 encoder to extract features from sampled frames, followed by a two-layer Transformer to produce contextualized frame features. Three pooling operations—masked mean, attention, and framedeviation pooling—aggregate complementary information across frames to obtain spatial features h¯, ha, and hd. Context from the global caption is incorporated by using AGBG to fuse video features vs. global caption feature zs = (1 − αs)vs + αsc.

** Temporal Quality Assessment:**

This assessment combines whole-video and segment-level motion features with caption features at corresponding temporal scales. Whole-video motion is extracted using a SlowFast-R50 encoder, and segment-level motion is derived from the segments used for caption construction. Semantic context is provided by BLIP features of the global caption and segment captions. AGBG is applied at whole-video and segment levels to produce fused features ut and uk. Contextualized segment features are produced using a temporal Transformer. The resulting segment-level temporal feature rseg is obtained by aggregating the segment features through three pooling operations. Finally, the whole-video fused feature and segment-level temporal feature are combined using AGBG to produce zt.

** Text-Video Alignment Assessment:**

This measures whether the realized visual content matches the intended semantics of the source prompt. It combines va (spatial feature) with BLIP features of the source prompt and global caption, fusing them as za = wV va + wP p + wC c. The alignment quality predictor receives za to produce the alignment score.

Training and Evaluation

The spatial, temporal, and text-video alignment branches are trained independently using their corresponding mean opinion score (MOS) labels. To optimize score accuracy and ranking consistency, the loss function is defined as L(b) = L(b)reg + λrL(b)rank.

Improvements for AI systems

Improve content awareness by constructing visually grounded captions through two methods: a global caption obtained by minimally revising the source prompt based on multi-frame visual evidence and prompt-independent segment captions that describe localized temporal content. This aligns textual evidence more accurately with generated video content, addressing the limitation where current methods rely solely on potentially inaccurate source prompts.
Enhance spatial quality assessment by combining spatial video features with the global caption using an Adaptive Global Blend Gate (AGBG) to fuse them, as described by the formula: zs = (1 − αs)vs + αsc, where αs ∈ [0, 1] controls the contribution of the global caption feature. This allows the spatial predictor to leverage both frame-level visual data and semantic context from the revised prompt.
Boost temporal quality prediction by incorporating both whole-video motion and localized segment features, specifically by applying AGBG at temporal scales: ut = (1 − αt)vt + αtc for whole-video features and uk = (1 − βt)mk + βtsk for segment-level features. This fusion, coupled with a subsequent temporal Transformer to produce contextualized segment features, enables a more nuanced understanding of motion dynamics across the video.
Strengthen text-video alignment by fusing three distinct textual inputs—spatial features from the video, the original source prompt, and the newly constructed global caption—using a weighted AGBG fusion: za = wV va + wP p + wC c. This provides a comprehensive textual evidence base for assessing whether the realized visual content matches the intended semantics of the source prompt.
Develop an offline, decoupled training pipeline by performing all caption construction offline, which involves using models like Qwen3-VL-8B [16] and VideoLLaMA3-7B [17] to generate captions independently before integrating them into the quality assessment predictors. This modular approach maintains the integrity of the captioning process separate from predictor training and inference.