Grounded Human-Attributed Description and Activity Recognition in Videos

arXiv:2610.22295 · cs.CV · Submitted 2026-10-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Grounded Human-Attributed Description and Activity Recognition in Videos".

Jane: The gist The authors introduce Grounded Human-Attributed Description and Activity Recognition (GHADAR), a task involving describing every person's characteristics and activity from video location,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper now, "Grounded Human-Attributed Description and Activity Recognition in Videos". The big idea here is that instead of just recognizing what a group is doing, or what one person is doing alone, they want to describe every single person in a video based on where they are.

Jane: That makes sense. It sounds like they're moving away from those simpler tasks to something much more detailed and practical for real-world stuff.

Lu: Exactly. They define this whole task as Grounded Human-Attributed Description and Activity Recognition, or GHADAR, where you get the spatial location of people in a video, usually through a bounding box, and then you have to generate two open-set captions for each person.

Meng: So for every person in that scene—like an elderly man on the street—you need to describe his attributes and what he's doing, like attempting to cross a road.

Tom: Right. And they tackle this by creating a new dataset based on AVA-Actions where they generate these open-set captions specifically for the person description and activity.

Jane: That sounds like a way to get the data needed for this more granular task, extending what's already out there.

Lalam: I see how that helps with the data generation part.

Tom: Then they propose a new training method called CAMP, which is Constrained Attention Masking-based Pretraining, to make sure the Vision Language Models actually use that grounding information during training.

Lu: So it’s a two-stage strategy. Phase one involves constraining the attention mask of the output tokens to only look at visual tokens within an expanded bounding box around the target person, which really forces localized reasoning.

Meng: That sounds like a direct way to stop the model from getting distracted by other people in the scene when focusing on one individual.

Jane: And phase two just removes those constraints and fine-tunes at a reduced learning rate so it can still pick up that broader context without losing its focus on the specific person.

Tom: The results they show, they demonstrate that this method outperforms state-of-the-art Vision Language Models on both the AVA-Captions dataset and another one called HC-STVG v2.

Lu: They show consistent gains over strong VLM baselines across both standard and their proposed metrics.

Meng: That's good for scaling up, because they focus on small to medium scale models which is more realistic for things like home assistants or surveillance systems.

Jane: And they also address how we evaluate this stuff by critiquing existing metrics, saying that relying too much on human annotations and exact text-text matching isn't very robust.

Tom: They propose a VLM-based evaluation schema where you compare concepts directly between the raw video content and the model's predictions.

Lu: That way, you look at things like completeness, uniqueness, and correctness of the caption against what's actually in the video.

Meng: It sounds like a more holistic way to check if the AI is actually understanding what it's supposed to be doing.

Jane: So this paper lays out a whole framework: a dataset called AVA-Captions, their CAMP method, and this new evaluation schema for GHADAR.

Tom: It sets up a pretty solid foundation for future research in human-attributed video understanding. What we're talking about now is the conclusion of "Grounded Human-Attributed Description and Activity Recognition in Videos".

Jane: The authors Jay Paranjape and his team are really pushing this idea that understanding characteristics and activities of individuals in complex multi-person environments is crucial for real-world applications >

Conclusion: Tom: So, to wrap up, this paper introduces Grounded Human-Attributed Description and Activity Recognition in Videos, or GHADAR, which is about describing every single person in a video given their location.

Jane: Right. It's taking those complex group scenes and breaking them down into individual human descriptions and what they are doing, using their spatial position as the starting point for everything.

Tom: The authors, Jay Paranjape and his team, they've put together a whole framework here that includes a new dataset called AVA-Captions, a training method called CAMP, and even this new way to evaluate the results.

Jane: They're really showing how you can move past just recognizing general group actions and get down to describing specific people in detail across different settings.

Tom: The main thing I’m taking away from this is that they aren't just proposing a new model; they’re building the whole pipeline—data, method, evaluation—for this specific task.

Jane: And it suggests that for real-world applications, like surveillance or assistants, knowing *who* is doing *what* and where they are is way more useful than just knowing a general activity happened.

Tom: It seems like the authors are really focused on making sure the AI actually learns to use that grounding information during training instead of just guessing.

Jane: Exactly. They’re using this new CAMP method to force the Vision Language Models to pay attention to that person's specific area in the video when they are generating those descriptions.

Tom: And they’ve done some solid work there, showing that their approach beats the current state-of-the-art Vision Language Models on this task.

Jane: It sets up a really interesting direction for future research into how we can make AI understand human presence and behavior in complex scenes with more precision.

Jay N. Paranjape, Jurijs Nazarovs, Rohan Sarkar, Yue Guo, Lipeng Ke, Sankar Venkataraman, Vishal M. Patel, Nataraj Jammalamadaka

Johns Hopkins University, USA · Amazon

cs.CV

Submitted: 2026-10-15

Updated: 2026-10-15

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: The gist The authors introduce Grounded Human-Attributed Description and Activity Recognition (GHADAR), a task involving describing every person's characteristics and activity from video location,

Key concepts

Grounded Human-Attributed Description and Activity Recognition (GHADAR): This is a task where the goal is to describe both the physical attributes (characteristics) and actions of every individual present in a video. Crucially, the input provides the spatial location of each person, making it practical for real-world use rather than just recognizing general group activities.
AVA-Captions Dataset: This new dataset extends AVA-Actions by creating open-set captions specifically for each person. These captions describe both the person's attributes (like age or clothing) and their specific activity, allowing the model to learn detailed, per-person descriptions in multi-person scenes.
Constrained Attention Masking (CAMP): This is a two-stage training method for Vision Language Models. In the first stage, it forces the model's attention mechanism to focus only on visual information within a specific bounding box around the target person. The second stage fine-tunes this by removing these constraints to allow the model to use broader context.
VLM-based Evaluation Schema: This is a new way to evaluate models that moves beyond simple text matching. It compares the generated predictions directly against the raw video content, assessing how well the model captures concepts like completeness, uniqueness, and correctness in relation to what is actually happening visually.

Terminology

Grounded Human-Attributed Description and Activity Recognition — 2× in this paper
Generalizability — 1× in this paper · explained in 6 episodes
Grounding — 1× in this paper · explained in 3 episodes

Summary

The gist The authors introduce Grounded Human-Attributed Description and Activity Recognition (GHADAR), a task involving describing every person's characteristics and activity from video location, and propose a novel method called Constrained Attention Masking-based Pretraining (CAMP) to improve performance on this task.

Task Definition

The GHADAR task is formally defined as: given the spatial location of people in a video (via bounding box), the goal is to generate two open-set captions — one describing each person’s attributes (e.g., an elderly man in white on the street, a child in blue uniform) and one describing their activity (e.g., attempting to cross a road, playing on the footpath)

Dataset AVA-Captions

The authors introduce AVA-Captions, which extends AVA-Actions by generating open-set, per-person captions for both attributes and activities, involving multi-person interactions

Method CAMP

The proposed method is Constrained Attention Masking-based Pretraining (CAMP), a two-stage training strategy for VLMs that explicitly leverages grounding information

Phase 1 of CAMP involves constraining the attention mask of the output tokens to the visual tokens within an expanded bounding box around the target person, enforcing localized reasoning
Phase 2 removes these constraints and fine-tunes at a reduced learning rate, allowing the model to integrate global context without eroding its person-specific focus

Evaluation Schema

The authors critique existing evaluation metrics in person and activity description, which are often overly reliant on human-annotations and exact text-text matching

To address this limitation, they propose a VLM-driven evaluation framework that directly compares conceptual alignment between the raw video content and the model prediction from the lens of completeness, uniqueness and correctness of the caption

Results

Experiments on HC-STVG and AVA-Captions show that their method outperforms SOTA VLMs on this task The results demonstrate consistent gains over strong VLM baselines across both standard and proposed metrics The study focuses on small- to medium-scale models, which are more practical for applications like home assistants and surveillance systems

Limitations

A limitation in the VLM-based metric is that it is highly dependent on the ability of the VQA to understand the video and the instruction Furthermore, VLMs can introduce inherent bias in their results Therefore, they recommend using VLM-based metrics for relative comparison between models rather than the absolute performance of a single model

Conclusion

The paper develops a complete framework for GHADAR, including a dataset, a novel method and an evaluation schema The authors encourage future research in human-attributed video understanding tasks that can be useful for various applications

--- Page 1 ---

Grounded Human-Attributed Description and Activity Recognition in Videos<ref:2610.22295#pg2>

Abstract. Understanding characteristics and activities of individuals in complex multi-person environments is crucial for real-world applications.<ref:2610.22295#pg2> However, existing datasets often simplify this problem to grounded group activity recognition, single-person activity recognition, or close-set activity recognition, thus limiting the generalizability of models trained on these datasets.<ref:2610.22295#pg2> In this work, we introduce the task of Grounded Human-Attributed Description and Activity Recognition (GHADAR), which involves describing the characteristics and activity descriptions of every person in a video, provided the location of the person in the video, which is more practical for real-world applications.<ref:2610.22295#pg2> To facilitate this, we introduce a new dataset derived from AVA-Actions by generating open-set captions for the person description and activity.<ref:2610.22295#pg2> In addition, we propose a novel method to effectively utilize the information contained in grounding during training by constraining the cross-attention masks during training in VLMs to improve performance for this task.<ref:2610.22295#pg2> Our experiments show that our method outperforms SOTA VLMs on this task.<ref:2610.22295#pg2> Finally, we demonstrate the limitations of existing evaluation metrics, which are overly reliant on human-annotations and exact texttext matching.<ref:2610.22295#pg2> As an added video-based evaluation, we propose a holistic VLM-based evaluation schema that compares concepts directly between the video and the generated predictions.<ref:2610.22295#pg2> Thus, in this work, we develop a complete framework for GHADAR, including a dataset, a novel method and an evaluation schema <ref:2610.22295#pg2> Keywords: Video Understanding · Activity Recognition · Vision Language Models<ref:2610.22295#pg2>

--- Page 3 ---

Fig. 1: Our proposed task with respect to existing tasks. While existing tasks are more relaxed, we propose an open-set human description and activity recognition that is more generic.

--- Page 4 ---

Table 1: Existing datasets. Most of the video clips in these datasets are short (0-20s) except for AVA-Actions and MEVA, which have 15 min long and 5 min long annotated videos, respectively.

--- Page 6 ---

Fig. 2: Examples from the AVA-Captions dataset. The captions in black are generated using expert-VLM while red action labels are provided in the original AVA-Actions dataset.

--- Page 7 ---

Fig. 3: The two phases of CAMP. In the first phase, the modified causal mask for the answer tokens makes sure they can only attend to the vision tokens corresponding to the relaxed bounding box around the person for which the caption is being generated.

--- Page 8 ---

Fig. 3: The two phases of CAMP. In phase 2 of CAMP, we remove constraints and fine-tune at a reduced learning rate, allowing the model to incorporate global context without eroding its person-specific focus.

--- Page 9 ---

Fig. 2: Examples from the AVA-Captions dataset.

Improvements for AI systems

Grounded Human-Attributed Description and Activity Recognition (GHADAR) task definition: The system can now generate "two open-set captions — one describing each person’s attributes (e.g., an elderly man in white on the street, a child in blue uniform) and one describing their activity (e.g., attempting to cross a road, playing on the footpath)" for every person given their spatial location in a video.
Constrained Attention Masking-based Pretraining (CAMP): The VLM can be trained to explicitly leverage grounding information, specifically by constraining the attention mask during decoding in Phase 1, forcing it to focus only at tokens that are related to the region around the person for which the caption is being generated.
Two-stage training strategy: The model benefits from a transition from localized reasoning to global context integration by employing Phase 2, where the constraints are removed and [the model] is fine-tuned with the default attention mask at a reduced learning rate, enabling it to incorporate global contextual cues while learning to generate outputs under the standard attention mechanism.
Novel evaluation schema: The system can be evaluated using a VLM-based video–prediction alignment protocol that directly compares conceptual alignment between the raw video content and the model prediction from the lens of completeness, uniqueness and correctness of the caption, complementing classical text-matching metrics.
Subject/Activity Metric Calculation: Instead of relying solely on lexical overlap, the system can be assessed using VLM-based metrics like Completeness (CMP), Correctness (CRT), and Uniqueness (UNQ) to verify factual alignment of the prediction with the video content and assess how discriminative is the predicted subject description in identifying the subject.

Abstract

Understanding characteristics and activities of individuals in complex multi-person environments is crucial for real-world applications. However, existing datasets often simplify this problem to grounded group activity recognition, single-person activity recognition, or close-set activity recognition, thus limiting the generalizability of models trained on these datasets. In this work, we introduce the task of Grounded Human-Attributed Description and Activity Recognition (GHADAR), which involves describing the characteristics and activity descriptions of every person in a video, provided the location of the person in the video, which is more practical for real-world applications. To facilitate this, we introduce a new dataset derived from AVA-Actions by generating open-set captions for the person description and activity. In addition, we propose a novel method to effectively utilize the information contained in grounding during training by constraining the cross-attention masks during training in VLMs to improve performance for this task. Our experiments show that our method outperforms SOTA VLMs on this task. Finally, we demonstrate the limitations of existing evaluation metrics, which are overly reliant on human-annotations and exact text-text matching. As an added video-based evaluation, we propose a holistic VLM-based evaluation schema that compares concepts directly between the video and the generated predictions. Thus, in this work, we develop a complete framework for GHADAR, including a dataset, a novel method and an evaluation schema, thereby establishing a strong foundation for future research in this domain.