Enhancing Gaze Reasoning in Vision Foundation Models for Gaze Following

arXiv:2605.22607 · cs.CV · Submitted 2026-05-21 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Enhancing Gaze Reasoning in Vision Foundation Models for Gaze Following".

Jane: Here's the gist of the paper. The gist: The proposed method enhances gaze reasoning in Vision Foundation Models by introducing a head-conditioned local LoRA and an out-of-cone penalty to improve performance when gaze targets are not semantically salient.

Tom: Key Limitation Identified Vision foundation models (VFMs) show strong scene understanding but contribute little to gaze reasoning, causing existing methods to rely on "semantically salient objects rather than true gaze cues".

Jane: This reliance on semantic priors leads to "degraded performance when targets are not salient".

Tom: The paper observes that existing VFM-based methods can often make plausible predictions by exploiting scene priors, such as "focusing on visually prominent objects, interaction centers, or semantically meaningful regions".

Title and authors: Tom: So we're looking at the paper "Enhancing Gaze Reasoning in Vision Foundation Models for Gaze Following," and basically, it shows that vision foundation models can be made much better at actually figuring out where someone is looking by focusing on geometry instead of just what looks semantically important.

Jane: Right. It confirms that by using head-conditioned local LoRA and the out-of-cone penalty, we can make these models more reliable when things aren't perfectly clear in a picture.

Lu: The mechanism itself is quite clever; it’s not just adding a filter, it’s fundamentally changing how the model learns to associate head position with gaze direction. It points towards a representation learning approach that is highly localized.

Meng: I'm still worried about the deployment cost of these two components working together at scale, but if this accuracy boost on inconsistent subsets translates to fewer bugs in production environments, it’s worth looking into.

Lalam: My assessment is high confidence here; the evidence shows that this method significantly reduces reliance on visual plausibility and improves the localization of the correct gaze direction. I'm very sure it’s a reliable improvement for gaze tasks.

Tom: So, we have this paper on HCLoRA and OOC penalty for GazeReasoning in VFM, and it proves localized adaptation is a strong way to enhance gaze reasoning without having to redesign the entire backbone-decoder pipeline.

Jane: It’s a really important piece of work because it tackles that gap between strong scene understanding and actual gaze tracking, showing how targeted modifications can help models perform better when things get tricky.

Lu: The implication here is that we might be able to build more reliable gaze-following systems by integrating these kinds of targeted modifications into existing VFM architectures.

Meng: If this works as well on the inconsistent subset, it means we don't have to re-engineer the whole thing just because some edge cases are hard to handle.

Lalam: The work provides a clear direction: use head conditioning and geometric penalties to make models prioritize true gaze cues over visual plausibility, which is what they successfully achieved with these results.

Tom: Alright, we've covered the mechanics and the results for "Enhancing Gaze Reasoning in Vision Foundation Models for Gaze Following." It’s been great exploring this paper with you all.

Jane: It has been a really insightful discussion on how we can make AI models better at understanding the world around us.

Lu: Thanks for the deep dive into the representation learning aspect of this work.

Meng: Thanks for bringing in the production concerns we need to keep in mind.

Lalam: Glad I could connect these results to what they mean for gaze tracking systems.

The paper's summary: Tom: So we’re looking at the summary of "Enhancing Gaze Reasoning in Vision Foundation Models for Gaze Following" again, and basically, it boils down to this: these vision foundation models are great at seeing a scene but they have this weakness where they rely on what *looks* important instead of actually figuring out where someone is looking.

Jane: Right. The paper explains that because the models are so good at general scene understanding, they often make guesses about gaze targets based on what’s semantically salient in the image, even when those things aren't the true gaze target.

Tom: They call this "semantic shortcut." So, existing methods just follow what looks visually plausible instead of performing actual gaze reasoning. They point out that this is a problem because it makes performance worse when the targets aren't semantically clear at all.

Jane: To fix this, they introduce two main things: first, a head-conditioned local LoRA that adjusts the model locally around the person's head position while keeping its main scene understanding intact.

Tom: I like that part about "localized adaptation." It’s like giving a specific training tweak only to that part of the network handling head movements, which preserves all those big scene representations they learned during pretraining.

Jane: And the second thing is this out-of-cone penalty. This penalizes any probability mass outside what should be geometrically possible for where someone can look from their position. It forces the model to rely on true geometric cues instead of just guessing based on what looks like a chair or a window.

Tom: That penalty sounds like it directly counters that semantic shortcut by injecting physical constraints into the gaze prediction, which is exactly what we need for reliable tracking when things aren't obvious <ref:2605.22607#pg4>. The experiments on GazeFollow and VAT show they get state-of-the-art performance, especially on those hard cases where things aren't semantically clear <ref:2605.22607#pg3>.

Jane: It’s interesting that they split their test set into two groups—consistent and inconsistent—and showed a big gap between them using a semantic prober <ref:2605.22607#pg3>. They found that this method consistently improved over the baseline, even when the semantic saliency was unreliable in those hard cases <ref:2605.22607#pg3>.

Tom: The ablation study confirms that both mechanisms are necessary; without either one, the gains on those hard cases just disappear <ref:2605.22607#pg4>. That’s a strong signal that this isn't just a small tweak; it fundamentally changes how the model reasons about gaze direction.

Jane: It means the system becomes much less likely to get confused when someone is looking at something subtle, like off-camera or at a small detail <ref:2605.22607#pg3>. This moves us closer to models that actually reason about physics, not just image patterns.

Tom: And I’m still curious about the real-world deployment cost. How much overhead does adding this head-conditioned LoRA mechanism add to inference speed when we’re trying to run these models fast enough for a live feed <ref:2605.22607#pg1>?

Jane: That's a practical question, Tom. Because if the adaptation adds too much computational cost, it might not be useful for real-time applications like autonomous vehicles or live monitoring systems. We’ll keep an eye on that trade-off <ref:2605.22607#pg3>.

Tom: Right, exactly. The mechanics are interesting on paper, but in production we need to know if the speed trade-off is worth this extra accuracy gain on those tricky inconsistent frames <ref:2605.22607#pg3>. Now, let's pivot completely and see what else is happening in the world of three dee modeling with these vision-language models.

The paper's improvements: Tom: So we’re talking about the specific improvements they introduced to make this gaze reasoning better than what was there before <ref:2605.22607#pg4>. It’s not just one fix; they have two distinct mechanisms working together for this task.

Jane: That’s right. The first is that head-conditioned local LoRA we talked about earlier, which lets the model adapt its features specifically for the head position <ref:2605.22607#pg1>. It improves how it learns about looking without hurting its overall understanding of the whole scene.

Tom: It’s like giving a specific training tweak only to that part of the network that handles head movements, preserving all those big scene representations they learned during pretraining <ref:2605.22607#pg4>. That separation is key for keeping things clean.

Jane: And the second improvement is this out-of-cone penalty. This part doesn't just adapt features; it actively injects gaze cues into the head tokens by penalizing any probability mass that isn't geometrically possible for a real gaze <ref:2605.22607#pg4>.

Tom: Exactly, it regularizes those auxiliary maps so they actually align with the scene tokens, pushing the model to rely on actual geometry instead of just what looks plausible <ref:2605.22607#pg4>. It’s a very direct way to counter that semantic shortcut we discussed earlier.

Jane: So what does this mean practically? It means the system is much less likely to get confused when someone isn't looking at something obvious, like when they are just looking off-camera or at a small detail <ref:2605.22607#pg3>.

Tom: If I’m shipping this to production, it means we get state-of-the-art results even on the inconsistent test subset where semantic saliency fails <ref:2605.22607#pg3>. The ablation study confirmed that both mechanisms are necessary; without either one, the gains on those hard cases just disappear <ref:two thousand six hundred five point two two six zero seven#pg4.

Jane: It’s about making the AI robust enough to handle real-world ambiguity, which is a huge hurdle when you’re trying to build systems that need to be reliable everywhere, not just in perfect training sets <ref:2605.22607#pg3>.

Tom: The authors even compared this penalty against other supervision strategies and found that the out-of-cone penalty was the best performer on those inconsistent subsets compared to direct gaze-vector regression <ref:two thousand six hundred five point two two six zero seven#pg4>. That’s a strong data point showing the value of geometric regularization.

Jane: It really shows that forcing a physical constraint is more effective than just letting the model guess based on what it sees as important. This is moving us closer to models that actually reason about physics, not just image patterns.

Tom: And I’m still curious about the real-world deployment cost. How much overhead does adding this head-conditioned LoRA mechanism add to inference speed when we’re trying to run these models fast enough for a live feed <ref:two thousand six hundred five point two two six zero seven#pg1>?

Jane: That's a practical question, Tom. Because if the adaptation adds too much computational cost, it might not be useful for real-time applications like autonomous vehicles or live monitoring systems <ref:2605.22607#pg3>.

Tom: Right, exactly. The mechanics are interesting on paper, but in production we need to know if the speed trade-off is worth this extra accuracy gain on those tricky inconsistent frames <ref:2605.22607#pg3>. Now, let's pivot completely and see what else is happening in the world of three dee modeling with these vision-language models.

Conclusion: Tom: So we’ve covered the mechanics and results for "Enhancing Gaze Reasoning in Vision Foundation Models for Gaze Following," and basically, this paper shows that vision foundation models can be made much better at actually figuring out where someone is looking by focusing on geometry instead of just what looks semantically important.

Jane: Right. It confirms that by using head-conditioned local LoRA and the out-of-cone penalty, we can make these models more reliable when things aren't perfectly clear in a picture.

Lu: The mechanism itself is quite clever; it’s not just adding a filter, it’s fundamentally changing how the model learns to associate head position with gaze direction <ref:2605.22607#pg4>. It points towards a representation learning approach that is highly localized.

Meng: I'm still worried about the deployment cost of these two components working together at scale, but if this accuracy boost on inconsistent subsets translates to fewer bugs in production environments, it’s worth looking into <ref:2605.22607#pg3>.

Lalam: My assessment is high confidence here; the evidence shows that this method significantly reduces reliance on visual plausibility and improves the localization of the correct gaze direction <ref:2605.22607#pg4>. I'm very sure it’s a reliable improvement for gaze tasks.

Tom: So, we have this paper on HCLoRA and OOC penalty for GazeReasoning in VFM <ref:2605.22607#pg1>. It’s been an interesting look at how we can use localized adaptation to fix a real weakness in vision models.

Jane: Definitely. This work is important because it shows that we don't have to just rely on the general scene understanding of foundation models; we can fine-tune them with specific geometric constraints for better reasoning.

Lu: It sets a precedent for how we should think about integrating auxiliary supervision into large pre-trained models going forward.

Meng: I'll keep an eye on the speed implications, but the accuracy gains on those hard cases are definitely something that could matter when we start shipping these kinds of systems <ref:2605.22607#pg3>.

Lalam: I’m confident this is a key direction for gaze research moving forward.

Tom: Alright, we've covered the mechanics and the results for "Enhancing Gaze Reasoning in Vision Foundation Models for Gaze Following" <ref:2605.22607#pg1>. It's been great exploring this paper with you all.

Jane: It has been a really insightful discussion on how we can make AI models better at understanding the world around us <ref:2605.22607#pg4>.

Lu: Thanks for the deep dive into the representation learning aspect of this work.

Meng: Thanks for bringing in the production concerns we need to keep in mind.

Lalam: Glad I could connect these results to what they mean for gaze tracking systems.

Shijing Wang, Yaping Huang, *Chaoqun Cui, *David Wong, Yihua Cheng, Alexandros Neophytou

Beijing Jiaotong University · University of Birmingham · MAIS, Institute of Automation, Chinese Academy of Sciences · Microsoft, UK

cs.CV

Submitted: 2026-05-21

Updated: 2026-10-17

Comments: Accepted at NeurIPS 2026. 17 pages, 11 figures

Code: https://github.com/shijing-wang/HCloRA

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 91/100

The gist: The gist: The proposed method enhances gaze reasoning in Vision Foundation Models by introducing a head-conditioned local LoRA and an out-of-cone penalty to improve performance when gaze targets are

Key concepts

Semantic Shortcut
This occurs when a vision model predicts where someone is looking based on what looks visually prominent in the scene, like a bright object or an interaction center, instead of actually following the person's eyes. This reliance on visual plausibility creates errors when the true gaze target isn't semantically obvious.
Head-Conditioned Local LoRA (HCLoRA)
This is a training technique that uses the query head position to adjust only specific parts of the vision model. It allows the model to adapt its features related to where the head is, while keeping its general scene understanding intact, which helps it learn how to reason about gaze more effectively.
Out-of-Cone Penalty
This is a regularization method that forces the model's gaze evidence maps to concentrate probability mass only within the geometrically possible area of where someone can be looking. It acts as a constraint, pushing the model to prioritize real gaze cues over irrelevant visual information.

Terminology

Summary

The gist: The proposed method enhances gaze reasoning in Vision Foundation Models by introducing a head-conditioned local LoRA and an out-of-cone penalty to improve performance when gaze targets are not semantically salient.

Key Limitation Identified

Vision foundation models (VFMs) show strong scene understanding but contribute little to gaze reasoning, causing existing methods to rely on semantically salient objects rather than true gaze cues. This reliance on semantic priors leads to degraded performance when targets are not salient. The paper observes that existing VFM-based methods can often make plausible predictions by exploiting scene priors, such as focusing on visually prominent objects, interaction centers, or semantically meaningful regions. This behavior is referred to as a semantic shortcut, where the model relies on semantic plausibility as a proxy for gaze following rather than performing gaze reasoning.

Proposed Training Mechanism

To address the issue of insufficient gaze reasoning, the authors propose a novel training mechanism consisting of two complementary components. First, they introduce a head-conditioned local LoRA, which uses the queried head position to modulate low-rank residual updates inside the VFM, enabling localized adaptation of head-relevant features while preserving the backbone’s scene-level representations. This mechanism is formulated as Y = W(X) + ∆(X, b), where the head-conditioned residual update is defined as ∆(X, b) = A(X) ⊙ M(b). Second, they propose an out-of-cone penalty, which regularizes auxiliary gaze-evidence maps by penalizing probability mass outside the geometrically plausible gaze cone. This penalty flexibly injects gaze cues into head features rather than imposing a rigid prior.

Experimental Validation and Results

Experiments on the GazeFollow and VAT datasets demonstrate that the method achieves state-of-the-art performance, with particularly strong improvements when gaze targets are not semantically salient. The authors validate their hypothesis by splitting the test dataset into consistent and inconsistent subsets using a semantic prober. They find that GazeLLE exhibits a significant performance gap between the two subsets, which supports the hypothesis that the semantic shortcut is a common issue across gaze following approaches. The results show that their method consistently improves over GazeLLE under the same ViT-B backbone, especially on the inconsistent subset where semantic saliency is unreliable.

Component Ablation and Analysis

A component ablation study confirms the necessity of both proposed designs. Adding HCLoRA leads to substantial gains on the inconsistent subset, while the out-of-cone penalty further improves all metrics on that subset. Furthermore, comparing different gaze-aware supervision strategies shows that the out-of-cone penalty achieves the best performance on the inconsistent subset compared to direct gaze-vector regression or rigid gaze-cone supervision. The out-of-cone penalty is shown to suppress gaze-inconsistent responses and concentrate more clearly on the gaze-consistent target.

Conclusion

The study concludes that strong benchmark performance in VFM-based gaze following can be driven by semantic shortcuts rather than reliable gaze reasoning. The authors successfully propose HCLoRA and the OOC penalty to enhance gaze reasoning, achieving state-of-the-art performance while significantly improving gaze reasoning ability. This demonstrates that our method robustly improves gaze reasoning through localized adaptation. The results show that both HCLoRA and OOC improve gaze reasoning ability under the 30◦ threshold. The proposed method effectively enhances gaze reasoning ability, consistent with their original motivation. The visualization results show that the method reduces reliance on semantic saliency and improves localization of the correct gaze direction. This shows that their method reduces reliance on semantic saliency and improves localization of the correct gaze direction. The final results show that our method robustly improves gaze reasoning through localized adaptation. This shows that our method robustly improves gaze reasoning through localized adaptation.

--- Page 1 ---

Enhancing Gaze Reasoning in Vision Foundation Models for Gaze Following

Shijing Wang1 Yaping Huang1∗ Chaoqun Cui3 David Wong4 Yihua Cheng2∗ Alexandros Neophytou4 Hyung Jin Chang2 1Beijing Jiaotong University 2University of Birmingham 3MAIS, Institute of Automation, Chinese Academy of Sciences 4Microsoft, UK Abstract Gaze following requires both scene understanding and gaze reasoning to localize the gaze target of an in-scene person. Recently, vision foundation models (VFMs) have demonstrated strong performance on this task, enabling simpler architectures while outperforming prior methods. However, we observe a key limitation of VFM-based approaches: while VFMs substantially improve scene understanding, they contribute little to gaze reasoning. As a result, existing methods often rely on semantically salient objects rather than true gaze cues, leading to degraded performance when targets are not salient. To address this, we propose a novel training mechanism to enhance gaze reasoning in VFMs for gaze following. Our method includes: (1) a head-conditioned local LoRA, which enables localized adaptation to preserve scene token learning while improving head token learning for gaze reasoning; and (2) an out-of-cone penalty, which injects gaze cues into head tokens while aligning them with scene tokens. Experiments on the GazeFollow and VAT datasets demonstrate that our method achieves state-of-the-art performance, with particularly strong improvements when gaze targets are not semantically salient. Our findings offer valuable insights for advancing future gaze following research. We will release the code once the paper is accepted.

--- Page 2 ---

GT GazeLLE Ours Figure 1: Left: Given an input image and the query head marked by the red box, existing VFM-based methods such as GazeLLE often predict gaze targets around semantically salient objects or interaction centers, even when they are not the true gaze target. Right: Comparison on the GazeFollow benchmark, where lower Min. L2 and Min. Ang. indicate better performance. semantically salient objects rather than true gaze cues, assigning high confidence to visually plausible but incorrect regions and thus degrading localization performance.

--- Page 3 ---

GazeLLE Ours Chong et.al Miao et.al Chong et.al 0.05 0.10 Avg L2 0.15 0.20 Incons semantically salient objects rather than true gaze cues, assigning high confidence to visually plausible but incorrect regions and thus degrading localization performance.

--- Page 4 ---

Figure 4: Overview of our proposed framework. Given an input image and the target person’s head box, we construct a head-position prior to modulate the low-rank residual branch inside a frozen pretrained vision backbone. The proposed head-conditioned local LoRA selectively adapts head-relevant features while preserving scene-level representations.

Improvements for AI systems

  1. The proposed HCLoRA mechanism enables localized adaptation of head-relevant features while preserving the backbone’s scene-level representations, allowing models to preserve scene token learning while improving head token learning for gaze reasoning.

  2. The out-of-cone penalty injects gaze cues by penalizing probability mass outside the geometrically plausible gaze cone, which regularizes auxiliary gazeevidence maps and encourages the model to rely on true gaze cues rather than insufficient semantic saliency.

  3. The improved system achieves state-of-the-art performance, particularly when targets are not semantically salient, by addressing the semantic shortcut observed in existing models where they often make plausible predictions by exploiting scene priors.

Sources

Related papers