Hierarchical Semantic Tree Anchoring for CLIP-Based Class-Incremental Learning

arXiv:2511.15633 · cs.CV, cs.LG · Submitted 2025-11-19 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Hierarchical Semantic Tree Anchoring for CLIP-Based Class-Incremental Learning".

Jane: Here's the gist of the paper. The gist The proposed HASTEN framework anchors hierarchical information into Class-Incremental Learning to reduce catastrophic forgetting by preserving feature drift through structured representations.

Tom: Class-Incremental Learning (CIL) enables models to learn new classes continually while preserving past knowledge, yet existing CLIP-based CIL methods fail to explicitly capture the inherent hierarchy in real-world visual and linguistic concepts, leading to fine-grained class features drift during incremental updates and ultimately to catastrophic forgetting.

Jane: This failure occurs because CLIP’s Euclidean feature space treats all points alike, which compresses distances and inadvertently pulls in unrelated specific concepts when broad concepts become too close to many specific ones.

Tom: Correspondingly, without hierarchy, fine-grained class features gradually drift across incremental tasks as new classes emerge and previous classes shift from their original positions.

Paper summary: Tom: We’ve covered how Hierarchical Semantic Tree Anchoring for CLIP-Based Class-Incremental Learning uses a hierarchical tree and hyperbolic space to anchor features, and how they stabilize the model against catastrophic forgetting <ref:2511.15633#pg2>.

Jane: So, to wrap up, this paper shows that by explicitly anchoring hierarchical information into CIL using structured supervision, you can maintain feature stability across incremental updates <ref:2511.15633#pg2>.

Lu: It’s about providing a stronger inductive bias toward hierarchy so the model retains accumulated knowledge and uses that structure to stabilize features <ref:2511.15633#pg2>.

Meng: For deployment, it means you can deploy a CLIP-based CIL model that is less likely to suffer from feature drift during continuous learning <ref:2511.15633#pg2>.

Tom: The authors show that HASTEN consistently outperformed existing methods on nine benchmarks while reducing both feature drift and catastrophic forgetting <ref:2511.15633#pg2>.

Jane: It’s a significant direction because it suggests that for building robust incremental learning systems, incorporating explicit semantic structure is a key way to preserve past knowledge <ref:2511.15633#pg2>.

Conclusion: Tom: So, we’re talking about HASTEN, that’s the name of this paper: "Hierarchical Semantic Tree Anchoring for CLIP-Based Class-Incremental Learning."

Jane: It sounds like they took a problem where models forget old knowledge when they learn new classes and tried to fix it by using structure. What exactly is this hierarchy they're talking about?

Tom: They’re not just looking at the classes as isolated labels. They’re using a GPT-five generated hierarchical semantic tree as external supervision. So, instead of just saying "this is a dog," they have parent nodes for "dog," and then children like "Labrador" or "Golden Retriever."

Lu: That structure is being used to organize the visual and textual features in hyperbolic space. The idea there is that you keep those parent-child relationships intact even when you’re learning something new.

Jane: Hyperbolic space? Like, instead of a flat map where everything is just distance from a center, they’re using a curved space for the features themselves. How does that help them keep the hierarchy stable?

Tom: Well, they have this text-to-text hierarchical loss. It forces each child node to stay inside an aperture cone centered around its parent node. That geometric constraint keeps things organized in that hyperbolic space.

Meng: From a deployment standpoint, that sounds like a lot of training overhead. How much does this extra structure actually cost us when we’re running this on real traffic? Does it make the model slow?

Lu: The authors mention they use gradient projection onto the null space of their shared hyperbolic mapper to stop old tasks from interfering. This is what they call gradient projection for stability. It keeps the updates focused on new things without messing up what already worked.

Jane: So, you’re essentially giving the model a map that knows its own structure, and then you’re telling it to only move in ways that respect that map? That sounds like a solid way to fight feature drift.

Tom: Exactly. And the results show they beat existing methods on nine different benchmarks while actually reducing both feature drift and catastrophic forgetting. That’s a big win for incremental learning.

Jane: It means for someone just listening, it changes how we think about how AI learns new things over time—it’s not just about adding data; it’s about organizing that data smartly. Where does this leave us next?

Tao Hu, Lan Li, Zhen-Hao Xie, Da-Wei Zhou

School of Artificial Intelligence, Nanjing University · State Key Laboratory for Novel Software Technology, Nanjing University

cs.CV, cs.LG

Submitted: 2025-11-19

Updated: 2026-10-20

Comments: Accepted to NeurlPS 2026. Code is available at https://github.com/LAMDA-CL/NeurIPS2026-HASTEN

Code: https://github.com/LAMDA-CL/NeurIPS2026-HASTEN

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 86/100

The gist: The gist The proposed HASTEN framework anchors hierarchical information into Class-Incremental Learning to reduce catastrophic forgetting by preserving feature drift through structured

Key concepts

Hierarchical Semantic Tree: This is an external knowledge source, generated by GPT-5, that defines the semantic relationships between concepts. It structures how broad categories like 'dog' relate to specific types like 'Labrador.' This tree provides the necessary supervision to organize features in a meaningful way.
Hyperbolic Space: Instead of standard Euclidean space where distances are flat, hyperbolic space is used because it naturally accommodates hierarchical structures. It allows the model to maintain parent-child relationships geometrically, making it easier to preserve class structure when learning new classes.
Gradient Projection for Stability: This technique ensures that updates made for a new class do not interfere with knowledge learned from previous classes. By projecting gradients onto a specific subspace (the null space of the shared mapper), the method stabilizes the model's weights, reducing catastrophic forgetting while still allowing adaptation to new information.

Terminology

Catastrophic Forgetting — 3× in this paper · explained in 9 episodes
Class-Incremental Learning — 2× in this paper
Class-Incremental Learning (CIL) — 1× in this paper · explained in 2 episodes
Hierarchical Structure — 1× in this paper · explained in 3 episodes
Knowledge Graph — 1× in this paper · explained in 8 episodes
Vision-language models — 1× in this paper · explained in 2 episodes

Summary

The gist The proposed HASTEN framework anchors hierarchical information into Class-Incremental Learning to reduce catastrophic forgetting by preserving feature drift through structured representations.

Motivation and Problem

Class-Incremental Learning (CIL) enables models to learn new classes continually while preserving past knowledge, yet existing CLIP-based CIL methods fail to explicitly capture the inherent hierarchy in real-world visual and linguistic concepts, leading to fine-grained class features drift during incremental updates and ultimately to catastrophic forgetting. This failure occurs because CLIP’s Euclidean feature space treats all points alike, which compresses distances and inadvertently pulls in unrelated specific concepts when broad concepts become too close to many specific ones. Correspondingly, without hierarchy, fine-grained class features gradually drift across incremental tasks as new classes emerge and previous classes shift from their original positions.

Hierarchy-Aware Anchoring: This involves training two hierarchy-aware modules for each task supervised by a GPT-5-generated hierarchical semantic tree. These features are then embedded in hyperbolic space via a globally shared hyperbolic mapper.
Gradient Projection for Stability: To mitigate catastrophic forgetting, the method projects gradients onto the null space of the shared hyperbolic mapper, preventing interference with prior tasks. This is achieved by maintaining an uncentered covariance C(b) and projecting the raw gradient ∆w onto the approximate null space V⊥V⊤⊥.

Hierarchical Loss and Alignment

The model's training objective is defined by combining several loss terms:

**- Hierarchical Loss: The Text-to-Text Hierarchical Loss for a parent-child edge (p, c) ∈ E is Ltxt→txt hier = Lentail(z p t, z c t) + SPδ − dB(z p t, z c t). This loss enforces the geometric constraint that each child node must lie within an aperture cone centered at its parent. The Text-to-Image Hierarchical Loss aligns the visual embedding zv with its corresponding class text node z c t: Ltxt→img hier = Lentailz c t, zv. The overall hierarchical objective is Lhier = λ1Ltxt→txt hier + λ2Ltxt→img hier. This loss stabilizes features by anchoring children within parent-centered cones with sufficient separation, preserving a hierarchy-respecting geometry across tasks. Hyperbolic Contrastive Loss further tightens crossmodal alignment using the negative hyperbolic distance from Eq. (4) as the similarity metric. This loss pulls matched pairs together and separates mismatches in B d. Hyperbolic Contrastive Loss is defined as Lcontrast = − 1/2 log(P I j) + log(P T j). Hyperbolic Contrastive Loss pulls matched pairs together and separates mismatches in B d, thereby stabilizing cross-task features and reinforcing cross-modal clusters. Hyperbolic Anchoring utilizes virtual-class anchoring to cache and freeze the hyperbolic text embeddings of all virtual nodes, while representing past classes k by drawing features from N(µk, Σk) into the hyperbolic space. This process creates class-level surrogates that mitigate cross-modal feature drift during training. Hyperbolic Contrastive Loss complements the conebased hierarchical supervision by further tightening crossmodal alignment. Hyperbolic Contrastive Loss is defined as Lcontrast = − 1/2 log(P I j) + log(P T j). Hyperbolic Contrastive Loss pulls matched pairs together and separates mismatches in B d, thereby stabilizing cross-task features and reinforcing cross-modal clusters. Hyperbolic Contrastive Loss is defined as Lcontrast = − 1/2 log(P I j) + log(P T j). Hyperbolic Contrastive Loss pulls matched pairs together and separates mismatches in B d, thereby stabilizing cross-task features and reinforcing cross-modal clusters. Hyperbolic Contrastive Loss is defined as Lcontrast = − 1/2 log(P I j) + log(P T j). Hyperbolic Contrastive Loss pulls matched pairs together and separates mismatches in B d, thereby stabilizing cross-task features and reinforcing cross-modal clusters.

Improvements for AI systems

Improved AI system can maintain hierarchical knowledge during continual learning by building a dataset-conditioned semantic tree using GPT-5 to provide structural supervision to feature learning. This allows for a dataset-specific hierarchical semantic tree that guides subsequent feature learning.
Improved AI system can preserve prior knowledge and resist catastrophic forgetting by projecting gradients onto the null space of features from previous tasks, as described in the paper: we project gradient updates onto the null space from prior-task features, preserving old-task outputs while learning new mappings.
Improved AI system can create unified text and image representations that respect semantic hierarchy by embedding features in hyperbolic space using a shared mapper, which naturally represents hierarchies. This is achieved by ensuring that the child embedding must lie inside a parent-centered cone of the parent through the entailment loss.
Improved AI system can ensure stable feature representations across incremental updates by using task-specific linear hierarchy-aware modules that are trained only for the new task while keeping all earlier modules frozen to preserve prior knowledge. This is done by aggregating outputs: We cumulatively aggregate the outputs of all modules to obtain the task-aware Euclidean features before mapping to the hyperbolic space.
Improved AI system can enhance cross-modal alignment and stability by employing a hyperbolic contrastive loss, which uses the negative hyperbolic distance from Eq. (4) as the similarity metric to pull matched pairs together in the hyperbolic space.
Improved AI system can improve robustness against hyperparameter choices by demonstrating that HASTEN's performance is robust to changes in both lambda1 and beta, making the framework practical for deployment with default settings of lambda1 = 0.5 and beta = 0.1.