FearCaut-Qwen: Affective Steering in a Vision-Language Model Shifts the Decision Criterion for Hazard Assessment

The gist The study examines that VLM hazard decisions can be adjusted at inference time without retraining and demonstrates how mechanistic interpretability can be used to diagnose and control VLM decision behaviors >

How it works

The research investigates the reluctance of Vision–language models (VLMs) to declare a hazard, which is addressed by decomposing the decision behavior into perceptual capability and decision-criterion placement using signal detection theory (SDT) > The authors propose a novel method, FearCaut-Qwen, inspired by human findings that fear makes humans risk-averse, to correct this overconservative decision policy > They localize an affective circuit in the model using mechanistic interpretability and then use activation steering to manipulate it while observing the effect on downstream prediction >

The methodology involves several distinct stages for localization and intervention >

**- The affective direction is localized on emotion-rich natural scenes containing no structural damage, causally validated by sparse-neuron knockout and distributed steering on held-out emotion data > This process involves layer-wise probing to identify a candidate band of decoder layers, followed by sparse neuron selection using four complementary selectors to rank neurons for each target emotion against the neutral reference > The final set of neurons, denoted as K, is then frozen based on validation balanced accuracy > The distributed component is computed as a class-mean difference in the residual stream at each candidate layer using the contrastive-activation construction established for behavioural steering > The intervention operators are implemented as a sparse operator that multiplies selected coordinates by a scalar s, and a distributed operator that adds a scaled unit direction to the residual stream > These interventions are installed as forward pre-hooks on the downprojection and forward hooks on the block output, respectively > The analysis then uses SDT to decompose every resulting behavioural change into discrimination, d', and decision criterion, c > This allows them to attribute a gain in recall to the mechanism that produced it > The study tests this on a two-stage SeisMLLM pipeline built on Qwen2.5-VL-7B-Instruct, which flags only 27.0% of genuinely unsafe buildings on the published SeisMLLM1K test split and never issues a false Red > The method is tested on a two-stage SeisMLLM pipeline built on Qwen2.5-VL-7B-Instruct, which flags only 27.0% of genuinely unsafe buildings on the published SeisMLLM1K test split and never issues a false Red > The decomposition of sensitivity and decision criterion is important because, as shown in Figure 3, if a benchmark contains many Red cases, a liberal criterion can raise recall, accuracy and macro-F1 while discrimination is constant or worse > The final comparison shows that the positive distributed fear/threat steering raised Red recall from 0.270 to 0.757, a paired change of +0.486 (95% CI [+0.306, +0.667], p Subtracting the same direction was directionally symmetric: Red recall fell to 0.000, a change of −0.270 (95% CI [−0.419, −0.133], p The decomposition identifies the mechanism unambiguously: the criterion moved by −1.515 while discrimination was not improved (Δd' = −0.493), a ratio of 3.1 > The second result is that the sparse fear/threat neurons are demonstrably necessary for source emotion recognition but do not carry the building criterion shift > A residual-stream direction pools many weakly aligned features, including components well outside any selector’s top ranks, and it is that pooled object which reaches the downstream computation > The most plausible mechanism is a distributed representational shift in the residual stream during perception, which changes what the perceptual report says and then propagates through an unmodified reasoning stage > The final conclusion is that a model’s willingness to declare a building unsafe is a distinct system property from its ability to tell unsafe from safe > Criterion placement nonetheless remains an engineering decision that must be made against an explicit loss function, the available capacity for human review, and the expected prevalence [73,74] > The final finding is that reporting d' and c alongside conventional metrics costs nothing and prevents that error > The paper concludes with the statement that the internal route is not merely a re-description of output calibration, but rather a shift in decision policy > This demonstrates how mechanistic interpretability can be used to diagnose and control VLM decision behaviors in engineering applications > The paper is bounded by the scope of the experiments, as they use one backbone, one affect corpus and one building benchmark > The statistical power limitation means they rest on the large bidirectional criterion change and its stage localization instead of on small accuracy differences > The paper suggests that activation patching is the natural next step to trace where the causal effect is actually transmitted > The paper concludes with the statement that a model’s willingness to declare a building unsafe is a distinct system property from its ability to tell unsafe from safe > The final finding is that reporting d' and c alongside conventional metrics costs nothing and prevents that error > The paper concludes with the statement that a model’s willingness to declare a building unsafe is a distinct system property from its ability to tell unsafe from safe > The final finding is that reporting d' and c alongside conventional metrics costs nothing and prevents that error > The paper concludes with the statement that a model’s willingness to declare a building unsafe is a distinct system property from its ability to tell unsafe from safe > The final finding is that reporting d' and c alongside conventional metrics costs nothing and prevents that error > The paper concludes with the statement that a model’s willingness to declare a building unsafe is a distinct system property from its ability to tell unsafe from safe > The final finding is that reporting d' and c alongside conventional metrics costs nothing and prevents that error > The paper concludes with the statement that a model’s willingness to declare a building unsafe is a distinct system property from its ability to tell unsafe from safe > The final finding is that reporting d' and c alongside conventional metrics costs nothing and prevents that error > The paper concludes with the statement that a model’s willingness to declare a building unsafe is a distinct system property from its ability to tell unsafe from safe > The final finding is that reporting d' and c alongside conventional metrics costs nothing and prevents that error > The paper concludes with the statement that a model’s willingness to declare a building unsafe is a distinct system property from its ability to tell unsafe from safe > The final finding is that reporting d' and c alongside conventional metrics costs nothing and prevents that error > The paper concludes with the statement that a model’s willingness to declare a building unsafe is a distinct system property from its ability to tell unsafe from safe > The final finding is that reporting d' and c alongside conventional metrics

1. textbfShift Decision Criterion via Affective Steering at Inference Time (FearCaut-Qwen): Act as a criterion lever by injecting an affective direction into the residual stream to shift decision boundaries, as demonstrated by FearCaut-Qwen, which moves that criterion from inside the model at inference time using an affective direction localized on decontaminated natural scenes. This allows for hazard assessment adjustments without retraining and demonstrate how mechanistic interpretability can be used to diagnose and control VLM decision behaviors in engineering applications.

2. textbfDecompose Decision Behavior via Signal Detection Theory (SDT): Explicitly separate perceptual capability from decision criterion to quantify the effect, using the formula d! = Φ−1(H) − Φ−1(F) c = −½[Φ−1(H) + Φ). This provides an engineering metric to distinguish between a model that has really learned something and one that has simply become readier to flag everything Red.

3. textbfIdentify Causal Affective Circuits: Localize fear/threat circuits using mechanistic interpretability: Use sparse neuron selectors and Borda consensus to identify sparse set N of emotion-sensitive coordinates, validating their necessity via knockout tests, ensuring the intervention is targeted at the causally necessary circuit rather than a generic perturbation.

4. textbfImplement Stage-Specific Intervention: Apply perception-only steering for maximum effect: Since Perception-only steering recovers 94.4% of the effect, this should be prioritized in modular pipelines where the goal is to change the character of the perceptual report before reasoning occurs, as Reasoning-only steering [increased Red recall by only 0.027 (95% CI [+0.000, +0.091]) is insufficient."

5. textbfEnsure Source Specificity: Validate interventions against four null control families: Since Knockout of the identified fear-responsive units produced an observed effect of +0.129, exceeding the 95% empirical null interval for all four control families, this ensures that the observed behavioral change is specific to the fear circuit and not a generic consequence of removing neurons.