Move by Move: Measuring and Steering How LLMs Conduct Psychotherapy

arXiv:2608.21325 · cs.CL · Submitted 2026-08-21 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Move by Move".

Jane: Detailed Research Summary of "Move by Move: Measuring and Steering How LLMs Conduct Psychotherapy" As a diligent researcher,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: The paper is titled "Move by Move: Measuring and Steering How LLMs Conduct Psychotherapy," and the authors are Afonso Baldo, Hugo Pitorro, Areti Vassilopoulos, Anabela C. Areias, Maya D’Eon Fabíola Costa, Ricardo Rei, and Nuno M. Guerreiro from Sword Health at Yale University. It sounds very focused on the mechanics of therapeutic interaction.

Jane: It sounds like they are trying to bridge that gap between what an LLM *says* and what a human therapist actually *does* in a session. They’re not just measuring if it’s good therapy; they’re mapping the specific steps taken during that therapy process.

Lu: The implication is that instead of just looking at the final dialogue, we can analyze the sequence of therapeutic actions to see where the AI deviates from established clinical practice. It suggests a way to audit model conduct systematically rather than just observing output quality.

Meng: So it’s less about making the AI sound like a therapist and more about ensuring its underlying decision-making process aligns with proven clinical steps. That shifts the focus from surface level performance to structural competence, which is what matters when you deploy something in a sensitive area like this.

Lalam: I think that structure is key because it gives us something concrete to measure against. If we can define the moves, we can then see exactly where our generation process misses those defined functions.

The paper's summary: Tom: So, in the summary of "Move by Move: Measuring and Steering How LLMs Conduct Psychotherapy," they apply this ten-move ontology to real counseling transcripts and compare them against how human clinicians actually conduct sessions. They found some pretty consistent differences between the two groups.

Jane: What are those differences? I remember reading that LLMs have a really high reliance on questioning, almost three times faster than a human therapist, but they also seem to skip some important parts of therapy entirely.

Lu: Specifically, they noted that models show a tendency to neglect psychoeducation, which is something the authors highlight as being critical for effective therapy. That’s a big gap in their autonomous skill production.

Meng: And another point they made was about context anchoring: the LLMs are very strongly context-anchored. They can carry forward strategies that a human clinician started, but they rarely initiate new ones on their own when leading the session independently. That points to a limitation in true autonomy.

Lalam: So it’s not that they can't remember what happened, it’s more about the initiative and the balance of those actions. They are good at carrying things forward, but not necessarily generating new therapeutic direction on their own.

The paper's improvements: Tom: The paper suggests a few ways to use this ontology to actually improve these models. One big idea is that by treating each move as a tool, you can guide the generation process directly. You can define what happens when the model calls a specific function, like `do skill building`.

Jane: So it’s about using agentic training where each therapeutic move is declared as a tool, and its description specifies exactly when that move is clinically appropriate and what the response should look like. That exposes the model to structured clinical decision-making.

Lu: They suggest that by exposing these tools, you can roughly halve the mean deviation from how human clinicians use their moves. And they claim this intervention improves turn-level alignment with human therapist choices by seven to nine percentage points without needing any fine-tuning of the underlying model weights <ref:2608.21325#pg2>.

Meng: That part about improving alignment without requiring fine-tuning is huge for us at scale. It suggests that guiding the behavior through these defined tools can be a very efficient way to steer performance toward clinical competence.

Lalam: It’s interesting because it moves away from just retraining the whole model and towards injecting a specific, structured set of rules into the generation process itself using those tools.

Conclusion: Tom: So to wrap up "Move by Move: Measuring and Steering How LLMs Conduct Psychotherapy," the authors have established this structured, empirically validated taxonomy for analyzing AI psychotherapy behavior. They’ve shown that while LLMs can mimic certain interaction aspects, they still lack the nuanced initiation and balance seen in human practice regarding things like psychoeducation and autonomous skill application.

Jane: It’s a powerful mechanism here because it gives both clinicians and developers a shared, auditable vocabulary for specifying and evaluating what an LLM is actually doing during a session. It helps quantify that gap we talked about earlier.

Lu: The implication is that this framework offers both clinicians for evaluation of current models and developers a precise toolset to guide the creation of more therapeutically competent agentic models going forward, specifically addressing those deficits in autonomous skill application.

Meng: For deployment, it means we have a clearer target for improvement. We can use this ontology to test if our agents are actually developing the skills they need when they're supposed to be leading sessions on their own.

Lalam: I think the core finding is that the structure matters more than just generating fluent text; defining these discrete functions gives us a path to measure and steer those functional deficits.

Tom: So, that’s what we’ve covered on "Move by Move: Measuring and Steering How LLMs Conduct Psychotherapy." It sets up a real conversation about how we build models that are not just conversational, but actually competent in complex human interactions. Next up, we're looking at some papers on vision and language adaptation.

Sword Health Yale University Instituto Superior Técnico

cs.CL

Submitted: 2026-08-21

Updated: 2026-10-19

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: Detailed Research Summary of "Move by Move: Measuring and Steering How LLMs Conduct Psychotherapy" As a diligent researcher, I have thoroughly analyzed the provided text snippets pertaining to the

Key concepts

Therapeutic Ontology: A standardized vocabulary of ten specific therapeutic moves derived from clinical literature. This provides a shared, auditable language for clinicians and developers to precisely define and evaluate the behavioral actions an LLM takes during therapy sessions.
Incessant Inquirers: A characteristic found in LLMs where they question patients at rates up to three times faster than human clinicians. This highlights a pattern of excessive questioning that needs adjustment for more natural therapeutic flow.
Context-Anchoring: The ability of LLMs to carry forward strategies initiated by a human therapist but rarely initiate new ones independently. While helpful, this shows LLMs struggle with autonomous strategy generation during independent sessions.
Move-Based Steering Mechanism: A method to guide LLM generation by treating each therapeutic move as a tool with specific rules for appropriateness and output. This allows developers to program structured clinical decision-making into the model's real-time responses.

Terminology

Steering — 1× in this paper · explained in 2 episodes

Summary

Detailed Research Summary of Move by Move: Measuring and Steering How LLMs Conduct Psychotherapy

As a diligent researcher, I have thoroughly analyzed the provided text snippets pertaining to the paper, Move by Move: Measuring and Steering How LLMs Conduct Psychotherapy. The core contribution of this work lies in establishing a rigorous framework for understanding and evaluating the therapeutic behavior exhibited by Large Language Models (LLMs) during simulated psychotherapy interactions.

Core Contribution: The Therapeutic Ontology

The central innovation of the paper is the introduction of a comprehensive ontology of ten therapeutic moves. This ontology is not arbitrary; it is meticulously constructed to be grounded in established clinical psychology literature, specifically drawing its foundation from the MULTI-60 inventory (McCarthy and Barber, 2009). These ten moves are defined as compact, function-based categories designed to be distinguishable from local linguistic context. Crucially, this ontology has been subjected to rigorous validation through an annotation campaign involving five licensed psychologists, lending significant clinical credibility to the proposed framework.

This ontology serves as a shared, auditable vocabulary that provides both clinicians and developers with a standardized means of specifying and evaluating the therapeutic behavior enacted by LLMs in therapy sessions.

Empirical Findings: Comparing LLM vs. Human Clinicians

The research proceeds by applying this ten-move ontology to real counseling transcripts alongside model-led sessions, allowing for a systematic comparison between how human clinicians and frontier models conduct psychotherapy. The analysis reveals several consistent and significant differences:

Inquiry Patterns: LLMs exhibit an excessive reliance on questioning. They are characterized as incessant inquirers, probing patients at rates up to three times the human rate.
Neglect of Psychoeducation: Models demonstrate a noticeable tendency to neglect psychoeducation, which is a critical component of effective therapy.
Contextual Anchoring: LLMs are strongly context-anchored. They possess the ability to carry forward strategies that were explicitly initiated by a human clinician but rarely initiate new strategies on their own when leading the session independently.
Skill Production Deficiencies: A notable gap exists in autonomous skill production; models fail to produce certain moves, such as Skill Building, when tasked with leading therapy sessions entirely on their own.

Impact of the Ontology: Bridging the Gap

The primary utility of this ontology is demonstrated by its effect on quantifying and narrowing the behavioral gap between LLMs and human therapists. By framing these clinical skill sets as discrete, measurable tools, the authors quantify how this framework impacts model performance:

Quantifiable Improvement: Exposing the ontology as a set of tools roughly halves the mean deviation from the human move distribution.
Alignment Enhancement: This intervention improves turn-level alignment with human therapist choices by 7–9 percentage points, achieved without requiring any fine-tuning of the underlying model.

Conclusion and Future Implications

In summary, Move by Move successfully establishes a structured, empirically validated taxonomy for analyzing LLM psychotherapy. The research highlights that while LLMs can mimic certain aspects of interaction (like carrying forward strategies), they lack the nuanced initiation and balance seen in human clinical practice—specifically regarding psychoeducation and autonomous skill application. The ontology provides a powerful mechanism to measure this deviation, offering both clinicians (for evaluation) and developers (for targeted improvement) a precise toolset to guide the development of more therapeutically competent agentic models.

Improvements for AI systems

A move-based steering mechanism will be integrated into LLM generation to improve alignment with human clinicians by roughly halving the mean deviation from the human move distribution and [improving] turn-level alignment with human therapist by 7–9 percentage points, without any fine-tuning.
LLMs can be explicitly guided using an ontology as a set of tools, where each therapeutic move is declared as a tool whose description specifies when the move is clinically appropriate, and its return value specifies how the response should be generated. This leverages agentic training to expose models to structured clinical decision-making.
The system prompt for synthetic members will include mandatory tool calls for every turn, forcing the model to call 1 or more moves every turn, directly implementing the ontology into real-time behavior.
The move classification system allows for a shared vocabulary, giving clinicians and developers a shared, auditable vocabulary for specifying and evaluating therapeutic behavior in LLMs. This enables systematic analysis of model conduct.
The model's response generation will be governed by strict rules derived from the ontology, such as Function over wording style, ensuring that moves are selected based on their therapeutic function rather than syntax.

Abstract

As the demand for mental health services outpaces clinical workforce capacity, users are increasingly turning to large language models (LLMs) for accessible and private support. Although recent evaluations suggest LLM care can reduce clinical symptoms, how models actually conduct therapeutic interactions compared to trained human clinicians remains largely unknown. Drawing on established clinical psychology frameworks that conceptualize therapy as a sequential decision-making process, we can systematically analyze the conversational interventions used in AI-driven sessions. To achieve this, we introduce an original, clinically validated ontology of ten distinct therapeutic moves to descriptively categorize strategies LLMs favor or neglect. Applying this ontology at scale reveals that models drastically over-use probing, disregard common CBT-favored techniques and are strongly context-anchored. Moreover, we propose exposing these categories as callable tools, bringing models significantly closer to the human clinical baseline.