MICA: Multi-granularity Intertemporal Credit Assignment for Long-Horizon Emotional Support Dialogue

arXiv:2603.06194 · cs.CL, cs.AI · Submitted 2026-03-06 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "MICA: Multi-granularity Intertemporal Credit Assignment for Long-Horizon Emotional Support Dialogue".

Jane: Here's the gist of the paper. The gist The MICA framework proposes a critic-free Reinforcement Learning approach for multi-turn emotional support tasks by deriving both immediate and delayed credit from a shared potential function over the user’s structured support state, enabling stable per-turn optimization without matched-state comparisons or learned critics.

Tom: MICA Framework Overview MICA is designed to address the challenge of sparse rewards and poor per-turn credit assignment in multi-turn emotional support dialogues by deriving both immediate and delayed credit from a shared potential function over the user’s structured support state The key idea is a multi-granularity credit assignment scheme that decomposes the value of a response into two coupled signals: its immediate effect on the user’s current support state and its delayed effect on the future trajectory Concretely, MICA derives both signals from a shared potential over the user’s structured support state: an Incremental Distance Reward (IDR) that measures turn-wise reduction in residual distance to a supported state, and the Monte Carlo return of IDR that captures future progress induced by the current response Because these two signals exhibit different systematic variation, MICA normalizes them at different scopes—Monte Carlo returns across samples at the same turn index, and immediate IDR across turns within a rollout group—yielding a practical critic-free advantage for stable per-turn optimization without matched-state comparisons, tree-based rollouts, or a learned value critic Reward Design and Normalization The framework utilizes two primary signals for credit assignment: the Incremental Distance Reward (IDR) and the Monte Carlo return The IDR is defined as the change in distance between consecutive turns, where rt = ϕ(xt−1, yt−1, zt−1) − ϕ(xt, yt, zt), which is positive when the assistant reduces the user’s empathetic distance The Monte Carlo return captures delayed effects by aggregating all future rewards after turn t To create a mixed advantage, MICA combines these signals via a convex mixture: Aˆ(i,j)t = αAˆ(i,j)t,turn + (1 − α)Aˆ(i,j)t,group This formulation integrates long-horizon returns with local dense feedback while avoiding exponential rollout cost and coarse trajectory-only credit assignment Optimization Scheme The policy gradient objective is optimized using the mixed advantage estimator, which combines turn-level normalized Monte Carlo returns with group-level normalized immediate rewards The turn-level advantage is computed by normalizing returns across samples at the same turn, accounting for turn-dependent return statistics The group-level advantage normalizes immediate rewards over a rollout group, emphasizing strong local signals while keeping gradients stable The final advantage is a convex combination of these two terms, and the policy parameters are updated via the on-policy objective L(θ) = E Aˆ(i,j)t log πθ(a(i,j)t h(i,j)t) Empirical Results and Performance MICA consistently outperforms GRPO and REINFORCE++ variants across various emotional benchmarks On EMPA, MICA improves the score by +42.5 on Qwen2.5-7B-Instruct and +28.2 on Qwen3-8B, raising Qwen3-32B to an EMPA Score of 84.2 These results demonstrate that MICA enables smaller models to reach performance levels that are competitive with strong closed-source baselines Furthermore, the mixed advantage estimator achieves the highest converged reward and avoids gradient norm explosion on both Qwen3-8B and Qwen2.5-7B-Instruct, suggesting it provides a more effective learning signal for long-horizon emotional support Ablation Studies The study shows that moderate turn-level weights, specifically α ∈

0.5, 0.7: , provide the highest mean EMPA Pass count on Qwen2.5-7B-Instruct Increasing α correlates with longer and more stable EMPA dialogues, indicating a behavioral shift toward longer and more stable dialogues under larger α Additionally, the Incremental Distance Reward (IDR) consistently outperforms the Absolute Distance Reward (ADR) across all three benchmarks under both Group-level and turn-level advantage settings The analysis of Judger effects shows that MICA is limited in sensitivity to the Judger choice, suggesting that the Judgers may differ in absolute reward calibration while preserving broadly consistent scoring directions MICA presents a critic-free RL framework for multi-turn interactive conversations by combining Incremental Distance Reward with a Mixed Advantage estimator that integrates turn-level Monte Carlo returns and group-level immediate rewards Experiments on EMPA, EQ-Bench, and EmoBench demonstrate that our approach consistently outperforms GRPO, yielding more stable and effective policies in open-ended, subjective settings Despite these gains, a key limitation is the reliance on environment-provided dense feedback which remains a challenge to obtain in many real-world scenarios Page 1 --- MICA: Multi-granularity Intertemporal Credit Assignment for Long-Horizon Emotional Support Dialogue Optimization Scheme The policy gradient objective is optimized using the mixed advantage estimator, which combines turn-level normalized Monte Carlo returns with group-level normalized immediate rewards The turn-level advantage is computed by normalizing returns across samples at the same turn, accounting for turn-dependent return statistics <ref:2603.

Paper summary: Tom: , so we're talking about this paper called MICA: Multi-granularity Intertemporal Credit Assignment for Long-Horizon Emotional Support Dialogue. The main idea here is that reinforcement learning works okay for single turns in large language models, but making it work for multi-turn emotional support is hard because the rewards are too sparse and you can't easily figure out which specific turn caused a good outcome.

Jane: , so if I'm driving and someone texts me something sad, my response affects how they feel later. But with this paper, they point out that we can't just compare states from one turn to the next because things are too complex, and looking at the whole conversation trajectory isn't enough for training.

Lu: , what MICA proposes is a critic-free reinforcement learning framework designed specifically for these multi-turn emotional support tasks. They claim this approach derives both immediate and delayed credit from a shared potential function that looks at the user’s structured support state across all turns.

Meng: , so it’s trying to get those credits—the reward—from two different angles simultaneously: what happened right now, and what will happen later in the conversation. That sounds complicated, but I need to know if it actually solves the credit assignment problem without needing some expensive learned critic.

Tom: , exactly. They introduce two specific signals for this: an Incremental Distance Reward, which measures how much the distance to a target state drops turn by turn, and a Monte Carlo return that captures those delayed effects over time.

Jane: , so it’s like they have one signal that tells you if you did a good job right now, and another signal that tells you if your actions set up the user for success in the long run. And then they combine those two using some normalization to get a mixed advantage.

Lu: , what makes it interesting is how they normalize these two signals at different scopes. They normalize the Monte Carlo return across samples at the same turn index, and they normalize the Incremental Distance Reward across turns within a rollout group.

Tom: , I like that idea of scope-specific normalization because it lets you get local feedback while still capturing those long-term effects without having to do massive trajectory rollouts every single time. And they call this mixed advantage estimator for stable per-turn optimization.

Jane: , so the paper suggests that by combining these two signals in a specific way, you can get a good signal for updating the policy turn by turn, even without needing matched-state comparisons or some kind of learned critic to tell you what's valuable.

Meng: , if this works without a learned critic and without expensive rollout costs, that’s huge for deployment because it keeps the learning process much cheaper and more stable on real dialogue data. What about the actual performance numbers they put out?

Paper summary: Lu: , they tested MICA on several emotional benchmarks, including EMPA, EQ-Bench, and EmoBench using models like Qwen2 point 5-7B-Instruct and Qwen3 models up to 32B parameters. The results show that MICA consistently outperforms GRPO and REINFORCE++.

Tom: , they claim a significant improvement on EMPA, achieving up to +forty-two point five on the Qwen2 point 5-7B-Instruct model, with some of those Qwen3 models reaching an EMPA Score of eighty-four point two compared to their previous scores. That’s a pretty solid number for a critic-free method.

Jane: , so what does this mean for someone who just listens to the show? It means that for interactive AI, especially in sensitive areas like emotional support, we might finally have a practical way to train them effectively across multiple turns without relying on super complex state comparisons.

Lu: , the authors also looked at how much weight they put on those two signals and found that moderate turn-level weights, specifically alpha between zero point five and zero point seven, gave the highest mean EMPA Pass count on Qwen2 point 5-7B-Instruct, and increasing alpha led to longer and more stable dialogues with larger values of alpha.

Tom: , that suggests there’s a sweet spot where you get both good local feedback and enough long-term guidance to keep the conversation going effectively, without letting it become chaotic. Also, they found that the Incremental Distance Reward was better than the Absolute Distance Reward on all three benchmarks under both advantage settings.

Meng: , I wonder what their stated limitations are because getting dense feedback in real-world scenarios is tough. What exactly does MICA stop doing? Is there a scenario where this framework just falls apart when we try to deploy it?

Jane: , the paper notes that while MICA works well, it still relies on environment-provided dense feedback, which is something that remains a challenge to obtain in many real-world scenarios. It’s not a perfect solution for every deployment situation.

Lu: , so the study shows MICA is effective and practical for multi-turn RL, but the authors themselves admit that relying on dense feedback from the environment is still a hurdle when trying to apply this in many real-world settings.

Tom: , so to wrap up on this paper, MICA is a critic-free reinforcement learning framework that uses a mixed advantage estimator combining turn-level Monte Carlo returns and group-level immediate rewards derived from a shared potential function over the user’s support state.

Jane: , it’s about finding a way to give large language models better credit for their long-term empathetic actions in conversations, even when we can't perfectly track every single state transition.

Lu: , yeah, it moves us toward having methods that don't rely on perfect matched-state comparisons or expensive rollout trees to handle those multi-turn challenges.

Meng: , from an engineering standpoint, if this means we can train these models on longer emotional dialogues more reliably without a massive computational overhead for the credit assignment part, that’s definitely worth paying attention to.

Tom: , so the paper shows that turn-aware credit assignment enables a way to make multi-turn RL for interactive LLMs more effective and practical by using this mixed advantage estimator.

Conclusion: Tom: So we've been looking at MICA, this new approach for reinforcement learning in multi-turn emotional support dialogues, and now we need to talk about what that title actually means for us.

Jane: Yeah, the title is "MICA: Multi-granularity Intertemporal Credit Assignment for Long-Horizon Emotional Support Dialogue." It’s trying to solve those credit assignment problems we talked about earlier.

Lu: Exactly. The core idea is this multi-granularity and intertemporal aspect, meaning it looks at different time scales—immediate and delayed—to figure out what the AI did right in a long conversation.

Meng: So instead of just looking at the next turn, it’s trying to calculate credit based on how the user's overall emotional support state is changing over time.

Tom: Right. It uses this shared potential function over the user's structured support state to derive both immediate and delayed credit signals, which is pretty clever for a critic-free setup.

Jane: The implication here is that we don't necessarily need those complicated matched-state comparisons or expensive rollout trees to get stable per-turn optimization in these kinds of interactive tasks.

Lu: If it works as they claim, it means we can train these models on longer emotional dialogues without incurring the massive computational cost of planning out every future step.

Meng: That would make deploying these models to handle real, ongoing conversations much more feasible at scale because the learning signal itself isn't exploding in complexity.

Tom: It’s about making the credit assignment practical and stable for long-horizon interactions, which is what we really needed for emotional support applications.

Jane: So it suggests that turn-aware credit assignment is a viable path toward building more effective multi-turn RL systems for interactive AI.

Lu: And the results they showed on EMPA, EQ-Bench, and EmoBench are quite compelling because MICA consistently outperforms other methods like GRPO and REINFORCE++.

Meng: But we have to remember their caveat—they still rely on environment-provided dense feedback, which is a limitation when we try to apply this in many real-world scenarios outside of controlled benchmarks.

Tom: That’s the catch. It works great on the test set, but getting that perfect feedback loop in messy real life is still a tough engineering problem for us to solve.

Jane: So MICA gives us a strong theoretical framework and good performance numbers, but it doesn't magically solve all deployment headaches yet.

Lu: We’re going to look at how they specifically tuned the alpha parameter, which controls the mix between immediate and delayed rewards, because that seems like a key lever for controlling dialogue length.

Meng: Yeah, understanding that tuning knob is crucial if we want to actually deploy this on a production system without making it run forever.

Naifan Zhang, Ruihan Sun, Jinwei Su, Hengjie Yang, Zhengyuan Pan, Zhaohan Chen

Tsinghua University · South China Normal University · Xiamen University

cs.CL, cs.AI

Submitted: 2026-03-06

Updated: 2026-10-21

Comments: The authors have identified that additional experiments and corrections are required. To prevent the dissemination of potentially inaccurate results, we request the withdrawal of this paper. A revised version will be submitted once the additional work is complete

Code: https://github.com/2200xiaohu/MAPO

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 89/100

The gist: The gist The MICA framework proposes a critic-free Reinforcement Learning approach for multi-turn emotional support tasks by deriving both immediate and delayed credit from a shared potential

Key concepts

Incremental Distance Reward (IDR)

This measures how much the assistant reduces the user's 'empathetic distance' between turns. It quantifies the positive change in support quality at each step by calculating the difference in a potential function value between consecutive states. A higher IDR means a better response for that specific turn.

Monte Carlo Return

This signal captures the delayed impact of an action by aggregating all rewards received from that point forward. It helps the model understand how its current response influences the user's state over many future turns, providing a long-term view of success rather than just immediate feedback.

Mixed Advantage Estimator

MICA combines two different types of credit signals—turn-level normalized Monte Carlo returns and group-level immediate rewards—using a convex mixture. This creates a single, stable advantage signal that optimizes the policy effectively by balancing short-term gains with long-term trajectory progress.

Structured Support State

This refers to the user's current emotional or support condition, which is represented by a potential function. Instead of looking at every possible state transition separately, MICA uses this shared structure to derive credit for both immediate and delayed actions efficiently.

Terminology

• Credit assignment — 4× in this paper · explained in 3 episodes
• Multi-granularity Intertemporal Credit Assignment — 2× in this paper
• GRPO — 1× in this paper · explained in 6 episodes
• Language Models (LLMs) — 1× in this paper · explained in 2 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 8 episodes
• Reinforcement Learning — 1× in this paper · explained in 4 episodes
• Reinforcement Learning (RL) — 1× in this paper · explained in 26 episodes

Summary

Improvements for AI systems

Abstract

Sources

• Affective Computing in the Era of Large Language Models: A Survey from the NLP Perspective
• Empathy and the Right to Be an Exception: What LLMs Can and Cannot Do
• Measuring Mathematical Problem Solving With the MATH Dataset
• Training Verifiers to Solve Math Word Problems
• Towards Open-Ended Emotional Support Conversations in LLMs via Reinforcement Learning with Future-Oriented Rewards
• RLVER: Reinforcement Learning with Verifiable Emotion Rewards for Empathetic Agents
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
• Group-in-Group Policy Optimization for LLM Agent Training
• EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models
• EmoBench: Evaluating the Emotional Intelligence of Large Language Models
• Qwen2.5 Technical Report
• Qwen3 Technical Report
• Cause-Aware Empathetic Response Generation via Chain-of-Thought Fine-Tuning
• Chain of Strategy Optimization Makes Large Language Models Better Emotional Supporter
• Echo-N1: Affective RL Frontier
• Agentic Reasoning: A Streamlined Framework for Enhancing LLM Reasoning with Agentic Tools
• ReTool: Reinforcement Learning for Strategic Tool Use in LLMs
• WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning
• REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization
• Proximal Policy Optimization Algorithms

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering