Mental Health AI Safety Claims Must Preserve Temporal Evidence

The gist Mental health AI safety claims must preserve temporal evidence because endpoint-style evaluations often fail to certify safety properties that depend on sequence, timing, accumulation, or recovery across extended interactions.

Temporal Safety Non-Identifiability

This formalization addresses the claim-evidence mismatch where an endpoint-style protocol observes an interaction only through a compression function ϕ: T → Z, and safety becomes non-identifiable when two temporally distinct interactions with different safety statuses produce the same compressed representation Temporal Safety Non-Identifiability. Proposition 1 states that if such trajectories exist, no deterministic evaluator that depends only on ϕ(τ) can correctly determine S for all trajectories in T, and likewise, no randomized evaluator whose output distribution depends only on ϕ(τ) can guarantee correct determination for both Compression-Induced Non-Identifiability of Temporal Safety. This means a safety property is identifiable from an evaluation representation only if the property is constant over the equivalence classes induced by that representation

SCOPE: Safety Claims Over Preserved Evidence

SCOPE is introduced as a general principle for aligning safety claims with the evidence an evaluation actually retains, and it is instantiated as SCOPE-MH for mental health AI. This standard requires evaluators to make explicit the relationship between the safety claim being asserted and the evidence the evaluation actually retains It mandates reporting on five diagnostic dimensions that endpoint-style evaluation commonly fails to certify, including longitudinal consistency, harm accumulation, intervention timing, recovery capability, and referral correctness The minimal operationalization for SCOPE-MH includes fields such as Safety claim and Preserved evidence

Operationalizing SCOPE-MH

SCOPE-MH is instantiated through five diagnostic dimensions to identify temporal safety properties that endpoint-style evaluation misses These dimensions are longitudinal consistency, harm accumulation, intervention timing, recovery capability, and referral correctness For instance, Longitudinal consistency requires tracking Track stance shifts, contradictions, boundary drift, and policy inconsistency Intervention timing requires measuring Measure escalation latency from first risk cue to appropriate intervention

Empirical Proof-of-Concept on Anno-MI

A proof-of-concept was conducted on the Anno-MI dataset of expert-annotated motivational interviewing conversations to audit what different evidence sources can observe and validly claim The analysis compared a per-turn behavior baseline with temporal client-language signals derived from change-talk and sustain-talk over the conversation Table 7 reports that the per-turn behavior baseline is the strongest classifier in this dataset, separating high- and low-quality conversations with d = 1.599 and p < 0.001 The temporal signal, sustain-talk delta (∆s), was found to be the strongest temporal signal and separates high- and low-quality conversations under a parametric test with p = 0.037

Limitations of Temporal Signals

The empirical results show that simple metrics do not capture all temporal failure modes; for example, Conversation 53 illustrates stagnation without overt resistance growth, which requires a separate stagnation-duration measure Conversation 131 demonstrates backsliding where change-talk drops even when sustain-talk remains acceptable, necessitating a backsliding index This confirms the SCOPE claim that Sustain-talk delta preserves evidence about one temporal property: resistance accumulation but does not preserve all temporal failure modes

Conclusion and Ethical Considerations

The paper concludes that endpoint-style evidence can support invalid safety conclusions when it is used to certify temporal safety properties whose determining evidence has been discarded The final position is that claims about delayed escalation, cumulative reinforcement, dependency formation, failed repair, or referral correctness require temporal evidence Deployers should treat temporal signals as one input among several and acknowledge the evaluation horizon as a first-class reporting requirement Privacy-preserving protocols are necessary complements for extended interaction logs containing sensitive mental health information Extended interaction logs contain sensitive mental health information, privacy-preserving protocols are necessary complements for extended interaction logs containing sensitive mental health information The question is not whether temporal evidence always improves prediction, but whether safety claims about temporal properties can be valid without preserving temporal evidence, and they cannot

References

Mental health AI safety claims must preserve temporal evidence.

SCOPE: Safety Claims Over Preserved Evidence.

A proof-of-concept on expert-annotated dialogues.

Conclusion and Ethical Considerations.

Ethical Considerations.

--- Page 1 ---

Mental Health AI Safety Claims Must Preserve Temporal Evidence Srimonti Dutta WAI USA Research Labs Texas, USA srimonti@womeninai.co Ratna Kandala University of Kansas Kansas, USA ratnanirupama@gmail.com Abstract The safety of mental health AI is often judged at the wrong temporal scale. Current evaluations typically score isolated responses, endpoint outcomes, or aggregate dialogue quality, while clinically consequential failures may arise from the order and accumulation of interactions themselves, including delayed escalation, repeated reinforcement, dependency formation, failed repair, and gradual deterioration across turns. This paper argues that this mismatch is not merely a limitation of evaluation coverage but a source of invalid safety conclusions. We introduce Temporal Safety Non-Identifiability, a formal account of why safety properties that depend on sequence, timing, accumulation, or recovery cannot be certified by protocols that discard those features. From this formalization, we develop SCOPE (Safety Claims Over Preserved Evidence) as a general principle for aligning safety claims with the evidence an evaluation actually retains, and instantiate it as SCOPE-MH, a mental-health instantiation of this reporting standard. We operationalize SCOPE-MH through a proof-of-concept on the Anno-MI dataset of expert-annotated motivational interviewing conversations, which reveals mechanisms of failure that per-turn behavior scoring does not represent. We propose SCOPE-MH as a diagnostic complement to existing evaluation infrastructure and argue that evaluation preserving temporal evidence is necessary, not optional, for safety-critical mental health AI deployment.

--- Page 2 ---

The scale of deployment makes this problem urgent. McBain et al. [2025] find that nearly one in eight US adolescents and young adults now use AI chatbots for mental health advice, with one in five among those aged 18 to 21 Rousmaniere et al. [2025] report that 48.7% of adults with ongoing mental health conditions use major LLMs therapeutically Hua et al. [2025] document that LLM-based architectures surged to 45% of mental health AI studies in 2024, yet only 16% underwent clinical efficacy testing This direction is consistent with broader mental health research emphasizing personalized temporal monitoring and dynamic recovery rather than isolated symptom states [Stringer, 2026, Bosl et al., 2025, Ngabo-Woods et al., 2025] A system that acts over time cannot be adequately assessed only by methods that observe final outcomes, isolated responses, or aggregate scores Mental health AI inherits evaluation practices from two mature traditions, neither designed to validate temporally extended interactive systems Clinical evaluation compresses interaction histories into endpoint outcomes such as symptom reduction, adherence, satisfaction, and adverse events; the Woebot randomized controlled trial, for example, evaluated a conversational Cognitive Behavioral Therapy (CBT) intervention using PHQ-9, GAD-7, and PANAS at baseline and follow-up [Fitzpatrick et al., 2017], and systematic reviews similarly synthesize evidence through depression, anxiety, distress, and well-being outcomes [Li et al., 2023] Machine learning evaluation introduces a different compression: fixed prompts, independent test instances, rubric judgments, preference scores, and aggregate benchmark performance, as in MMLU and HELM [Hendrycks et al., 2020, Liang et al., 2023] These methods are valuable, but mental health support rarely has the structure of an independent input-output task The meaning of a user’s message may depend on earlier hesitation, repeated reassurance-seeking, gradual disclosure, or escalating risk; the appropriateness of a response depends on where it occurs in the interaction Recent work has begun to address this gap through safety-oriented benchmark questions, LLM-based evaluation tools, realistic counseling scenarios, multi-turn mental health support benchmarks, riskspecific dashboards, role-aware adversarial interaction evaluation, and memory-based detection of manipulative patterns [Park et al., 2025, Li et al., 2025, Pombal et al., 2025, Zhang et al., 2026, Lee et al., 2026, Kandala et al., 2016]. These are important steps, but multi-turn coverage alone does not solve the validity problem. A benchmark may contain dialogue and still fail to support the intended safety claim if it reports only final responses, average scores, or event counts while discarding the temporal evidence on which the claim depends The problem is not merely benchmark coverage. It is claim validity.

--- Page 3 ---

We formalize this interaction mismatch as Temporal Safety Non-Identifiability.

1. Temporal Safety Non-Identifiability implementation: The AI system should be designed with an evaluation protocol that explicitly preserves temporal structure, ensuring that safety properties that depend on sequence, timing, accumulation, or recovery cannot be certified by protocols that discard those features. This means moving beyond endpoint outcomes to capture the full interaction trajectory.

2. SCOPE-MH reporting integration: The AI system's monitoring and reporting layer must adopt the SCOPE framework to explicitly state the safety claim being made, the temporal determinants of the claim, and whether it preserves evidence for that claim, thereby preventing safety claims that exceed their evidential scope.

3. Longitudinal consistency tracking: The system should incorporate a diagnostic dimension to track longitudinal consistency by monitoring track stance shifts, contradictions, boundary drift, and policy inconsistency to ensure the system maintains a coherent and safe stance as context evolves across extended dialogues.

4. Harm accumulation detection: Implement mechanisms to track harm accumulation by monitoring repeated reinforcement, dependency signals, cumulative risk markers, or worsening user-state indicators to prevent localized acceptable responses from compounding into systemic risk.

5. Intervention timing assessment: Develop a metric to measure intervention timing by tracking escalation latency from first risk cue to appropriate intervention, ensuring the system intervenes when risk first becomes detectable.

6. Recovery capability scoring: The AI should be assessed on recovery capability by tracking indicators such as acknowledgment, correction, redirection, stabilization, and subsequent user-state change following a miscalibrated or unsafe response.

7. Referral correctness verification: For systems involving referrals, the evaluation must assess referral correctness by monitoring referral timing, specificity, severity alignment, continuity, ensuring referrals occur at the right severity and time.

The safety of mental health AI is often judged at the wrong temporal scale. Current evaluations typically score isolated responses, endpoint outcomes, or aggregate dialogue quality, while clinically consequential failures may arise from the order and accumulation of interactions themselves, including delayed escalation, repeated reinforcement, dependency formation, failed repair, and gradual deterioration across turns. This paper argues that this mismatch is not merely a limitation of evaluation coverage but a source of invalid safety conclusions. We introduce Temporal Safety Non-Identifiability, a formal account of why safety properties that depend on sequence, timing, accumulation, or recovery cannot be certified by protocols that discard those features. From this formalization, we develop SCOPE (Safety Claims Over Preserved Evidence) as a general principle for aligning safety claims with the evidence an evaluation actually retains, and instantiate it as SCOPE-MH, a mental-health instantiation of this reporting standard. We operationalize SCOPE-MH through a proof-of-concept on the AnnoMI dataset of expert-annotated motivational interviewing conversations, which reveals mechanisms of failure that per-turn behavior scoring does not represent. We propose SCOPE-MH as a diagnostic complement to existing evaluation infrastructure and argue that evaluation preserving temporal evidence is necessary, not optional, for safety-critical mental health AI deployment.