From Black-Box to Clinical Insight: A Multi-Stage Explainable Framework for Speech-Based Cognitive Impairment Detection

The gist The framework proposes a multi-stage explainability system that translates black-box transformer predictions into clinically grounded narratives by integrating SHAP attribution, linguistic features, and an LLM reasoning pipeline.

Model Performance

The SpeechCARE-Adaptive Gating Network multimodal screening model achieved an Area Under the Curve (AUC) of 86.83% and an F1-score of 72.11% on the official Test set released by the PREPARE challenge, confirming its balanced effectiveness in accurately detecting cognitive impairment The model achieved an Area Under the Curve (AUC) of 86.83% and an F1-score of 72.11% on the official Test set released by the PREPARE challenge organizers>.

Explainability Framework Components

The proposed framework systematically connects transformer predictions to clinically interpretable linguistic evidence through three complementary components The framework integrates three complementary components: (1) SHAP adapted for transformer architectures with hierarchical aggregation of subword attributions to word-level explanations; (2) complementary linguistic features capturing clinically relevant dimensions such as lexical richness and syntactic complexity; and (3) sequential Large Language Model (LLM) orchestration, a four-stage reasoning pipeline using LLaMA-3.1-70B-Instruct [16] to translate model attributions and linguistic evidence into structured, plain-language clinical explanations>.

SHAP Adaptation and Linguistic Features

The SHAP adaptation for transformer models addresses subword tokenization by implementing a model wrapper that encodes input text using the tokenizer, computes transformer embeddings, and outputs class probabilities To account for subword tokenization, we applied hierarchical SHAP value aggregation to map subword-level attributions back to interpretable wordlevel representations>. Furthermore, theory-informed linguistic features are extracted across four clinically grounded domains: (1) Lexical richness (e.g. type–token ratios, Brunet’s and Honore’s indices); (2) Syntactic complexity (e.g. average clause length, part-of-speech diversity); (3) Disfluencies and repetition (e.g. speech rate, prolonged pauses); and (4) Semantic coherence While SHAP values provide local token-level explanations, they lack explicit grounding in linguistically grounded interpretability>.

Multi-Stage LLM Reasoning Pipeline

The multi-stage LLM reasoning pipeline uses LLaMA-3.1-70B-Instruct [16] to translate model attributions and linguistic evidence into structured, plain-language clinical explanations Our pipeline chained four specialized LLMs, each configured with distinct prompts and knowledge sources>. The stages are as follows:

1. Stage 1 - token-level SHAP interpretation: This LLM received SHAP scores as structured input (token, SHAP value) alongside the transcript and the model’s prediction (control, MCI, AD) An instruction prompt guided the LLM to map token importance to six cognitive-linguistic dimensions: lexical richness, syntactic complexity, disfluencies and repetition, semantic coherence, spatial reasoning difficulty, and executive function>.

2. Stage 2 - feature-level linguistic interpretation: Theoryinformed linguistic features corresponding to the six cognitivelinguistic dimensions were provided as structured inputs The LLM was prompted to interpret these quantitative patterns in relation to cognitive-linguistic functioning>.

3. Stage 3 - cross-source interpretation aggregation: This stage received the outputs of Stages 1 and 2 as structured input and was prompted to compare SHAP-based token attributions with feature-level metrics This stage received the outputs of Stages 1 and 2 as structured input and was prompted to compare SHAP-based token attributions with feature-level metrics, integrating convergent evidence into a single interpretation>.

4. Stage 4 - structured report generation: The final stage in the sequence performed a ”polish and summarize” workflow For each dimension, the model generated concise bullet-point explanations, providing a clinically interpretable report accessible to support clinical readability>.

Clinical Validation and Utility

Two primary care physicians independently evaluated the structured reports under blinded conditions, with the structured reports found to align closely with each patient’s individual cognitive-linguistic biomarkers Quantitatively, physicians agreed on 98% of cases, with Cohen’s κ = 0.85>. Furthermore, a usability study showed that the assessment achieved an overall usability score of 82 out of 100 The clinicians reported that the provided linguistic summaries offer actionable evidence that can enhance decision-making for the patient evaluation for cognitive impairment>.

Future Directions

The paper notes that future work should extend to acoustic transformer explanations by translating mHuBERT attention patterns and acoustic representations into clinically interpretable narratives This framework extends the SpeechCARE Adaptive Gating Network (AGF) [17], a multimodal screening model that integrates acoustic and linguistic transformer representations via a Mixture-of-Experts–inspired fusion architecture>.

The current framework focuses on linguistic explainability; future work should extend to acoustic transformer explanations by translating mHuBERT attention patterns and acoustic representations into clinically interpretable narratives This paper is the first to present a multi-stage explainability framework showing that sequentially combining SHAP attributions, theory-informed linguistic features, and LLM-based reasoning can bridge the gap between black-box transformer predictions and clinically meaningful narratives>.

Speech analysis provides a functional complement to recently approved biomarker technologies such as plasma assays (pTau217 and βamyloid approved by FDA in 2025), which reflect neuropathology but do not capture its functional impact on communication In contrast, speech quantifies everyday cognitive-linguistic function through prosody, acoustic parameters, and semantic organization Clinician evaluation supports the feasibility and clinical utility of individualized explanations within diagnostic workflows, positioning sequential LLM orchestration as a promising paradigm for XAI-enabled cognitive screening.

The paper is the first to present a multi-stage explainability framework showing that sequentially combining SHAP attributions, theory-informed linguistic features, and LLM-based reasoning can bridge the gap between black-box transformer predictions and clinically meaningful narratives Speech analysis provides a functional complement to recently approved biomarker technologies such as plasma assays (pTau217 and βamyloid approved by FDA in 2025), which reflect neuropathology but do not capture its functional impact on communication.

The framework maps model outputs to four cognitive-linguistic dimensions, including lexical richness, syntactic complexity, and semantic coherence Physician evaluation on 70 stratified English samples demonstrated strong alignment with patient-level cognitive profiles, and a System Usability Scale score of 82/100 indicated high potential for clinical workflow integration.

The framework maps model outputs to four cognitive-linguistic dimensions, including lexical richness, syntactic complexity, and semantic coherence <ref:2606.

1. Bold header: Translation of SHAP attributions into clinically meaningful narratives

This improvement allows "translates black-box transformer predictions into clinically grounded narratives by integrating SHapley Additive exPlanations (SHAP)-based token attribution, theory-informed linguistic features, and a four-stage LLM reasoning pipeline. This enables the system to generate structured, plain-language clinical explanations" that link specific model outputs to patient profiles.

2. Bold header: Integration of theory-informed linguistic feature extraction

The system can now explicitly extract features such as (1) Lexical richness (e.g. type–token ratios, Brunet’s and Honore’s indices) and (2) Syntactic complexity (e.g., average clause length, part-of-speech diversity), moving beyond token-level explanations to capture clinically relevant dimensions like lexical richness and syntactic complexity.

3. Bold header: Sequential LLM orchestration for progressive interpretive refinement

The four-stage pipeline allows the system to perform (Stage 1 - token-level SHAP interpretation) followed by (Stage 2 - feature-level linguistic interpretation), enabling a progressive refinement of interpretations, as described in the paper's manual orchestration sequential pipeline.