Hearsay: Can an Auditor Trust the Record a Deployed Agent Harness Writes?
cs.CR, cs.AI, cs.CE, cs.SE
Submitted: 2026-09-26
Updated: 2026-09-26
Comments: 48 pages (9-page main text), 5 figures. Under review. Code and data: https://anonymous.4open.science/r/hearsay-4E7C
Code: https://github.com/open-telemetry/semantic-conventions-genai
License: http://creativecommons.org/licenses/by/4.0/
The gist: An agent harness, the code that turns a model into an agent, writes its own record of each run, and that record is all a later reader gets when a run is disputed, investigated or audited.
Terminology
Abstract
An agent harness, the code that turns a model into an agent, writes its own record of each run, and that record is all a later reader gets when a run is disputed, investigated or audited. We call a record evidentiary when a reader who was not there can check it without trusting the writer. Across sixteen deployed frameworks, none writes one in full. Hearsay examines the record, not the task: five harnesses run fourteen tasks, three blinded LLM examiners and a human panel read the records, and every excerpt an examiner quotes is checked mechanically for who wrote it. First, the record lets a reader name the fault but not prove how the run went. Examiners name the right fault in 74 to 91% of 140 runs, but the fault can be proved only from two files the benchmark adds; for what happened in between, fewer than one citation in ten lands on anything the harness did not write, and the examiner with the fewest false alarms catches half of the entries we delete, rewrite or fabricate. Second, the remedy is a second author, not a stronger seal on the first. An append-only log of what passes between harness and model, kept outside the harness and read against the record in both directions, reports all 28 omissions and fabrications we made a harness commit as it ran, where a hash chain over the harness's own record passes all 28. Handed the log, examiners keep their fault verdicts but rest more of their citations on what the harness did not write. What makes a record evidence is who writes it, not what is captured.
Sources
- Black-Box Access is Insufficient for Rigorous AI Audits
- Why Do Multi-Agent LLM Systems Fail?
- Visibility into AI Agents
- Seeing the Whole Elephant: A Benchmark for Failure Attribution in LLM-based Multi-Agent Systems
- TRAIL: Trace Reasoning and Agentic Issue Localization
- Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agents
- Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses
- Dive into Claude Code: The Design Space of Today's and Future AI Agent Systems
- Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?
- Auditable Agents
- Agent-Diff: Benchmarking LLM Agents on Enterprise API Tasks via Code Execution with State-Diff-Based Evaluation
- Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architectures
- DEMM-Bench: A Cross-Regime Benchmark for Agent-Runtime Governance-Evidence Sufficiency
- AgentTrace: Causal Graph Tracing for Root Cause Analysis in Deployed Multi-Agent Systems
- Architectural Design Decisions in AI Agent Harnesses
- $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
- AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?
- Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems
- AgentSight: System-Level Observability for AI Agents Using eBPF
Related papers
- SoK: AI-Augmented Binary Reversing
- Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
- Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
- Estimating the Social Cost of Corporate Data Breaches
- Efficient Fuzzy PSI under One-Sided Assumptions
- Sealing the Audit-Runtime Gap for LLM Skills