Source or It Didn't Happen: A Multi-Agent Framework for Citation Hallucination Detection
cs.CL
Submitted: 2026-05-09
Updated: 2026-10-13
Code: https://github.com/aaFrostnova/CiteTracer
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large language models are increasingly used in scientific writing, yet they can fabricate citation-shaped references that appear plausible but fail bibliographic verification.
Terminology
Abstract
Large language models are increasingly used in scientific writing, yet they can fabricate citation-shaped references that appear plausible but fail bibliographic verification. Existing detectors often reduce verification to binary found/not-found decisions and rely on brittle parsing or incomplete retrieval, offering little field-level signal to auditors. We reframe citation hallucination detection as taxonomy-aligned field-level adjudication and introduce a 12-code taxonomy spanning Real, Potential, and Hallucinated citations. Based on this taxonomy, we build CiteTracer, a cascading multi-agent detector that extracts structured citations from PDF and BibTeX, retrieves evidence through cache lookup, URL fetch, scholar connectors, and web search, applies deterministic field matching, and routes ambiguous cases to class-specialist judgers. We release a benchmark of 2,450 synthetic citations built from real seeds with controlled LLM mutations, paired with 957 real-world fabricated citations drawn from desk-rejected submissions at ICLR 2026 and an anonymous conference. CiteTracer reaches 97.1% accuracy on the synthetic benchmark, with class-level F1 scores of 97.0, 95.8, and 98.5 for Real, Potential, and Hallucinated, respectively, and detects 97.1% of fabrications on the real-world set without abstaining. Code: https://github.com/aaFrostnova/CiteTracer.
Sources
- Learning Soft Linear Constraints with Application to Citation Field Extraction
- Qwen3-VL Technical Report
- Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory
- AutoData: A Multi-Agent System for Open Web Data Collection
- CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era
- Web Retrieval Agents for Evidence-Based Misinformation Detection
- A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models
- DeepSeek-OCR 2: Visual Causal Flow
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering