SWE-Journey: Towards More Realistic Evaluation of Coding Assistants through Long-Horizon, Multi-Turn Interaction

arXiv:2610.11559 · cs.CL, cs.AI, cs.MA, cs.SE · Submitted 2026-10-08 · Read on arXiv

cs.CL, cs.AI, cs.MA, cs.SE

Submitted: 2026-10-08

Updated: 2026-10-08

Code: https://github.com/SWE-agent/mini-swe-agent

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Verification — 56× in this paper · explained in 2 episodes
• Benchmark — 49× in this paper · explained in 2 episodes
• Benchmarking — 7× in this paper · explained in 4 episodes
• Accuracy — 6× in this paper · explained in 1 episode
• Grounding — 5× in this paper · explained in 3 episodes
• Trace — 5× in this paper · explained in 2 episodes
• Verifier — 5× in this paper · explained in 2 episodes
• Adaptation — 4× in this paper · explained in 1 episode
• Alignment — 4× in this paper
• Core — 3× in this paper · explained in 2 episodes
• Abstraction — 2× in this paper · explained in 1 episode
• Agentic Coding — 2× in this paper · explained in 1 episode
• Decode — 2× in this paper · explained in 2 episodes
• Earth Rotation Angle — 2× in this paper
• Embeddings — 2× in this paper · explained in 2 episodes
• LLM-as-a-Judge — 2× in this paper · explained in 3 episodes
• RAG — 2× in this paper · explained in 3 episodes
• Robustness — 2× in this paper · explained in 5 episodes
• Steering — 2× in this paper · explained in 2 episodes
• AI Agents — 1× in this paper · explained in 2 episodes
• Adam — 1× in this paper · explained in 2 episodes
• Agent Interactions — 1× in this paper
• Clustering — 1× in this paper · explained in 2 episodes
• Consistency — 1× in this paper · explained in 3 episodes
• Decoupling — 1× in this paper · explained in 2 episodes
• Failure Modes — 1× in this paper · explained in 3 episodes
• Fidelity — 1× in this paper · explained in 2 episodes
• Generative Model — 1× in this paper · explained in 2 episodes
• Generative Models — 1× in this paper · explained in 3 episodes
• Hysteresis — 1× in this paper · explained in 3 episodes
• Provenance — 1× in this paper · explained in 5 episodes
• Reasoning Traces — 1× in this paper · explained in 3 episodes
• Redundancy — 1× in this paper · explained in 2 episodes
• Semantic Alignment — 1× in this paper · explained in 4 episodes
• State Management — 1× in this paper · explained in 3 episodes
• Transformer — 1× in this paper · explained in 2 episodes

Sources

• MirrorCode: AI can rebuild entire programs from behavior alone
• SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents
• Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation
• SWE-chat: Coding Agent Interactions From Real Users in the Wild
• FeatBench: Towards More Realistic Evaluation of Feature-level Code Generation
• SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration
• Doc2Feat-Bench: Evaluating Documentation-Driven Feature Addition
• SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
• SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?
• NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents
• nnMobileNet++: Towards Efficient Hybrid Networks for Retinal Image Analysis
• An Empirical Study on Developers Shared Conversations with ChatGPT in GitHub Pull Requests and Issues
• SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories?
• R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents
• Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents
• Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety
• SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades
• RefineBench: Evaluating Refinement Capability of Language Models via Checklists
• RACE-Bench: A Reasoning-Augmented Benchmark for Repository-Level Code Agents on Feature Addition
• SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering