JetSpec: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Drafting
cs.CL
Submitted: 2026-06-16
Updated: 2026-10-03
Code: https://github.com/hao-ai-lab/JetSpec
License: http://creativecommons.org/licenses/by/4.0/
The gist: Speculative decoding (SD) accelerates autoregressive Large Language Models (LLMs) by drafting multiple tokens and verifying them in parallel, but it faces a scaling limitation: increasing the draft
Terminology
Abstract
Speculative decoding (SD) accelerates autoregressive Large Language Models (LLMs) by drafting multiple tokens and verifying them in parallel, but it faces a scaling limitation: increasing the draft budget improves speed only when acceptance remains high and drafting overhead stays low. This ceiling has been difficult to break because prior head-based SD methods face a causality-efficiency dilemma. Autoregressive drafters produce path-conditioned candidates that are effective for tree speculative decoding with higher acceptance length, but their drafting cost grows with tree depth. Bidirectional block-diffusion drafters generate all positions in one pass, but their branch-agnostic marginals can form individually plausible yet mutually inconsistent trees, wasting budget and reducing acceptance. We propose JetSpec, a head-based SD framework that combines one-forward drafting efficiency with branch-wise causal conditioning. JetSpec trains a causal parallel draft head over fused hidden states from the frozen target model, producing candidate trees whose scores align with the target model's autoregressive factorization. This enables JetSpec to convert larger draft budgets into longer accepted prefixes and higher end-to-end speedup. Across math, coding, and chat benchmarks on dense and MoE Qwen3 models, JetSpec consistently outperforms bidirectional-head and tree-based SD baselines. On H100 GPUs, JetSpec achieves up to 9.64x speedup on MATH-500 and 4.58x on open-ended conversational workloads, with further latency gains demonstrated through vLLM integration under realistic serving loads. Our code and models are available at https://github.com/hao-ai-lab/JetSpec.
Sources
- Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters
- Program Synthesis with Large Language Models
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
- Accelerating Large Language Model Decoding with Speculative Sampling
- DFlash: Block Diffusion for Flash Speculative Decoding
- Evaluating Large Language Models Trained on Code
- Training Verifiers to Solve Math Word Problems
- DeepSeek-V3 Technical Report
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- Better & Faster Large Language Models via Multi-token Prediction
- Direct Alignment of Draft Model for Speculative Decoding with Chat-Fine-Tuned LLMs
- Fast and Accurate Causal Parallel Decoding using Jacobi Forcing
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees
- EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
- EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test
- CD4LM: Consistency Distillation and aDaptive Decoding for Diffusion Language Models
- SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications
- d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillation
- Accelerating Speculative Decoding with Block Diffusion Draft Trees
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering