Training-Free Speculative Decoding & Token Reuse
Aug 2026 – PresentAdvisor: Prof. Jiacheng Shen
Optimized the draft-proposal path for training-free LLM inference. Multi-source suffix retrieval, trie-based candidate fusion, and confidence-calibrated, cost-aware selection reduced decode-phase wall time by 20.4% against the original Suffix proposer on Llama 3.1-8B with an NVIDIA A40, across five full benchmark runs.
Reworked sequential per-request speculation into batched, cross-request parallel proposal with a persistent C++ thread pool and GIL release. At batch size 16, CPU proposal time fell from 280.6 to 102.9 μs (2.73× speedup) with bitwise-identical drafts.