Entropy Adaptive Decoding Dynamic Model Switching for Efficient Inference
Extensive Reading Author Info Background The Problem: Standard decoding applies the same computational power to every token generated. However, text generation has heterogeneous complexity. A complex logical deduction in a mathematical proof requires significantly more “intelligence” than generating routine connecting phrases (e.g., “therefore,” “it follows that”). The Limitation of Existing Solutions: Current optimization techniques, such as Speculative Decoding, are conservative. They prioritize perfect output fidelity, ensuring the output matches the large model exactly by verifying every token. The authors argue this is unnecessary for many applications. Insights The paper’s Proposal: Entropy Adaptive Decoding (EAD). Dynamically switches between a small model ($M_S$) and a large model ($M_L$) during generation. Unlike speculative decoding, EAD accepts controlled output divergence—meaning the output might differ from what the large model would have produced alone, provided the reasoning remains sound. So why not use EAD when divergence occurs in Speculative Decoding? ...