CAS-Spec Cascade Adaptive Self-Speculative Decoding for On-the-Fly Lossless Inference Acceleration of LLMs

Extensive Reading

01Author Info#

02Background#

• Existing "Self-Speculative Decoding" (SSD) methods are easy to use (training-free) but often slower than methods that rely on training specialized draft models.
• "Cascade Speculative Decoding" (using a hierarchy of draft models) offers high speed but is impractical because it requires training and maintaining multiple draft models.

03Insights#

The paper proposes Cascade Adaptive Self-Speculative Decoding (CAS-Spec).

This framework constructs a "virtual" hierarchy of draft models directly from the target model itself, without needing extra training. It effectively combines

• the convenience of self-speculation
• the high performance of cascade decoding.

04Challenges#

• Can existing training-free self-speculative decoding methods be used to construct an effective cascade of draft models above retrieval-based methods for on-the-fly speculative decoding?
• Can we achieve further speedup by adaptively routing the draft models and assigning the draft lengths, with regards to the characteristics of different DSIA strategies?

05Approaches#

5.1EWIF#

The basic formula for EWIF (denoted as ) typically looks like this:

In the paper, it is formalized as:

Where:

• (Alpha): The Acceptance Rate. It represents the probability that the Target Model accepts a token generated by the Draft Model.

  • High High EWIF (Good).

• (Cost Coefficient): The relative cost of the Draft Model compared to the Target Model.

  • Low High EWIF (Good).
• : The number of draft tokens generated in one step.

分子是等比数列求和：

分母是本次操作消耗的归一化时间：

5.2DSIA#

Dynamically Switchable Inference Acceleration (DSIA): These are techniques that can be toggled during inference to trade accuracy for speed. By applying different DSIA configurations, the system creates a hierarchy of draft models ().

• Layer Sparsity: Skipping a subset of transformer layers or blocks to generate tokens faster.
• Activation Quantization: Using lower precision (e.g., INT4) for activations and KV caches during draft generation.
• Early-Exiting: Using predictions from intermediate layers.
• Bottom Draft Model: The hierarchy typically ends with a "base" draft model that is extremely fast and computationally negligible, such as Prompt Lookup Decoding (PLD).

Cascade Strategies: The paper outlines three ways to combine these strategies into a hierarchy:

• Mixing-DSIA: Combining orthogonal strategies (e.g., Layer Sparsity + Activation Sparsity).
• Replacing-DSIA: Switching between conflicting strategies (e.g., different attention mechanisms).
• Scaling-DSIA: Using the same strategy with different intensities (e.g., different degrees of sparsity).

5.3Dynamic Tree Cascade (DyTC) Algorithm#

Since there are multiple "virtual" draft models available, the system needs a smart way to decide which model to use and how many tokens to generate at any given step. The DyTC algorithm handles this scheduling.

DyTC dynamically selects the optimal draft model and draft length based on real-time heuristics rather than fixed schedules. It aims to maximize the "Expected Walltime Improvement Factor" (EWIF).

Key heuristics:

• Acceptance Rate Estimation: It tracks the acceptance rate () of each draft configuration using an Exponential Moving Average (EMA) to adapt to changing generation contexts

• Hardware-Aware Latency Prediction: It estimates the cost (latency) of each DSIA strategy using a regression model based on the hardware's roofline performance.

Instead of a single sequence, DyTC constructs a draft token tree:

• It expands the tree by selecting the "leaf" node with the highest accumulated acceptance probability (using acceptance rate)
• It uses Tree Attention to verify multiple branches in parallel, increasing the chance of accepting a longer sequence of tokens

在构建 Attention Tree 时

• 使用 Acceptance Rate Estimation 来寻找扩展点和计算 EWIF 的分子部分
• 使用 Hardware-Aware Latency Prediction 计算 EWIF 的分母部分
• Vertical Cascade 可以用于构建树的不同分支
• Horizontal Cascade 可以用于构建同一个分支的不同 token

06Evaluation#

07Thoughts#

7.1When Reading#

08Related Works#