AIConfigurator Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving

AIConfigurator profiles GEMM, attention, communication and memory primitives once per GPU and composes them to predict serving performance, searching static, aggregated and disaggregated deployments without benchmarking each one.

Doc no.
KKZ-TN-127
Date
2026-02-05
GMT day
2026:036
Est. read
3 min
AI-Aided

01Author Info#

02Background#

The primary challenges identified in optimizing LLM inference in production are the combinatorial explosion of configuration parameters (parallelism strategies, batch sizes, quantization) and the diversity of inference frameworks (TensorRT-LLM, vLLM, SGLang), which makes manual tuning or exhaustive GPU benchmarking prohibitively expensive and slow.

03Insights#

Instead of modeling an entire neural network as a black box, the system breaks down LLM inference into fundamental, reusable operations called primitives, profile these primitives then combine the statistics to model the LLM inference process.

04Challenges#

05Approaches#

5.1Primitive Decomposition and Offline Profiling#

Inference is modeled as a sequence of specific kernels:

  • Computation: GEMM (General Matrix Multiply) operations parameterized by dimensions and precision (FP16, FP8, INT8).
  • Attention: Compute-bound context attention and memory-bound generation attention.
  • Communication: Operations like AllReduce, AllGather, and Point-to-Point transfers.
  • Memory: Data movement and cache operations.
pasted-image-20260205104341
Figure 1

Performance Database (PerfDatabase): The system builds a hardware-specific database by profiling these primitives once on target hardware (e.g., NVIDIA H100).

For configurations not explicitly profiled, it uses interpolation or theoretical roofline models (Speed-of-Light estimation) to predict latency.

5.2Simulation and Performance Modeling#

pasted-image-20260205104503
Figure 2

The system uses the database to reconstruct end-to-end performance estimates. It explicitly models three distinct serving modes to capture complex runtime behaviors:

  • Static Mode: Models sequential processing with fixed batch sizes. This serves as a baseline where Time-To-First-Token (TTFT) equals prefill latency, and Time-Per-Output-Token (TPOT) is the average decode step latency.
  • Aggregated Mode (Continuous Batching): Models modern engines that mix prefill and decode phases within the same iteration. The algorithm accounts for resource contention and uses a rate-matching heuristic to throttle decode requests if the prefill phase dominates, preventing "starvation" of new requests.
  • Disaggregated Mode: Models architectures where prefill and decode occur on separate GPU pools. The model calculates the optimal ratio of prefill-to-decode workers and explicitly accounts for the network overhead required to transfer the KV cache between nodes.

5.3Automated Search and Generation#

The optimization process follows a multi-step workflow:

  • TaskRunner: Constructs a valid search space based on user constraints (e.g., TTFT < 100ms) and hardware availability.
  • InferenceSession: Iterates through candidate configurations (varying tensor parallelism, pipeline parallelism, expert parallelism), querying the database to estimate performance for each.
  • Pareto Analyzer: Ranks configurations to find the optimal trade-off between system throughput (tokens/second/GPU) and generation speed (tokens/second/user).
  • Generator: Automatically translates the theoretical optimal configuration into framework-specific launch flags (e.g., setting --enable_cuda_graph or specific memory fractions) for the target engine (TRT-LLM, vLLM, etc.).
pasted-image-20260205104554
Figure 3

06Evaluation#

pasted-image-20260205104121
Figure 4
pasted-image-20260205104153
Figure 5
  • Accuracy: The approach achieves high fidelity (low error rates compared to ground truth) by accounting for complex behaviors like expert load imbalance and network overhead in disaggregated setups.
  • Speed: The simulation-based search is significantly faster.

07Thoughts#

7.1When Reading#

Referenced by
  1. KKZ-TN-076
    LLM Inference Papers Index
    2025-07-28