#import "@tola/source:0.0.0": tola-meta #import "@tola-modules/nasapunk:0.0.0": img #tola-meta(( title: "AIConfigurator Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving", date: "2026-02-05", summary: "AIConfigurator profiles GEMM, attention, communication and memory primitives once per GPU and composes them to predict serving performance, searching static, aggregated and disaggregated deployments without benchmarking each one.", tags: ("arXiv-2601", "llm-serving", "auto-tuning"), )) #quote(block: true)[ AI-Aided ] = Author Info = Background The primary challenges identified in optimizing LLM inference in production are the #strong[combinatorial explosion of configuration parameters] (parallelism strategies, batch sizes, quantization) and the diversity of inference frameworks (TensorRT-LLM, vLLM, SGLang), which makes manual tuning or exhaustive GPU benchmarking prohibitively #strong[expensive and slow]. = Insights Instead of modeling an entire neural network as a black box, the system breaks down LLM inference into fundamental, reusable operations called primitives, profile these primitives then combine the statistics to model the LLM inference process. = Challenges = Approaches == Primitive Decomposition and Offline Profiling Inference is modeled as a sequence of specific kernels: - #strong[Computation]: GEMM (General Matrix Multiply) operations parameterized by dimensions and precision (FP16, FP8, INT8). - #strong[Attention]: Compute-bound context attention and memory-bound generation attention. - #strong[Communication]: Operations like AllReduce, AllGather, and Point-to-Point transfers. - #strong[Memory]: Data movement and cache operations. #img("/assets/images/pasted-image-20260205104341.png", alt: "pasted-image-20260205104341") #strong[Performance Database (PerfDatabase)]: The system builds a hardware-specific database by profiling these primitives once on target hardware (e.g., NVIDIA H100). #quote(block: true)[ For configurations not explicitly profiled, it uses interpolation or theoretical roofline models (Speed-of-Light estimation) to predict latency. ] == Simulation and Performance Modeling #img("/assets/images/pasted-image-20260205104503.png", alt: "pasted-image-20260205104503") The system uses the database to reconstruct end-to-end performance estimates. It explicitly models three distinct serving modes to capture complex runtime behaviors: - #strong[Static Mode]: Models sequential processing with fixed batch sizes. This serves as a baseline where Time-To-First-Token (TTFT) equals prefill latency, and Time-Per-Output-Token (TPOT) is the average decode step latency. - #strong[Aggregated Mode (Continuous Batching)]: Models modern engines that mix prefill and decode phases within the same iteration. The algorithm accounts for resource contention and uses a rate-matching heuristic to throttle decode requests if the prefill phase dominates, preventing \"starvation\" of new requests. - #strong[Disaggregated Mode]: Models architectures where prefill and decode occur on separate GPU pools. The model calculates the optimal ratio of prefill-to-decode workers and explicitly accounts for the network overhead required to transfer the KV cache between nodes. == Automated Search and Generation The optimization process follows a multi-step workflow: - #strong[TaskRunner]: Constructs a valid search space based on user constraints (e.g., TTFT \< 100ms) and hardware availability. - #strong[InferenceSession]: Iterates through candidate configurations (varying tensor parallelism, pipeline parallelism, expert parallelism), querying the database to estimate performance for each. - #strong[Pareto Analyzer]: Ranks configurations to find the optimal trade-off between system throughput (tokens/second/GPU) and generation speed (tokens/second/user). - #strong[Generator]: Automatically translates the theoretical optimal configuration into framework-specific launch flags (e.g., setting `--enable_cuda_graph` or specific memory fractions) for the target engine (TRT-LLM, vLLM, etc.). #img("/assets/images/pasted-image-20260205104554.png", alt: "pasted-image-20260205104554") = Evaluation #img("/assets/images/pasted-image-20260205104121.png", alt: "pasted-image-20260205104121") #img("/assets/images/pasted-image-20260205104153.png", alt: "pasted-image-20260205104153") - #strong[Accuracy]: The approach achieves high fidelity (low error rates compared to ground truth) by accounting for complex behaviors like expert load imbalance and network overhead in disaggregated setups. - #strong[Speed]: The simulation-based search is #strong[significantly faster]. = Thoughts == When Reading = Related Works