Posts

Intensive Reading Author Info Zichang Liu：Research Scientist at Meta. Jue Wang, Ph.D: Founder & President of Stylar AI (stylar.ai). Tri Dao: Assistant Professor of Computer Science at Princeton University. Chief Scientist at Together AI. Background LLM Inference Latency Breakdown Challenges Speeding up inference-time sparse LLMs in wall-clock time while maintaining quality and in-context learning abilities remains a challenging problem. While sparsity and pruning have been well-studied, they have not seen wide adoption on LLMs due to the poor quality and efficiency trade-offs on modern hardware such as GPUs: ...

Math Vector-Matrix Multiplication 从三个不同的角度分析向量乘以矩阵的运算过程 $xW$。 假设向量 $x$ 的形状是 $(1, 3)$，矩阵 $W$ 的形状是 $(3, 6)$。 $$x = \begin{bmatrix} x_1 & x_2 & x_3 \end{bmatrix}$$$$ W = \begin{bmatrix} w_{11} & w_{12} & w_{13} & w_{14} & w_{15} & w_{16} \\\\ w_{21} & w_{22} & w_{23} & w_{24} & w_{25} & w_{26} \\\\ w_{31} & w_{32} & w_{33} & w_{34} & w_{35} & w_{36} \end{bmatrix} $$根据矩阵乘法规则，结果 $y = xW$ 的形状将是 $(1, 6)$。 角度一：将 W 视为元素的二维集合 这是最基本、最微观的视角。我们将矩阵 $W$ 看作是一个 $3 \times 6$ 的数字网格。结果向量 $y$ 中的每一个元素 $y_j$，都是通过将向量 $x$ 的每个元素与其在矩阵 $W$ 中对应列的每个元素相乘，然后将结果相加得到的。 ...

My reading notes. 2025 2023 && 2024 bigtable cap-twelve-years-later zab mapreduce chubby chain-replication time, clocks, and the ordering farm zookeeper

My reading notes. 2025 0715-0721 Concurrency Control as a Service Sonata Multi-Database Transactions Made Fast and Serializable Uncategorized WIP 🚧 towards-transaction-as-a-service grit taking-omid-to-the-clouds epoxy ad-hoc-transactions-in-web-applications omid-reloaded data-management-in-microservices scalable-distributed-transactions-across-heterogeneous-stores cobra

Intensive Reading Author Info ‪Keivan Alizadeh-Vahid‬ - ‪Google Scholar‬ Iman Mirzadeh: An ML Research Engineer at Apple. Background LLM is hard for personal devices to load. The standard approach is to load the entire model into DRAM (Dynamic Random Access Memory) for inference. However, this severely limits the maximum model size that can be run. Challenges The primary challenge is that the memory footprint of large language models (LLMs) often exceeds the limited DRAM capacity of personal devices. While storing models on high-capacity flash memory is a potential solution, it introduces two new major challenges: ...

Intensive Reading Author Info Zhenliang Xue: From IPADS. Yixin Song: First author of PowerInfer. Zeyu Mi (糜泽羽): He is an associate professor at School of Software, Shanghai Jiao Tong University (SJTU). Haibo Chen [IPADS]: Director of Institute of Parallel and Distributed Systems. Background Sparsity FFN 的参数占比大，稀疏化特征也明显（特别是在使用 ReLU 时），所以可以在执行计算前利用一个 predictor 来预测哪些神经元会被激活，从而降低计算和 I/O 开销。 PowerInfer2 还探索了 LLM 推理过程中的动态稀疏性： 当批次很大时，对于任何一个神经元，只要它被输入中的至少一个激活，它在这一步的计算中就不是稀疏的。由于不同输入会激活不同神经元，其聚合效应导致大量神经元被激活，形成稳定、密集的“热点”，整体稀疏度显著降低。 由于某些序列会更早终止，所以有效批次的大小也会动态波动。这个实时变化导致了模型的计算模式在一个任务的生命周期内，会从一个接近稠密的模式平滑地过渡到一个高度稀疏的模式。 Mobile Hardware Characteristics 与 PC 相比，手机的硬件有两个特点： Heterogeneous computing capabilities with distinct sparse computation characteristics. CPU 更擅长稀疏计算 NPU 更擅长稠密计算 GPU 比 CPU 和 NPU 都更慢，而且在推理中使用 GPU 会影响设备的渲染帧率 移动 LLM 推理框架应同时利用异构处理器，以最大限度地利用共享内存带宽 Distinct storage architecture with unique I/O characteristics. 读的块大小越大，吞吐量越高 数据范围越小，吞吐量越高 频率越高的 CPU core 读取时吞吐量越高 UFS 并发能力有限 ...

Extensive Reading Author Info Ji Lin’s Homepage Jiaming Tang Shang Yang | MIT EECS Song Han - Associate Professor, MIT EECS Background Quantization is vital for running LLM on edge devices. Challenges Quantization-aware training (QAT) is not efficient due to the high training cost. Post-training quantization (PTQ) suffers from large accuracy degradation under a low-bit setting. Insights Not all weights in an LLM are equally important. Protecting only 1% salient weights can greatly reduce quantization error. To identify salient weight channels, we should refer to the activation distribution, not weights. Mixed-precision format is not hardware-efficient, we can employ activation-aware scaling. Approaches Activation-aware Weight Quantization ...

My reading notes. 2026 0203-0209 Cascade Speculative Drafting for Even Faster LLM Inference CAS-Spec Cascade Adaptive Self-Speculative Decoding for On-the-Fly Lossless Inference Acceleration of LLMs Draft & Verify Lossless Large Language Model Acceleration via Self-Speculative Decoding Swift On-the-fly Self-speculative Decoding For LLM Inference Acceleration 3-Model Speculative Decoding Hierarchical Speculative Decoding with Dynamic Windows for Efficient Language Model Inference LayerSkip Enabling Early Exit Inference and Self-Speculative Decoding AIConfigurator Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving Revati Transparent GPU-Free Time-Warp Emulation for LLM Serving 0127-0202 FlexPrefill A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference XAttention Block Sparse Attention with Antidiagonal Scoring SLED A Speculative LLM Decoding Framework for Efficient Edge Serving R-Stitch Dynamic Trajectory Stitching for Efficient Reasoning Estimating LLM Uncertainty with Evidence Entropy Adaptive Decoding Dynamic Model Switching for Efficient Inference Think Big, Generate Quick LLM-to-SLM for Fast Autoregressive Decoding 2025 Remaining Beyond the 80 20 Rule High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning KVCache Cache in the Wild Characterizing and Optimizing KVCache Cache at a Large Cloud Provider 1111-1117 LServe Efficient Long-sequence LLM Serving with Unified Sparse Attention QServe W4A8KV4 Quantization and System Co-design for Efficient LLM Serving Quest Query-Aware Sparsity for Efficient Long-Context LLM Inference Dynamic Sparse Attention on Mobile SoCs A dynamic parallel method for performance optimization on hybrid CPUs SmoothQuant Accurate and Efficient Post-Training Quantization for Large Language Models DuoAttention Efficient Long-Context LLM Inference with Retrieval and Streaming Heads Efficient Streaming Language Models with Attention Sinks KTransformers Unleashing the Full Potential of CPU GPU Hybrid Inference for MoE Models 1104-1110 EAGLE Speculative Sampling Requires Rethinking Feature Uncertainty 1028-1103 Aegaeon Effective GPU Pooling for Concurrent LLM Serving on the Market DistServe Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving Splitwise Efficient Generative LLM Inference Using Phase Splitting Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve 0826-0901 ELMS Elasticized Large Language Models On Mobile Devices Scaling Up On-Device LLMs via Active-Weight Swapping Between DRAM and Flash 0819-0825 STI Turbocharge NLP Inference at the Edge via Elastic Pipelining EdgeMoE Empowering Sparse Large Language Models on Mobile Devices LLM as a System Service on Mobile Devices SmallThinker A Family of Efficient Large Language Models Natively Trained for Local Deployment HeteroLLM Accelerating Large Language Model Inference on Mobile SoCs with Heterogeneous AI Accelerators A Survey of Resource-efficient LLM and Multimodal Foundation Models H2O Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models 0812-0818 KV-Runahead Scalable Causal LLM Inference by Parallel Key-Value Cache Generation Striped Attention Faster Ring Attention for Causal Transformers Ring Attention with Blockwise Transformers for Near-Infinite Context TPI-LLM Serving 70B-scale LLMs Efficiently on Low-resource Mobile Devices LLM.int8() 8-bit Matrix Multiplication for Transformers at Scale 0729-0804 Fast On-device LLM Inference with NPUs Deja Vu Contextual Sparsity for Efficient LLMs at Inference Time PowerInfer-2 Fast Large Language Model Inference on a Smartphone LLM in a flash Efficient Large Language Model Inference with Limited Memory PowerInfer Fast Large Language Model Serving with a Consumer-grade GPU 0722-0728 AWQ Activation-aware Weight Quantization for LLM Compression and Acceleration FlexGen High-Throughput Generative Inference of Large Language Models with a Single GPU LoRA Low-Rank Adaptation of Large Language Models SpecInfer Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification EdgeLLM Fast On-Device LLM Inference With Speculative Decoding Efficient Memory Management for Large Language Model Serving with PagedAttention 0715-0721 A Survey on Efficient Inference for Large Language Models -0714 Orca A Distributed Serving System for Transformer-Based Generative Models EdgeShard Efficient LLM Inference via Collaborative Edge Computing ServerlessLLM Locality-Enhanced Serverless Inference for Large Language Models Uncategorized WIP 🚧 ...

Intensive Reading Author Info ‪Yixin Song‬ - ‪Google Scholar‬ Zeyu Mi (糜泽羽): He is an associate professor at School of Software, Shanghai Jiao Tong University (SJTU). Haotong Xie (谢昊彤) Haibo Chen [IPADS]: Director of Institute of Parallel and Distributed Systems. Background Local deployments focus on low latency in processing small batches. LLM inference exhibits notable sparsity in neuron activation, a phenomenon observed in both self-attention and MLP blocks. The offloading technique leverages the CPU’s additional computational and memory resources. GPU-centric offloading utilizes CPU memory to store portions of the model parameters that exceed the GPU’s capacity. Lead to substantial per-token latency mainly due to frequent data transfers between GPU and CPU. Over 99.5% of processing time is consumed by transferring LLM weights from CPU to GPU. Hybrid offloading distributes model parameters between GPU and CPU, splitting them at the Transformer layer level. The CPU processes its layers first, then sends intermediate results to the GPU for token generation. The CPU, with higher memory but lower computational power, ends up handling 98% of the total computational time. ...

Extensive Reading Author Info Ying Sheng: She got her Ph.D. in Computer Science at Stanford University (Centaur), where she was advised by Clark Barrett. Before that, she received an M.S. in Computer Science from Columbia University in 2017 and a B.E. in Computer Science and Technology from ACM Honored Class, Shanghai Jiao Tong University in 2016. Lianmin Zheng: He is a member of technical staff at xAI. His research interests include machine learning systems, large language models, compilers, and distributed systems. Previously, he completed his Ph.D. at UC Berkeley, where he was advised by Ion Stoica and Joseph E. Gonzalez. Binhang Yuan(袁彬航) – Assistant Profossor@CSE HKUST: He is an assistant professor in the Department of Computer Science & Engineering (CSE), also affiliated with World Sustainable Development Institute, at the Hong Kong University of Science and Technology (HKUST). He is leading the Relaxed System Lab. Background Prior efforts to lower resource requirements of LLM inference correspond to three directions: ...

Extensive Reading Author Info About | Edward Hu: Edward Hu is a founding partner in a stealth AI company in Woodside, CA. He was a researcher at OpenAI and received his research training as a Ph.D. student advised by Yoshua Bengio, a recipient of the 2018 A.M. Turing Award. Before graduate school, Edward was a researcher at Microsoft, where he invented LoRA and μTransfer. Yelong Shen - Microsoft | AMiner Background The dominant paradigm in modern NLP is ...