Archive

vLLM Hybrid KV Cache

TorchInductor Pipeline 与算子融合

Learning TileLang with GEMV

NVLink vs PCIe in Practice: Bandwidth and LLM Serving

GPU Server Topology: NVLink, PCIe and NUMA

ggml Q4_0 vec_dot Walkthrough

KTransformers: CPU/GPU Hybrid Inference for MoE Models

torch.compile Walkthrough

Qwen3.5 Dense Model Architecture

TileLang Programming Model

Production Ready TileLang Ops Framework

ML Compilers Overview

vLLM and torch.compile

Piecewise CUDA Graph

Math for ML

ggml Memory Layout and Allocation

C++ Notes

[Sequence Modeling 01] RNN

[Sequence Modeling 03] Mamba

[Sequence Modeling 02] LSTM

[Sequence Modeling 04] Linear Attention

Cascade Speculative Drafting for Even Faster LLM Inference

LayerSkip Enabling Early Exit Inference and Self-Speculative Decoding

3-Model Speculative Decoding

Swift On-the-fly Self-speculative Decoding For LLM Inference Acceleration

Draft & Verify Lossless Large Language Model Acceleration via Self-Speculative Decoding

Hierarchical Speculative Decoding with Dynamic Windows for Efficient Language Model Inference

CAS-Spec Cascade Adaptive Self-Speculative Decoding for On-the-Fly Lossless Inference Acceleration of LLMs

Revati Transparent GPU-Free Time-Warp Emulation for LLM Serving

AIConfigurator Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving

R-Stitch Dynamic Trajectory Stitching for Efficient Reasoning

Estimating LLM Uncertainty with Evidence

XAttention Block Sparse Attention with Antidiagonal Scoring

FlexPrefill A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference

PyTorch Notes

Think Big, Generate Quick LLM-to-SLM for Fast Autoregressive Decoding

Entropy Adaptive Decoding Dynamic Model Switching for Efficient Inference

Beyond the 80 20 Rule High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning

Triton Notes: Tutorials and Kernel Paradigms

SLED A Speculative LLM Decoding Framework for Efficient Edge Serving

KVCache Cache in the Wild Characterizing and Optimizing KVCache Cache at a Large Cloud Provider

Token Level Routing Inference System for Edge Devices

SmoothQuant Accurate and Efficient Post-Training Quantization for Large Language Models

QServe W4A8KV4 Quantization and System Co-design for Efficient LLM Serving

LServe Efficient Long-sequence LLM Serving with Unified Sparse Attention

Quest Query-Aware Sparsity for Efficient Long-Context LLM Inference

Efficient Streaming Language Models with Attention Sinks

DuoAttention Efficient Long-Context LLM Inference with Retrieval and Streaming Heads

Dynamic Sparse Attention on Mobile SoCs

A dynamic parallel method for performance optimization on hybrid CPUs

EAGLE Speculative Sampling Requires Rethinking Feature Uncertainty

Splitwise Efficient Generative LLM Inference Using Phase Splitting

DistServe Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving

Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve

Sync Dot Files with Chezmoi

Scaling Up On-Device LLMs via Active-Weight Swapping Between DRAM and Flash

ELMS Elasticized Large Language Models On Mobile Devices

STI Turbocharge NLP Inference at the Edge via Elastic Pipelining

SmallThinker A Family of Efficient Large Language Models Natively Trained for Local Deployment

HeteroLLM Accelerating Large Language Model Inference on Mobile SoCs with Heterogeneous AI Accelerators

EdgeMoE Empowering Sparse Large Language Models on Mobile Devices

H2O Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models

A Survey of Resource-efficient LLM and Multimodal Foundation Models

LLM as a System Service on Mobile Devices

TPI-LLM Serving 70B-scale LLMs Efficiently on Low-resource Mobile Devices

Striped Attention Faster Ring Attention for Causal Transformers

Ring Attention with Blockwise Transformers for Near-Infinite Context

KV-Runahead Scalable Causal LLM Inference by Parallel Key-Value Cache Generation

LLM.int8() 8-bit Matrix Multiplication for Transformers at Scale

Git Essentials

LLM Generated Content

VS Code Essentials

LLM Preliminaries

Fast On-device LLM Inference with NPUs

Deja Vu Contextual Sparsity for Efficient LLMs at Inference Time

Transactions Papers Index

Distributed Papers Index

LLM in a flash Efficient Large Language Model Inference with Limited Memory

PowerInfer-2 Fast Large Language Model Inference on a Smartphone

PowerInfer Fast Large Language Model Serving with a Consumer-grade GPU

LLM Inference Papers Index

AWQ Activation-aware Weight Quantization for LLM Compression and Acceleration

SpecInfer Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification

LoRA Low-Rank Adaptation of Large Language Models

FlexGen High-Throughput Generative Inference of Large Language Models with a Single GPU

Efficient Memory Management for Large Language Model Serving with PagedAttention

EdgeLLM Fast On-Device LLM Inference With Speculative Decoding

A Survey on Efficient Inference for Large Language Models

Sonata Multi-Database Transactions Made Fast and Serializable

Concurrency Control as a Service

Orca A Distributed Serving System for Transformer-Based Generative Models

EdgeShard Efficient LLM Inference via Collaborative Edge Computing

ServerlessLLM Locality-Enhanced Serverless Inference for Large Language Models

NumPy Notes

Build a Large Language Model (From Scratch) Reading Note

Useful Websites and Extensions

Common Golang Mistakes

CLI & Server Notes

Jujutsu (jj) Tutorial

Tips and Guidance for Writing

Pass the Wall

Docker Network Introduction

Using Rust

[100 Mistakes in Golang 08-09] Chapter 8 & 9

[100 Mistakes in Golang 07] Chapter 7

[100 Mistakes in Golang 06] Chapter 6

[100 Mistakes in Golang 05] Chapter 5

[100 Mistakes in Golang 03] Chapter 3

[100 Mistakes in Golang 04] Chapter 4

[100 Mistakes in Golang 02] Chapter 2

Handling Golang Errors

Golang Struct and Interfaces

Rust Exercises Notes

From Clicks to Commands: A Windows User's Survival Guide

macOS Setup & Workflow

A Piece of: Golang Profile

[Missing Semester 05] Lecture 5

[Missing Semester 04] Lecture 4

[Missing Semester 02] Lecture 2

[Missing Semester 01] Lecture 1

Watchdogs in OpenFaaS

A Piece of: Logs

Data Management in Microservices: State of the Practice, Challenges, and Research Directions

Transaction Anomalies

Paper Note: Taking Omid to the Clouds: Fast, Scalable Transactions for Real-Time Cloud Analytics

Paper Note: Omid, Reloaded: Scalable and Highly-Available Transaction Processing

Programming Rust Book Note

Paper Note: Towards Transaction as a Service

Paper Note: Scalable Distributed Transactions across Heterogeneous Stores

Paper Note: GRIT: Consistent Distributed Transactions across Polyglot Microservices with Multiple Databases

Paper Note: How to Read a Paper

Talk about Postgres Visibility Check Rules

Paper Note: Ad Hoc Transactions in Web Applications: The Good, the Bad, and the Ugly

A Piece of: ThreadLocal

Note: WSL2 Mirrored 网络模式下异常情况总结

Paper Note: Cobra: Making Transactional Key-Value Stores Verifiably Serializable

Paper Note: Epoxy: ACID Transactions Across Diverse Data Stores

Paper Note: Zab: High-performance broadcast for primary-backup systems

[DDIA 09] Chapter 9 Consistency and Consensus

Paper Note: CAP Twelve Years Later: How the "Rules" have Changed

Research: 6.824 Lab2B 中异常情况的分析

Talk about Consistency and Consensus

Paper Note: Chain Replication

Paper Note: Time, clocks, and the ordering

Talk about Redlock

[DDIA 08] Chapter 8 The Trouble with Distributed Systems

Paper Note: Chubby

Paper Note: ZooKeeper

Paper Note: MapReduce

Paper Note: BigTable

[DDIA 07] Chapter 7 Transactions

[DDIA 06] Chapter 6 Partitioning

[DDIA 05] Chapter 5 Replication

[DDIA 04] Chapter 4 Encoding and Evolution

[DDIA 03] Chapter 3 Storage and Retrieval

[DDIA 02] Chapter 2 Data Models and Query Languages