Tags

KTransformers: CPU/GPU Hybrid Inference for MoE Models

ggml with C++

LServe Efficient Long-sequence LLM Serving with Unified Sparse Attention

DuoAttention Efficient Long-Context LLM Inference with Retrieval and Streaming Heads

EAGLE Speculative Sampling Requires Rethinking Feature Uncertainty

Scaling Up On-Device LLMs via Active-Weight Swapping Between DRAM and Flash

ELMS Elasticized Large Language Models On Mobile Devices

STI Turbocharge NLP Inference at the Edge via Elastic Pipelining

HeteroLLM Accelerating Large Language Model Inference on Mobile SoCs with Heterogeneous AI Accelerators

EdgeMoE Empowering Sparse Large Language Models on Mobile Devices

TPI-LLM Serving 70B-scale LLMs Efficiently on Low-resource Mobile Devices

LLM Preliminaries

Fast On-device LLM Inference with NPUs

Deja Vu Contextual Sparsity for Efficient LLMs at Inference Time

LLM in a flash Efficient Large Language Model Inference with Limited Memory

PowerInfer-2 Fast Large Language Model Inference on a Smartphone

PowerInfer Fast Large Language Model Serving with a Consumer-grade GPU

AWQ Activation-aware Weight Quantization for LLM Compression and Acceleration

SpecInfer Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification

FlexGen High-Throughput Generative Inference of Large Language Models with a Single GPU

Efficient Memory Management for Large Language Model Serving with PagedAttention

EdgeLLM Fast On-Device LLM Inference With Speculative Decoding

A Survey on Efficient Inference for Large Language Models

Orca A Distributed Serving System for Transformer-Based Generative Models

EdgeShard Efficient LLM Inference via Collaborative Edge Computing

ServerlessLLM Locality-Enhanced Serverless Inference for Large Language Models

ggml with C++

CLI Notes

Server Management

Software Router

Sync Dot Files with Chezmoi

Clang on Apple

Git Essentials

VS Code Essentials

Deep Learning Basic

NumPy Notes

Useful Websites

Common Golang Mistakes

Effective macOS Shortcuts

Dev Operations

Jujutsu (jj) Tutorial

Chrome Plugin Recommendations

Pass the Wall

Docker Network Introduction

Handling Golang Errors

Golang Struct and Interfaces

Aerospace

From Clicks to Commands: A Windows User's Survival Guide

Mac Development Environment Setup

A Piece of: Logs

Note: WSL2 Mirrored 网络模式下异常情况总结

Data Management in Microservices: State of the Practice, Challenges, and Research Directions

Paper Note: Taking Omid to the Clouds: Fast, Scalable Transactions for Real-Time Cloud Analytics

Paper Note: Omid, Reloaded: Scalable and Highly-Available Transaction Processing

Paper Note: Towards Transaction as a Service

Paper Note: Scalable Distributed Transactions across Heterogeneous Stores

Paper Note: GRIT: Consistent Distributed Transactions across Polyglot Microservices with Multiple Databases

Paper Note: How to Read a Paper

Paper Note: Ad Hoc Transactions in Web Applications: The Good, the Bad, and the Ugly

Paper Note: Cobra: Making Transactional Key-Value Stores Verifiably Serializable

Paper Note: Epoxy: ACID Transactions Across Diverse Data Stores

Paper Note: Zab: High-performance broadcast for primary-backup systems

Paper Note: CAP Twelve Years Later: How the "Rules" have Changed

Paper Note: Chain Replication

Paper Note: Time, clocks, and the ordering

Paper Note: Chubby

Paper Note: ZooKeeper

Paper Note: MapReduce

Paper Note: BigTable

Build a Large Language Model (From Scratch) Reading Note

Tips and Guidance for Writing

[100 Mistakes in Golang 08-09] Chapter 8 & 9

[100 Mistakes in Golang 07] Chapter 7

[100 Mistakes in Golang 06] Chapter 6

[100 Mistakes in Golang 05] Chapter 5

[100 Mistakes in Golang 03] Chapter 3

[100 Mistakes in Golang 04] Chapter 4

[100 Mistakes in Golang 02] Chapter 2

Programming Rust Book Note

[DDIA 09] Chapter 9 Consistency and Consensus

[DDIA 08] Chapter 8 The Trouble with Distributed Systems

[DDIA 07] Chapter 7 Transactions

[DDIA 06] Chapter 6 Partitioning

[DDIA 05] Chapter 5 Replication

[DDIA 04] Chapter 4 Encoding and Evolution

[DDIA 03] Chapter 3 Storage and Retrieval

[DDIA 02] Chapter 2 Data Models and Query Languages

[DDIA 09] Chapter 9 Consistency and Consensus

[DDIA 08] Chapter 8 The Trouble with Distributed Systems

[DDIA 07] Chapter 7 Transactions

[DDIA 06] Chapter 6 Partitioning

[DDIA 05] Chapter 5 Replication

[DDIA 04] Chapter 4 Encoding and Evolution

[DDIA 03] Chapter 3 Storage and Retrieval

[DDIA 02] Chapter 2 Data Models and Query Languages

Dynamic Sparse Attention on Mobile SoCs

Scaling Up On-Device LLMs via Active-Weight Swapping Between DRAM and Flash

ELMS Elasticized Large Language Models On Mobile Devices

SmallThinker A Family of Efficient Large Language Models Natively Trained for Local Deployment

Fast On-device LLM Inference with NPUs

PowerInfer-2 Fast Large Language Model Inference on a Smartphone

EdgeLLM Fast On-Device LLM Inference With Speculative Decoding

EdgeShard Efficient LLM Inference via Collaborative Edge Computing

[100 Mistakes in Golang 08-09] Chapter 8 & 9

[100 Mistakes in Golang 07] Chapter 7

[100 Mistakes in Golang 06] Chapter 6

[100 Mistakes in Golang 05] Chapter 5

[100 Mistakes in Golang 03] Chapter 3

[100 Mistakes in Golang 04] Chapter 4

[100 Mistakes in Golang 02] Chapter 2

A dynamic parallel method for performance optimization on hybrid CPUs

ELMS Elasticized Large Language Models On Mobile Devices

A Survey of Resource-efficient LLM and Multimodal Foundation Models

LLM as a System Service on Mobile Devices

Striped Attention Faster Ring Attention for Causal Transformers

PowerInfer-2 Fast Large Language Model Inference on a Smartphone

A Survey on Efficient Inference for Large Language Models

Estimating LLM Uncertainty with Evidence

LServe Efficient Long-sequence LLM Serving with Unified Sparse Attention

Dynamic Sparse Attention on Mobile SoCs

Scaling Up On-Device LLMs via Active-Weight Swapping Between DRAM and Flash

SmallThinker A Family of Efficient Large Language Models Natively Trained for Local Deployment

HeteroLLM Accelerating Large Language Model Inference on Mobile SoCs with Heterogeneous AI Accelerators

KVCache Cache in the Wild Characterizing and Optimizing KVCache Cache at a Large Cloud Provider

LServe Efficient Long-sequence LLM Serving with Unified Sparse Attention

Quest Query-Aware Sparsity for Efficient Long-Context LLM Inference

Efficient Streaming Language Models with Attention Sinks

DuoAttention Efficient Long-Context LLM Inference with Retrieval and Streaming Heads

H2O Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models

Revati Transparent GPU-Free Time-Warp Emulation for LLM Serving

AIConfigurator Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving

Splitwise Efficient Generative LLM Inference Using Phase Splitting

DistServe Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving

Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve

Transaction Anomalies

Talk about Postgres Visibility Check Rules

Talk about Consistency and Consensus

About CAP

Talk about Redlock

Ring Attention with Blockwise Transformers for Near-Infinite Context

Deja Vu Contextual Sparsity for Efficient LLMs at Inference Time

AWQ Activation-aware Weight Quantization for LLM Compression and Acceleration

Efficient Memory Management for Large Language Model Serving with PagedAttention

[Missing Semester 05] Lecture 5

[Missing Semester 04] Lecture 4

[Missing Semester 02] Lecture 2

[Missing Semester 01] Lecture 1

Research: 6.824 Lab2B 中异常情况的分析

Talk about Consistency and Consensus

About CAP

Talk about Redlock

torch.compile Walkthrough

ML Compilers Overview

vLLM and torch.compile

Piecewise CUDA Graph

SmoothQuant Accurate and Efficient Post-Training Quantization for Large Language Models

QServe W4A8KV4 Quantization and System Co-design for Efficient LLM Serving

LLM.int8() 8-bit Matrix Multiplication for Transformers at Scale

AWQ Activation-aware Weight Quantization for LLM Compression and Acceleration

Using Rust

100 Exercises to Learn Rust Note

Rustlings Note

Programming Rust Book Note

XAttention Block Sparse Attention with Antidiagonal Scoring

FlexPrefill A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference

LServe Efficient Long-sequence LLM Serving with Unified Sparse Attention

Quest Query-Aware Sparsity for Efficient Long-Context LLM Inference

[Missing Semester 05] Lecture 5

[Missing Semester 04] Lecture 4

[Missing Semester 02] Lecture 2

[Missing Semester 01] Lecture 1

LayerSkip Enabling Early Exit Inference and Self-Speculative Decoding

Draft & Verify Lossless Large Language Model Acceleration via Self-Speculative Decoding

LLM in a flash Efficient Large Language Model Inference with Limited Memory

LLM in a flash Efficient Large Language Model Inference with Limited Memory

PowerInfer Fast Large Language Model Serving with a Consumer-grade GPU

FlexGen High-Throughput Generative Inference of Large Language Models with a Single GPU

Triton: Notes and Paradigms

Triton: Built-in Tutorial

HeteroLLM Accelerating Large Language Model Inference on Mobile SoCs with Heterogeneous AI Accelerators

Quest Query-Aware Sparsity for Efficient Long-Context LLM Inference

EAGLE Speculative Sampling Requires Rethinking Feature Uncertainty

KV-Runahead Scalable Causal LLM Inference by Parallel Key-Value Cache Generation

Quest Query-Aware Sparsity for Efficient Long-Context LLM Inference

Efficient Streaming Language Models with Attention Sinks

DuoAttention Efficient Long-Context LLM Inference with Retrieval and Streaming Heads

DistServe Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving

Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve

ServerlessLLM Locality-Enhanced Serverless Inference for Large Language Models

Cascade Speculative Drafting for Even Faster LLM Inference

SpecInfer Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification

EdgeLLM Fast On-Device LLM Inference With Speculative Decoding

TPI-LLM Serving 70B-scale LLMs Efficiently on Low-resource Mobile Devices

Striped Attention Faster Ring Attention for Causal Transformers

KV-Runahead Scalable Causal LLM Inference by Parallel Key-Value Cache Generation

torch.compile Walkthrough

ML Compilers Overview

vLLM and torch.compile

Revati Transparent GPU-Free Time-Warp Emulation for LLM Serving

AIConfigurator Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving

Revati Transparent GPU-Free Time-Warp Emulation for LLM Serving

AIConfigurator Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving

Sonata Multi-Database Transactions Made Fast and Serializable

Concurrency Control as a Service

A Piece of: Golang Profile

A Piece of: Go Tests

SmoothQuant Accurate and Efficient Post-Training Quantization for Large Language Models

FlexGen High-Throughput Generative Inference of Large Language Models with a Single GPU

XAttention Block Sparse Attention with Antidiagonal Scoring

FlexPrefill A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference

EdgeMoE Empowering Sparse Large Language Models on Mobile Devices

EdgeLLM Fast On-Device LLM Inference With Speculative Decoding

Probability and Statistics

Matrix Math

CAS-Spec Cascade Adaptive Self-Speculative Decoding for On-the-Fly Lossless Inference Acceleration of LLMs

Beyond the 80 20 Rule High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning

HeteroLLM Accelerating Large Language Model Inference on Mobile SoCs with Heterogeneous AI Accelerators

Fast On-device LLM Inference with NPUs

A Survey of Resource-efficient LLM and Multimodal Foundation Models

A Survey on Efficient Inference for Large Language Models

TileLang Programming Model

Production Ready TileLang Ops Framework

Triton: Notes and Paradigms

Triton: Built-in Tutorial

Sonata Multi-Database Transactions Made Fast and Serializable

Concurrency Control as a Service

Research: 6.824 Lab2B 中异常情况的分析

Hierarchical Speculative Decoding with Dynamic Windows for Efficient Language Model Inference

LoRA Low-Rank Adaptation of Large Language Models

Entropy Adaptive Decoding Dynamic Model Switching for Efficient Inference

Token Level Routing Inference System for Edge Devices

R-Stitch Dynamic Trajectory Stitching for Efficient Reasoning

STI Turbocharge NLP Inference at the Edge via Elastic Pipelining

SpecInfer Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification

KVCache Cache in the Wild Characterizing and Optimizing KVCache Cache at a Large Cloud Provider

A Piece of: ThreadLocal

Watchdogs in OpenFaaS

Efficient Streaming Language Models with Attention Sinks

Swift On-the-fly Self-speculative Decoding For LLM Inference Acceleration

Think Big, Generate Quick LLM-to-SLM for Fast Autoregressive Decoding

EdgeShard Efficient LLM Inference via Collaborative Edge Computing

Splitwise Efficient Generative LLM Inference Using Phase Splitting

A Piece of: ThreadLocal

LoRA Low-Rank Adaptation of Large Language Models

Efficient Memory Management for Large Language Model Serving with PagedAttention

LLM Generated Content

QServe W4A8KV4 Quantization and System Co-design for Efficient LLM Serving

Qwen3.5 Dense Model Architecture

EdgeMoE Empowering Sparse Large Language Models on Mobile Devices

Cascade Speculative Drafting for Even Faster LLM Inference

3-Model Speculative Decoding

H2O Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models

LLM.int8() 8-bit Matrix Multiplication for Transformers at Scale

Orca A Distributed Serving System for Transformer-Based Generative Models

PyTorch Notes

Orca A Distributed Serving System for Transformer-Based Generative Models

SLED A Speculative LLM Decoding Framework for Efficient Edge Serving

Ring Attention with Blockwise Transformers for Near-Infinite Context

ServerlessLLM Locality-Enhanced Serverless Inference for Large Language Models

PowerInfer Fast Large Language Model Serving with a Consumer-grade GPU

KTransformers: CPU/GPU Hybrid Inference for MoE Models

Scaling Up On-Device LLMs via Active-Weight Swapping Between DRAM and Flash

Deja Vu Contextual Sparsity for Efficient LLMs at Inference Time

TPI-LLM Serving 70B-scale LLMs Efficiently on Low-resource Mobile Devices

vLLM and torch.compile