KKKZOZ's Blog

Notes on research, systems and engineering.

by kkkzoz
156 posts
Recent posts
  1. /156
    2026-11-04

    vLLM Hybrid KV Cache

    Why vLLM's "GPU KV cache size" for Qwen3.8-27B is not a count of token slots and changes with max_model_len: GDN state, attention pages and one shared block pool.

    vLLMLLM-Inference
  2. /155
    2026-11-04

    TorchInductor Pipeline 与算子融合

    跟着一个 10 个算子的例子走完 TorchInductor 的四个阶段：Graph Preprocessing、Lowering、Scheduling、Codegen，重点是 Scheduler 怎么用 can_fuse、score_fusion 和 fuse_nodes 决定融合，并与 TVM 的 FuseOps 对比。

    ML Compilerstorch.compile
  3. /154
    2026-11-04

    Learning TileLang with GEMV

    用 GEMV 入门 TileLang：同一个 FP16 GEMV 写十四遍，依次用到 T.Kernel、线程绑定、shared / local / fragment 三种分配、T.copy、T.vectorized、T.Pipelined 和 reducer，以及 warp、合并访存、128-bit 加载这些与 CPU 不同的 GPU 访存特性。

    TileLangGPU
  4. /153
    2026-11-02

    NVLink vs PCIe in Practice: Bandwidth and LLM Serving

    Peer-to-peer and NCCL bandwidth measured on an RTX A6000 and an A100 server, and what NVLink and PCIe change when vLLM serves Qwen3.8-27B with tensor parallelism.

    GPUHardwareLLM-Inference
  5. /152
    2026-10-30

    GPU Server Topology: NVLink, PCIe and NUMA

    What NV#, PIX, PXB, PHB, NODE and SYS in nvidia-smi topo -m mean on the board of a two-socket GPU server, with two real machines.

    GPUHardware
  6. /151
    2026-10-29

    ggml Q4_0 vec_dot Walkthrough

    How ggml's CPU backend computes a Q4_0 matrix-vector product during decode, split across threads, with and without repacking.

    LLM-Inferenceggml
  7. /150
    2026-10-24

    KTransformers: CPU/GPU Hybrid Inference for MoE Models

    How KTransformers (SOSP '25) runs the 671B DeepSeek-V3 on one GPU plus a dual-socket Xeon: AMX-aware CPU kernels, a single CUDA Graph for the whole decode step, NUMA-aware tensor parallelism, and Expert Deferral, which overlaps CPU and GPU by letting some experts arrive one layer late.

    SOSP-25LLM-Inference
  8. /149
    2026-10-20

    torch.compile Walkthrough

    用 ((a + b) * c.relu()).sum(dim=-1) 跟踪 torch.compile 的每个阶段：Dynamo FX Graph、AOT ATen Graph、TorchInductor IR（TensorBox / StorageBox / Pointwise / Reduction / realize）、Scheduler 与生成的 kernel。

    ML Compilerstorch.compile
  9. /148
    2026-08-19

    Qwen3.5 Dense Model Architecture

    基于 transformers 源码拆解 Qwen3.5 文本主干：每 4 层中 3 层 Gated DeltaNet、1 层 gated full attention 的 hybrid 结构，两类 attention 的工作机制与 cache 组织，以及端到端数据流。

    Model-Architecture
  10. /147
    2026-08-18

    TileLang Programming Model

    用 TileLang 写 kernel 的 Block 视角与 Thread 视角、常用原语的区别、内存分配，以及各类操作最终落在哪一层执行。

    TileLang