Notes on research, systems and engineering.

by kkkzoz
Latest doc
KKZ-TN-158
Entries
158
Latest entry
2026-11-19
Activity / 12 mo
  1. /158
    2026-11-19

    Decode GEMV on the CPU: Bandwidth, Compute or NUMA

    Decode GEMV on a Core Ultra 9, an M4 Pro and a dual-socket Xeon: how close each gets to its memory bandwidth, and what holds it back.

    LLM-InferenceHardwareggml
  2. /157
    2026-11-16

    CUDA Hardware Hierarchy: From SMs to Lanes

    Where the threads of one CUDA launch run on an RTX A6000, where their data lives, and why some memory access patterns are slow.

    GPUCUDAHardware
  3. /156
    2026-11-04

    vLLM Hybrid KV Cache

    Why vLLM's "GPU KV cache size" for Qwen3.8-27B is not a count of token slots and changes with max_model_len: GDN state, attention pages and one shared block pool.

    vLLMLLM-Inference
  4. /155
    2026-11-04

    TorchInductor Pipeline 与算子融合

    跟着一个 10 个算子的例子走完 TorchInductor 的四个阶段:Graph Preprocessing、Lowering、Scheduling、Codegen,重点是 Scheduler 怎么用 can_fuse、score_fusion 和 fuse_nodes 决定融合,并与 TVM 的 FuseOps 对比。

    ML Compilerstorch.compile
  5. /154
    2026-11-04

    Learning TileLang with GEMV

    用 GEMV 入门 TileLang:同一个 FP16 GEMV 写十四遍,依次用到 T.Kernel、线程绑定、shared / local / fragment 三种分配、T.copy、T.vectorized、T.Pipelined 和 reducer,以及 warp、合并访存、128-bit 加载这些与 CPU 不同的 GPU 访存特性。

    TileLangGPU
  6. /153
    2026-11-02

    NVLink vs PCIe in Practice: Bandwidth and LLM Serving

    Peer-to-peer and NCCL bandwidth measured on an RTX A6000 and an A100 server, and what NVLink and PCIe change when vLLM serves Qwen3.8-27B with tensor parallelism.

    GPUHardwareLLM-Inference
  7. /152
    2026-10-30

    GPU Server Topology: NVLink, PCIe and NUMA

    What NV#, PIX, PXB, PHB, NODE and SYS in nvidia-smi topo -m mean on the board of a two-socket GPU server, with two real machines.

    GPUHardware
  8. /151
    2026-10-29

    ggml Q4_0 vec_dot Walkthrough

    How ggml's CPU backend computes a Q4_0 matrix-vector product during decode, split across threads, with and without repacking.

    LLM-Inferenceggml
  9. /150
    2026-10-24

    KTransformers: CPU/GPU Hybrid Inference for MoE Models

    How KTransformers (SOSP '25) runs the 671B DeepSeek-V3 on one GPU and a dual-socket Xeon, and why the CPU side is the hard part.

    SOSP-25LLM-Inference
  10. /149
    2026-10-20

    torch.compile Walkthrough

    用 ((a + b) * c.relu()).sum(dim=-1) 跟踪 torch.compile 的每个阶段:Dynamo FX Graph、AOT ATen Graph、TorchInductor IR(TensorBox / StorageBox / Pointwise / Reduction / realize)、Scheduler 与生成的 kernel。

    ML Compilerstorch.compile