KKKZOZ's Blog

Notes on research, systems and engineering.

by kkkzoz
152 posts
Recent posts
  1. /152
    2026-11-02

    NVLink vs PCIe in Practice: Bandwidth and LLM Serving

    Peer-to-peer and NCCL bandwidth measured on an RTX A6000 and an A100 server, and what NVLink and PCIe change when vLLM serves Qwen3.8-27B with tensor parallelism.

    GPUHardwareLLM-Inference
  2. /151
    2026-10-30

    GPU Server Topology: NVLink, PCIe and NUMA

    What NV#, PIX, PXB, PHB, NODE and SYS in nvidia-smi topo -m mean on the board of a two-socket GPU server, with two real machines.

    GPUHardware
  3. /150
    2026-10-29

    ggml Q4_0 vec_dot Walkthrough

    How ggml's CPU backend computes a Q4_0 matrix-vector product during decode, split across threads, with and without repacking.

    LLM-Inferenceggml
  4. /149
    2026-10-24

    KTransformers: CPU/GPU Hybrid Inference for MoE Models

    How KTransformers (SOSP '25) runs the 671B DeepSeek-V3 on one GPU plus a dual-socket Xeon: AMX-aware CPU kernels, a single CUDA Graph for the whole decode step, NUMA-aware tensor parallelism, and Expert Deferral, which overlaps CPU and GPU by letting some experts arrive one layer late.

    SOSP-25LLM-Inference
  5. /148
    2026-10-20

    torch.compile Walkthrough

    用 ((a + b) * c.relu()).sum(dim=-1) 跟踪 torch.compile 的每个阶段：Dynamo FX Graph、AOT ATen Graph、TorchInductor IR（TensorBox / StorageBox / Pointwise / Reduction / realize）、Scheduler 与生成的 kernel。

    ML Compilerstorch.compile
  6. /147
    2026-08-19

    Qwen3.5 Dense Model Architecture

    基于 transformers 源码拆解 Qwen3.5 文本主干：每 4 层中 3 层 Gated DeltaNet、1 层 gated full attention 的 hybrid 结构，两类 attention 的工作机制与 cache 组织，以及端到端数据流。

    Model-Architecture
  7. /146
    2026-08-18

    TileLang Programming Model

    用 TileLang 写 kernel 的 Block 视角与 Thread 视角、常用原语的区别、内存分配，以及各类操作最终落在哪一层执行。

    TileLang
  8. /145
    2026-08-17

    Production Ready TileLang Ops Framework

    在生产环境使用 TileLang：三层算子缓存，用 custom_op 与 register_fake 兼容 torch.compile，以及它与 Triton 在缓存和 PyTorch 集成方式上的异同。

    TileLang
  9. /144
    2026-08-10

    vLLM and torch.compile

    vLLM 中与 FX 和 torch.compile 相关的几条路径：Transformers backend 的 FX 改写，以及 vLLM 自定义 backend 的图分区、shape 管理、custom pass、缓存与 CUDA Graph 协同。

    ML Compilerstorch.compilevLLM
  10. /143
    2026-08-09

    Piecewise CUDA Graph

    vLLM 的 Piecewise CUDA Graph 如何以 Attention 为边界拆分模型、只 capture CUDA-Graph-safe 的区域，FX-level split 与 Inductor-level partition 两种实现，以及它与 Full CUDA Graph 的关系。

    ML Compilers