Decode GEMV on the CPU: Bandwidth, Compute or NUMA

Decode GEMV on a Core Ultra 9, an M4 Pro and a dual-socket Xeon: how close each gets to its memory bandwidth, and what holds it back.

Doc no.
KKZ-TN-158
Date
2026-11-19
GMT day
2026:323
Est. read
14 min

In decode, every weight matrix is multiplied by one vector: a GEMV (General Matrix-Vector multiply). Each weight is read once per token, for one multiply-add.

The example here is the up and gate projections of one Qwen3-8B layer: two 12288 × 4096 matrices, 31.5 MiB in Q2_K and 54 MiB in Q4_0. ggml Q4_0 vec_dot Walkthrough shows how ggml computes them.

With n threads, the time has two floors:

  • , the bandwidth floor: weight bytes over the rate n cores read DRAM (Dynamic Random-Access Memory, the main memory). No kernel is faster, so this post calls it the optimal latency.
  • , the compute floor: one thread’s time, mostly compute, split over n cores.

On three CPUs, Q2_K is held back by something different each time:

Table 1
MachineCPU read ceilingQ2_K is limited byFixBest 2-bit / 4-bit
Core Ultra 9 285K67 GB/sbandwidthsoftware prefetch, 16 threads0.50 / 0.85 ms
Apple M4 Pro223 GB/scomputeT-MAC table lookups0.13 / 0.24 ms
2 × Xeon Gold 6248R216 GB/s, each socket reading its own memoryNUMA layoutone shard per socket0.145 / 0.27 ms

With each fix, all three come within about 10% of the optimal latency.

01Memory Bandwidth#

What the bus can carry at best:

Each factor has its own limit, and some numbers on a spec sheet, like CAS (Column Address Strobe) latency, are not one of them.

1.1Transfers per Clock#

Figure 1 — DDR moves data on both edges of the I/O clock: a 2400 MHz clock gives 4800 MT/s.

DDR (Double Data Rate) memory moves data on both edges of the clock. DDR5-4800, its fifth generation, makes 4800 Mega-Transfers per Second (MT/s) from a 2400 MHz I/O clock. A channel is 64 bits wide, so each transfer moves 8 bytes:

1.2Channels#

A channel is an independent 64-bit path from the memory controller to the DIMMs (Dual In-line Memory Modules, the memory sticks). The Core Ultra 9 285K has two.

Figure 2 — Same capacity, twice the bandwidth: with one stick per channel, both channels transfer at the same time.

One 64 GB stick leaves channel B idle: 38.4 GB/s. Two 32 GB sticks, one on each channel, give 76.8 GB/s. The sticks must sit on different channels; two on the same channel are still single channel.

Adding two more sticks adds capacity, not bandwidth:

Figure 3 — Four sticks share the same two channels: twice the capacity, the same peak.

On the 285K it even costs bandwidth: with two sticks per channel, Intel’s supported speed drops from 5600–6400 to 4400–4800 MT/s. More bandwidth needs more channels, that is, a wider bus.

1.3Transfer Rate#

Figure 4 — Dual-channel peak for each DDR5 speed grade, and where the 285K sits.

The 285K’s official speed depends on the kind of DIMM:

Table 2
2 DIMMs, one per channelSupported speed
CUDIMMDDR5-6400
UDIMMDDR5-5600

On a UDIMM (Unbuffered DIMM) the clock goes straight from the CPU to the DRAM chips, and arrives distorted at high rates. A CUDIMM (Clocked UDIMM) regenerates it with a Client Clock Driver (CKD) on the stick:

Figure 5 — UDIMM vs. CUDIMM: a clock driver on the stick regenerates the clock that reaches the DRAM chips.

The machine in this post runs DDR5-4800: 2 × 38.4 = 76.8 GB/s peak.

1.4CAS Latency#

Figure 6 — CL40: 40 clock cycles from a READ command to the first data.

“C40” is the CAS Latency (CL): 40 clock cycles from a READ command to the first data, 16.7 ns at DDR5-4800. Across speeds, compare in nanoseconds:

CL does not change the peak. Decode streams weights, so what matters is MT/s and bus width.

1.5Bus Width#

Apple’s M4 Pro uses LPDDR5X (Low-Power DDR5, eXtended), soldered next to the chip. It gets more than twice the 285K’s best, mostly from a wider bus:

Table 3
MemoryTotal bus widthPeak
Core Ultra 9 285K (this post)DDR5-4800128-bit76.8 GB/s
Core Ultra 9 285K (official max)DDR5-6400128-bit102.4 GB/s
M4 ProLPDDR5X-8533256-bit273 GB/s
Figure 7 — From 102.4 to 273 GB/s: twice the bus width, then a 1.33× faster rate.

The width is set by the memory controller, not by the memory type:

Figure 8 — The 285K’s controller drives two 64-bit channels; the M4 Pro’s drives four 64-bit LPDDR5X packages next to the SoC (System on a Chip).
  • Neither LPDDR5X nor DDR5 fixes the total width: 4- and 8-channel DDR5 workstations reach 256 and 512 bits.
  • Memory in the package makes a wide bus practical, but its capacity cannot be upgraded.
  • The CPU and GPU of the M4 Pro share the controller, and with it the 273 GB/s.

The dual-socket Xeon server goes wide the other way: each Xeon Gold 6248R has its own 6-channel controller, and the two sockets are joined by UPI (Ultra Path Interconnect, Intel’s link between sockets).

Figure 10 — 2 × Xeon Gold 6248R with 12 × 32 GB DDR4-2666, one DIMM on each channel.

DDR4-2666 gives 21.3 GB/s per channel, 128 GB/s per socket, 256 GB/s in total, but only if each socket reads its own memory.

The three machines:

Table 4
MachineMemoryBus widthPeak
Core Ultra 9 285KDDR5-4800, 2 channels128-bit76.8 GB/s
M4 ProLPDDR5X-8533256-bit273 GB/s
2 × Xeon Gold 6248RDDR4-2666, 6 channels per socket2 × 384-bit2 × 128 GB/s

The peak is what the bus can carry. What the cores can pull through it is lower, and has to be measured.

02Core Ultra 9: Bandwidth-Bound#

The Core Ultra 9 285K has 8 Performance (P) and 16 Efficiency (E) cores, with 2 × DDR5-4800.

2.1The Read Ceiling#

Figure 11 — Read-only probe on the Core Ultra 9: 67 GB/s from 8 threads, 87% of the DDR5-4800 peak.
  • One core reads 19 GB/s. 8 P-cores reach 67 GB/s, 87% of the 76.8 GB/s peak.
  • More threads add nothing: 57 at 16, 66 at 24.

STREAM Triad (a[i] = b[i] + s * c[i]) reports 52 GB/s at 8 threads. It counts 3 arrays of traffic, but the core also reads a[i]‘s line before writing it (write-allocate): × 4/3 ≈ 70 GB/s.

The optimal latency follows: 33.0 MB / 67 GB/s = 493 µs for Q2_K, and 56.6 MB / 67 GB/s = 845 µs for Q4_0.

2.2Baseline: ggml#

Effective bandwidth = weight bytes / measured latency. Plotted on the same axes:

Figure 12 — ggml’s effective bandwidth for Q2_K and Q4_0 against the read ceiling. Error bars: range over runs.
  • Q2_K and Q4_0 read at the same rate at every thread count, so Q4_0 takes 1.7× as long, its weight size over Q2_K’s (54 / 31.5 MiB).
  • At 24 threads both reach 62–67 GB/s, the probe’s ceiling.
Figure 13 — Latency per layer against the optimal. Ranges are over runs; the 24-thread runs have rare 6 ms outliers.

With 24 threads, ggml is within 2–4% of the optimal. The M4 Pro rows are a preview of the next section: its Q2_K and Q4_0 take the same time though Q4_0 reads 1.7× the bytes.

Two problems remain:

  • At 24 threads, latency spikes.
  • On 8 P-cores, ggml reads 38 GB/s; the probe reads 67. ggml keeps too few loads in flight per core, and the E-cores close the gap only by adding more loads.

2.3Fix 1: Leave One Core Free#

Each thread pinned to one core (0 … n − 1), 600 calls per run:

Table 5
ThreadsQ2_K, µsQ4_0, µsSlow calls, poll = 50 / 0
8 (P only)82314610 / 0
1264211360 / 0
165489510 / 0
204908550 / 4
224948540 / 0
234968510 / 0
24 (all)49084147 / 21

A slow call takes over 3× the median (Q2_K). poll is how long ggml’s threads busy-wait before they sleep; 0 turns it off. Latencies are with poll = 50.

Figure 14 — With every core busy, an OS task preempts one thread and the other 23 wait at the barrier.
  • With one core free the spikes go away. At 24 threads the 99th percentile is 3.5–5.9 ms, near multiples of 6 ms, which looks like scheduler time slices.
  • Turning busy-waiting off halves the slow calls but does not remove them (47 → 21), and gemv-probe’s own thread pool spikes too, so busy-waiting is not the cause.
  • 20 threads (8 P + 12 E) already reach 67 GB/s, with the 99th percentile under 0.94 ms.

2.4Fix 2: Software Prefetch#

More threads reach the ceiling, but the E-cores are wanted elsewhere: a GPU inference server needs host threads to drive the GPU. The goal is the ceiling from the 8 P-cores alone.

Each core needs more loads in flight. Software prefetch asks for the next rows before they are needed:

Figure 15 — While a thread computes row r, it prefetches row r + D.

A sketch of the loop in gemv-probe:

c
// one thread's chunk of rows [r0, r1); D = 2
for (int r = r0; r < r1; r++) {
    const char *ahead = W + (size_t)(r + D) * row_bytes;
    for (size_t off = 0; off < row_bytes; off += 64)
        __builtin_prefetch(ahead + off);                 // one hint per cache line
    y[r] = vec_dot(K, W + (size_t)r * row_bytes, x);    // ggml's own dot product
}

A Q2_K row of 4096 weights is 1344 bytes, 21 cache lines; a Q4_0 row is 2304 bytes, 36 lines.

  • The distance barely matters: D = 1, 2, 4 and 8 all give 59–62 GB/s on 8 P-cores. D = 2 is the most stable.
  • We keep ggml’s dynamic split (64-row chunks from a shared counter). With a static split the P-cores wait for the E-cores: 590 vs. 497 µs (Q2_K, 16 threads).
Figure 16 — Effective bandwidth with and without software prefetch (Q2_K; Q4_0 is within 2 GB/s).
Table 6
8 P-cores, µs per layerQ2_KQ4_0
ggml8601490
gemv-probe + prefetch535930
Change−38%−38%
  • One core gains nothing: 17.5 GB/s with or without prefetch, near the probe’s 19 GB/s. The hardware prefetcher keeps up.
  • Many cores raise DRAM latency under load, and the hardware prefetcher falls behind. Software prefetch restores the loads in flight: 8 cores go from 38 to 61 GB/s.

2.5Best Achievable#

Figure 17 — Every configuration against the optimal latency; “free cores” are cores left for other work.

16 threads with prefetch is within 1% of the optimal and leaves 8 E-cores free. The M4 Pro is still 1.5× (Q2_K) and 2.6× (Q4_0) faster; that is bandwidth, not kernel.

Tried, with no effect: pinning threads to P- or E-cores, huge pages (MADV_HUGEPAGE), and turning off ggml’s busy-wait.

03M4 Pro: Compute-Bound#

The M4 Pro has 10 P-cores and 4 E-cores, with LPDDR5X-8533 on a 256-bit bus. The same benchmark with 8 threads:

Figure 18 — Q4_0 reads 1.7× the bytes of Q2_K. On the Ultra 9 it takes 1.7× as long; on the M4 Pro, the same time.

If bytes do not set the latency here, something else does.

3.1The Read Ceiling#

Figure 19 — Read-only probe on the M4 Pro: 223 GB/s at 10 threads, 82% of the nominal 273 GB/s.
  • The probe saturates at 10 threads (all P-cores): 223 GB/s, 82% of nominal, 3.3× the Ultra 9.
  • One core reads 86 GB/s, 4.5× an Ultra 9 core. A few M4 Pro cores already pull most of the bandwidth.

3.2Baseline: ggml#

Figure 20 — Left: effective bandwidth against the read ceiling. Right: share of the ceiling at 8 threads.
  • Near 100% of the ceiling, a kernel is bandwidth-bound.
  • Far below it, the kernel is compute-bound: memory idles while the cores unpack and multiply.

At 8 threads, a pure read of Q2_K’s 33 MB takes 0.162 ms; ggml takes 0.332 ms. About half of the time is compute.

3.3Diagnosis: Compute-Bound#

A call takes the longer of two times:

  • Read time: weight bytes / read ceiling. Q4_0 has 1.7× the bytes, so it reads longer.
  • Compute time: unpacking the weights and multiply-adding them.

To measure compute time, we run one thread. Memory is not the limit there: one core can read 86 GB/s, and ggml uses only 14 (Q2_K) and 25 GB/s (Q4_0).

Figure 21 — One thread, time to compute one layer.

The two formats take the same compute time. Q2_K has fewer bits per weight but more work to unpack each:

  • Q4_0: blocks of 32 weights with one FP16 (16-bit Floating Point) scale; a weight is d × (q − 8).
  • Q2_K: super-blocks of 256 weights in 16 sub-blocks, each with a 4-bit scale and a 4-bit minimum, on top of two FP16 factors; a weight is d × scale × q − dmin × min.

More threads split the compute time, and the read ceiling grows with the thread count. Per layer, in ms:

Figure 22 — Compute time (one-thread time / threads), read time (“bandwidth”: bytes / read ceiling) and measured latency on the M4 Pro.
  • Q2_K computes longer than it reads, at both thread counts: it is compute-bound, and its fewer bytes do not help.
  • At 8 threads, both formats come down to about the same compute time, so they take the same time. This answers the puzzle at the top of the section.
  • At 10 threads, Q4_0′s read time is the longer one: it turns bandwidth-bound.

3.4Fix: T-MAC#

T-MAC replaces unpack-and-multiply with table lookups:

  1. Split each 2-bit weight into two 1-bit planes.
  2. For every 4 activations, precompute the 16 possible sums, once per token.
  3. Each 4 bits of a plane index that table; one tbl instruction of NEON, Arm’s SIMD (Single Instruction, Multiple Data) extension, does 16 lookups.
  4. Combine the planes with a shift, then apply the scales.
Figure 23 — Left: one thread, weights processed per second. Right: effective bandwidth as a share of the read ceiling at the same thread count.

One core computes a layer in 0.88 ms instead of ggml Q2_K’s 2.36 ms: 2.7× the weights per second (114 vs. 42.6 G/s). 2-bit becomes bandwidth-bound: 96% of the ceiling at 14 threads, against ggml’s 62%.

3.5Best Achievable#

Figure 24 — Measured / optimal latency at 10 threads; optimal = bytes the kernel reads / read ceiling.
  • 2-bit: ggml is 1.7–1.9× the optimal; T-MAC is within 2–8%, about 2× faster (0.265 → 0.133 ms).
  • 4-bit: everything is within 7%. ggml Q4_0 was already bandwidth-bound.

04Dual Xeon: NUMA-Bound#

2 × Xeon Gold 6248R, 24 cores and 6 × DDR4-2666 per socket.

  • One socket, 16 cores: Q2_K 0.56 ms, Q4_0 0.64 ms.
  • Both sockets, 48 cores: Q2_K 0.99 ms, Q4_0 0.50 ms. Q2_K is 2× slower while reading 42% fewer bytes.

4.1The Read Ceiling#

4.1.1Four Layouts#

On a two-socket machine, each socket has its own memory, a NUMA node (Non-Uniform Memory Access: a core reaches its own node faster than the other). A socket reads its own node directly; it reaches the other node over UPI and through the other socket’s memory controller.

Figure 25 — Four ways to place threads and weights. Solid: local read. Dashed: remote read, over UPI and through the other socket’s memory controller.

One thread pool over both sockets with numactl --interleave=all is the third case: about half of all reads cross UPI.

4.1.2Ceiling by Layout#

The same read-only probe, with each layout:

Figure 26 — Read ceiling by layout, GB/s, mean of 2 runs.
  • Remote memory gives half the local bandwidth: 32 vs. 61 GB/s at 8 cores.
  • One pool interleaved over both sockets gets 116 GB/s, barely more than one socket alone.
  • Each socket reading only its own node gets 216 GB/s, both sockets’ worth.

4.2Baseline: One Thread Pool#

gemv-probe with one thread pool and one shared queue of 64-row chunks, pages interleaved across the sockets:

Table 7
One thread poolRead ceilingOptimalMeasuredMeasured / optimal
Q2_K · 1 socket, 24 cores109 GB/s304 µs600 µs1.98×
Q4_0 · 1 socket, 24 cores109 GB/s520 µs662 µs1.27×
Q2_K · 2 sockets, 48 cores116 GB/s285 µs989 µs3.47×
Q4_0 · 2 sockets, 48 cores116 GB/s489 µs504 µs1.03×

Q4_0 on two sockets sits at its ceiling. Q2_K does not, and gets slower with the second socket: 600 → 989 µs.

4.3Diagnosis: Not Compute-Bound#

Figure 27 — Compute floor, bandwidth floor and measured latency on the Xeon, with the shared pool.

On two sockets Q2_K is far above both floors: 989 µs against 285 and 97. The time goes to coordination across cores and sockets. (The compute floor is generous: one Xeon core reads only 8.7 GB/s, so includes memory stalls.)

4.4Fix: Local Shards#

Figure 28 — Left: one pool over both sockets, pages interleaved. Right: the output rows split in half, each half’s weights on its own socket’s memory.
  • We split the output rows in two and bind each half’s weights to one socket’s memory.
  • Each socket gets its own threads and its own copy of the input vector, splits its rows statically, and the two join at the end.
Figure 29 — gemv-probe Q2_K, 48 cores (24 per socket), µs per layer. Q4_0: 503 → 462 → 269 µs.
  • Every bar runs the same vec_dot, so the lost time was not unpacking arithmetic.
  • A static split alone gives 2.4×; local shards give another 2.6×. The second step changes page placement and thread coordination at once, so the two are not separated here.

4.5Best Achievable#

With local shards every format reaches the optimal:

Table 8
48 cores, local shardsBytesOptimalMeasured× optimalShared layout
gemv-probe Q2_K31.5 MiB153 µs160 µs1.05×994 µs
gemv-probe Q4_054 MiB262 µs269 µs1.03×503 µs
T-MAC W2 g12827 MiB131 µs145 µs1.11×270 µs
T-MAC W4 g12851 MiB247 µs269 µs1.09×480 µs

Optimal = bytes / 216 GB/s.

  • T-MAC W2 is 3.7× faster than Q2_K with the shared layout, and only 1.1× with local shards, about what its 14% fewer bytes give.
  • 32 cores (16 per socket) are nearly enough: Q2_K 172 µs, W2 151 µs. The last 16 cores add 4–7%.

05Summary#

Table 9
Core Ultra 9M4 Pro2 × Xeon
CPU read ceiling67 GB/s223 GB/s216 GB/s (local shards)
One-core read19 GB/s86 GB/s8.7 GB/s
Q2_K is limited bybandwidthcomputeNUMA layout
Fixprefetch, 16 threadsT-MAClocal shards
Best 2-bit latency0.50 ms0.13 ms0.145 ms (T-MAC W2)
Best 4-bit latency0.85 ms0.24 ms0.27 ms (Q4_0 / W4)

Up and gate fused, per layer, 36 weight copies rotated. Xeon: 48 cores, gemv-probe.

To find out what limits a decode GEMV on a given CPU:

  1. Measure the read ceiling with the kernel’s access pattern: weights larger than the cache, rotated, every thread count.
  2. Divide the weight bytes by it: that is the optimal latency.
  3. Compare the kernel’s latency to the optimal. Near 1×, the kernel is done; only fewer bytes help.
  4. Far from it, time one thread for the compute floor. Above it: unpacking costs too much (M4 Pro). Above both floors: look at thread coordination and where the pages live (Xeon).

All three machines end at the same place: latency ≈ bytes / read ceiling.

Terms
STREAM
A standard benchmark of sustainable memory bandwidth, by John McCalpin. It times four loops over arrays far larger than the cache, Copy, Scale, Add and Triad, and divides the bytes it counts by the time.↩