ggml Memory Layout and Allocation

This is one layer’s self-attention in ggml’s GPT-2 example, examples/gpt-2/main-backend.cpp, trimmed:

// cur: [3*n_embd, N], the output of the QKV projection for N new tokens
// off: byte offset of layer il in the KV cache (il*n_ctx*n_embd*sizeof(float))
struct ggml_tensor * Qcur = ggml_view_2d(ctx, cur, n_embd, N, cur->nb[1], 0);

struct ggml_tensor * Q = ggml_permute(ctx,
        ggml_cont_3d(ctx, Qcur, n_embd/n_head, n_head, N),
        0, 2, 1, 3);

struct ggml_tensor * K = ggml_permute(ctx,
        ggml_reshape_3d(ctx,
            ggml_view_1d(ctx, model.memory_k, (n_past + N)*n_embd, off),
            n_embd/n_head, n_head, n_past + N),
        0, 2, 1, 3);

struct ggml_tensor * KQ = ggml_mul_mat(ctx, K, Q);
// ... scale, causal mask, softmax -> KQ_soft_max

struct ggml_tensor * V_trans = ggml_cont_3d(ctx,
        ggml_permute(ctx,
            ggml_reshape_3d(ctx,
                ggml_view_1d(ctx, model.memory_v, (n_past + N)*n_embd, off),
                n_embd/n_head, n_head, n_past + N),
            1, 2, 0, 3),
        n_past + N, n_embd/n_head, n_head);

struct ggml_tensor * KQV = ggml_mul_mat(ctx, V_trans, KQ_soft_max);
cur = ggml_cont_2d(ctx, ggml_permute(ctx, KQV, 0, 2, 1, 3), n_embd, N);

Only the two ggml_mul_mat calls do arithmetic. view, reshape and permute change how bytes already in memory are read, and cont copies them into a new layout. And no call here computes anything yet: each one adds a node to a graph, which runs later, in memory that something else allocates.

So two questions:

1. Why does K go into ggml_mul_mat as it is, while V has to go through ggml_cont first?
2. Where does the memory for KQ come from?

The first is answered by how a tensor describes its memory, the second by ggml’s allocators, which the four GPT-2 examples introduce one at a time. llama.cpp, at the end, puts the same pieces together.

Versions: ggml 83835ff (Aug 2025). The llama.cpp part follows the code as I read it in late 2025, with a few points checked against 1e6f04a (Oct 2026), as noted.

01Memory Layout#

From include/ggml.h, trimmed:

struct ggml_tensor {
    enum ggml_type type;
    struct ggml_backend_buffer * buffer;   // where data lives

    int64_t ne[GGML_MAX_DIMS];  // number of elements
    size_t  nb[GGML_MAX_DIMS];  // stride in bytes:
                                // nb[0] = ggml_type_size(type)
                                // nb[1] = nb[0]   * (ne[0] / ggml_blck_size(type)) + padding
                                // nb[i] = nb[i-1] * ne[i-1]

    enum ggml_op op;                        // the op that produces this tensor
    struct ggml_tensor * src[GGML_MAX_SRC]; // its inputs

    struct ggml_tensor * view_src;          // for views: the tensor viewed
    size_t               view_offs;

    void * data;
    char name[GGML_MAX_NAME];
    // ...
};

GGML_MAX_DIMS is 4. For each dimension i:

• ne[i] is how many elements it has;
• nb[i] is how many bytes to move to reach the next element along it.

Dimension 0 is the innermost: in a matrix, ne[0] is the number of columns, and a row is ne[0] elements next to each other. That is the reverse of PyTorch’s order: a ggml tensor with ne = {K, M} is a PyTorch tensor of shape (M, K).

The address of an element is a dot product of its index with the strides:

(char *) t->data + i0*t->nb[0] + i1*t->nb[1] + i2*t->nb[2] + i3*t->nb[3]

A tensor is contiguous when the strides are the ones the formula in the struct comment gives: nb[0] is the element size and each nb[i] is nb[i-1] * ne[i-1], so the elements sit one after another in row-major order with no gaps. ggml_is_contiguous checks exactly this.

1.1ggml_mul_mat#

// a: ne = {K, M},  b: ne = {K, N}  ->  c: ne = {M, N}
struct ggml_tensor * c = ggml_mul_mat(ctx, a, b);

Element (m, n) of c is the dot product of row m of a and row n of b. In matrix notation that is c = b · aᵀ, not a · b.

Both operands are read along dimension 0, so every dot product walks two runs of adjacent memory. For a linear layer this matches how weights are stored: W has ne = {n_in, n_out}, the activations x have ne = {n_in, N}, and ggml_mul_mat(ctx, W, x) gives ne = {n_out, N}, which is x · Wᵀ.

Dimensions 2 and 3 are batch dimensions: the product is taken for each index in them, and a is broadcast if it has fewer.

02View, Reshape, Permute and Cont#

The first three only build a new ggml_tensor header with different ne, nb or data, pointing at the same bytes. ggml_cont is the only one that copies.

2.1ggml_view#

struct ggml_tensor * ggml_view_2d(
        struct ggml_context * ctx,
        struct ggml_tensor  * a,
        int64_t               ne0,
        int64_t               ne1,
        size_t                nb1,      // stride of dim 1, from the caller
        size_t                offset) { // byte offset into a
    const int64_t ne[2] = { ne0, ne1 };

    struct ggml_tensor * result = ggml_view_impl(ctx, a, 2, ne, offset);

    result->nb[1] = nb1;
    result->nb[2] = result->nb[1]*ne1;
    result->nb[3] = result->nb[2];

    return result;
}

The view starts offset bytes into a and takes nb1 from the caller, so a view can skip bytes between rows. Qcur above is one: cur holds Q, K and V side by side in each row, 2304 floats in all, and Qcur takes the first 768 floats of every row, with nb[1] still 9216 bytes.

The KV cache is the main use. In the GPT-2 examples, memory_k is one flat F32 tensor of n_layer * n_ctx * n_embd elements, and each step writes the new tokens’ K into a view of it:

// K of the N new tokens goes to positions n_past .. n_past+N-1 of layer il
struct ggml_tensor * k = ggml_view_1d(ctx, model.memory_k, N*n_embd,
        (ggml_element_size(model.memory_k)*n_embd)*(il*n_ctx + n_past));
ggml_build_forward_expand(gf, ggml_cpy(ctx, Kcur, k));

ggml_cpy copies Kcur into the bytes the view points at. Attention then reads positions 0 .. n_past+N-1 of the layer through a second view, the ggml_view_1d inside K above. Nothing is copied to read the cache.

2.2ggml_reshape#

ggml_reshape_* starts with GGML_ASSERT(ggml_is_contiguous(a)), then sets a new ne and computes nb from it as for a contiguous tensor. The bytes keep their order, so the new strides are only right if the old ones were the contiguous ones.

That is why the code builds Q with ggml_cont_3d rather than ggml_reshape_3d: Qcur is the strided view from above, so it has to be copied before it can take the shape {64, 12, N}. K comes from ggml_view_1d, which is a plain contiguous slice of the cache, so ggml_reshape_3d is enough.

2.3ggml_permute#

ggml_permute(ctx, a, ax0, ax1, ax2, ax3) sends dimension i of a to position ax_i, and moves ne[i] and nb[i] there together. From src/ggml.c, trimmed:

struct ggml_tensor * result = ggml_view_tensor(ctx, a);  // same data

ne[axis0] = a->ne[0];   nb[axis0] = a->nb[0];
ne[axis1] = a->ne[1];   nb[axis1] = a->nb[1];
ne[axis2] = a->ne[2];   nb[axis2] = a->nb[2];
ne[axis3] = a->ne[3];   nb[axis3] = a->nb[3];

// copy ne and nb into result
result->op     = GGML_OP_PERMUTE;
result->src[0] = a;

For a 2 × 2 F32 matrix A, ggml_permute(ctx, A, 1, 0, 2, 3) swaps the two strides:

ne = {2, 2}   nb = {4, 8}

A[0][1] = data[0*8 + 1*4] = data[4] -> b
A[1][0] = data[1*8 + 0*4] = data[8] -> c

ne = {2, 2}   nb = {8, 4}

P[0][1] = data[0*4 + 1*8] = data[8] -> c
P[1][0] = data[1*4 + 0*8] = data[4] -> b

(P[i1][i0] is at i1*nb1 + i0*nb0.) P is the transpose of A, and ggml_transpose is this call.

2.4ggml_cont#

ggml_cont always creates a new tensor with the same shape, op = GGML_OP_CONT and src[0] the input, even when the input is already contiguous. Its bytes are filled when the graph runs. On the CPU, ggml_compute_forward_dup_f32 (src/ggml-cpu/ops.cpp) copies in one of two ways:

• If each row of the source is contiguous (nb[0] is the element size), it copies row by row with memcpy.
• Otherwise it copies element by element, with the full address formula for each.

The row path, simplified:

char       * d = (char *) dst->data;
const char * s = (const char *) src->data;

for (int64_t i3 = 0; i3 < src->ne[3]; i3++)
for (int64_t i2 = 0; i2 < src->ne[2]; i2++)
for (int64_t i1 = 0; i1 < src->ne[1]; i1++) {
    // source row: jump with the strides; destination: just append
    const char * row = s + i1*src->nb[1] + i2*src->nb[2] + i3*src->nb[3];
    memcpy(d, row, src->ne[0]*src->nb[0]);
    d += src->ne[0]*src->nb[0];
}

ggml_cont_2d, ggml_cont_3d and ggml_cont_4d do the copy and give the result a new shape in the same step, as a copy followed by a reshape would.

2.5Attention#

For GPT-2 small (n_embd = 768, 12 heads of 64, F32), with L = n_past + N:

Neither Q nor K is contiguous: the heads are interleaved in memory, 256 bytes apart. But dimension 0 of each is a contiguous run of 64 floats, one head of one token, and that is all ggml_mul_mat needs. It asserts !ggml_is_transposed(a), that is nb[0] <= nb[1], and the CPU kernel asserts that nb[0] of both inputs is the element size; the other strides can be anything. So KQ = ggml_mul_mat(K, Q) has ne = {L, N, 12}, and for each head, element (l, n) is k_l · q_n, as attention needs.

The second product needs, for each head, output dimension d and query n:

KQV[d][n] = Σ_l w[n][l] · V[l][d]

ggml_mul_mat takes dot products along dimension 0 of both operands, so l has to be dimension 0 of both. For the weights KQ_soft_max it already is. For V it means the transpose, ne = {L, 64, 12}, which permute(1, 2, 0, 3) gives in shape but not in memory: nb = {3072, 4, 256}. Now nb[0] > nb[1], ggml_mul_mat stops at its assert while the graph is still being built, and so the code copies V with ggml_cont_3d into a layout where each row is one d across all L positions.

The last line does the same for the output: KQV has ne = {64, N, 12}, permute(0, 2, 1, 3) makes it {64, 12, N}, and ggml_cont_2d copies it into {768, N}, the 12 heads of each token side by side, ready for the output projection.

03Memory Allocation#

3.1Overview#

• ggml_tensor: the header above. It is also a node of the compute graph, through op and src.
• ggml_context: a memory pool of fixed size, set in ggml_init, that holds tensor headers, and with no_alloc = false their data too.
• ggml_cgraph: arrays of pointers to the tensors of one graph.
• ggml_backend: one device, such as CPU, CUDA or Metal. It allocates buffers and runs the ops.
• ggml_backend_buffer: memory a backend allocated, in RAM or in VRAM. With no_alloc = true, tensor->data points into one of these.

So a tensor has two parts with different needs:

• The header has a fixed size, ggml_tensor_overhead(), and always lives in a context.
• The data has a size that depends on ne and type. It may live in the context or in a backend buffer, and several tensors may share it, as views do.

3.2Compute Graph#

A graph has two kinds of tensors:

• Leaves: weights and inputs. Their data has to be there before the graph runs.
• Nodes: intermediates, produced while the graph runs.

struct ggml_cgraph {
    int size;    // maximum number of nodes/leafs/grads/grad_accs
    int n_nodes; // number of nodes currently in use
    int n_leafs; // number of leafs currently in use

    struct ggml_tensor ** nodes;     // tensors with data that can change if the graph is evaluated
    struct ggml_tensor ** leafs;     // tensors with constant data
    int32_t             * use_counts;// number of uses of each tensor, indexed by hash table slot

    struct ggml_hash_set visited_hash_set;
    enum ggml_cgraph_eval_order order;
};

ggml builds the whole graph before it computes anything, so all memory can be assigned up front and the computation itself never allocates.

The two kinds also live for different lengths of time. During generation the same graph runs once per token:

• Weights are needed from load to exit. They are allocated and filled once.
• Intermediates are needed only during one run of the graph, and their sizes depend on how many tokens the run takes.

The KV cache sits in between: it lives across runs like a weight but is written every step. The GPT-2 examples allocate it with the weights.

3.3gpt-2 Examples#

ggml/examples/gpt-2 has four versions of the same model, and each one hands one more part of memory management to ggml:

3.3.1main-ctx#

Weights. The loader adds up the size of every weight tensor itself:

size_t ctx_size = 0;
// ...
ctx_size += ggml_row_size(GGML_TYPE_F32, n_embd); // ln_f_g
ctx_size += ggml_row_size(GGML_TYPE_F32, n_embd); // ln_f_b

ctx_size += ggml_row_size(wtype, n_vocab * n_embd);       // wte
ctx_size += ggml_row_size(GGML_TYPE_F32, n_ctx * n_embd); // wpe
ctx_size += ggml_row_size(wtype, n_vocab * n_embd);       // lm_head
// ...

and creates a context of that size with no_alloc = false:

struct ggml_init_params params = {
    /*.mem_size   =*/ ctx_size,
    /*.mem_buffer =*/ NULL,
    /*.no_alloc   =*/ false,
};
model.ctx_w = ggml_init(params);

In such a context, every ggml_new_tensor_* takes room for the data from the pool, right after the header (ggml_new_tensor_impl in src/ggml.c):

size_t obj_alloc_size = 0;

if (view_src == NULL && !ctx->no_alloc) {
    // allocate tensor data in the context's memory pool
    obj_alloc_size = data_size;
}

struct ggml_object * const obj_new = ggml_new_object(ctx, GGML_OBJECT_TYPE_TENSOR, GGML_TENSOR_SIZE + obj_alloc_size);
// ...
    /*.data =*/ obj_alloc_size > 0 ? (void *)(result + 1) : data,

Intermediates. Every call to gpt2_eval creates a new context on a static buffer, builds the graph in it, computes, and frees the context with ggml_free(ctx0):

static size_t buf_size = 256u*1024*1024;
static void * buf = malloc(buf_size);

if (mem_per_token > 0 && mem_per_token*N > buf_size) {
    const size_t buf_size_new = 1.1*(mem_per_token*N); // add 10% to account for ggml object overhead
    buf_size = buf_size_new;
    buf = realloc(buf, buf_size);
}

struct ggml_init_params params = {
    /*.mem_size   =*/ buf_size,
    /*.mem_buffer =*/ buf,
    /*.no_alloc   =*/ false,
};
struct ggml_context * ctx0 = ggml_init(params);

256 MB is a guess. After the first run the program measures mem_per_token = ggml_used_mem(ctx0)/N and grows the buffer when a later run needs more. Every intermediate gets its own memory, even after the last op that reads it has run.

3.3.2main-alloc#

main-alloc keeps the weights as they were and replaces the per-run context with a graph allocator, ggml_gallocr. The graph context now has no_alloc = true, so building a graph creates headers only, and the allocator assigns the data:

ggml_gallocr_t allocr = ggml_gallocr_new(ggml_backend_cpu_buffer_type());

// create the worst case graph for memory usage estimation
int n_tokens = std::min(model.hparams.n_ctx, params.n_batch);
int n_past = model.hparams.n_ctx - n_tokens;
struct ggml_cgraph * gf = gpt2_graph(model, n_past, n_tokens);

// pre-allocate the compute buffer for the worst case (optional)
ggml_gallocr_reserve(allocr, gf);

Then, in every gpt2_eval:

struct ggml_cgraph * gf = gpt2_graph(model, n_past, embd_inp.size());
// set tensor->data for every tensor in the graph
ggml_gallocr_alloc_graph(allocr, gf);
// copy the token ids and positions into the input tensors
ggml_graph_compute(gf, &plan);

Why the worst case uses n_batch. The size of the intermediates grows with the number of tokens in a run, N: activations are {n_embd, N}, and KQ is {n_past + N, N, 12}. A 1000-token prompt in one run would need a buffer sized for 1000 tokens, and that buffer would then sit mostly unused while generating one token at a time. So the program caps a run at n_batch tokens. A longer prompt goes in over several runs, and since each run leaves its K and V in the cache, nothing is computed twice. The worst case is then a full batch at the end of the context: n_tokens = n_batch and n_past = n_ctx - n_batch.

A larger n_batch makes the buffer larger; a smaller one makes a long prompt take more runs.

What reserve does. ggml_gallocr_reserve walks the graph in execution order and assigns each tensor an offset in a buffer, without touching memory. It frees a node’s block once every node that reads it has been assigned, and lets an op write in place of its input when it can. It records each offset and the peak, then allocates buffers of the peak size.