metastable blog
1 min readinferencesystems

Where inference latency hides

A small, interactive model for reasoning about dispatch, memory, compute, and batching.

By Placeholder Name

Inference latency is easy to compress into one number. That number is useful for dashboards and not much else. To reason about a system, we need to pull it apart.

At a useful level of abstraction, a request pays for three things:

  1. Dispatch — scheduling, launch, and framework overhead.
  2. Memory — moving weights, activations, and KV state.
  3. Compute — the arithmetic that advances the model.
FIG. 01

A toy model of batched inference latency

interactive
4
Latency9.6 ms
Throughput417 req/s
Inference latency compositionA stacked timeline of dispatch, memory, and compute costs.END-TO-END LATENCYdispatchmemorycompute0 ms12 ms
Illustrative model, not measurements. Drag batch size to inspect the latency/throughput tradeoff.

Batching changes the question

A larger batch amortizes fixed dispatch costs and usually improves hardware utilization. It also asks each request to wait for more work to finish. Throughput rises while latency rises with it.

The toy model in the figure is intentionally simple:

latency = dispatch + memory(batch) + compute(batch)
throughput = batch / latency

Real systems add queueing, topology, kernels, cache behavior, and contention. The point of a small model is not to predict the answer—it is to make the tradeoff visible before adding those details.