How Local LLM Throughput Scales with Multiple Agents

I came across a Reddit post showing a 128 GB Strix Halo machine running Qwen3.8-27B at 31.4 tok/s with speculative decoding.^[1] The number puzzled me. My ordinary 24 GB base-M4 MacBook gets about 20 tok/s on a 27B with a short context: a 4-bit quant, plus every speed trick that costs no further quality, speculative decoding included. A box built for local AI, barely ahead of a laptop?

It also made me wonder what a box like that would give five agents working in parallel. The obvious math: 31.4 ÷ 5 ≈ 6 tok/s per agent, so keeping all five near full speed would take something like five boxes.

The mistake is in the division. It treats solo speed as a fixed total that concurrent requests must share, and that is not how a GPU under load behaves.

Why solo tok/s is not total capacity

To the model server, five agents are just five concurrent requests. While generating each token, a transformer multiplies activation vectors by large weight matrices, layer after layer. At one layer, the calculation for request A is:

y_A = W × x_A

x_A is request A's activation vector; W is the layer's weight matrix, shared by every request.

W is far too large for the GPU's fast on-chip cache, so it has to be streamed in from memory again for every token. With one request, each weight that arrives is used in exactly one multiply-add. On almost any GPU at batch size one, the compute units finish that work faster than memory can deliver the weights.^[2] The GPU spends most of each step waiting, not multiplying.

This is also the answer to the laptop puzzle: at a single request, solo speed tracks memory bandwidth, not compute — so the AI box's bigger GPU barely shows.

Now suppose four requests reach this layer during the same decoding step. Each has its own activation vector, but all four need the same W:

[y_A y_B y_C y_D] = W × [x_A x_B x_C x_D]

The runtime stacks the four activation vectors as columns and computes Y = WX in one pass. This is not free (it is four times the multiply-adds), but the expensive part barely changes: one read of W now feeds four requests instead of one.^[3] The GPU was waiting on memory anyway; batching gives that idle compute something to do.

What this looks like on Strix Halo

A concurrency test tracks two numbers: total tok/s across all requests and the average per request.

The chart below is from a community benchmark of the same model the Reddit post ran: Qwen3.8-27B on a Strix Halo machine with llama-server and four slots.^[4] This run uses a heavier 8-bit quant and no speculative decoding, so its solo rate sits far below the Reddit run's 31.4; what matters here is the shape as requests are added.

Total output rises with each added request

Dividing the solo rate across four requests predicts 7.1 / 4 ≈ 1.8 tok/s each. Measured: 5.4 to 6.2 tok/s each, depending on whether you divide the wall-clock total or time each request over its own run — either way a modest dip, not a four-way split. Total output reached 21.8 tok/s, 3.06× the solo run.

The same benchmark also ran the model's built-in speculative decoder. Alone, it more than doubles output: 15.5 tok/s against 7.1. By four requests its total falls behind plain batching, 16.9 against 21.8. The trick feeds on the same idle compute that batching does, so its advantage shrinks with every added request and is gone by the fourth.

Four slots is where this sweep stops, short of the ceiling. A longer sweep on the same silicon — a mixture-of-experts model, Qwen3.6-35B-A3B — shows the whole curve:^[5]

Total output rises, then levels off

Total output climbs to 2.74× solo at eight requests (about 20 tok/s each), then nearly stops: sixteen requests add only 2.5% more. Past that point the naive division finally comes true — sixteen requests split a fixed total, about 10 tok/s each — but the total they split is 166 tok/s, not 59. This climb is shallower than the 27B's: at these batch sizes, concurrent requests in a mixture-of-experts wake different experts and share fewer weight reads.

The math I should have done

Back to my five agents. Against the plain solo rate, four concurrent requests kept 85% of it on the 27B and about half on the MoE sweep — nowhere near the fifth I calculated — and the box puts out two to three times its solo total. Against the 31.4 anchor the correction cuts the other way: that number has the speculative bonus baked in, and under concurrency the bonus melts. Both corrections point at one error: the total I divided is not fixed. Neither sweep ran five requests, so the exact rate for five agents is one I would have to measure.

Three caveats. The requests must actually share one server: llama-server takes --parallel N, which opens N request lanes called slots; Ollama has OLLAMA_NUM_PARALLEL. Batching shares the weights, not each request's private context: every slot keeps its own KV cache, the request's conversation state. That cache costs memory and per-step reads that grow with context length, so long agent sessions flatten the curve earlier than these short-context tests, which ran at a few thousand tokens or less. And these are decode-only numbers: agent turns also pay for prompt processing (prefill), a separate cost the charts do not show.

The dependable way to size a box is not arithmetic: run your own model at your target concurrency and read the per-agent rate off the sweep. Both benchmarks here did that by firing N simultaneous requests at llama-server and timing them — an ordinary load test, pointed at your own GPU.

Serving many users from one weight read is the economics every inference provider runs on — a box on your desk just lets you watch it happen.

Sources

1. Qwen3.8-27B at 31.4 tok/s on Strix Halo, using DFlash2 speculative decoding.
2. NVIDIA's matrix multiplication guide: why matrix multiplication can be limited by memory bandwidth; Efficiently Scaling Transformer Inference, p. 3: why loading weights can dominate LLM inference at small batch sizes.
3. NanoFlow, p. 4: why one weight read can serve several activation vectors in a batch.
4. Qwen3.8-27B concurrency benchmark on Strix Halo: Unsloth UD-Q8_K_XL on a Ryzen AI Max+ 395 with llama-server (Vulkan, four slots), 256 generated tokens per request, greedy sampling; plain and speculative (MTP) runs at one to four concurrent requests.
5. Strix Halo Guide multi-user sweep: Qwen3.6-35B-A3B UD-Q4_K_M on a Beelink GTR9 Pro with llama-server and Vulkan/RADV, about 4,096 context tokens per slot, 128 generated tokens per request, one warm-up request, then three runs at each concurrency level.