Inference Clusters

Chapter 1 described throughput and capability clusters. Inference uses both patterns. Independent model replicas can answer more requests, which is throughput scaling. A model that does not fit or run quickly enough on one device can be divided across several devices or nodes, which is capability scaling. A large inference installation usually combines the two: each replica is itself a small cooperating cluster, and many replicas serve independent traffic.

Inference uses a trained model to produce a result from new input. The input may be an image, a sensor record, a group of database features, an audio stream, or a sequence of tokens. This appendix gives language-model serving additional attention because it makes memory growth, streaming responses, and multi-stage scheduling particularly visible. The broader method applies to other models as well: define the model and quality target, follow one request through the service, identify the state and communication it creates, and measure the complete result at the load the service must carry.

An inference cluster therefore includes more than accelerator servers. It includes the endpoint that receives a request, admission and routing, model artifacts, preprocessing and postprocessing, the serving engine, network and storage paths, identity and policy, telemetry, and the machinery that replaces or upgrades a replica without losing the service. A fast kernel inside an unavailable or overloaded endpoint is not a successful inference system.

The inference service boundary

Inference workloads first divide by the relationship between a request and time. An online service receives work as requests arrive and returns each result to a waiting caller. Interactive use is the most visible example, but machine-to-machine APIs can have equally strict deadlines. An offline or batch workload starts with a finite collection of inputs and usually values completion rate and cost more than the latency of one item. A scheduled fraud-scoring run and a public chat endpoint may use the same model and hardware while requiring different batching, admission, and failure policies.

The service path begins before the model. A gateway authenticates the caller, applies size and policy limits, and identifies the model or version. Admission control decides whether the request can enter a queue without violating a deadline or exhausting memory. A router selects a compatible replica or model-parallel group. Preprocessing creates tensors or tokens, the model executes, and postprocessing turns outputs into the response contract. Streaming systems return partial output while generation continues.

The control path is separate from this request path. It registers model artifacts, creates serving groups, assigns devices, distributes configuration and credentials, checks readiness, changes traffic weights, and drains old replicas. A process can be alive before its model is loaded or its collective group is usable, so a useful readiness check runs through the model path. A deployment is complete only after the router sends ordinary requests to the new replica and can stop sending them again if the result is wrong.

Latency, throughput, and useful results

No single inference number describes a service. For a language model, time to first token (TTFT) measures the wait through queueing and prompt processing until generation begins. Time per output token (TPOT), or its inverse in tokens per second, describes the pace after the first token. End-to-end latency includes the complete response. Aggregate throughput counts work across all callers. These values must be reported with input and output lengths, because a short classification request and a long prompt with a long answer are different jobs.

Concurrency links latency and throughput. Batching more work can improve accelerator use and total throughput, but waiting to form a batch adds delay. Once offered load approaches the service's sustainable capacity, the accelerators may remain fully occupied while new work accumulates in queues and tail latency rises sharply. A load test should therefore sweep arrival rate or concurrency and plot the performance surface, not report the single point that produces the largest token count. MLPerf Endpoints follows this approach by reporting throughput, TTFT, interactivity, and latency across concurrency.

Percentiles matter because averages hide the requests users remember. Report at least median and high-percentile behavior over a stated interval, and retain the input and output length distributions that produced them. Separate queue time from model execution where the stack allows it. A high p99 may come from a long prompt, a cold model, cache transfer, network loss, another tenant's batch, or a failed worker; the remedy depends on which stage consumed the time.

Goodput counts only the work that satisfies the declared quality and service objectives. A system that accepts more requests but lets half of them miss the deadline has increased raw throughput without increasing useful service. Quality belongs in the same definition. Quantization, speculative decoding, a different tokenizer, altered sampling, or a substituted model can change speed and output. Compare systems only after fixing the model or documenting the substitution and rerunning the quality evaluation.

Cost and energy also need a boundary. Include the complete serving group, required hosts and network, and the idle capacity needed to meet bursts or failures. A device thermal-design rating is not wall power. MLPerf Inference, when power is submitted, measures full-system AC power for the accompanying benchmark run. An internal test should state whether gateways, storage, fabric, cooling allocation, and standby replicas are inside or outside its number.

What has to fit in memory

Model parameters establish the first memory floor. A rough dense-model estimate multiplies the parameter count by bytes per stored parameter, but a deployable image can also contain scales, metadata, multiple experts, adapters, tokenizer data, and engine-specific layouts. Runtime memory adds kernels, communication buffers, temporary workspaces, graph captures, activations, and allocator reserve. The usable model capacity is therefore lower than accelerator capacity printed on the device label.

Autoregressive transformer serving adds a key-value cache. Attention layers retain key and value tensors for tokens that have already been processed so the model does not recompute the entire prefix for every new token. For conventional attention, the cache for one request can be approximated from:

KV bytes ≈ tokens × layers × 2 × KV heads × head dimension × bytes per element

The factor of two represents keys and values. Multiply again by active requests, then account for sharding, paging, alignment, and implementation reserve. Architectures with grouped-query attention, multi-head latent attention, sliding windows, state-space layers, or compressed cache formats change the expression. The reliable value comes from the exact model and serving engine, but the formula reveals why context length and concurrency consume the same finite memory pool.

Cache allocation also changes throughput. Reserving each request's maximum possible sequence wastes memory when actual lengths vary. Paged allocation divides the cache into blocks that can be assigned as sequences grow. The PagedAttention work behind vLLM demonstrated how reducing fragmentation and duplicate cache storage allows more requests to remain active. Prefix caching can reuse a shared prompt, but the router must send a matching request to a replica that owns the cache, and multi-tenant systems must prevent cache identity or hash choices from crossing an authorization boundary.

Quantizing model weights or the KV cache can release memory and reduce traffic. It can also change output quality, require calibration or scale data, and select different kernels. The comparison must keep numerical format beside the result. A platform that fits a larger batch only after changing the model representation has made a potentially useful trade, not run the identical case.

Prefill, decode, and batching

A language-model request has two visible compute phases. During prefill, the engine processes the prompt, creates the initial KV cache, and produces the first output token. The prompt tokens can be processed with substantial parallel work, so prefill often has a stronger arithmetic demand. During decode, the engine generates one token per request per iteration while repeatedly reading model state and the growing cache. Decode often emphasizes memory bandwidth, communication delay, and the number of active sequences that can be batched. These are tendencies, not universal classifications; model architecture, sequence length, batch, kernels, and hardware can move the boundary.

Ordinary dynamic batching groups requests before an execution. Autoregressive serving benefits from continuous or iteration-level batching, in which completed sequences leave and new sequences can join between decode iterations. Orca formalized iteration-level scheduling for generative models. The scheduler still has a tradeoff: admitting a long prefill can delay active decodes, while protecting every decode can leave arithmetic capacity unused. Sarathi-Serve showed how chunking long prefills can reduce those stalls. These are scheduling mechanisms, so their benefit must be measured against the site's prompt and response distribution rather than assumed from one paper result.

Prefill and decode can also be placed on separate worker groups. DistServe described this as a way to choose different parallel and resource plans for TTFT and TPOT. Disaggregation is not free. The decode group needs the cache created during prefill, and that transfer consumes bandwidth, memory, and time. It also creates a handoff that can fail. A colocated system may be preferable when the transfer dominates or when traffic is too small to keep both pools balanced.

Parallelism, traffic, and routing

The first scaling method is replication. Each replica owns a complete model or complete model-parallel group and answers independent requests. Replication increases aggregate capacity and provides a unit that can be drained or replaced. It also duplicates weights and may duplicate prefix caches. A router that ignores active sequences, cache state, model version, and request length can send work to an apparently idle replica that cannot admit it.

When one model instance needs several devices, the parallel plan defines the traffic. Place the most frequent and latency-sensitive communication on the fastest local paths when possible. Crossing a node boundary adds network adapters, switches, routing, and more failure domains. The model can still scale well across nodes, but only if the communication pattern, message size, software collective, and fabric match.

Parallel dimensions can be combined. A service might use tensor parallelism within an eight-device node, pipeline parallelism across two nodes, and several copies of that group behind a router. The scheduler has to allocate the whole group together, and the health system has to treat it as one replica. If one worker loses its fabric path, keeping the remaining processes alive does not preserve a usable model.

Routing becomes stateful even when the API appears stateless. Prefix caching favors a replica that already holds common context. Long-running sessions may retain cache or adapter state. Different models, numerical formats, context limits, or service objectives form distinct pools. The router should admit a request only to a pool that can finish it, and it needs a defined response when no pool has capacity: wait within a bounded queue, reject with a retry signal, route to a smaller or remote service where policy permits, or shed lower-priority work.

The Linux serving environment

Linux connects the model process to the hardware and to the service. Firmware and the kernel expose processors, memory, accelerators, PCIe, network adapters, and storage. Device drivers and user-space runtimes provide the execution path. Communication libraries form model-parallel groups. The serving engine manages model loading, cache, scheduling, and kernels. Containers or packages carry that engine and its dependencies. A gateway, router, telemetry system, identity service, and deployment controller turn the process into an endpoint.

The compatibility chain is as important here as it is in Chapter 6: model format, tokenizer, serving engine, framework, kernels, collective library, accelerator runtime, driver, and firmware have to agree. A container does not carry the host driver or repair an unsupported device kernel. Record the complete serving image and model digest, not merely a marketing name for the model.

Host and container settings can also determine whether a compatible image performs correctly. Communication libraries and serving engines may use shared memory for interprocess exchange, so the container's /dev/shm allocation must match the runtime's actual demand instead of inheriting an arbitrarily small default. On multi-socket hosts, CPU workers, host memory, accelerators, and their PCIe or network paths should be placed with NUMA locality in mind. Measure the topology and pin only where the result is understood; an incorrect affinity policy can be worse than allowing Linux to schedule freely.

Kubernetes commonly operates continuously available endpoints. Device plugins advertise accelerators to the kubelet, and an inference controller or ordinary deployment machinery can create replica groups, services, and rollout policy. Kubernetes does not understand a model's cache or collective topology unless the serving stack and scheduler expose those needs. A pod can be scheduled while model weights are still loading, and a generic process check can pass before the endpoint is ready for traffic.

Slurm is often a better fit for offline inference, finite campaigns, or a cluster whose primary contract is scheduled HPC work. GPUs can be allocated through generic resources, and a job can reserve all devices and nodes needed by one model-parallel group. Long-running endpoints can be launched under Slurm, but service discovery, rolling updates, traffic routing, and always-on availability then need explicit owners. Chapter 7 compares the two operating models; neither removes the inference-aware scheduler inside the serving engine.

Model distribution is a storage and startup problem. Large artifacts should have an authoritative digest and controlled access, then be staged or cached close enough that a fleet restart does not overload shared storage. A partial download must never become ready. Local copies need capacity, eviction, integrity, and cleanup policy. Keeping every model loaded reduces start latency but consumes expensive memory while idle; loading on demand saves memory but can turn a burst into a queue of cold starts.

Telemetry should follow one request and one serving group. A correlation identifier needs to cross the gateway, queue, router, model, and response path. Record admission, queue, routing, prefill, decode, cache allocation, model and runtime errors, output completion, cancellation, and retry. Correlate those events with accelerator memory and activity, host memory and CPU, fabric counters, power, thermals, and replica lifecycle. Prompt and output contents may be sensitive or regulated, so useful tracing records identifiers, lengths, timing, and policy decisions without copying content unless an approved diagnostic procedure requires it.

Failure, change, and multitenancy

Replication changes the effect of failure. Losing one independent replica need not interrupt the endpoint if the router stops new admissions and the remaining replicas have headroom. Losing one worker in a tensor- or pipeline-parallel group usually removes the whole group. The service objective must include the capacity left after that loss, not only whether an HTTP listener remains available.

A retry is not automatically safe. A classification request may be repeated without consequence, while an inference call embedded in an external transaction may cause duplicate action. A streaming response can fail after the caller has received part of it. Define request identifiers, deadlines, cancellation, replay, and partial-response behavior at the API boundary. When a caller disconnects, propagate cancellation far enough to release cache and compute instead of finishing output no one can consume.

Changes should separate artifact correctness from traffic exposure. Validate a model and serving image offline, load them into a bounded canary group, run fixed quality and performance probes, then shift a small traffic fraction. Compare output, errors, cache use, TTFT, TPOT, and resource demand before continuing. Draining means stopping new admissions while bounded existing requests complete or reach a documented cutoff. Rollback requires the previous model, image, configuration, and router rule to remain available.

Multitenancy adds resource and information boundaries. One tenant's long prompts or large batches can consume cache and delay another tenant. Per-tenant queues, budgets, rate limits, context limits, and priority may be needed before accelerator scheduling. Model artifacts, adapters, prompts, outputs, caches, logs, and traces all have owners and retention rules. Device sharing can improve utilization, but memory visibility, side channels, reset behavior, and failure isolation must match the data classification. A utilization gain does not justify an isolation claim the platform cannot enforce.

Specialization and access

Two independent axes help classify current systems. The first is specialization. General-purpose accelerator clusters can train, fine-tune, and serve many model families. An inference-oriented system narrows some combination of numerical formats, operators, model structures, memory movement, or software in exchange for latency, throughput, power, or cost. The second axis is access. A system may be purchased for on-premises operation, consumed as managed capacity, exposed through a public cloud, or used mainly inside the organization that designed it.

AMD Helios and NVIDIA Vera Rubin NVL72 illustrate general-purpose rack-scale accelerators that can run both training and inference. Etched's announced inference cluster, SambaNova SambaRack, Cerebras systems, and NVIDIA Groq 3 LPX illustrate different forms of inference emphasis. These products should not define the appendix: model support, shipping status, software access, and measured evidence can change faster than the cluster mechanisms. Their value here is to show that specialization can occur at the chip, memory, rack, serving-stage, or service boundary.

Access changes what the operator must prove. An on-premises system exposes rack power, cooling, fabric integration, spares, firmware, root access, and lifecycle support. A managed endpoint replaces some of that work with quotas, region and capacity availability, data movement, egress, API behavior, provider observability, and price. A cloud-hosted appliance may sit between them. Linux can run inside all three while only the on-premises operator sees the host environment.

A reproducible comparison plan

Begin with the service, not the accelerator. Select the models, numerical representations, quality checks, input and output distributions, arrival pattern, concurrency range, deadlines, and availability target. Include both the ordinary case and the case that constrains design: long context, a burst, a large model, a strict first-token limit, or an offline deadline. State whether the test includes the gateway, tokenizer, network, storage, and model-loading path.

Run the comparison in controlled stages:

1. Prove the model once. Load the exact artifact on one serving group, run fixed inputs, verify output and quality, and preserve the model, tokenizer, engine, kernel, driver, and configuration identities.
2. Map memory. Record model residency, runtime reserve, cache capacity, maximum admitted context, and the effect of the selected numerical format. Confirm that an oversized request is rejected or bounded cleanly.
3. Measure a load curve. Sweep arrival rate or concurrency from idle through saturation. Report latency percentiles, throughput, goodput, queue growth, rejection, and memory at each point.
4. Exercise the parallel path. Verify device and rank placement, local and scale-out collectives, cache or activation transfers, and performance with the exact topology. Remove a worker from a test group and confirm the complete replica is withdrawn.
5. Test mixed traffic. Combine short and long inputs, short and long outputs, cache hits and misses, or the site's equivalent workload classes. Check that one class does not silently consume every admission slot.
6. Test cold and changed state. Restart a replica with cold model storage, roll to a canary version, drain it, and roll back. Measure load time, readiness, traffic loss, and the storage or network burst.
7. Test service failure. Stop a replica, a router instance, and an approved dependency in separate trials. Verify retry and partial-response behavior, capacity after failure, alerts, and recovery.
8. Measure the whole boundary. Capture wall energy and all required hosts or services, then calculate cost using the actual utilization and reserve needed to meet the objective.
9. Publish the conditions. Keep raw request records without sensitive content, configuration, run interval, versions, topology, exclusions, quality evidence, and failures so the result can be repeated or challenged.

An inference cluster is still a Linux compute cluster. The workload has unusual request timing and fast-growing state, but the design method remains familiar: understand the work, fit its state, place frequent communication on suitable paths, make shared resources explicit, and measure the result users receive. Appendix 2 follows the same hardware into distributed training, where the model changes on every step and the workers synchronize around one long-running job.

References and further reading

1. MLCommons, "MLPerf Inference: Datacenter."
2. MLCommons, "MLPerf Endpoints."
3. MLCommons, "MLPerf Endpoints v0.7: A Foundation Release," July 2026.
4. Yu et al., "Orca: A Distributed Serving System for Transformer-Based Generative Models," OSDI 2022.
5. Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention," SOSP 2023.
6. Agrawal et al., "Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve," OSDI 2024.
7. Zhong et al., "DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving," OSDI 2024.
8. vLLM, "Data Parallel Deployment."
9. vLLM, "Context Parallel Deployment."
10. vLLM, "Quantized KV Cache."
11. NVIDIA Triton Inference Server, "Batchers."
12. Kubernetes, "Device Plugins."
13. SchedMD, "Generic Resource Scheduling."