Building a Cluster from the Ground Up

By Chander Kant Updated July 31, 2026

A request for a new cluster often arrives in the form of a shopping list: a processor generation, a number of GPUs, an interconnect speed, and a storage capacity. None of these tells us whether the system will solve the problem that justified it. The project must begin one step earlier, with evidence about the work, and end one step later, with users completing that work through a supportable production service.

The preceding chapters examined each major layer. We studied node balance, interconnect behavior, storage paths, facilities, software, scheduling, administration, and service operation. This chapter joins those layers in project order. Its purpose is not to prescribe one cluster architecture. It is to show how an organization can make its assumptions visible, test uncertain choices before they become expensive, and accept a delivered system against the reasons it was purchased.

We will use a representative replacement project. The current service runs a mixture of independent CPU jobs, multi-node MPI simulations, GPU analysis and training, and data-intensive workflows. Demand has grown, some queues are persistently delayed, active data is approaching an operational limit, and the facility has finite power and cooling headroom. The organization has a budget and a desired service date, but neither the architecture nor the number of nodes has been decided. This is the right point to begin.

The project case and its owners

A cluster project needs a named outcome. “Replace the old cluster” describes an event, not a result. A useful case identifies the communities and work to be served, the present constraint, the improvement sought, the planning horizon, the consequences of doing nothing, and the operating resources available after installation. It also states what is outside the project. A new compute cluster may not include a permanent archive, a building electrical upgrade, a new identity platform, or support for every legacy application unless those items are explicitly funded and scheduled.

The project sponsor owns the reason for spending the money. A service owner accepts the delivered system into operation. Technical leads own compute, accelerators, fabric, storage, facilities, software, security, and migration. Procurement and legal staff own the acquisition process and contract form. User representatives identify workload behavior and validate outcomes. A vendor or integrator may perform much of the implementation, but it cannot own the customer's requirement or declare its own work accepted without the customer's evidence.

Governance should be light enough to make decisions and strong enough to preserve them. Establish who can approve requirements, architecture, budget changes, substitutions, exceptions, and acceptance. Keep one risk register with an owner, next action, decision date, and effect on cost, schedule, or service. “GPU software may not be ready” is not yet a useful risk. Name the framework and application, the version or capability in question, the date by which it must be demonstrated, the fallback, and the person responsible for obtaining evidence.

The first schedule should contain decision gates rather than one delivery date. Typical gates include approved needs and scope, workload evidence complete enough for design, requirements baseline, architecture review, facility authorization, solicitation release, contract award, detailed design, site readiness, factory test, delivery, integrated acceptance, service transition, and project close. Long-lead power equipment, optics, storage devices, accelerators, and network services may need action before the complete system contract, but early commitment should be tied to an approved risk rather than hidden in the schedule.

Measure the work before designing the machine

Chapter 9 established the service measures and project records that now become design input. Scheduler accounting shows requested and allocated processors, memory, accelerators, duration, queue delay, exit state, and measured use where collection is enabled. Storage telemetry shows capacity, file count, throughput, metadata pressure, and transfer demand. Facility meters show actual power and environmental behavior. Support records reveal missing software, difficult workflows, and recurring operational limits that a utilization graph cannot show.

Historical accounting is not a complete demand forecast. It contains only work that users submitted to the available system. Jobs may have been reduced, delayed, sent elsewhere, or never attempted because the current cluster lacked memory, accelerators, software, storage, or a suitable queue. Interview project leads and application owners. Ask about pending grants and contracts, new instruments, model sizes, data-retention obligations, deadlines, license limits, and work that is currently impossible.

Group workloads by the resource pattern that changes the design. Useful dimensions include:

serial, threaded, distributed-memory, accelerator, or mixed execution;
typical and maximum processors, memory, accelerator memory, node count, and duration;
communication pattern, message sizes, collectives, locality, and tolerance for oversubscription;
read, write, checkpoint, metadata, local-storage, and external-transfer behavior;
software, compiler, driver, kernel, container, and license dependencies;
concurrency, deadline, preemption, restart, and interactive needs;
data classification, collaboration, retention, audit, and network restrictions.

Do not infer all of this from requested resources. A job can request an entire node because policy requires exclusive placement while using little memory bandwidth. A user may request twice the observed memory after one unexplained out-of-memory failure. Profile a representative run. HPCToolkit can relate CPU and accelerator activity to source regions, Darshan can characterize application I/O, and vendor tools can expose device and communication timelines. Use the smallest measurement that answers the design question; indiscriminate tracing of a large production run can create more data than insight.

Evidence is useful when it changes a requirement or an acceptance test
Observed need	Evidence to collect	Possible requirement and test
Large MPI simulation misses a deadline	Strong-scaling curve, rank placement, collective and point-to-point profile, memory use, current fabric counters	Complete the fixed validated case within a stated time on a stated node count; retain correctness and communication evidence
GPU workflow cannot increase model or problem size	Device and host memory high-water marks, precision, sharding, CPU-GPU and GPU-GPU transfers, software compatibility	Run the selected case at the required size without out-of-memory fallback on the proposed device topology
Many short jobs wait longer than they run	Eligible, start, and end times by size; submission rate; scheduler cycle and RPC behavior; launch overhead	Meet a percentile expansion-factor objective for the declared short-job class under a representative mixed load
Checkpoint activity disrupts other projects	Checkpoint size, interval, concurrent writers, file count, sustained and burst rate, restart behavior, storage-client load	Complete the coordinated checkpoint within a threshold while a background service workload stays within its latency limit
Users cannot reproduce an old result	Application source, build record, container or package manifest, input identity, scheduler request, output validation	Reconstruct the supported environment and reproduce the selected validation result from retained project records

Select a small portfolio of cases rather than one favorite application. Include important production work, emerging work that affects architecture, and a control case that should not regress. Preserve inputs, versions, build and run instructions, expected output, measurement method, and permission to disclose results during a competitive process. If a dataset cannot leave the organization, decide whether bidders can run in a controlled environment, receive a synthetic equivalent, or submit a system for local testing.

Write requirements that can survive a proposal

A requirement should describe needed behavior or an interface without unnecessarily selecting the product that provides it. “The fabric shall be InfiniBand” may be justified by an established software, operational, or compatibility requirement. If the actual need is application communication performance, stating the application cases, scale, topology constraints, and measurable threshold allows competing designs to address it. Product names are appropriate where interoperability or support depends on them; they should not substitute for the underlying reason.

Good requirements are singular, clear, bounded, and verifiable. Include the operating condition and measurement boundary. “The storage system shall deliver 1 TB/s” omits readers and writers, block or request pattern, file count, persistence semantics, client count, dataset size, duration, cache state, and whether the result is aggregate or per job. A precise requirement may be longer, but it prevents bidder and customer from testing different systems under the same sentence.

Distinguish requirements from goals and evaluation preferences. A mandatory requirement is a condition of an acceptable solution. A scored objective rewards better value without making every ambitious number a contractual minimum. A design target guides engineering before complete information is available. Record which is which. An RFP full of maximum goals labeled “shall” can eliminate workable proposals, inflate risk pricing, or create exceptions before the project starts.

Each requirement should carry a stable identifier, source, rationale, owner, priority, verification method, acceptance stage, and change history. NASA systems-engineering guidance distinguishes verification—showing that specified requirements were met—from validation—showing that the integrated system serves the intended need. The distinction is particularly useful for clusters. A storage array can pass its component bandwidth requirement while the user's workflow remains slower because metadata, clients, network placement, or software differs from the proposed path.

Figure 10.1: Traceability prevents the project from accepting a collection of good component results that does not satisfy the original need. Evidence can revise a weak requirement before the contract baseline; later changes need an explicit effect on design, cost, schedule, test, and operation.

The trace should run in both directions. Every requirement needs a source and a verification method. Every architecture feature and contract deliverable should satisfy one or more requirements. Every acceptance test should identify what it proves. An expensive feature with no requirement deserves scrutiny. A mandatory requirement with no design owner or test is not ready for solicitation.

Build a capacity model with time in it

Peak arithmetic is a poor capacity model. Multiplying devices by advertised operations per second assumes that all workloads use the relevant operation, precision, memory path, and software efficiency at once. A service delivers a portfolio of work over time. Capacity planning has to account for resource demand, concurrency, deadlines, utilization, maintenance, failure, scheduling fit, data growth, and the periods when demand peaks.

Begin with measured work units. For each workload class, estimate runs per planning interval, resource quantity per run, elapsed time, and growth. CPU demand may be expressed in node-hours for a defined node class; accelerator demand in device-hours for a defined model and memory capacity; storage in active bytes, retained bytes, files, metadata operations, and transfer rate. Keep unlike resources separate until there is an explicit conversion based on measured application performance.

Annual demand alone can hide a deadline. Ten thousand node-hours fit easily into a year but not necessarily into a two-day campaign. Record required concurrency and completion windows. For a synchronized job, the cluster must supply the entire gang at once. For a throughput campaign, many smaller slots may be sufficient. Scheduler simulations or replay from historical jobs can reveal fragmentation and queue effects that a spreadsheet of total hours misses.

A capacity model keeps demand, service loss, and growth visible
Model element	Question	Evidence or treatment
Base demand	How much validated work is currently completed?	Accounting and application results by resource class, not hardware peak
Suppressed demand	What work is delayed, reduced, moved, or impossible?	Queue distributions, rejected requests, user forecasts, external usage, support records
Concurrency and deadlines	How much capacity must be available at the same time?	Campaign windows, gang size, reservations, interactive need, workflow dependencies
Growth	Which projects, datasets, models, or instruments will change demand?	Low, expected, and high scenarios with dates and named assumptions
Operating loss	What capacity is unavailable to useful scheduling?	Maintenance, failures, drains, facility limits, provisioning time, fragmentation
Useful utilization	How much occupancy can be sustained while preserving service objectives?	Queue delay and expansion factor by class, not a universal maximum percentage
System balance	Will data, fabric, memory, software, power, or licenses limit compute?	Cross-layer model and representative concurrent tests
Expansion path	Can the design grow without stranding or degrading the first purchase?	Ports, rack positions, power, cooling, address space, licenses, topology, support horizon

Use ranges. A base, expected, and high scenario makes uncertainty visible and helps identify decisions that remain good across several futures. Separate growth that is already contracted from speculative opportunity. Apply failure and maintenance margin based on observed service history or a stated design objective. Do not add the same margin repeatedly at project, subsystem, and vendor levels without naming it; hidden stacked contingencies can buy an unbalanced system.

Storage needs at least four views: active capacity, protected or retained capacity, performance and concurrency, and namespace scale. Data reduction, compression, purge, and archive policies can change capacity, but they require project and service action. Network models need endpoint bandwidth, fabric topology, simultaneous traffic, external transfer, and management paths. Facility models need measured workload power, equipment ratings, redundancy state, startup behavior, environmental range, and planned expansion. Chapter 5 develops the facility load model.

For accelerator workloads, model memory before arithmetic throughput. A device that cannot hold the required working set may need sharding, offload, recomputation, or smaller batches, changing network and storage demand. Record precision, model or problem size, parallel strategy, host memory, interconnect, and software versions beside performance. A projected speedup from one kernel does not establish end-to-end training time, inference behavior, or simulation throughput.

Benchmarks, prototypes, and uncertainty

A benchmark is a controlled question. The project should know what decision each benchmark informs. HPL measures dense double-precision linear-system performance and is useful for stressing a large machine, but it is not a general measure of application value. HPCG exercises a different combination of sparse computation, memory, and communication. STREAM measures memory bandwidth under defined conditions. OSU micro-benchmarks can characterize MPI communication. IO500 combines data and metadata tests under published rules. MLPerf suites define AI workloads and quality or scenario conditions. None replaces the organization's representative applications.

Use several levels of evidence:

Inspection and micro-benchmarks verify topology and isolate processor, memory, accelerator, network, and storage paths.
Component and system benchmarks exercise a repeatable property across nodes or at scale, often useful for performance uniformity and stress.
Mini-applications and kernels reproduce the important behavior of a larger application with code and data that can be shared.
Applications and workflows validate time, correctness, scaling, data movement, and usability at the boundary that justified the purchase.

Record the benchmark source and revision, patches, compiler and options, libraries, driver and firmware, dataset, precision, process and thread mapping, node count, topology, warm-up, repeats, measurement interval, correctness rule, and allowed tuning. Require bidders to label a result as measured, modeled, projected, or extrapolated. A projected result can be useful for unreleased hardware, but it should not quietly become measured evidence in the evaluation table.

Set acceptance thresholds before bids reveal which number is convenient. Include an expected variability rule and a repeat procedure. Some tests need a minimum; others need a range or uniformity bound. A single excellent node does not establish fleet quality. Test distributions across hardware cohorts and racks. Preserve individual results so an average cannot hide a group of slow, thermally limited, or incorrectly configured nodes.

Correctness is part of performance. Verify residuals, checksums, expected records, model quality, or application-specific invariants. Confirm that storage data is persistent under the stated semantics, not acknowledged only in cache. A tuned benchmark that changes the mathematical problem, dataset, precision, or durability condition is a different test unless the requirement expressly allows it.

Prototype the decisions with the largest uncertainty or switching cost. That may be application portability to a new accelerator, multi-rail communication, GPU-direct storage, a liquid-cooled rack interface, an unfamiliar filesystem, a disaggregated architecture, or a cloud workflow. Argonne's evaluation testbeds illustrate the value of exposing emerging hardware before a production commitment. A prototype should end with a recorded decision: accept the approach, reject it, narrow its scope, or carry a quantified risk and fallback.

Keep the benchmark definitions usable after acceptance. Benchpark and ReFrame are examples of tools that can describe reproducible HPC benchmarks and regression tests across systems. The exact framework is less important than preserving the test as an operating asset. A benchmark that runs only during a vendor-led acceptance week cannot help diagnose a regression six months later.

Compare architectures as complete services

Develop more than one credible architecture before selecting a bill of materials. Options may vary node balance, accelerator concentration, fabric topology, storage design, rack density, deployment phasing, integration boundary, or use of external capacity. Compare them against the same requirement set and cost horizon. An option that wins every row probably reflects scoring written after the answer was chosen.

Identify hard interfaces early. The node's PCIe and NUMA topology constrains accelerator and network placement. The fabric connects compute, storage, management, and perhaps external resources with different traffic needs. Rack density constrains power, cooling, service space, cable length, and failure domain. Software support connects the Linux distribution, kernel, drivers, MPI, communication libraries, containers, scheduler, applications, and security lifecycle. A change in one layer should trigger review of the interfaces it touches.

Hold a preliminary design review while meaningful alternatives remain. Confirm that the architecture addresses every mandatory requirement, fits the facility and schedule, has an operating owner, and identifies unresolved proof. Later, hold a detailed design review against actual part numbers, firmware and software sets, rack elevations, cable and port maps, power and cooling connections, management addresses, provisioning profiles, storage layout, security zones, and acceptance procedures. Marketing architecture diagrams are not detailed designs.

Freeze only what needs control. A contract that names every minor component can make harmless substitutions expensive, while a contract that permits “equivalent” parts without criteria can change performance, power, reliability, or support. Define controlled characteristics and an approval path. A storage-device substitution may require endurance, firmware, performance, failure, and rebuild evidence. A network optic may require reach, thermal, cable, firmware, and switch compatibility. The customer should receive an updated bill and test effect before approving the change.

Build, buy, and integration boundaries

Building from components gives the organization direct control and can be appropriate when the team already operates the relevant hardware at scale. An integrated cluster can transfer assembly, interoperability, installation, and support work to a supplier. Neither approach removes integration. It changes who performs it, who carries schedule and compatibility risk, and where the customer must retain enough knowledge to operate and verify the result.

Assign ownership before a fault falls between suppliers
Responsibility	Questions to settle	Required evidence or handoff
Component compatibility	Who qualifies server, memory, accelerator, adapter, optics, switch, storage, firmware, kernel, and driver combinations?	Qualified bill, version matrix, exceptions, regression results, change process
Rack integration	Who owns assembly, busbar or PDU, manifolds, cable routing, labels, weight, shipment bracing, and reassembly?	Rack elevation, as-built diagrams, inspection and delivery checklist
Facility interface	Where do vendor and site responsibility meet for power, cooling, network, floor, alarms, and safety?	Interface-control documents, operating envelope, approved connection and test procedure
Cluster software	Who supplies Linux, firmware tools, provisioning, scheduler, drivers, libraries, containers, monitoring, and updates?	Source or packages, configuration, licenses, SBOM where required, build and recovery procedure
Acceptance	Who prepares, runs, witnesses, diagnoses, reruns, and signs each test?	Approved procedure, raw results, defect record, retest, acceptance authority
Production support	Who receives a cross-layer incident and coordinates component suppliers?	Severity definitions, response and escalation, diagnostic entitlement, parts, hours, support term

Support boundaries matter most when the fault is ambiguous. An MPI regression might involve application code, library, kernel, adapter firmware, switch configuration, topology, or a failing cable. If each supplier accepts only a pre-isolated defect, the cluster operator becomes the unpaid integrator. The support plan should provide one accountable escalation path for the integrated scope, while allowing specialists from several organizations to participate.

Review lead times, end-of-sale dates, firmware and software support horizons, spare strategy, warranty start, and replacement equivalence. A three-year project can receive a component that is already close to end of support. Keep critical cold or on-site spares where replacement delay exceeds service tolerance, but account for firmware aging, batteries, media retention, and parts that require vendor enrollment. Define whether a failed part is returned with data-bearing media and what sanitization or retention option applies.

Supply-chain review belongs in acquisition and operation. NIST SP 800-161 treats hardware, software, firmware, suppliers, integrators, delivery, maintenance, and disposal as one risk-management lifecycle. Request provenance and component transparency appropriate to the organization's risk. An SBOM can help identify software components, but it is useful only when versions are mapped to the delivered and updated system and someone acts on relevant vulnerability information.

Total cost and a schedule that includes operation

Compare options over the same service horizon and workload. Acquisition price is joined by design and procurement labor, facility construction, power distribution, cooling, racks, network services, storage, software, licenses, support, staffing, training, migration, data movement, downtime, spares, financing, energy, water, external service charges, and decommissioning. Include taxes and currency exposure where they apply. Avoid false precision by showing ranges and the assumptions that dominate them.

The lowest equipment price is not necessarily the lowest service cost
Cost category	Commonly missed items	Useful comparison basis
Acquisition	Optics, cables, management switches, licenses, shipping, taxes, acceptance support, initial spares	Delivered and accepted scope, not headline compute price
Facility	Electrical study, switchgear, transformers, containment, CDU and piping, floor work, controls, permits, commissioning	Incremental work and reusable infrastructure over its own life
Operation	Administrators, software specialists, user support, monitoring, security, backup, on-call, training	Staff time and service responsibility by year and scenario
Energy and cooling	Idle and workload profile, facility overhead, demand charges, water, seasonal limits, growth	Measured or modeled energy for the same useful work and service window
Software and support	Subscriptions, commercial applications, license servers, accelerator stack, extended warranty, escalation	Compatible versions and support period needed by the application portfolio
Transition and retirement	Parallel operation, data copy, application porting, lost productivity, media sanitization, recycling	Complete change from old service to stable new service
External capacity	Storage, requests, network, egress, idle instances, commitments, failed work, engineering	Validated workflow completion under low, expected, and high demand

Power estimates deserve scenarios. Equipment nameplate ratings protect electrical design but can overstate normal energy; one benchmark run can understate a future workload. Use measured current-system behavior, vendor configuration and power-limit data, prototype results, and Chapter 5's design rules. State whether the model includes only IT equipment or facility overhead. A lower-power design that takes much longer may consume more energy per completed result.

The schedule must include design reviews, facility work, network circuits, manufacturing, software readiness, benchmark access, security review, shipping, installation, defect correction, acceptance, migration, training, and operational stabilization. Add dependencies. A system delivered before electrical service is ready is inventory, not schedule success. A facility ready before final rack data can also be expensive if assumptions force rework.

Maintain schedule contingency around known uncertainty, not an unexplained percentage at the end. Prototype completion may gate architecture. Utility work may gate energization. A firmware release may gate accelerator qualification. Data movement may gate retirement. When a milestone slips, update the service date and parallel-operation cost rather than preserving the headline date by silently shortening acceptance.

From market inquiry to contract

A request for information can test whether the market can meet the need, expose alternative architectures, refine facility assumptions, and identify long-lead or proprietary dependencies. It is not an award. Ask respondents to separate currently orderable products from roadmaps and to identify the maturity of software, benchmark results, and delivery claims. The public LLNL Future Generation HPC Center RFI is one example of using an inquiry to explore a multivendor center before a procurement.

The request for proposal should give every bidder the same problem and evaluation rules. Include scope, service and technical requirements, workload and benchmark package, facility interfaces, security and supply-chain obligations, documentation, training, project schedule, design reviews, bill-of-material detail, support, warranty, acceptance, data rights, disclosure rules, change control, and contract remedies. State which information is mandatory, scored, or requested for planning.

Ask for a compliance matrix in which each requirement is answered with complies, does not comply, or a clearly described exception, plus the proposed design evidence. Narrative proposals can otherwise spread one exception across several sections. Require actual, projected, and extrapolated performance to be labeled. The NERSC-10 technical requirements, for example, ask respondents to identify benchmark version, official-submission status, modifications, and whether workflow performance is measured or projected.

Evaluation should consider technical fit, measured performance, correctness, architecture risk, facility fit, software maturity, support, delivery, lifecycle, security, total cost, and the quality of exceptions. Avoid converting every field into a weighted decimal that implies more certainty than the evidence. Some requirements are gates. Some risks need a prototype or contract milestone. Record the basis for the decision so the project can later distinguish what was promised from what evaluators merely hoped.

Put acceptance in the contract, not in a document written after delivery. Define test versions, inputs, scale, environment, permitted tuning, thresholds, repeat and variance rules, raw evidence, witness rights, defect classification, correction period, retest, partial acceptance, milestone payment, warranty start, and the authority to approve an exception. LLNL's published CTS-2 procurement material illustrates the distinction between supplier delivery checks and the customer's subsequent functionality, performance, and stability acceptance.

Prepare the site and receive the actual system

Facility preparation proceeds from the controlled design, not an early sales estimate. Reconcile rack dimensions and weight, shipping sections, loading dock and route, floor loading, seismic requirements, service clearances, electrical feeds, connectors, phase balance, grounding, cooling water or airflow, heat rejection, controls, fire protection, leak detection, network circuits, security, staging, and waste handling. Qualified facility professionals and local codes govern the work.

Chapter 5 describes physical commissioning: document inspection, unloaded checks, staged load, sustained representative work, controlled transitions, approved failure cases, and baseline capture. This chapter uses those results as one gate in a larger project. Passing the cooling loop does not accept the cluster, but the cluster cannot begin thermal and workload acceptance on an uncommissioned cooling path.

Factory acceptance can find assembly, firmware, software, and performance defects before shipping. Define the tested configuration and how it represents the delivered system. A small scalable unit may prove node behavior but not full-fabric routing, filesystem scale, control-plane load, or rack thermal interaction. Preserve the factory bill, versions, configurations, logs, benchmark results, exceptions, and serial numbers. These records provide a baseline after disassembly and transport.

Receiving is a controlled technical step. Inspect packaging and shock or tilt indicators where used, record damage before unpacking, reconcile racks and components with shipping and asset records, verify seals and serials, and quarantine discrepancies. Confirm that substitutions were approved. Track data-bearing media and cryptographic devices. Do not power equipment simply because it has arrived; the approved energization sequence should begin only after electrical, cooling, grounding, network, and safety release.

Update the as-built record during installation. Cable maps, rack elevations, port assignments, management addresses, firmware inventory, power-feed mapping, coolant branches, asset tags, and photographs should describe what is present, not what the proposal depicted. Installation defects found later are much easier to isolate when every endpoint and physical path has a durable identity.

Commission in layers, then cross the layers

Commissioning moves from low-risk foundations toward complete work. A failed stage stops dependent stages while allowing independent work to continue. The exact sequence varies, but the dependencies are consistent: safe facility service precedes sustained load; management access precedes unattended fleet work; nodes, fabric, and storage precede distributed applications; a stable control plane precedes user acceptance.

Figure 10.2: Component tests are repeated at the integrated boundaries that depend on them. A defect returns to the earliest affected layer, then its dependent tests are rerun. The final baseline records not only passing results but also the exact build, approved exceptions, owners, and recovery information.

Start with management identity and a small node cohort. Verify BMC access, console, power, inventory, firmware, network boot, image, credentials, monitoring, and scheduler enrollment. Reconcile serial numbers, MAC addresses, switch ports, rack positions, and node names. Run node-level processor, memory, accelerator, PCIe, local-storage, thermal, and power tests long enough to expose early failures. Compare the distribution across the fleet rather than validating only a sample average.

Bring up the fabric by topology. Verify link speed and width, error counters, routing, path diversity, management separation, and expected locality. Test point-to-point latency and bandwidth, collectives, message rate, congestion, and failure behavior at increasing scale. Chapter 3 gives the fabric acceptance sequence. Preserve the physical and logical topology used by each result.

Commission storage with data integrity first, then capacity, metadata, throughput, concurrency, recovery, and client behavior. Test the paths applications will use, including MPI-IO, HDF5, object access, checkpoints, small files, transfer services, snapshots, and archive handoff where included. A peak write test against an empty system does not establish mixed production service. Chapter 4 develops these storage tests.

Only then should the complete scheduler, accounting, software, identity, telemetry, security, and support paths be treated as a service. Test job arrays, dependencies, interactive work, large gangs, accelerator resources, limits, reservations, accounting, failure states, node drain and return, software modules, containers, license tokens, data permissions, and user portals. Before early access, verify the approved security baseline: administrative roles, service accounts, network paths, firewall policy, image and container provenance, vulnerability status, logging, and documented exceptions. Restore selected control-plane state from backup. Exercise the maintenance and incident communication path rather than assuming it will emerge during the first outage.

Run the representative workload portfolio alone and under a mixed background load. Confirm correctness, time, scaling, energy, data movement, queue behavior, and repeatability. Include checkpoint and restart, node or link loss where supported, and a full user path from transfer and software selection through result retention. A benchmark team can tune an application into a result that ordinary users cannot reproduce; require documented builds and launch methods through the production interfaces.

Every failure receives an identifier, affected requirement, observed configuration, evidence, severity, owner, disposition, correction, and retest. A workaround is not automatically a correction. An accepted exception should state residual effect, operational procedure, owner, expiration or review date, and any price, warranty, or capacity adjustment. Do not erase failed results after a passing rerun; they explain the delivered risk and may predict later behavior.

Migrate the service, not only the data

Migration begins during design because application portability, data movement, identity, and user training can determine the service date. Inventory supported applications and environments, current projects, datasets, workflows, scheduler behavior, network dependencies, licenses, portals, automation, and local conventions. Decide what will be ported, rebuilt, retired, archived, or temporarily left on the old system.

Offer an early-access period with production-like policy and a limited user group. Start with maintainers of important applications and projects that represent different resource patterns. Track every manual fix. If an expert has to set an undocumented environment variable, change a container, or choose a hidden queue option, the service is not ready for broad migration. Turn repeated work into the supported environment or documentation.

Move data in planned stages. Identify authoritative copies, transfer tools, expected duration, bandwidth controls, checksums, open-file behavior, incremental synchronization, freeze or cutover window, ownership, permissions, quotas, and retained rollback copy. A successful byte transfer can still produce an unusable tree if UID/GID mapping, ACLs, symlinks, extended attributes, sparse files, or application metadata changes.

Run old and new services in parallel long enough to validate real work, but give the overlap an owner, budget, and end condition. Parallel operation consumes staff, licenses, facility capacity, and user attention. State where new jobs should run, how allocations translate, which data copy is authoritative, and how users report differences. Avoid moving a project back and forth after its output path has diverged.

Cutover criteria should include accepted infrastructure, supported software, migrated data, user-path probes, help coverage, status communication, security authorization, backup and restore, spare and escalation readiness, and successful representative projects. Freeze risky unrelated changes around the transition. After cutover, review queue behavior, failures, performance variance, storage growth, support load, and facility measurements daily until they become predictable.

Retire the old cluster only after required data, logs, configurations, licenses, keys, and records are handled. Remove credentials and network paths, sanitize media under organizational policy, update asset and support records, recover reusable parts where appropriate, and use an approved recycling or disposal path. Preserve enough configuration and benchmark history to explain results produced on the former system.

The requirements-to-acceptance matrix

The matrix is the project's compact memory. It should be started with requirements, maintained through design and contract, and completed during acceptance. The form can be simple, but the columns need to make ambiguity visible:

Sample entries connect business and workload needs to objective evidence
ID and source	Requirement	Verification and threshold	Owner and result
PERF-01 MPI project deadline	The validated simulation case shall complete on the declared production node count within the approved elapsed-time limit.	Run the fixed source, input, compiler, MPI, mapping, and correctness check three times; apply the stated variance rule.	Application lead; measured result, logs, output validation, exception if any
NODE-04 Fleet consistency	Nodes in each hardware class shall expose the approved topology and remain within the performance-uniformity bound.	Inspect every node; run cohort tests for memory, accelerator, PCIe, network, local storage, thermal, and power behavior.	Integration lead; distribution and quarantined-node record
DATA-03 Checkpoint service	The storage path shall complete the representative concurrent checkpoint without violating the background-workload limit.	Run fixed writer count, bytes, files, persistence condition, and background case for the specified interval.	Storage lead; raw client/server results and integrity check
FAC-02 Single approved fault	The selected rack group shall sustain the required workload under the documented redundant-component fault.	Qualified team removes the approved feed or cooling component; record load, temperature, alarms, action, throughput, and recovery.	Facility lead; signed procedure and measurements
OPS-05 Failed-node recovery	An operator shall detect, drain, repair or reimage, validate, and return a node through the production control path.	Inject the approved fault on a test node; preserve incident evidence and complete Chapter 8's node return-to-service test.	Operations lead; incident timeline and accepted node state
SVC-02 Project first run	An eligible project member shall complete the documented first workflow through supported service paths.	Authenticate, transfer, build or select software, submit, start, read, write, validate, account, and retain the result.	Service owner and user representative; end-to-end record

For each entry, retain the exact procedure, test environment, raw result, analysis, date, witnesses, configuration identity, defect links, and approval. Passing a demonstration by hand does not prove an automation requirement. Passing by analysis does not prove a test requirement unless the approved verification method permits it. If a requirement changes, review the architecture, contract, test, cost, schedule, and already accepted evidence affected by that change.

Acceptance is a decision, not the moment the last command exits zero. Confirm that mandatory requirements are satisfied or have approved exceptions; defects have dispositions; documentation, source, licenses, credentials, training, spares, and support are handed over; as-built records match the system; warranty and service dates are known; final payments follow the contract; and the operating owner accepts the baseline. User validation should confirm that the complete system is suitable for the intended work, not merely that each contractual test passed.

After thirty, sixty, or ninety days of representative operation, compare the service with the project case. Review workload completion, queue delay, usable capacity, failures, energy, storage growth, support load, application portability, and actual cost. Some acceptance tests should become periodic regressions. ReFrame, Benchpark, or a local framework can run them after firmware, driver, kernel, network, storage, or scheduler changes and can compare the result with the accepted baseline.

The matrix also closes the book's design loop. Chapter 1 began with the reason for clustering and the kinds of work a cluster performs. Chapters 2 through 9 supplied the component and service knowledge needed to make credible decisions. The final project should be able to point from every major purchase back to useful work, and from every important operating claim forward to evidence. Chapter 11 turns to seismic imaging, where application behavior ties node, fabric, storage, software, and service choices back to useful work.