Running a Production Cluster Service

Chapter 8 followed a node into scheduler service. A production cluster has to carry a larger promise. It must take a person who has data and an application, establish the right access and resource policy, help that application run at useful scale, preserve the result for the required time, and respond predictably when any part of the path fails.

This distinction is easy to miss because the machine is so visible. Racks, accelerators, switches, filesystems, and queue displays provide concrete signs of progress. A user can still have a poor service while all of those components are up. The login host may answer while identity mapping is broken. Compute nodes may be idle while the software needed by the project has been retired without notice. A job may exit successfully while writing an incomplete result to scratch space that is purged the next morning.

We will follow a representative project through this chapter. The project begins with a program tested on a researcher's laptop. It has a few terabytes of input, hundreds of independent preparation tasks, a multi-node simulation, and a GPU-assisted analysis stage. The group expects several people to collaborate for one year and then retain selected results. No single detail is unusual. What makes the work a production service is the set of agreements and operating practices that connect all of them.

Define the service before measuring it

A cluster service is the collection of paths by which work becomes possible. It includes identity, project membership, allocations, login and portal access, compute and accelerator classes, the scheduler, supported software, data storage, transfer, documentation, support, maintenance communication, and the people who operate them. Some of these functions may be supplied by other teams. They still belong in the service description because a user cannot route around a failed dependency merely because it has a different owner.

The first useful document is therefore a service catalog, not a hardware inventory. For each function, it says who may use it, what behavior they may rely on, which limits apply, who owns it, how a problem is reported, and where current status is published. The language needs to be specific enough to guide a decision. "Large scratch space" is not a service statement. Capacity, quota, purge interval, backup status, performance intent, and the procedure for requesting an exception are.

Availability objectives should be attached to user actions. "The cluster is 99.9 percent available" leaves unanswered what was tested. A better statement might cover the fraction of scheduled service hours in which an eligible user can authenticate, submit a documented validation job, start it within a stated service class, read its input, and write its result. Storage durability, interactive latency, and transfer completion need different indicators because one percentage cannot describe all three.

Planned maintenance should be named in the objective rather than quietly removed from the denominator after the fact. So should service hours, external dependencies, and declared exclusions. An ambitious number without a reliable measurement path creates an argument, not an objective. Start with a small number of important actions, measure them from the user side, and add detail when the service can act on it.

The service boundary also states what the cluster is not. It may not be a permanent repository, an unrestricted Internet host, a high-availability transactional database platform, or an appropriate environment for regulated data. These limits are useful when they are visible before a project commits to the system. They become operational failures when they are discovered after data has arrived.

The project journey

Our example group should not begin by requesting a large allocation. The first objective is to establish fit. What program will run? Which parts are serial, threaded, distributed, or accelerated? How large are the inputs and outputs? Which licenses apply? Does the data have restrictions? How quickly must a result return? Which step failed or took too long on the current system? The answers connect the project to the architecture and policies described in earlier chapters.

The service can offer a small exploratory allocation with short jobs and modest storage. This gives both sides evidence. The group learns the login, transfer, software, scheduler, and support paths. The operations team sees actual memory use, I/O behavior, accelerator compatibility, message pattern, job duration, failure modes, and support demand. A larger request can then refer to measured work rather than a rough multiplication of laptop time.

Project membership should express responsibility as well as permission. A project owner or principal investigator approves members and accepts data, license, cost, and acceptable-use obligations. Technical leads may manage software or workflows. Ordinary members run work without gaining administrative control over the group. Service identities need their own owner, purpose, credential lifetime, and review date. Shared personal accounts make all of these distinctions disappear and should not be used.

An allocation translates a project decision into resource permission. It may be denominated in node-hours, CPU-hours, GPU-hours, storage, service credits, or a local unit. None is a perfect measure of value. The unit should be understandable, tied to accounting evidence, and adjusted when different resources have very different cost or scarcity. An allocation is a planning and fairness mechanism; it is not a promise that every submitted job will begin immediately.

Onboarding is complete when a member can perform the whole first workflow, not when an account appears in a directory. A useful first-run path authenticates, transfers a small input, selects or builds the intended software, submits a batch job, observes its placement, verifies its output, records its resource use, and moves the result to the proper data tier. The support team should run this path itself after important changes. Documentation that no operator has recently followed is an untested interface.

Access and software presented to users

Command-line SSH remains an efficient interface for experienced cluster users. It should not be the only way to present the service. Open OnDemand can provide browser-based file access, shells, jobs, desktops, notebooks, and applications while still placing computational work through the scheduler. Science gateways and project portals can narrow the interface further, allowing a community to submit a known workflow without learning every scheduler option. APIs support automation and integration with laboratory systems.

These paths should converge on the same identity, allocation, data, and resource policy. A notebook launched from a portal must not escape accounting merely because the user did not type sbatch. A gateway service account should preserve the project and, where required, the initiating user in job and audit records. Interactive applications also need limits. A browser session that quietly occupies a GPU for a week is still a scheduled workload.

Login nodes are for editing, compilation, submission, lightweight inspection, and moving into an allocated interactive session. They are not spare compute nodes. Publish practical process, memory, time, and I/O limits, provide a clear command for obtaining an interactive allocation, and enforce the boundary consistently. A policy that exists only in an angry message after a login node becomes overloaded teaches users to hide work instead of placing it correctly.

Chapter 6 described the Linux software and parallel programming environment. The production service adds a lifecycle around it. A software catalog should distinguish at least three levels:

• Supported environments are installed, documented, security-maintained, and exercised by validation jobs. The team states compatible hardware classes and how long a release will remain available.
• Community or project environments are shared by a group and may use modules, Spack environments, Conda environments, Apptainer images, or another recorded build. The project owns scientific validation while the service supplies safe storage and execution boundaries.
• Individual experiments allow users to build in their own space. They receive best-effort support and must not become an undocumented production dependency.

Support level is separate from installation method. A container can be an officially qualified environment, and a centrally installed module can be unsupported legacy software. The important record connects application version, source or image digest, compiler, MPI, numerical libraries, accelerator runtime, host driver, build options, patches, and validation result. For licensed software, it also records entitlement, license-server dependency, token limit, and who may run it.

Module names should make change visible. Moving application/default to a new incompatible release without notice can alter hundreds of jobs. Keep versioned modules, announce retirement dates, identify affected recent users from accounting where policy permits, provide a tested migration example, and retain the build recipe or container digest required to reconstruct important work. Old software cannot remain exposed indefinitely, especially when it contains known vulnerabilities or depends on an obsolete operating system. Retirement is therefore a managed transition, not a promise of permanence.

Our example project first builds its program in a development environment, then records a production environment after the small run is correct. The production record includes the source revision, dependency lock or package manifest, compiler and MPI versions, container digest if used, input identity, launch command, and validation checksum or scientific test. The group may improve the program later, but it can still distinguish a software change from a change in the underlying cluster.

Data has a lifecycle, not just a path

A filename says where data is now. It does not say why it is there, how long it will remain, whether another person may read it, or whether it can be recovered after loss. Those meanings belong to the service. Home, project, scratch, archive, object, transfer, and node-local spaces should each have a declared purpose. Chapter 4 explains the storage paths and filesystems themselves; this chapter defines the contract presented to a project.

Scratch storage is valuable because it can be optimized for active computation without carrying every durability obligation of a repository. The word scratch, however, is not permission for an unpredictable purge. Publish the selection rule, warning schedule, exclusion process, and action taken on open files or active checkpoints. Base age on an explicitly named timestamp. Test the purge mechanism against a non-production namespace before it is allowed to remove project data.

Backup and replication should not be confused. Replication can keep a storage service available when a device or server fails, while also replicating accidental deletion or corruption. A backup or versioned snapshot may recover older state, but only within its retention and restore limits. A scratch filesystem may offer component redundancy with no backup at all. The project needs these distinctions before it decides where the only copy of an input or result will live.

Large transfers should use a managed path that can retry, verify, resume, and report independently of a laptop session. Globus is one widely used example: it separates a collection, which presents data and access policy, from the endpoint systems that provide storage access, and its transfer service tracks asynchronous tasks. Other tools can provide equivalent behavior. The operating requirement is a durable transfer record, integrity verification, bounded credentials, and a clear answer when either side is unavailable.

Data classification precedes ingestion. The project should identify contractual, personal, export-controlled, proprietary, health, or other regulated material before copying it to the cluster. The answer determines eligible storage, collaborators, transfer paths, logs, encryption, retention, and incident obligations. Discovering sensitive data by scanning an unrestricted scratch filesystem after the project has begun is an expensive substitute for intake.

The example group separates reusable inputs from generated intermediates. Reference datasets and source code live in protected project space. Active working data and checkpoints live on scratch. Each run records the source input identifier rather than relying on a mutable directory name. Selected results, software records, and explanatory metadata move to archive after validation. The remaining intermediates can expire without making the published result impossible to understand.

Reliable workflows and experiment records

A production workload is rarely one batch script. It can include ingest, validation, preprocessing, parameter generation, many independent tasks, a synchronized simulation, analysis, visualization, and retention. A workflow engine or an explicit dependency graph makes those relationships visible. Snakemake, Nextflow, Pegasus, and other systems can submit work to schedulers, while ordinary scheduler dependencies are sufficient for many smaller pipelines. The choice should match the scale and failure behavior rather than the fashion of the tool.

Each task should declare its inputs, outputs, resources, software, and success condition. A zero process exit code is useful but incomplete. The task may also need to confirm that expected files exist, can be parsed, contain the intended records, and correspond to the right inputs. Write outputs to a temporary name and publish them atomically where the filesystem permits. A retry must not treat a half-written file from the previous attempt as a valid result.

Retries are appropriate for transient failures, not for every failure. A transfer interruption, reclaimed external instance, or temporary service error may succeed later. An invalid input, deterministic program defect, exhausted memory request, or license denial usually needs a changed condition. Bound the attempt count, add delay or backoff, retain the reason, and make repeated failure visible. Unlimited immediate retries can turn one bad task into a scheduler and support incident.

Checkpointing trades additional I/O and implementation complexity for a smaller amount of lost work. Its value depends on job duration, failure and preemption rate, checkpoint interval, checkpoint size, write bandwidth, restart time, and whether the saved state is actually complete. A checkpoint policy therefore belongs to the workflow and service together. The storage team must know the burst pattern, the scheduler must preserve enough time for a final checkpoint when preempting, and the project must test restart from a checkpoint produced by the production software version.

Workflow automation can overload a healthy control plane. A shell loop that calls squeue every second, thousands of one-second jobs, one dependency per tiny file, or many workers simultaneously listing a directory can consume scheduler RPC, database, metadata, and login-node capacity out of proportion to the computation. Use job arrays, bundled tasks, event or accounting interfaces, local manifests, staggered polling, and bounded submission rates. Measure the complete workload shape, including control operations and file count.

Reproducibility begins with identity. Record the workflow definition and revision, application and environment, input dataset and parameters, scheduler request, cluster and partition, hardware class, job identifiers, timestamps, and output checks. For distributed or accelerator work, add rank and device placement, communication-library versions, precision and determinism settings, and the random seeds that are meaningful to the application. A research object format such as RO-Crate can package data and contextual metadata, but a simple, versioned manifest is already much better than reconstructing the run from shell history.

Our example pipeline bundles its preparation work into job arrays, places the multi-node simulation behind validated inputs, checkpoints to scratch, and runs analysis only after the expected simulation outputs pass structural checks. The workflow records every scheduler job identifier. When one GPU task fails, it retries only the affected partition of work and preserves the first failure record. This is less dramatic than rerunning the whole project, and that is precisely the point.

Measure whether useful work completes

Utilization is necessary but not sufficient. A cluster can report nearly full accelerator use while projects wait too long, jobs fail repeatedly, or users consume expensive nodes on work that scales poorly. It can also report low utilization because a critical campaign has finished early, because demand is seasonal, or because a facility constraint removed nodes from service. A useful review combines demand, delivery, efficiency, reliability, support, and outcome measures.

Scheduler accounting supplies submit, eligible, start, end, requested resources, allocated resources, state, exit code, and measured use when collection is configured. Slurm's sacct and reporting tools can expose much of this record. A COMPLETED state means that the batch script returned zero; it does not know whether a simulation converged or an output contains the expected data. Preserve system and application success as separate fields.

Report distributions rather than averages alone. The average queue delay can remain stable while the largest jobs or one project class waits much longer. Percentiles should still be grouped by relevant job size and resource class; a single percentile across one-minute CPU jobs and week-long accelerator jobs is difficult to act on. Show the observation interval and sample count, and retain the raw definition when dashboards change.

Utilization should use usable capacity in its denominator. A drained node is not an idle opportunity for the scheduler, although it remains installed equipment for financial reporting. Keep both views. Likewise, separate requested from allocated and measured use. Users often request extra wall time or memory as protection against eviction, while a scheduler may allocate complete nodes even when a job uses only part of each one. The gap identifies an opportunity for policy, education, or application tuning, not automatically misconduct.

Training and inference services need workload-specific measures in addition to cluster measures. Distributed training may track completed steps, samples per second, model-flop utilization, checkpoint overhead, time to a target quality, and restart loss. Inference may track successful requests, time to first token or result, per-token or per-request latency, throughput, queueing, availability, and cost under a declared model and request distribution. Appendix 1 and Appendix 2 develop those workloads. Their measurements should not be imposed on unrelated simulations merely because the cluster contains accelerators.

Change, incidents, support, and learning

A maintenance window is a user-facing event before it is an engineering event. Publish its scope, start, expected end, affected services, job handling, data risk, preparation required from users, and status location. Long-running jobs may need reservations or a submission cutoff well before the window. A storage interruption can affect jobs that appear computationally independent. The notice should describe consequences rather than list only the devices being changed.

The change record connects purpose, owner, reviewed plan, affected service objectives, canary and validation path, user communication, rollback, and final result. Chapter 8 described canaries for node and fleet changes. At the service level, the canary must cross the user path. A scheduler upgrade is not fully validated by daemon health; submit representative CPU, accelerator, array, dependency, interactive, accounting, and cancellation cases through supported access paths.

During an incident, assign roles explicitly. One person coordinates and keeps the current plan; another investigates or executes; another communicates when the impact is broad enough to require it. Record a timeline in a durable shared location. Status updates should state observed impact, what remains available, the next action or decision point, and when another update will arrive. Users can plan around an uncertain recovery more effectively than around silence.

Restoration and diagnosis have different clocks. The immediate objective may be to stop data loss, isolate failed nodes, route around a service, or restore a known-good version. Root-cause investigation can continue after the service is stable. Avoid changing several layers at once unless containment requires it. Every unrecorded experiment makes the timeline harder to interpret and the successful action harder to reproduce.

A post-incident review should explain conditions and decisions rather than find a person to blame. Include impact, detection, response, communication, recovery, contributing technical and organizational factors, what worked, and concrete follow-up with owners and dates. An incident can justify improvements in detection, mitigation, documentation, training, capacity, testing, or communication even when no single permanent fix exists. Review recurring low-severity events as well as spectacular outages; they often consume more project time.

Support records are another source of service telemetry. Categorize requests by the step the user was trying to complete: access, transfer, build, submission, queueing, runtime, performance, output, retention, or policy. Track recurrence and affected project class, not just ticket count. Ten users asking the same question are evidence that the service interface or documentation needs repair. Closing ten tickets individually does not remove that work from the next ten users.

Documentation should contain runnable paths for common work, short explanations of local policy, resource-selection examples, and precise data semantics. Training can begin with shell, scheduler, transfer, and software basics, then progress to profiling, scaling, workflows, and reproducibility. Office hours are valuable for ambiguous problems and can reveal emerging demand before formal requests do. They should complement a tracked support system rather than becoming an invisible queue known only to well-connected users.

Capacity and service review

Capacity planning starts with demand that could not be served as well as demand that ran. Scheduler records show submitted work, but not jobs users withheld because the queue was too long, data that could not fit, software that was unavailable, or projects that never onboarded. Combine accounting with allocation requests, rejected or reduced requests, support cases, project forecasts, instrument schedules, grant milestones, and office-hour conversations.

Segment demand by the resource that constrains it. CPU cores alone do not describe a job that needs large memory, a particular GPU, fast local storage, high-bandwidth fabric placement, licensed tokens, or a six-hour contiguous reservation. Track the queue and usable capacity for each meaningful class. Growth in one accelerator model does not automatically justify more general CPU nodes, and a filesystem metadata limit is not solved by unused data capacity.

Look at time. Academic terms, weather campaigns, instrument runs, financial cycles, design milestones, model-training campaigns, and publication deadlines create peaks. A monthly average can hide them. Ask projects for a range rather than a single exact forecast, record the assumptions, and compare forecast with observed use at the next review. Forecasting improves when there is no penalty for being honestly uncertain.

Service review can occur at several cadences. Daily operations examine incidents, failed probes, degraded capacity, and urgent communication. A weekly review follows changes, recurring tickets, storage headroom, long drains, and near-term campaigns. Monthly or quarterly reviews examine service objectives, queue distributions, application success, allocation consumption, project outcomes, energy and cost, forecast demand, risk, and work that should be stopped. The interval matters less than connecting evidence to named decisions and owners.

Capacity action is not always a purchase. The service may tune an application, correct oversized requests, add a short-job queue, bundle tiny work, improve data staging, retire unused software, recover drained nodes, adjust reservation policy, or move a suitable campaign to another resource. Expansion becomes justified when the unmet demand remains after these changes and the proposed resource matches the actual constraint. Chapter 10 will turn that evidence into a project and acceptance plan.

Multitenancy, sensitive work, and licenses

A shared cluster brings together users who do not necessarily trust one another, projects with competing priorities, and data with different obligations. Scheduler isolation, Linux accounts, filesystem permissions, containers, and network policy each cover part of this problem. None should be presented as a complete boundary without a threat model and testing.

Define collaboration at the project level. Group membership, shared directories, service accounts, data collections, repositories, and allocation roles should derive from the same approved project record where practical. Review membership periodically and when a person changes role. Revoke interactive credentials, scheduler access, transfer grants, portal sessions, API tokens, project groups, and licensed-software entitlement together; removing an SSH key alone can leave several valid paths.

Sensitive workloads may require dedicated partitions, nodes, filesystems, networks, access hosts, logs, administrators, or an entirely separate cluster. The answer depends on the data and control requirements, not the label HPC. NIST SP 800-171, for example, specifies requirements for protecting controlled unclassified information in nonfederal systems, while an organization may have additional contractual or national rules. Map each applicable requirement to an implemented control and retained evidence. Do not advertise compliance from the presence of encryption or multifactor authentication alone.

Temporary files and operational records deserve attention. Job command lines, environment variables, crash dumps, profiler output, scheduler accounting, container caches, and support attachments can reveal paths, dataset names, secrets, or proprietary parameters. Collect only what has an operating purpose, restrict access, set retention, and give projects a documented method for supplying sensitive diagnostic material.

Commercial software introduces another shared limit. A cluster may have far more compute capacity than license tokens, or a license may restrict geography, cloud execution, named users, or container use. Represent scarce tokens in scheduling where possible so jobs wait before consuming nodes. Monitor license-service availability separately from application behavior, and include entitlement in external-capacity decisions. Moving a binary to a cloud node does not move the legal right to execute it.

Reproducible research and security are often presented as competing goals. They need not be. A project can retain source revisions, manifests, checksums, parameters, derived metadata, and validation records without publishing protected inputs or credentials. The record should distinguish material that can be shared, material retained under controlled access, and material that must be destroyed. FAIR principles encourage data to be findable, accessible under stated conditions, interoperable, and reusable; accessible does not mean public without restriction.

External capacity, managed services, and federation

External resources extend the service only after identity, data, software, policy, and support have crossed the boundary. Acquiring an instance is the easy part. A useful job needs the correct image and drivers, scheduler or workflow integration, input placement, credentials, license permission, output return, monitoring, accounting, security controls, and a person who owns failure on both sides.

Slurm can resume and suspend nodes through site-supplied programs, including cloud instances added on demand, and can place local and cloud nodes in a hybrid configuration. The controller's ability to request a node does not establish that the node is ready. Resume timeouts, image identity, registration, configuration, mounts, credentials, monitoring, workload validation, and cleanup all remain part of the operating design. Rate limits also protect the provider quota, scheduler, image service, and budget from an uncontrolled launch wave.

A Slurm federation addresses a different problem. Jobs can have sibling records across eligible peer clusters, with the clusters coordinating so one runs. This can improve placement across established services, but it does not make the clusters identical. The application still needs compatible software and hardware, the user needs access, and the data must be present or moved. Current Slurm documentation also states limitations, including job-array behavior and a warning that the federation mechanism is not intended as a very high-throughput submission environment. Federation should be tested with the site's actual job patterns rather than treated as a universal global queue.

Cost comparison must include the whole path. Metered compute is joined by storage, requests, network, data egress, software licensing, support, reserved commitments, idle booted nodes, failed work, engineering time, and the local resources retained for the rest of the service. Conversely, an on-premises comparison includes facility work, power, cooling, staffing, spares, support, financing, downtime, and refresh. Use a range for uncertain workload and price assumptions, then reconcile estimates against invoices and accounting.

Data gravity is a practical constraint rather than a slogan. A short computation can be uneconomic to move when its input is large and the output must return. A long computation with a small input may be an excellent external candidate. Measure transfer time and sustained rate over the real endpoints, include preparation and checksum work, and state which copy becomes authoritative. For repeated campaigns, pre-position immutable reference data and move only changing inputs and selected results where policy permits.

The service should place guardrails before external launch: approved instance and accelerator types, region and network, image digests, maximum nodes and duration, project budget, required tags, idle and failure cleanup, data locations, encryption, license approval, and alerts at spend thresholds. A kill switch is not enough if terminating the instances destroys the only result. Teardown follows output validation and retention.

Completing or renewing a project

Projects end gradually. Members leave, funding changes, a paper is delayed, and active data becomes reference material. An explicit review date prevents accounts, allocations, data, licenses, and service identities from persisting forever because no one issued a final instruction. Notify the owner early enough to choose renewal, transfer, archive, or deletion.

The project should identify the results that must remain, the person or organization that will own them, the retention term, access conditions, metadata, software record, and retrieval path. Verify archived material before removing the active copy. A checksum establishes bit identity, not scientific completeness, so the project also needs to confirm that selected outputs, inputs or references, parameters, and explanatory files can support the intended future use.

Revoke people and machines according to the recorded decision. Remove project roles, interactive and portal access, scheduler associations, transfer shares, repository permissions, API and automation credentials, licensed-software entitlement, and external-provider access. Disable service identities whose workflows have ended. Preserve only the operational and audit records required by policy, with their own access and retention.

Close the allocation with a plain account of what happened. Record resource use, important variance from the original estimate, failed or abandoned approaches, support and service issues, retained data, outputs or milestones, and follow-on demand. This is useful even when the project did not meet its scientific or commercial objective. It improves the next allocation and the capacity model, and it separates a service failure from an uncertain research result.

Production-readiness review

A production cluster service is ready when a representative project can complete its path and the organization can operate that path repeatedly. The following review is intentionally end to end:

1. Publish the service boundary. Name user communities, supported workload and data classes, service hours, objectives, limits, owners, status path, support path, and explicit exclusions.
2. Onboard a test project. Approve a project owner and members, assign an allocation, create the required data spaces, and verify that every identity and role maps correctly.
3. Complete a first-run workflow. Authenticate through each supported access path, transfer input, select or build software, obtain an interactive allocation, submit batch work, verify output, inspect accounting, and retain the result.
4. Test software reconstruction. Rebuild or retrieve a supported environment from its recorded recipe or digest and run the qualification case on every declared hardware class.
5. Exercise data semantics. Test quota, sharing, snapshot or backup restore where promised, scratch purge warning, archive retrieval, checksum verification, and end-to-end external transfer.
6. Test workflow failure. Interrupt a transfer, fail one array task, exhaust a bounded retry, restart from a checkpoint, and prove that incomplete output cannot be accepted as final.
7. Verify accounting definitions. Reconcile scheduler, workflow, storage, support, energy, and project records for a sample. Confirm time zones, denominators, exclusions, and handling of requeue or resize.
8. Measure user paths. Run service probes from outside the control plane for login, submission, job start, storage read and write, result retrieval, and status communication.
9. Run a maintenance event. Announce impact and job policy, establish a reservation or cutoff, execute canaries, validate complete user paths, update status, and close the change record.
10. Simulate an incident. Assign coordination, technical, and communication roles; contain the condition; preserve a timeline; restore service; and produce reviewed follow-up actions.
11. Review multitenancy. Attempt prohibited project, filesystem, scheduler, container, network, portal, transfer, and support-record access. Verify logging and revocation across every path.
12. Exercise external capacity. Launch a bounded job on the selected external model, validate image and policy, move data, record cost, return and verify output, then remove resources and credentials.
13. Offboard the test project. Select and verify retained material, revoke users and service identities, remove allocations and shares, apply deletion policy, and confirm the final project record.
14. Hold an operating review. Examine objectives, queue and success distributions, usable capacity, incidents, recurring support, data growth, cost, project outcomes, risks, and forecast demand. Assign decisions rather than merely presenting charts.

The tests turn the cluster from installed equipment into an accountable service. They also produce the evidence needed for the next design. Chapter 10 will start with that evidence and follow a cluster project from workload requirements and capacity modeling through procurement, commissioning, migration, and acceptance.

References and further reading

1. Open OnDemand: Browser access to high-performance computing resources.
2. Open OnDemand: Architecture, configuration, security, and application documentation.
3. ACCESS: Getting a first project and resource allocation.
4. ACCESS: Allocations policy.
5. ACCESS: Common HPC paths and service references.
6. HPC Carpentry: Introduction to using remote high-performance computing systems.
7. Lmod documentation: Managing software environments with modules.
8. Spack: Reproducible software environments.
9. Apptainer user guide.
10. Globus: Collections, endpoints, storage gateways, and access.
11. Globus: Reliable Transfer API and task monitoring.
12. Snakemake: Workflow executors.
13. Snakemake: Slurm executor plugin.
14. Nextflow: Process errors and retry strategies.
15. Pegasus Workflow Management System documentation.
16. RO-Crate specification: Packaging research data and contextual metadata.
17. Wilkinson et al.: The FAIR Guiding Principles for scientific data management and stewardship.
18. SchedMD: Slurm job accounting with sacct.
19. SchedMD: Job, step, and derived exit codes.
20. SchedMD: Queue fields, pending time, and job state.
21. SchedMD: Accounting reports.
22. SchedMD: Scheduler performance and diagnostic statistics.
23. Open XDMoD: HPC metrics, accounting, and performance analysis.
24. Google SRE: Service level indicators and objectives.
25. Google SRE: Incident management guide.
26. Google SRE Workbook: Postmortem culture and learning.
27. Google: SRE best practices for capacity management.
28. NIST SP 800-171 Revision 3: Protecting controlled unclassified information.
29. CISA: Cross-sector cybersecurity performance goals.
30. SchedMD: Power saving, on-demand nodes, and hybrid cloud capacity.
31. SchedMD: Slurm federated scheduling.
32. SchedMD: Slurm network paths for clusters and federations.