Linux Software and Parallel Programming Environment

Chapter 2 described the processors, memory, and accelerators inside a node, while Chapter 3 followed data between nodes. Software joins those pieces into a computer that an application can actually use. A fast processor is of little value when an executable was built for the wrong instruction set, and a low-latency interconnect cannot help an MPI library that has fallen back to an unintended transport.

On a workstation, a developer may control almost the entire software environment. A cluster separates that control among application developers, software maintainers, system administrators, scheduler policy, and equipment vendors. The kernel and device drivers may come from the site image, the compiler and MPI library from a module, numerical libraries from another build, and the application itself from a container. Each layer can be correct in isolation while the assembled environment is wrong.

This chapter follows one application from source code to a reproducible multi-node run. It does not prescribe one Linux distribution, compiler, MPI implementation, or package manager. Those choices change. The more durable task is to identify the interfaces between them, decide which combinations the site supports, and retain enough evidence to reconstruct a successful run.

The software path to the hardware

A cluster application rarely talks to the hardware by itself. Source code is translated by a compiler and linker. The executable calls language runtimes and scientific libraries. MPI or another communication library passes messages through user-space network libraries and kernel drivers. Accelerator code crosses a similar boundary through a device runtime and driver. The scheduler supplies nodes and starts processes, while the operating system supplies memory, files, processes, and device access.

Figure 6.1 shows this path as a set of interfaces rather than a pile of products. The boundary between the application and a library is often an application binary interface, or ABI. The boundary between a user-space device library and its kernel driver has its own compatibility rules. A container can carry much of the upper part of the figure, but it normally continues to use the host kernel and selected host device support.

Early Beowulf installations often assembled this path by installing packages and applying local changes directly on each node. BOOTP or DHCP setup, custom boot scripts, and even kernel patches appeared in software-environment instructions because deployment and application support were tightly mixed. The underlying questions remain useful, but the responsibilities are now clearer. Images, network boot, configuration, and fleet administration belong to Chapter 8. Here we start with an installed node and ask what an application needs above and through that base.

Establishing a supported base

A software stack begins with a platform definition. Record the processor architecture, Linux distribution and release, kernel, firmware, device drivers, and the user-space libraries that communicate with those drivers. On an accelerator node, the platform definition also includes device firmware and the supported driver/runtime combination. On an RDMA network, it includes the adapter firmware, kernel driver, user-space provider, MPI build, and transport selected at run time.

This does not mean that every version must be frozen indefinitely. Security fixes and hardware support require change. It means that change should move a tested combination to another tested combination. If a driver update forces an MPI rebuild, or a new kernel changes an out-of-tree accelerator module, the dependency is known before production jobs discover it.

Linux makes an important distinction here. Its system-call interface to user programs is deliberately stable, but its internal kernel interfaces are not a permanent ABI for external modules. A vendor driver built outside the kernel tree may therefore have a narrower kernel compatibility range than an ordinary application. Distribution-supported kernel and driver packages reduce this burden by testing and rebuilding the combination together. A site that assembles its own kernel must take ownership of the same work.

The supported base also includes quiet services that parallel software assumes will work. Nodes need consistent identities and hostnames. Time synchronization makes logs and traces comparable, although it does not turn wall clocks into a substitute for MPI synchronization. User and group identities must resolve consistently where shared files are used. Filesystems and temporary directories must be mounted at the paths expected by the job. Security limits, locked memory, process counts, and device permissions have to match the chosen communication and accelerator stack.

Kernel tuning should follow a measured constraint. Changing page size, NUMA policy, interrupt affinity, network buffers, transparent huge pages, or I/O settings can help one workload and harm another. Start from a vendor- and distribution-supported configuration, measure the application, change one controlled variable, and retain a way back. Let the measurement define both the setting and the regression test that accompanies it.

The build environment is a contract

A compiler does more than translate syntax. It chooses an object format, calling conventions, language-runtime dependencies, instruction sets, optimization transformations, and sometimes an OpenMP or accelerator runtime. The linker then joins those decisions to the interfaces offered by MPI, numerical libraries, and the operating system. The resulting executable carries the choices made by that particular toolchain.

For C, C++, and Fortran MPI programs, use the MPI implementation's wrapper compilers unless there is a documented reason not to. Commands such as mpicc, mpicxx, and mpifort invoke an underlying compiler with the include paths, libraries, and link options required by that MPI installation. They do not constitute separate compilers. Record both the wrapper and the compiler beneath it, and rebuild all objects when changing compiler families.

Compiler flags belong in the build record. Optimization level, target architecture, floating-point transformations, debug information, OpenMP or offload options, and link mode can affect correctness as well as speed. Aggressive reassociation or reduced precision may be acceptable for one algorithm and invalidate another. Instructions selected for the newest login node may fail on older compute nodes. A cluster with several processor generations either needs a common target or separate builds matched to node features.

Modules and software hierarchies

Environment modules provide names for supported software combinations. Loading a module can set executable paths, library paths, manual paths, compiler variables, and product-specific settings. The module name should identify enough of the stack to distinguish incompatible choices. A bare module named mpi/latest is convenient today but provides poor evidence six months later.

A hierarchical module tree can expose dependent software only after its parent toolchain has been selected. Loading a compiler reveals MPI implementations built with that compiler; loading one of those MPI implementations then reveals libraries built against that pair. This arrangement reduces the chance that an application will combine a Fortran runtime from one compiler with a library built by another, or link to a different MPI than the one used at run time.

Modules describe and activate an installed stack, but they are not by themselves a complete build history. A modulefile can change while keeping the same name. Capture the resolved module list, the module collection or revision where the site supports one, and the concrete versions reported by compilers and libraries. Batch scripts should load their environment explicitly rather than inherit an unexplained login session.

Package and build tools

Cluster sites commonly use tools such as Spack or EasyBuild to express dependency graphs and produce consistent installations. Spack environments can keep root specifications in a spack.yaml file and concrete dependency decisions in a spack.lock file. EasyBuild recipes combine an easyconfig with an easyblock and a toolchain definition. These systems can make a large matrix manageable, but reproducibility still depends on retaining recipes, patches, source checksums, compiler versions, build options, and the repository or package index state used to resolve dependencies.

The application itself should also have an explicit build. CMake, Meson, Autotools, Make, language package tools, or a small site script can all work when the inputs and outputs are clear. Keep generated objects outside the source tree where practical, make configuration results visible, and test a clean build. The clean build exposes old objects and undeclared libraries before users have to find them in a failed job.

Build caches and precompiled packages reduce repeated work, especially when a dependency graph contains large compilers or accelerator libraries. A cache entry has to be keyed by the parts of the platform that affect its ABI and behavior. Reusing a binary because its package version matches is insufficient if the compiler, target architecture, MPI, or build options differ. Signed build artifacts and controlled repositories also help distinguish approved software from a directory that merely happens to be writable by several users.

Retain the run record

A useful run record answers four questions: what source was used, how it was built, where it ran, and how it was launched. At minimum, retain the source revision or release archive checksum; dependency lock or package recipe; compiler, MPI, and library versions; build flags; executable or container digest; node type; operating system and driver versions; scheduler request; process and thread placement; important environment variables; input-data identity; and application command line.

A short machine-readable manifest can sit beside human-readable notes and application output. Have the run or submission wrapper create it while the environment is still present. Retain identifiers, checksums, versions, and non-secret configuration, while excluding secrets and renewable credentials.

Choose the programming model at the memory boundary

The first programming question is where the data lives. Threads in one process can normally address the same virtual memory. Processes on different nodes cannot; data must move through messages, one-sided operations, a storage system, or a higher-level framework that performs that movement. Accelerators introduce another memory and execution boundary even when they are installed in the same node.

Shared-memory programming

OpenMP adds parallel regions, work sharing, tasks, synchronization, and accelerator directives to C, C++, and Fortran. A loop can often be parallelized with a small source change, but the directive does not prove that iterations are independent. Variables have sharing attributes, reductions need correct operators, and a missing synchronization can produce a result that changes with timing.

Memory placement matters on a multi-socket node. Linux normally maps a page near the processor that first touches it. If one initialization thread touches an entire array and later threads spread across sockets, many accesses may cross the socket interconnect. Parallel first-touch initialization, deliberate thread affinity, and an understanding of the node's NUMA topology can make the software layout follow the hardware layout discussed in Chapter 2.

Thread counts should come from the allocation rather than the largest number the node could theoretically start. Math libraries may create their own thread teams. Combining eight MPI ranks, eight OpenMP threads per rank, and an eight-thread BLAS call can turn a 64-core allocation into hundreds of runnable threads. Either give each layer a deliberate share of the cores or keep lower-level libraries serial inside an already parallel region.

Distributed-memory programming

MPI is the standard portable interface for process communication in cluster applications. Each MPI process has a rank within a communicator. Point-to-point operations move data between ranks; collective operations coordinate a group; derived datatypes describe non-contiguous layouts; topology interfaces help express communication structure; and MPI I/O coordinates access to files. The standard describes semantics. The implementation decides how to use shared memory, RDMA, TCP, or other transports on the installed system.

A correct decomposition assigns both computation and data to ranks. A three-dimensional mesh might be divided into subdomains, with neighboring ranks exchanging boundary cells before each iteration. Increasing the rank count makes each subdomain smaller, but the surface-to-volume ratio grows and the exchange becomes a larger fraction of the work. The program eventually reaches a point where more ranks add communication and synchronization faster than they remove computation.

Nonblocking communication can overlap useful work with data movement when the implementation and algorithm allow progress. It can also add complexity without overlap if the program immediately waits, sends buffers before their prior operation is complete, or assumes progress that the MPI implementation does not provide. Measure the actual timeline. An API whose name begins with I is an opportunity for overlap, not evidence that overlap occurred.

Collectives should be expressed as collectives rather than rebuilt from many point-to-point messages without a specific reason. Implementations can choose algorithms based on message size, communicator size, topology, and transport. The best choice can change at different scales. Site tuning should therefore exercise representative collectives and full applications at the intended scale, then interpret microbenchmark latency in that context.

Hybrid process, thread, and device mapping

A hybrid application uses fewer MPI ranks per node and multiple threads within each rank. This can reduce replicated memory and the number of inter-node messages. It may also reduce communication concurrency or expose thread-safety costs. There is no universally correct ratio. A useful starting point follows hardware boundaries: one rank per NUMA domain or accelerator, threads bound to nearby cores, and memory placed near the rank that uses it. The application then has to confirm that this mapping balances work.

Figure 6.2 is not a recipe for every two-node job. Some applications need one rank per core; others use one rank per node. A rank may drive several accelerators, or several ranks may share one device when the runtime supports it. The diagram's purpose is to make ownership visible. If two ranks accidentally select the same accelerator while another remains idle, the job can be correct and still perform very poorly.

Accelerator programming

CUDA and HIP expose host APIs and device kernels for NVIDIA and AMD accelerators respectively. SYCL provides a C++ programming model whose implementation targets one or more backends. OpenMP target and OpenACC use directives to identify regions and data for offload. These approaches differ in portability, language integration, tooling, library coverage, and access to device-specific features. The practical choice begins with the application's existing code and the stack supported on the intended cluster.

Offload has a data path as well as a compute path. A kernel that runs quickly can still lose to the CPU version if each small operation copies data across a device link. Keep data resident for a useful sequence of operations, make transfers explicit in the performance model, and test whether communication can proceed directly between devices where the hardware and software support it. Confirm the actual device-to-device path with transport diagnostics and a measured transfer.

NCCL and RCCL provide topology-aware collective operations for arrays on accelerators. They are communication libraries used by frameworks and applications, not complete replacements for MPI or for an application programming model. Appendix 2 follows these collectives through distributed AI training. The same software discipline applies there: framework, compiler, device runtime, collective library, driver, firmware, and fabric must form a tested combination.

Scientific and data libraries

A mature numerical library usually offers more than a convenient function call. Its algorithms have been studied for stability, its interfaces are used by many applications, and optimized implementations can exploit vector units, cache, threads, or accelerators. Reusing that work is normally preferable to writing a private matrix multiply or Fourier transform. The application still has to call the library in a way that fits its data and parallel decomposition.

The Basic Linear Algebra Subprograms, or BLAS, define common vector and matrix operations. Level 1 routines operate mainly on vectors, Level 2 on matrices and vectors, and Level 3 on matrices. The higher arithmetic intensity of many Level 3 operations gives optimized libraries more opportunity to reuse data and approach processor throughput. LAPACK builds dense linear algebra algorithms on BLAS. ScaLAPACK extends related operations to distributed-memory systems using block-cyclic data layouts and communication support.

Other libraries address different structures. FFTW computes one- and multi-dimensional Fourier transforms and includes MPI interfaces for distributed data. PETSc provides data structures and scalable solvers for systems arising from partial differential equations and related problems. HDF5 supplies a data model, file format, and APIs for structured data; Parallel HDF5 coordinates participating MPI processes through MPI I/O. Domain libraries add established algorithms and file conventions for particular sciences and engineering disciplines.

Several compatibility details deserve explicit checks:

• Integer width: LP64 library interfaces commonly use 32-bit integers even on 64-bit systems, while ILP64 variants use 64-bit integers for dimensions and indices. Mixing the interfaces can produce incorrect calls rather than a friendly error.
• Threading: A threaded BLAS can help a serial or lightly threaded application. Inside a large OpenMP or MPI job, uncontrolled library threads can oversubscribe cores and make performance erratic.
• Precision and numerical behavior: Single, double, mixed, and reduced-precision paths do not have interchangeable error properties. Optimized implementations may also choose different reduction orders, so bitwise identity is not always a reasonable portability test.
• Distributed layout: A library's communicator, process grid, data distribution, and collective-call rules must agree with the surrounding application. Conversion into the library's preferred layout can dominate a short solve.
• Accelerator backend: A library name may front several CPU and device implementations. Record which backend was selected and whether data remained on the device between calls.
• File and schema compatibility: A data library version is only part of the record. Dataset layout, chunking, compression, collective access, and format-version bounds affect portability and performance.

Parallel I/O makes the connection to Chapter 4's storage architecture. A file format cannot eliminate a poor access pattern. Thousands of ranks creating tiny files can overload metadata service. Thousands of independent writes into one shared layout can create lock or chunk contention. Collective buffering and data aggregation can turn many small requests into fewer large requests, but the application has to select dimensions, chunks, and aggregators that fit both its decomposition and the filesystem.

Containers on a cluster

A container packages a user-space filesystem and metadata describing how to run it. This is valuable on a cluster because an application can carry a consistent set of language runtimes and libraries from development into production. It does not package a new physical machine. Linux containers use the host kernel, and cluster jobs continue to depend on host devices, mounts, identities, scheduler controls, and site security.

OCI specifications define common image and runtime formats used by a broad container ecosystem. Apptainer is widely used in research computing because it can execute images without requiring users to operate a privileged daemon and because its single-file SIF format fits common cluster workflows. A site may support OCI images directly, convert them, or build a signed SIF image from a definition file. Retain the definition and immutable image digest so the record identifies exact image bytes even after a tag such as latest moves.

Drivers and accelerators

The host owns the kernel driver for an accelerator. The container supplies application libraries, while the runtime may bind selected host driver libraries and device files into the container. This arrangement allows one host driver to serve several compatible user-space stacks, but it does not make every combination valid. The container's C library, device runtime, application binary, host driver, and accelerator generation still need a documented compatibility path.

The same principle applies to high-speed networking. An image may contain MPI, communication plugins, and user-space RDMA libraries, but the host supplies the device and kernel support. Verify which libraries are taken from the image and which are injected from the host. A container that silently reaches the network over TCP can appear portable while losing the property for which the cluster was purchased.

Launching MPI containers

Apptainer documents two broad MPI arrangements. In the hybrid model, the host's MPI launcher starts processes and communicates with a compatible MPI inside the container. In a bind model, host MPI components are made available inside the container. Both require attention to implementation family, ABI, process-management interface, network plugins, and filesystem paths. Continue beyond the one-process test with an MPI integration run that exercises the host launcher, container MPI, and high-speed transport.

The scheduler should remain outside the image unless the site deliberately supports another model. It grants the allocation, constrains resources, and launches or authorizes job steps. The container should see only the devices, CPUs, memory, files, and credentials assigned to the job. This keeps accounting and isolation attached to the site control plane taken up in Chapter 7.

Containers improve repeatability above the kernel boundary, while performance still depends on the platform. Different processors may execute the same binary with different vector capabilities. Device code may require another target. MPI may choose another collective algorithm on a different topology. Filesystem mounts and data paths can change. Include the image in the platform test and retain the resulting hardware and transport measurements with it.

Debugging parallel programs

Debugging begins by reducing the number of things that can vary. Establish a correct serial or one-process result on a small input. Run the same problem with several threads on one node, then with several ranks on one node, then across two nodes. Increase the problem and node count only after each boundary works. This sequence cannot find every scale-dependent defect, but it gives a failure a much smaller neighborhood.

Compile a diagnostic build with warnings, debug symbols, assertions, and conservative optimization. Memory and undefined-behavior sanitizers can find invalid accesses and arithmetic assumptions in supported CPU code. Thread sanitizers can expose some shared-memory races. Their overhead and compatibility may limit large MPI or accelerator runs, so use them first on small cases and pair them with tools designed for the selected device runtime.

Print statements remain useful when they identify the process, thread, time, and program phase. They become misleading when buffered output from hundreds of ranks arrives in a different order from the events being investigated. Direct each rank to a controlled log only when the resulting file count is acceptable, or gather structured diagnostics through a smaller number of writers. Flush deliberately around a suspected failure and remember that printing changes timing.

GDB can run one process, attach to a selected rank, or inspect a core file. Parallel debuggers add coordinated process control, rank grouping, and comparison of variables across processes. Modern Linux systems can name and collect core files through the kernel's core-pattern mechanism and site tooling, superseding the special "named core" kernel patch used in some early clusters. Core collection still needs storage limits and privacy controls because a process image can be very large and can contain application data or credentials.

Distributed failures often appear as a hang rather than a crash. One rank may take a different branch and miss a collective, send a different element count, wait on a request whose buffer was reused, or fail before its peers reach the same operation. Obtain stack traces from all ranks, group identical states, and find the first point where their control flow diverges. A timeout identifies that progress stopped; it does not identify which rank first violated the protocol.

Numerical correctness needs its own tests. Compare physical invariants, residuals, conservation laws, convergence rates, or accepted error bounds rather than relying only on byte-for-byte output. Parallel reductions can change operation order, and floating-point addition is not associative. A difference within a justified tolerance may be correct. A tolerance chosen after seeing an inconvenient result is not a test.

Profiling and performance analysis

Optimization starts with elapsed time and completed work. Measure a representative workload, repeat it enough to understand variation, and retain the software and placement record. Judge a faster kernel or a higher accelerator-utilization percentage by its effect on the time and resources required to produce the complete scientific result.

Separate the job into computation, memory access, communication, synchronization, and I/O. These categories interact, but they suggest different evidence. CPU samples can locate hot functions. Hardware counters can show instructions, cycles, cache behavior, branches, vector activity, and memory traffic. MPI profiles can summarize message size, call count, collective time, and imbalance. Accelerator timelines can show kernels, transfers, and idle gaps. Filesystem and application counters can expose small requests or uneven writers.

Linux perf uses the kernel's perf_events interface for counting, sampling, and tracing. PAPI offers a portable programming interface over hardware and software counters. Counter names that look similar across processors can have different definitions, availability, and counting conditions, so record the event description and platform. Multiplexing more events than the hardware can count simultaneously also introduces estimation. Begin with a small set connected to a hypothesis.

MPI defines a profiling interface in which corresponding PMPI_ entry points allow tools to intercept MPI calls. It also defines a tools information interface for control and performance variables and events. These interfaces support portable instrumentation, but a profiler still has overhead. A full event trace from thousands of ranks can perturb the job and produce more data than can be interpreted. Start with summaries, narrow the interval or rank set, and collect a detailed trace only around the unexplained phase.

Time synchronization helps correlate node, network, storage, and scheduler records. Within a performance tool, use the clock and synchronization method it specifies. Across independent systems, retain clock offset and uncertainty. A timestamp ordering whose differences are smaller than that uncertainty cannot establish causality.

Strong and weak scaling

Strong scaling keeps the total problem fixed while increasing resources. If time on one node is T₁ and time on p nodes is T_p, speedup is T₁/T_p, and parallel efficiency is speedup/p. The serial portion, communication, synchronization, and load imbalance eventually prevent linear speedup. A baseline can use more than one node when the problem cannot fit on one; state that baseline rather than calling it a one-node result.

Weak scaling increases the total problem with the resources so that work per resource remains approximately fixed. Constant runtime would be ideal, but communication distance, global collectives, metadata, and imbalance often grow. Define what is held constant. Equal grid cells per rank and equal bytes per accelerator are different conditions if the algorithm or memory hierarchy changes at a node boundary.

Both tests should report more than a curve. Include problem dimensions, process and thread counts, placement, node type, compiler and libraries, communication transport, warm-up, number of repetitions, and whether I/O is included. Inspect rank distributions as well as averages. A few slow ranks can determine the completion time of a synchronized job while disappearing in the mean.

Performance work should end with an explanation that predicts another measurement. If the evidence says the solver is limited by memory bandwidth, increasing arithmetic throughput alone should have little effect. If time is dominated by a global reduction, changing local loop code should not restore scaling. The next experiment either supports the explanation or forces a better one.

Checkpoint and restart

A checkpoint records enough application state to continue useful work after interruption. The most portable method is application-level checkpointing: the program writes its logical state in a documented format and reconstructs processes, communication, and device state when it restarts. This requires development work, but it allows the format to omit temporary data, change the rank count where the algorithm permits, and survive software changes that a raw process image cannot.

Checkpoint design begins with consistency. If ranks write independently while some have advanced to the next iteration, the resulting collection may never have represented one valid application state. Coordinate the checkpoint epoch, record which pieces belong to it, write new data under temporary names, verify completion and checksums, and publish a small manifest last. Keep at least one previously verified checkpoint until the new one has been read successfully.

The storage load can be substantial. A job with 512 ranks writing 20 GiB per rank produces 10 TiB at each checkpoint. If all ranks begin together and must finish in five minutes, the data alone asks for roughly 34 GiB/s before metadata, contention, redundancy, or filesystem overhead. Reducing state, aggregating writes, staggering independent jobs, using incremental checkpoints, or placing a burst tier in the path can make the recovery plan compatible with the storage system described in Chapter 4.

System-level tools such as DMTCP can checkpoint unmodified user-space computations and support a range of threaded and distributed programs through coordination and plugins. This can be valuable for software that lacks application checkpoints. It is still necessary to validate the exact MPI, accelerator, RDMA, filesystem, scheduler, and restart environment. External services and device state may need explicit disconnect and reconnect handling.

A checkpoint that has never been restored is only a file. Test restart on a clean allocation, verify the continued result, and measure both checkpoint pause and recovery time. Retain the executable or image, input identity, checkpoint schema, rank layout, and software environment needed to read it. Scheduler preemption, planned maintenance, hardware failure, and long application phases may justify different checkpoint intervals; each should be based on failure cost and measured checkpoint behavior rather than a convenient round number.

Reconstructing a multi-node run

We can now follow the application through the complete path. The example is not tied to one package manager or scheduler. It is a sequence of evidence that can be implemented with the site's tools.

1. Identify the platform. Select the supported node type and record its Linux release, kernel, firmware, accelerator and network drivers, and relevant site services.
2. Resolve the toolchain. Choose the compiler, MPI, accelerator runtime, and numerical libraries as one compatible stack. Save the concrete module list or environment lock.
3. Build cleanly. Start from a named source revision, apply recorded patches, configure in a clean directory, and retain flags, dependency recipes, logs, and artifact checksums.
4. Prove the smallest case. Run a known input with one process. Check assertions, invariants, accepted error bounds, and output schema before parallelism hides a basic defect.
5. Cross one boundary at a time. Test threads on one node, ranks on one node, accelerators on one node, and then two nodes. Confirm device and network transports rather than inferring them from success.
6. Make placement explicit. Record ranks per node, threads per rank, CPU and memory binding, device visibility, and any communication-library topology settings.
7. Profile a representative phase. Begin with elapsed time and component summaries. Add CPU, MPI, device, or I/O detail in response to a specific unexplained cost.
8. Test at useful scale. Perform strong or weak scaling with stated problem sizes and baselines. Examine the slow ranks and variability, not only the best run.
9. Exercise checkpoint and failure paths. Restore from a checkpoint on a new allocation and verify the continued result. Test the behavior after a lost rank or device if the application claims to tolerate it.
10. Publish the run record. Store source and data identifiers, build provenance, executable or image digest, platform, allocation, placement, launch command, environment, results, and measurement method together.

This sequence turns "the application ran on the cluster" into a result that another person can examine and repeat. It also creates a practical support boundary. When a future update changes the result, the old and new records reveal which layer moved.

Chapter 7 takes up scheduling, allocation, placement, accounting, and job performance. Chapter 8 covers provisioning and fleet administration, including the work required to configure, observe, update, and repair the nodes that support this software path.

References and further reading

1. MPI Forum, MPI 4.1 Standard.
2. OpenMP Architecture Review Board, OpenMP Specifications.
3. Khronos Group, SYCL 2020 Specification.
4. NVIDIA, CUDA Programming Guide.
5. AMD, HIP Programming Model.
6. OpenACC Organization, OpenACC Specification.
7. NVIDIA, NCCL User Guide.
8. AMD, RCCL Documentation.
9. Linux Kernel Documentation, The Linux Kernel Driver Interface.
10. Linux RDMA Project, RDMA Core Userspace Libraries and Daemons.
11. Lmod Documentation, Software Module Hierarchy.
12. Spack Documentation, Environments.
13. EasyBuild Documentation, Terminology and Toolchains.
14. Netlib, LAPACK Users' Guide.
15. Netlib, ScaLAPACK.
16. FFTW, Distributed-memory FFTW with MPI.
17. PETSc Documentation, Overview.
18. The HDF Group, Introduction to Parallel HDF5.
19. Open Container Initiative, Runtime, Image, and Distribution Specifications.
20. Apptainer User Guide, Apptainer and MPI Applications.
21. Apptainer User Guide, GPU Support.
22. Linux Kernel Documentation, Discovering Kernel Subsystems Used by a Workload.
23. Innovative Computing Laboratory, PAPI Documentation.
24. MPI Forum, Tool Support and Profiling Interfaces.
25. DMTCP, Distributed MultiThreaded Checkpointing.