Scheduling, Resource Management, and Performance

By Chander Kant Updated July 31, 2026

Chapter 6 followed an application from source code to a reproducible executable. The next problem is to give that executable the right processors, memory, accelerators, network position, and time without allowing it to interfere with other work. A resource manager has to perform this job for many users at once, under policies that are understandable and measurable.

The queue is only the visible part of the system. Behind it, the resource manager validates a request, decides when the job is eligible, orders it against other work, finds a suitable set of nodes, launches processes, constrains them to the allocation, observes their use of resources, and records the result. A weakness at any one of these stages can appear to the user as a mysterious wait, an erratic run time, an unavailable accelerator, or a job that succeeds interactively and fails in batch.

This chapter follows one job through that path. Slurm is used for the detailed examples because it exposes the major scheduling and resource-control decisions found in present-day Linux clusters. The same questions apply to OpenPBS, Flux, LSF, HTCondor, and other workload managers even when their object names and commands differ.

Job lifecycle

A batch job begins as a description. The description names a program and its inputs, but it also states the resources and conditions under which the program may run. Those conditions may include a node count, task count, CPUs per task, memory, accelerator type and count, wall-clock limit, project account, partition, quality of service, reservation, node features, and dependencies on earlier jobs.

Submission does not make a job immediately runnable. The resource manager first checks whether the request is valid and whether the user is permitted to make it. A syntactically correct job can still be ineligible because it is held, depends on unfinished work, exceeds an account limit, requires a reservation that has not started, or asks for a resource combination that the cluster cannot supply. Eligibility should be established before priority is interpreted. A high-priority job that is not eligible cannot run.

Once eligible, the job competes with other work. The scheduler applies site policy, estimates future availability, and looks for an allocation that satisfies the request. If it finds one, it assigns resources to the job for a defined period. The node-side runtime then creates one or more job steps, launches tasks, binds them to hardware, and applies operating-system controls. At completion, the resource manager collects status and usage, runs any cleanup, releases the resources, and stores an accounting record.

Figure 7.1: The resource manager preserves the relationship between a request, the resources actually allocated, and the work that used them. A retry is a new scheduling attempt and is useful only when the application can resume safely.

Slurm reports this lifecycle through states such as PENDING, RUNNING, COMPLETED, FAILED, OUT_OF_MEMORY, NODE_FAIL, PREEMPTED, and TIMEOUT. A pending reason is especially valuable. Priority, Resources, Dependency, and an account limit describe different problems and require different responses. The reason is a diagnosis of the present state, however, rather than a promise that the job will start at a particular moment.

The objects being managed

Scheduling discussions become confusing when several words are used for the same object, or when one word is used for several objects. The following terms establish the meanings used in this chapter. They follow Slurm where a concrete name is needed, with the more general meaning beside it.

Scheduler objects describe work, resources, and policy
Object	Meaning	Distinction to preserve
Job	A request for resources for a bounded period, together with the work to run in that allocation	The executable is one part of the job, along with identity, policy, requested resources, and state.
Job step	A set of tasks launched within an existing job allocation	One allocation can run several steps, either in sequence or concurrently when resources remain.
Task	One scheduled program instance, commonly an MPI rank or another process launched as part of a step	A task may use several CPU threads. Task count and CPU count are therefore not interchangeable.
Node	A managed computer that contributes processors, memory, devices, and other resources	A node may be partly allocated or assigned as a whole, depending on site configuration and job request.
Partition or queue	A logical set of resources with access rules, limits, and scheduling policy	Partitions can overlap. A partition name does not necessarily identify separate physical hardware.
Account or project	The organization, grant, team, or other entity to which use is charged	A login identity answers who submitted the work; an account answers whose allocation or policy applies.
Quality of service	A named policy that can carry priority, limits, or preemption behavior	The name describes a policy class; application quality is unrelated.
Reservation	Resources set aside for specified users, accounts, or purposes during a time interval	A reservation protects future access. It does not by itself make every job in it high priority.
TRES	Dimensions that Slurm can track for allocation, limits, billing, and accounting, including CPUs, memory, nodes, energy, licenses, and configured generic resources	A trackable resource can be a count, a measured quantity, or another policy dimension; it need not be a node-local device.
GRES	Configured resources associated with nodes, commonly accelerators and other devices	A generic resource can also be tracked as a TRES. Cluster-wide licenses use Slurm's separate license resource mechanism.

Slurm control plane

Slurm divides responsibility among a small number of daemons and commands. The central slurmctld keeps the authoritative view of nodes and jobs, evaluates scheduling decisions, and grants allocations. A second controller can be configured as a backup. Each compute node runs slurmd, which reports node state and starts work that has been authorized by the controller. For each job step, a slurmstepd process supervises the local tasks, signals them, gathers status, and removes their execution context when they finish.

Accounting is commonly separated through slurmdbd. The database daemon accepts job records and policy data on behalf of the controller and user commands. This arrangement keeps database credentials away from ordinary clients and allows the controller to cache records through a short accounting-service interruption. A sufficiently long interruption can exhaust that cache, and the database also supplies associations, limits, and fair-share history used by policy. The accounting path therefore remains part of the operational control plane.

User commands present different views of this control plane. sbatch submits a script and normally returns before the job runs. salloc obtains an allocation for interactive or scripted use. srun can request an allocation in some contexts, but inside an existing job it normally launches a job step. squeue reports queued and running work, sinfo reports partitions and nodes, scontrol exposes detailed state and administrative controls, and sacct reads accounting records for running and completed jobs.

High availability has to be described by function rather than by counting servers. A backup controller can preserve scheduling control, but node daemons, authentication, name service, accounting, shared storage, and the controller state directory each have their own failure behavior. A controller interruption may allow already launched payloads to continue on compute nodes while new scheduling decisions stop. The site should test that behavior, including recovery of job state, instead of inferring it from a diagram with two controller names.

Control-plane authentication is also part of correctness. A node must be able to determine that a launch request was issued by the authorized controller for the identified user and allocation. Slurm commonly uses MUNGE for component authentication and signed job-step credentials for launch authorization. Consistent clocks, names, user identities, and keys are operational prerequisites. Their provisioning belongs to Chapter 8, but the scheduler depends on them for every job.

Resource requests and application requirements

A scheduler can make a good placement only from the information it has. A request should therefore describe the resources the application can use, not the largest allocation the user hopes will make it faster. Excess CPUs, memory, or accelerators can keep other jobs waiting while doing no work. An undersized request can lead to swapping, an out-of-memory termination, oversubscribed threads, or several ranks competing for one device.

The following Slurm script describes a four-node accelerator job. It is an example of the relationships among request fields, not a site-independent recipe. CPU numbering, memory policy, GPU names, partitions, and launch integration vary among clusters.

#!/bin/bash
#SBATCH --job-name=solver-scale
#SBATCH --account=project-a
#SBATCH --partition=gpu
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=8
#SBATCH --cpus-per-task=8
#SBATCH --gpus-per-node=8
#SBATCH --mem=480G
#SBATCH --time=02:00:00
#SBATCH --output=solver-%j.out

module purge
module load compiler
module load mpi
module load application

srun --cpu-bind=cores ./solver --input case-42.yaml

The request separates several quantities. Four nodes define the scale-out allocation. Eight tasks per node define the process layout. Eight CPUs per task provide room for each process's threads. Eight GPUs per node request devices, but the application or launch environment must still map ranks to those devices correctly. In Slurm, --mem=480G requests 480 GiB per node. The mutually exclusive --mem-per-cpu and --mem-per-gpu forms instead express memory per usable allocated CPU or per allocated GPU. The appropriate form follows the way the application consumes memory, and the site's consumable-resource configuration determines how the request participates in node selection. The two-hour limit tells the scheduler how long the resources may be occupied.

The word CPU deserves care. Depending on Slurm's resource-selection configuration, a schedulable CPU may represent a hardware thread, a core, or another configured unit. --cpus-per-task=8 means eight allocatable CPU units for each task; it does not guarantee that those units form one NUMA domain or that an OpenMP runtime will bind eight threads correctly. The site must publish its interpretation, and the run record should retain the resulting placement.

Wall time is both a protection and a planning input. The limit stops a runaway job, but the backfill scheduler also uses it to decide whether lower-priority work can finish before a higher-priority reservation. Habitually asking for the partition maximum deprives the scheduler of useful information. A practical estimate includes measured run time at the requested scale, expected input variation, checkpoint or shutdown time, and a modest safety margin. Production history should then improve the estimate.

Arrays, dependencies, and steps

A job array represents many similar batch jobs with a common initial request. It is appropriate for independent parameter values, input shards, frames, or samples. Slurm can represent a large array compactly and can cap the number of elements running at once. The array index should select an explicit input; it should not be used as an accidental source of uniqueness without a durable manifest that maps indexes to work.

Dependencies express a workflow without making a login session poll the queue. A preprocessing job can release a simulation only after successful completion, and a reduction can wait for an entire array. Failure semantics need to be selected deliberately. A dependency on successful completion should remain blocked when an upstream job fails, while a cleanup or diagnostic job may need to run after any terminal result.

A job step is useful when several commands share one allocation. Data can be staged, a parallel executable can run, and a post-processing step can follow without returning the nodes to the global queue between them. Concurrent steps must fit within the allocation. If each step assumes ownership of all CPUs or devices, putting them in one job does not prevent interference.

Scheduling policy

Queue order is not usually first-in, first-out. The scheduler first removes work that is not eligible, then orders eligible jobs using the policy chosen by the site. Slurm's multifactor priority can combine age, fair-share history, association, job size, partition, quality of service, trackable-resource factors, a site factor, and a user-controlled nice adjustment. The factors and their weights are local policy. A priority number copied from one cluster has no meaning on another.

Age allows an eligible job to gain priority while it waits. Fair share compares an account's assigned share with its recent resource consumption and raises work from accounts that have received less service. This priority adjustment differs from a hard monthly allotment. Limits provide that separate control: they can constrain submitted jobs, running jobs, wall time, or resource use at the user, account, quality-of-service, partition, or cluster level.

Quality of service is useful when the site can explain the service being offered. A short-job class may have a low wall-time maximum and quick access. A preemptible class may accept interruption in exchange for opportunities to use otherwise idle resources. An urgent class may be restricted to an operational process with explicit authorization. A name such as high without published limits and eligibility rules invites users to experiment against policy rather than work with it.

Several mechanisms contribute to the behavior users see as a queue
Mechanism	What it controls	What users need to know
Eligibility	Whether a job may be considered now	Holds, dependencies, invalid associations, reservations, and enforced limits can keep a job ineligible regardless of priority.
Priority	The order in which eligible jobs are considered	The enabled factors, their relative weights, and whether priority changes with age or usage.
Fair share	Queue preference based on assigned shares and prior consumption	The accounting hierarchy, usage window or decay, and resources included in billing.
Limits	Maximum submitted, running, time, or resource quantities	Which user, account, quality-of-service, and partition limits apply and how precedence is resolved.
Reservation	Future access to named resources during a time interval	Who may use it, whether jobs must request it, and the wall time required to finish inside it.
Backfill	Use of gaps that do not delay the scheduler's plan for higher-priority work	Accurate time limits can create earlier starts; the estimated start time can change as the queue and cluster change.
Preemption	Interruption of selected lower-priority work to release resources	Which jobs can be interrupted, the signal and grace period, and whether the result is cancel, requeue, suspend, or another configured action.

Backfill and reservations

Strict priority scheduling can leave processors idle while the first waiting job gathers enough nodes. Backfill uses those holes. It projects when running jobs will finish, protects the expected start of higher-priority work, and starts a lower-priority job when its requested time fits before that boundary. The lower-priority job uses only resources that remain available until the protected start time.

This calculation explains why wall-time requests affect other users. If a 20-minute job requests eight hours, it will not fit in a one-hour gap. If a running job requests two hours and actually needs six, the future plan can be invalidated by timeout or by a local extension. Scheduler estimates are never perfect, but measured and reasonably tight limits make the plan more useful.

An advanced reservation sets resources aside for a future interval. Reservations support planned experiments, teaching sessions, maintenance, and time-sensitive work. They also reduce what the ordinary queue can use, so their size and duration should be visible and justified. Before a maintenance reservation begins, the scheduler can stop launching jobs whose limits extend across its start time. This creates a deliberate drain without taking nodes offline prematurely.

Placement and fragmentation

An allocation with the correct resource counts can still have the wrong shape. A tightly coupled MPI job may perform well inside one group of leaf switches and poorly when spread across several. An accelerator job may need GPUs attached to the same high-bandwidth switch domain, CPUs and memory near each device, and network adapters with direct device paths. A storage-heavy job may need local scratch capacity or access to a particular data tier.

Slurm can represent node features, generic resources, and network topology. A tree topology can prefer the smallest group of switches that satisfies the request. Block-oriented placement can protect contiguous groups for larger jobs and reduce fragmentation. These policies trade immediate utilization against the ability to serve future jobs. Packing every small job into the first free cores may look efficient now while scattering the remaining GPUs or nodes into pieces that no pending distributed job can use.

Fragmentation is therefore different from simple idleness. A cluster can show idle CPUs while all suitable GPUs are occupied, or idle GPUs while no set of nodes has the required network relationship. Requests can also create fragmentation when they ask for an unusual ratio, such as one GPU and nearly all memory on each node. Operators need resource-shape reports, not only a total percentage busy.

The user should request constraints that are necessary for correctness or measured performance and omit preferences that merely describe a favorite node. A narrow feature expression reduces placement choices and can lengthen the wait. The administrator, in turn, should give stable meanings to features and remove stale labels. A request for a device generation or local capacity must select what its name promises.

Allocation, binding, and enforcement

Allocation answers which resources belong to the job. Binding answers where each task and thread should execute. Enforcement prevents the job from escaping those boundaries. These are related operations, but none of them automatically proves the others happened.

Consider a node with two processor sockets. The scheduler may allocate eight cores near one accelerator, yet an unbound process can migrate across every CPU the operating system permits it to see. That migration can move memory accesses across the NUMA interconnect and interfere with another allocation. CPU affinity maps each task to an intended subset, while a cpuset constraint limits the CPUs on which the job can execute. The application runtime must then create no more runnable threads than the allocation can support.

Linux cgroup v2 provides a unified hierarchy in which processes can be grouped and resource controllers applied. Slurm's cgroup plugins can constrain allocated cores, RAM, swap, and devices, while gathering usage from the same execution hierarchy. Device filtering in the cgroup v2 plugin uses eBPF programs attached to job cgroups, so the compute-node kernel must provide the required eBPF support. The cgroup is a boundary, not a placement algorithm. A task plugin or runtime affinity policy still determines which task uses which permitted CPU. Memory placement may require NUMA-aware first touch or an explicit policy in addition to a byte limit.

Figure 7.2: The controller decides what the job may use; node-side components turn that decision into process placement and Linux resource boundaries. Accounting then links observed use back to the allocation.

Accelerators, sharing, licenses, and energy

Accelerators are usually represented as generic resources and, when accounting is enabled for them, as trackable resources. The scheduler must know which device files belong to each resource and how devices relate to CPU cores and links. The launch environment can expose assigned device identifiers through variables such as CUDA_VISIBLE_DEVICES, while cgroup device control prevents access to unassigned devices. Environment variables alone are not an isolation boundary.

NVIDIA Multi-Instance GPU, or MIG, partitions supported devices into hardware-backed instances. Slurm can represent configured MIG instances as separately allocatable GPU resources and apply task binding and cgroup isolation. The scheduler does not dynamically invent the MIG layout for each job; the site establishes a supported partitioning and advertises it. Users should request the resource type that corresponds to that layout rather than depend on a physical device number.

Multi-Process Service, or MPS, addresses a different problem. It allows compatible CUDA processes to share execution capacity on a GPU. A scheduler can account for MPS shares, but a share is not equivalent to the memory and fault isolation of a separate physical GPU or MIG instance. The site must state whether sharing can cross jobs or users, how servers are started and stopped, and which limits are actually enforced.

Licenses and other cluster-wide resources can be scheduled even when they are not attached to one node. A job that needs a finite solver license should request it with the processors that will use it. Otherwise the scheduler may start the job and leave expensive nodes idle while the application waits for a license outside scheduler control. The same model can protect a temporarily unavailable service or filesystem by withholding its represented resource, provided the name and operational procedure remain accurate.

Energy can also be tracked, but its meaning depends on the measurement source and interval. Node power sampled from a management controller, processor energy counters, and accelerator telemetry cover different components and have different accuracy. An energy value should be stored with its source and scope. It can then support energy-to-solution comparisons without pretending that every watt consumed by a shared node belongs neatly to one job.

Resource leakage

Resource control fails when a process escapes the allocation. Common paths include direct SSH to a compute node, background processes left after a job, helper daemons launched outside the step, writable device files, and container runtimes that create processes in another cgroup. Compute-node login policy should either deny an unallocated session or adopt it into the caller's active job. Epilog cleanup is useful, but cleanup after leakage is weaker than placing every process correctly at creation.

The same rule applies to monitoring and support agents. Their CPU, memory, network, and device use must be reserved as system overhead or explicitly included in the node's allocatable capacity. A node advertised as having every byte and CPU available to jobs leaves nothing for the kernel, storage clients, network interrupts, or resource-manager daemons. The configured capacity should describe what workloads can safely consume.

Parallel jobs and failure

A tightly coupled job cannot make progress with an arbitrary subset of its ranks. The scheduler must secure the required resources before the parallel step starts. In general scheduling language this all-or-nothing admission is often called gang scheduling. Slurm also uses the term gang scheduling for time-slicing suspended jobs, so the intended meaning should be stated whenever the phrase is used.

MPI processes should be launched through the resource manager's supported integration rather than through an unmanaged chain of remote shells. The launcher then receives the allocation, task count, placement, credentials, environment, and signal path from one authority. It can create accounting records for the step and terminate the remaining ranks when the job cannot continue. Chapter 6 describes the MPI programming and runtime side of this launch.

Distributed AI training has the same allocation problem at a larger memory and device cost. All workers normally need to join before synchronized training can advance. A partially started job can occupy several accelerators while waiting indefinitely for the remainder. Admission should therefore consider the complete worker group, network and topology constraints, checkpoint interval, expected duration, and the cost of preemption. Appendix 2 follows the training workload itself.

Preemption, checkpoints, and retries

Preemption makes room for selected higher-priority work by interrupting lower-priority work. Slurm can cancel, requeue, or suspend jobs according to site policy. These actions have different costs. Cancellation discards uncheckpointed progress. Requeue releases the allocation and submits the job to scheduling again. Suspension may retain memory on the nodes, which means the apparently stopped job continues to consume a scarce resource.

A grace period and warning signal give the application an opportunity to checkpoint, but they do not create a checkpoint. The application must write a consistent restart state within the available time, place it on storage that survives allocation loss, and record enough information to resume. A checkpoint that takes twenty minutes cannot satisfy a five-minute warning. The scheduler policy and application design must agree on the interval and deadline.

Requeued work also needs idempotent output handling. The new attempt may repeat computation that completed just before the interruption. Output files should be written to attempt-specific paths or committed atomically, and the restart process should distinguish a complete checkpoint from a partially written one. A durable scientific run identity should connect every scheduling attempt that contributes to the same result.

Node failure requires similar discipline. A parallel step may terminate when one node disappears even if every other node remains healthy. The accounting record should preserve the failed node and state, while the application log preserves its own last consistent point. Automatic requeue is appropriate only for failures the application can recover from. Repeating a deterministic input error consumes the cluster again without improving the outcome.

Interactive allocations

Interactive access is useful for debugging, inspecting data, testing a short build, and measuring placement before a long production run. It should use the same resource request and enforcement path as batch work. With Slurm, salloc obtains the allocation and srun launches a managed job step. The allocation always carries an account and time limit, while CPU, memory, and device enforcement applies to processes launched into its steps. A site can configure LaunchParameters=use_interactive_step so that an salloc shell itself runs as a managed step on an allocated compute node. Without that configuration, the shell can remain on the submission host, and calculations should be launched from it with srun.

The login node is not a substitute for an interactive allocation. Compilation and lightweight editing may be permitted there, but an unbounded calculation competes with submission commands, file transfers, and other users' sessions. Sites should publish what is acceptable on login nodes and provide a short or interactive partition that makes the correct path convenient.

Some workloads are services rather than jobs that naturally terminate. A notebook server used for several hours can still run in a bounded allocation. A long-lived inference endpoint, workflow controller, or data service may fit better under a service-oriented control plane. The choice depends on lifecycle and policy, not on whether the executable is packaged in a container.

Slurm and Kubernetes

Slurm begins with a queue of requests for finite resources over time. Kubernetes begins with an API describing desired workload objects and continually reconciles their state. Both place work on Linux nodes, and both can manage CPUs, memory, and accelerators, but their primary abstractions lead to different operating behavior.

Choose a control plane from the workload lifecycle and policy requirements
Dimension	Slurm	Kubernetes
Primary unit	A job allocation containing one or more steps and tasks	A Pod scheduled to a node, usually managed by a higher-level object such as a Job, Deployment, or StatefulSet
First operating objective	Order queued work and grant bounded access to scarce shared resources	Reconcile declared workload state and keep the requested Pods and services operating
Batch admission	Partitions, associations, limits, priority, fair share, reservations, backfill, and optional preemption are integrated with job accounting	A Kubernetes Job manages completion and retries; workload queuing and quota admission can be added with Kueue or another batch layer
Parallel coordination	A multi-node allocation is granted before its step launches; MPI launch integration is established practice	Core gang scheduling based on PodGroup is alpha and disabled by default in Kubernetes 1.35 and 1.36; Kueue and workload operators provide deployed alternatives
Long-running services	Possible in a time-bounded job, but service discovery and continuous reconciliation are not its primary model	Deployments, StatefulSets, Services, probes, and rolling updates directly model long-running applications
Accelerators	GRES and TRES connect selection, binding, limits, accounting, and fair-share policy	Device plugins advertise integer extended resources that cannot be overcommitted; Dynamic Resource Allocation provides a newer, richer device path
Placement	Node features, consumable resources, and configured network topologies guide whole-job placement	The scheduler filters and scores nodes for Pods using requests, affinity, topology, and plugins; coordinated workload placement needs the applicable group mechanism
Identity	A job normally runs as the submitter's site-wide Unix UID and GID; scheduler associations apply project policy	A service account identifies the workload to the Kubernetes API, while a security context sets the Linux user and groups of container processes
Storage	Shared POSIX filesystems commonly present the same paths and permissions on every allocated node	Volumes and persistent volume claims describe mounted storage; CSI drivers can provision and attach it, while process UID and GID still govern POSIX access
Failure model	Records job and step states; retry or requeue must match application restart behavior	Controllers create replacement Pods and Jobs apply retry, failure, success, and deadline policies
Natural fit	MPI, scientific batch, throughput arrays, scheduled accelerator training, and other bounded shared-compute work	APIs, inference services, workflow controllers, cloud-native data systems, and applications needing continuous reconciliation

Kubernetes does include a batch Job object. A Job creates one or more Pods and tracks successful completions, with controls for parallelism, retry limits, failure policy, and an active deadline. The ordinary Kubernetes scheduler filters nodes that cannot satisfy one Pod, scores the feasible nodes, and binds the Pod to the selected node. That sequence is not by itself the same as admitting a complete tightly coupled workload under a shared queue policy.

This area is changing. Kubernetes 1.35 introduced core PodGroup gang scheduling as an alpha feature, disabled by default, and 1.36 documents alpha workload-aware preemption for those groups. Production designs should distinguish an available upstream experiment from a feature the cluster has enabled, tested, and committed to support. Kueue takes a different approach by making the complete-to-finish workload a unit of admission, calculating its requested quota from one or more Pod sets, and synchronizing that decision with Jobs and supported workload operators.

A mixed organization can use both control planes without making them compete for the same uncoordinated resources. Separate node pools or Slurm partitions can give each system clear ownership. Another design starts a Kubernetes environment inside a Slurm allocation for the life of one experiment, leaving global admission with Slurm. Whichever model is chosen, one authority must own each CPU, byte of memory, and device at a given time.

Identity and storage need equally clear ownership. A typical Slurm job runs under the submitting user's Unix UID and GID and expects the same numeric identities and POSIX paths on every allocated node. Kubernetes uses service accounts for API identity and a Pod security context for the user and groups of container processes. Persistent volumes and CSI drivers govern how storage is presented, but the process's numeric identity still determines access to a mounted POSIX filesystem. A hybrid design must therefore map identities and paths deliberately, apply one network policy at every boundary, and avoid charging or counting the same resource use in both systems.

Accounting and performance

Good accounting keeps three ledgers separate: requested resources, allocated resources, and consumed resources. The request shows what the user told the scheduler. The allocation shows what the resource manager granted. Consumption estimates what the processes actually used. Comparing only request with use can miss a scheduler rounding rule; comparing only allocation with use can hide a systematically inflated request.

Slurm can store records for a job and its steps. Depending on the enabled plugins and hardware sources, those records can include eligible, start, and end times; state and exit code; allocated nodes and CPUs; requested and allocated TRES; CPU time; memory high-water marks; I/O; accelerator memory and utilization; and consumed energy. sstat examines running steps, while sacct reads running and completed records. A zero or missing field may mean that the source was not collected, not that the application used none of the resource.

Sampling also limits interpretation. A short memory spike between accounting samples can be missed, and a job-level maximum may represent one task rather than simultaneous use by all tasks. Accelerator utilization reports how busy a device appeared during samples, not whether its operations were useful or numerically efficient. Accounting data is strong enough to identify patterns and candidates for investigation; application profiling remains necessary to explain them.

Service and application measures

Cluster utilization describes occupancy rather than useful output. A system can remain nearly full with oversized, stalled, or repeatedly failing jobs. Performance management should connect resource occupancy to completed scientific, engineering, or business work. Several measures expose different parts of that relationship:

Queue wait is start time minus eligible time. Using submission time without separating dependency or hold time can make the scheduler appear responsible for a wait it could not resolve.
Run time is end time minus start time. For a preempted or requeued workload, report attempt run time as well as total time across attempts.
Turnaround is completion time minus submission or eligibility, with the chosen definition stated. It reflects the user's end-to-end delay.
Expansion factor is turnaround divided by run time for a completed job. It helps compare waiting impact across short and long jobs, but becomes unstable for extremely short runs.
Throughput counts completed work in a period. Jobs per hour is useful only when the jobs are comparable; samples, simulations, images, or application-defined units can be better denominators.
Allocation occupancy compares observed active resource time with allocated resource time. It should be calculated separately for CPUs, memory, and accelerators because one job can use them in different proportions.
Failure rate separates application exits, timeouts, out-of-memory events, node failures, launch failures, cancellations, and preemptions. Combining them conceals the responsible layer.
Variability compares the distribution of run time or throughput for a controlled workload. Median, 95th percentile, node type, placement, and software revision are more informative than one average.

These measures should be segmented before conclusions are drawn. Interactive jobs, short throughput tasks, large MPI jobs, training runs, and services have different expectations. A six-hour queue wait may be acceptable for one scheduled campaign and disastrous for a ten-minute debugging job. Likewise, a low CPU occupancy can be correct for a memory-bandwidth-bound program or a job whose main work is on accelerators.

Accounting can improve requests. Historical run time by application, input class, scale, and software revision can suggest a wall limit. Peak memory can identify a safer request, with room for input variation and measurement interval. Repeated device underuse can prompt a smaller accelerator count or a profiling study. These recommendations should remain visible to the user and preserve an override; an opaque system that silently rewrites requests is difficult to debug.

Retention and access policy matter because job records reveal usernames, project names, paths, command lines, and resource behavior. Store only fields with an operational or reporting purpose, protect scripts and environment captures that may contain sensitive values, and define how long detailed and aggregated records remain. The accounting system is part of the cluster's operational history, not a convenient place to collect every environment variable indefinitely.

Scheduler acceptance tests

A parsed configuration establishes syntax. A small acceptance suite establishes behavior using jobs with known CPU, memory, accelerator, network, duration, and failure characteristics. Run the suite after scheduler, kernel, cgroup, driver, topology, or node-image changes.

Validate admission. Submit allowed and disallowed accounts, partitions, limits, dependencies, and reservations. Confirm that each rejection or pending reason explains the actual rule.
Confirm selection. Request each published node feature and accelerator type. Verify the allocated hardware rather than trusting the feature label.
Inspect task placement. Run several rank and thread layouts. Record task-to-node and task-to-CPU binding, NUMA locality, and the relationship between each rank and device.
Exercise enforcement. Attempt to exceed CPU, memory, and device boundaries. Confirm the intended throttling or termination and ensure another job cannot open the assigned device.
Launch across nodes. Start a representative MPI or other tightly coupled step through the supported launcher, then test signal propagation and cleanup after one task fails.
Test arrays and dependencies. Limit array concurrency, fail selected elements, and verify success and failure dependency semantics.
Demonstrate backfill. Create a protected future start and show that a short lower-priority job fits while a longer one waits. Repeat with truthful and inflated time limits.
Audit priority and fair share. Use controlled accounts and usage history to verify the configured factors and published policy, including decay or reset behavior.
Exercise reservations and maintenance. Confirm access boundaries, start and end transitions, and the draining behavior for jobs that would cross the reservation.
Test preemption and restart. Deliver the documented warning, measure checkpoint completion, requeue the job, and verify that output remains correct across attempts.
Reconcile accounting. Compare requested, allocated, bound, and observed resources for successful and failed jobs. Check which fields remain unavailable and document that limitation.
Fail control components. Exercise the approved controller and accounting failover procedures in a test window, observing running jobs, new submissions, cached records, and state recovery.
Measure scale. Submit a controlled burst of jobs and arrays, observe scheduling-cycle latency and command responsiveness, and retain a baseline for later upgrades.

The suite should include workloads that resemble the cluster's real demand. A one-node sleep process proves little about a topology-aware MPI allocation or an eight-node training job. At the same time, each test should isolate one policy decision where possible. Small deterministic tests locate a defect; full applications confirm that the repaired system serves its purpose.

The resource manager now has a tested path from request to accounting record. Chapter 8 follows the nodes underneath that path: how they are provisioned, secured, observed, changed, drained, repaired, and returned to service without breaking the assumptions the scheduler makes about them.