Cluster Administration, Automation, and Security

Chapter 7 followed a job through the resource manager. That path assumes that every allocated node has the advertised processors, memory, devices, software, identity, and network position. Cluster administration is the work required to make that assumption true, keep it true while the system changes, and withdraw a node quickly when it is no longer true.

A node is more than a running Linux installation. It is a physical asset, a baseboard management controller, a collection of firmware, a boot policy, a node image, local configuration, network identity, a scheduler record, a monitoring target, and a history of changes. All of these descriptions must point to the same machine. A correct operating system on the wrong switch port, or a healthy accelerator advertised under the wrong scheduler name, can be as damaging as a failed component.

This chapter follows one node from discovery through provisioning and service, then through failure, repair, validation, and return to the scheduler. The examples use common Linux and open infrastructure tools, but the operating method is more important than a particular product: record desired state, make changes through a controlled path, observe the result independently, and require evidence before restoring capacity.

Node lifecycle and source of truth

A node should enter the cluster through a defined state machine. The first state is not idle; it is unverified. Hardware arriving from a vendor may contain the expected model of processor and accelerator, yet differ in firmware, memory population, cabling, or device revision. The node becomes schedulable only after inventory, provisioning, configuration, and acceptance tests agree with the record the scheduler will publish.

The authoritative inventory should assign a durable identity before an operating system is installed. At minimum, that record connects the asset tag and chassis serial number with rack and unit position, BMC address and certificate, management and workload interface addresses, switch ports, node image or profile, scheduler node name, hardware class, ownership, warranty data, and current lifecycle state. MAC addresses and firmware-reported UUIDs are useful evidence, but either can change after a board replacement. The inventory needs an explicit procedure for transferring identity to replacement hardware.

Several systems will keep partial copies of this information: DHCP, DNS, the provisioner, configuration management, monitoring, the scheduler, a BMC manager, and perhaps a data-center infrastructure manager. One of them must be declared authoritative for each field. Synchronization then has a direction. Without that ownership, an administrator can fix a hostname in DNS while the provisioner silently restores the old value at the next boot.

Lifecycle state should describe operational permission, not just power. Useful states include discovered, provisioning, validating, available, drained, quarantined, under repair, and retired. A powered node can remain quarantined for forensic work. A node whose slurmd process answers can remain drained until its memory, accelerator, fabric, storage, and workload tests pass. Keeping these meanings separate prevents an automated recovery script from turning a communications success into an unsupported claim of health.

Management plane and console access

A management plane provides control when the workload plane is unavailable or untrustworthy. It includes the out-of-band network for BMCs, console and power access, provisioning services, administrative access hosts, and the APIs used to inventory and change hardware. It may also include separate service networks for the scheduler, identity, monitoring, DNS, DHCP, and image distribution. Calling all of these functions the management node hides their different failure and security boundaries.

The services can share hardware at small scale, but their state and recovery paths should remain distinct. A failed provisioning host should not erase the only copy of DHCP configuration. Loss of the monitoring interface should not prevent console access. A scheduler-controller failover should not depend on a single local filesystem mounted from the failed primary. The design begins by listing each administrative function, the state it owns, its upstream dependencies, and the way it is restored.

BMCs, Redfish, and remote control

A baseboard management controller is a computer inside the server that remains active independently of the host operating system. Depending on the platform, it can report inventory and sensors, power the host on or off, select a boot target, expose virtual media, update firmware, provide a graphical remote console, and carry the host's serial console over the network. Vendor interfaces differ, while the DMTF Redfish standard provides a resource-oriented HTTPS API for many of these operations. OpenBMC is an open Linux distribution for management controllers and implements substantial portions of Redfish on supported platforms.

This independence makes the BMC valuable and dangerous. It can remain reachable when the host kernel is down, but the same privilege can alter firmware, inspect a console, mount media, and reset the machine beneath operating-system security controls. The BMC network belongs on an isolated, non-routed or tightly routed management segment. Access should pass through a hardened administrative gateway with multifactor authentication, named accounts or centrally mapped roles, short sessions, command and event logging, and default-deny policy. BMC interfaces should never be exposed directly to the public Internet.

Factory credentials, obsolete TLS, unused IPMI or web services, permissive service discovery, and old BMC firmware are recurring risks. Commissioning should replace defaults, install a trusted certificate where the implementation permits it, restrict roles, disable protocols that are not required, register the firmware baseline, and test account recovery. Disable IPMI over LAN when Redfish and the selected console paths provide all required functions. On platforms where Serial over LAN is an IPMI payload, verify a Redfish or vendor console replacement before disabling that transport. If a legacy interface remains necessary, confine it to the management network and the narrowest available role. Firmware updates must be staged like host changes because a failed BMC update can remove the very recovery channel needed to repair the server.

Dedicated KVM switches and serial concentrators still have a place, especially around older or specialized equipment. On contemporary server nodes, remote KVM and Serial over LAN usually travel through the BMC network and remove much of that rack cabling. They do not remove the need for an independent path. A failed management switch, a lost BMC credential, or a firmware defect can still require a crash cart and a person at the rack.

The console chain

A headless server does not have one console configuration. It has a chain of producers and consumers, and each stage can succeed while the next stage remains silent. Troubleshooting is much faster when the boundary is explicit:

1. Platform firmware. UEFI or system firmware selects console redirection and presents setup, hardware initialization, and boot-manager output. The BMC must connect its Serial over LAN service to the same UART or expose a remote graphical console.
2. Boot loader. GRUB needs its own serial device, speed, parity, word length, and terminal selection. Firmware output followed by silence before the kernel starts points here.
3. Linux kernel. One or more console= parameters select kernel message destinations. For example, console=tty0 console=ttyS0,115200n8 sends output to both display and serial device; ordering also influences the preferred userspace console.
4. Userspace login. On a systemd system, a unit such as serial-getty@ttyS0.service supplies a login prompt. Kernel messages without a prompt indicate a userspace or policy problem rather than a boot-loader failure.

The device name and speed are platform facts, not universal constants. Some systems expose a different UART, and a BMC may fix or negotiate its Serial over LAN rate. The site should record the chosen console for every hardware profile and validate it after firmware changes. Multiple kernel consoles also deserve a test: the Linux kernel can write to several, while /dev/console and systemd may select one according to ordering and registration behavior.

A console transports keystrokes and output; it does not define authorization. The boot loader can require protection against unauthorized editing, the operating system controls login, and the management gateway controls who may open the console stream. Enabling direct root login on a serial line simply because it is called a console weakens the boundary. Named administrative access and auditable privilege escalation are preferable, with a separately protected recovery procedure for failures of central identity.

Node identity and provisioning

Provisioning turns inventory into a booted node. A common path begins when firmware performs PXE or UEFI network boot and obtains addressing and a boot target through DHCP. It then loads iPXE or another network loader and retrieves a kernel, initial RAM filesystem, and boot arguments. The early userspace discovers storage and network devices, installs or mounts the operating-system image, applies node-specific data, and starts the real system. HTTP or HTTPS is often used for larger artifacts even when TFTP remains in the first handoff.

Each handoff must preserve node identity. A request may initially be recognized by BMC inventory, switch port, client architecture, MAC address, or a firmware UUID. The provisioner maps that evidence to the intended node record and returns only the profile authorized for it. An unknown machine should enter a discovery or quarantine workflow rather than receive a production image and cluster credentials.

Stateful, stateless, and image-based nodes

A stateful node installs its root filesystem on local storage and retains it across boots. A stateless node retrieves most or all of its operating environment at boot and treats local media as cache, scratch space, or no requirement at all. Many clusters use a middle position: a versioned image is installed locally, while persistent identity and selected configuration are applied separately. The choice affects recovery time, network demand, update method, and the meaning of drift.

Warewulf is one current example designed for Linux cluster provisioning. It associates nodes and profiles with a node image and uses overlays to render per-node files around that common image. That separation is useful beyond Warewulf: the base image contains shared software, while the node record supplies hostname, interfaces, mounts, certificates, and other identity. A change to one node's address should not require rebuilding an image used by hundreds of peers.

Automated installation remains useful for stateful nodes. Kickstart can describe an unattended Red Hat family installation, including source, storage, packages, network, and completion behavior. Newer image-based operating modes can install a bootable OCI-derived system and support transactional updates and rollback. These approaches are not interchangeable merely because both produce a Linux root filesystem. The site should choose one ownership model for packages and configuration, then make rebuild and rollback follow that model.

Golden images require provenance. Store the build definition, base repository or image digest, package lock or manifest, kernel and driver versions, build logs, checksums, signatures where available, and the acceptance result. A mutable tag such as latest cannot identify what a node booted last month. Promotion should move an immutable digest through development, canary, and production policy without rebuilding different bits for each environment.

The artifact path also has to survive a coordinated boot or job launch. Hundreds of nodes fetching the same root image, container, or SquashFS file can overload a registry, metadata service, or shared filesystem even when steady-state traffic is small. Regional mirrors, node-local caches, and pre-staging can spread the load, but cache capacity, eviction, integrity checks, and the authoritative digest still need ownership. Measure the largest intended cohort instead of assuming that a successful one-node fetch will scale.

Secrets and durable host identity should normally be injected at enrollment rather than baked into the common image. A shared SSH host private key, BMC password, monitoring credential, or Slurm authentication key turns one copied artifact into a cluster-wide credential leak. The provisioned node can authenticate a one-time enrollment with inventory evidence or a protected bootstrap token, obtain its own host credentials, and discard the bootstrap material. Rotation then has a defined issuer and scope.

Desired state, drift, and staged change

Configuration management expresses what should be true after it runs. Packages have selected versions, services have declared states, files have controlled content and permissions, kernel parameters have known values, and mounts refer to intended storage. An idempotent operation reaches the same result when repeated. This property turns recovery from a sequence of remembered commands into convergence from a recorded definition.

Ansible, Salt, Puppet, and other systems can implement this model. Tool choice matters less than ownership and testability. Every managed field should have one controlling source, reviewable changes, a way to limit scope, and output that records what changed. Dry-run and diff modes help inspect a proposed change, but they cannot prove that a restart will succeed or that a driver will bind to hardware. Validation remains a separate stage.

Image construction and runtime configuration need a deliberate boundary. Files that affect early boot, kernel modules, accelerator drivers, and fundamental libraries often belong in a tested image. Node-specific network data, scheduler identity, mounts, certificates, and rapidly rotated secrets often belong in an overlay or enrollment step. Duplicating ownership across image build, cloud-init, configuration management, and administrator shell commands produces changes that overwrite one another.

Canaries, batches, and rollback

A cluster-wide command can turn a small mistake into simultaneous loss of service. Changes should move through representative canaries and bounded batches. A good canary includes the hardware and workload characteristics that make the change risky: accelerator model, network adapter, NUMA layout, local storage, rack generation, and node image. The first available node is not necessarily representative.

The operational sequence is straightforward. Drain the canary from new work, wait for or deliberately handle its running jobs, record the pre-change state, apply the change, reboot when required, run node and workload acceptance tests, observe it for a defined interval, and only then continue to a bounded batch. Stop conditions belong in the plan before rollout: failed boot, console loss, new hardware errors, fabric degradation, benchmark regression, or an unexpected configuration diff.

Rollback must describe compatible state, not just an older package command. A kernel rollback may require the matching accelerator and network modules. Firmware rollback may be blocked by the platform or may require restoring configuration defaults. A node image rollback can encounter data or /etc changes made by the newer deployment. Test the reverse path on the canary, keep required artifacts available, and record the last known-good combination as a set.

Firmware and driver sets

A compute node contains firmware in the BMC, platform, storage controllers, drives, network adapters, DPUs, accelerators, and sometimes retimers or switches. Linux adds a kernel, device firmware blobs, out-of-tree modules, user-space drivers, communication libraries, and management tools. Compatibility claims commonly apply to a matrix of these versions. Updating one row without the others can preserve boot while degrading RDMA, peer-to-peer transfers, error reporting, or reset behavior.

Maintain a qualified bill of versions for each hardware profile. Capture Redfish inventory before and after firmware work, verify signatures and source, apply updates in an order supported by the vendor, and preserve event logs. NIST describes platform firmware resilience in terms of protecting against unauthorized change, detecting change, and recovering to a state of integrity. Those goals make firmware recovery and known-good images procurement requirements, not tasks invented after a failed update.

Drift detection compares observed state with the approved record. It should find package and file changes, unexpected services, kernel arguments, firmware versions, interface names, mount options, scheduler configuration, and credential age. Some differences are legitimate emergency repairs. The repair still needs to be reconciled into desired state or explicitly reverted; otherwise the next automation run can undo the incident response without understanding why.

Identity, access, and secrets

Copying /etc/passwd, /etc/shadow, and SSH keys to every node does not scale safely. A centralized identity service provides users, groups, authentication, and authorization policy, while a Linux client such as SSSD resolves and caches that information. FreeIPA, Active Directory, LDAP and Kerberos deployments differ in protocol and administration, but the cluster still needs consistent numeric UID and GID values wherever a shared POSIX filesystem interprets ownership.

Authentication answers who presented a valid credential. Authorization answers whether that identity may enter a login node, administer a service, submit under an account, use a reservation, or become root. These decisions should be represented in groups, host access rules, scheduler associations, and privilege policy rather than scattered copies of an administrator's public key. Removing a person from the organization should close every ordinary path without waiting for a configuration sweep over hundreds of nodes.

OpenSSH public keys remain appropriate when issuance, expiry, and revocation are managed. OpenSSH certificates can replace distribution of many individual keys with trust in a user or host certificate authority. A certificate can carry principals, validity intervals, and restrictions, while host certificates reduce the temptation to accept an unverified host key after reimaging. The certificate authority itself is a high-value secret and should be protected separately from routine automation.

Login nodes are the normal administrative and user boundary. Direct SSH to compute nodes should be denied, restricted to operators, or adopted into an active scheduler job as discussed in Chapter 7. A scheduler allocation does not grant general BMC or root access. Conversely, an infrastructure administrator does not need to read every user's data merely because the role can repair hardware. Separate roles and audited elevation reduce both mistakes and the impact of a stolen credential.

Services also have identities. The scheduler, provisioner, monitoring collectors, image registry, BMC automation, and storage clients need keys, certificates, tokens, or passwords with explicit audiences and permissions. Store secret material outside ordinary source repositories, deliver it only to the nodes and services that need it, rotate it without a cluster-wide outage, and ensure that logs and configuration diffs redact it. A Slurm key copied to every component has a different rotation and exposure model from a short-lived monitoring token.

Central identity can fail, so recovery access must be designed rather than improvised. A break-glass account should have a narrowly defined purpose, protected credential, alert on use, periodic test, and immediate rotation after use. Cached logins may support continuity, but they also extend access after central revocation. The permitted cache interval should follow the site's threat model, and critical administrative actions may require a live authorization check.

Security zones and software provenance

A cluster has several trust zones because its networks carry different authority. The public or institutional network reaches login, web, API, and transfer services. Compute nodes communicate with the scheduler, storage, peers, and selected repositories or proxies. Storage servers accept data-plane and administrative traffic. BMCs, power devices, and management switches form an out-of-band zone with control below the operating system. Treating all of them as one trusted private network gives a compromised compute job a short route to the controls that can rewrite or power down the cluster.

Segmentation can use physically separate switches, virtual routing and forwarding instances, VLANs with enforced policy, host firewalls, or combinations of them. The essential test is whether an unauthorized path is actually blocked. A compute node should not route to BMC addresses merely because both ranges use private space. An administrator's laptop should reach the management gateway, not every controller directly. Egress from BMCs should be limited to required identity, time, DNS, event, and update destinations.

Service accounts and mutual authentication help within a zone, but they do not replace network policy. A compromised provisioning service can distribute code to every node it controls. A compromised telemetry collector should not gain the same authority simply because it shares a subnet. Place services according to the privilege they exercise, restrict their outbound paths, and use separate credentials for reading inventory, changing firmware, opening consoles, and controlling power where the platform supports those roles.

Images, packages, and boot integrity

The node image is an administrative command executed at cluster scale. Its inputs should come from approved repositories over authenticated channels, with package and image signatures verified according to site policy. Build provenance connects the output digest to the builder, recipe, and input materials. An inventory or software bill of materials then helps determine which node profiles contain a vulnerable component without scanning names from memory.

UEFI Secure Boot can verify signatures along the firmware, shim, boot-loader, and kernel path on supported systems. A trusted platform module can record measurements for later appraisal. These controls provide evidence only when keys, policy, and recovery are managed. They do not make an old signed kernel safe, and disabling verification to load an untracked driver defeats the assurance expected from the chain.

Vulnerability response starts with a reliable asset and software inventory. Determine affected hardware profiles and images, decide whether the exposure is reachable in each zone, prepare a corrected artifact, and use the ordinary canary and drain path. Emergency priority can shorten observation intervals, but it should not remove rollback, console access, or post-change validation. If compromise is suspected, preserve relevant logs and isolate the node before rebuilding it from known-good inputs.

Observability and operational evidence

No single signal can establish node health. The BMC sees power, temperature, fans, voltage, memory events, and firmware logs outside the host. Linux sees machine checks, driver resets, filesystems, processes, and network errors. Fabric managers see links and topology. Storage clients see latency and timeouts. The scheduler sees registration, job failures, and resource state. A useful monitoring system preserves these viewpoints and joins them by node identity and time.

Prometheus Node Exporter is one way to expose Linux hardware and kernel metrics, while IPMI, SNMP, Redfish, and vendor exporters can collect from devices that cannot run an agent. Performance Co-Pilot remains another Linux telemetry framework. OpenTelemetry collectors can receive, process, and export metrics, logs, and traces where that model fits. A site may choose different products, but metric names, labels, sampling intervals, retention, and expected cardinality need design. Per-process or per-job labels at cluster scale can overwhelm a system intended for node health.

Accelerators add another evidence path. NVIDIA DCGM provides GPU telemetry, passive health watches, topology information, and active diagnostics; AMD SMI exposes monitoring and reliability information for supported AMD GPUs. Their output should be joined to the same node, job, image, driver, and time records as kernel and scheduler events. Passive health data can identify a reset, memory error, thermal excursion, or degraded link, while an active diagnostic asks whether an idle device can complete a controlled test. Neither result, by itself, proves that the entire node is ready for a representative workload.

Facility telemetry also belongs on the incident timeline. In a liquid-cooled rack, loss of CDU flow, an abnormal supply or return temperature, or a leak alarm can affect several healthy hosts at once. The alert path may need to drain a node group before hardware protection shuts it down. Chapter 5 explains how facility and cluster measurements meet; the administrative system supplies the node identities, scheduler actions, and recovery record.

Logs need an off-node path because a failed or reimaged node may lose its local journal. Forward kernel, service, authentication, scheduler, provisioning, network, storage, and BMC events with the original host identity and timestamp. Preserve raw detail long enough for incident analysis, then retain aggregates according to operational and privacy needs. Command lines, usernames, paths, and environment data can be sensitive; collection should follow a stated purpose rather than the availability of a field.

Crash evidence requires preparation. Kernel crash dumps reserve memory and need a tested destination. Persistent storage such as pstore or a BMC event log may retain clues when the ordinary root filesystem does not. Netconsole can send kernel messages over the network when disk logging fails, though it has different reliability and security properties from a serial console. Test each path by causing a controlled event on a non-production node; a configured crash collector that has never received a dump remains an assumption.

Time, alerts, and capacity

Time synchronization is part of authentication and diagnosis. Kerberos, certificates, distributed logs, scheduler records, and performance traces all depend on credible clocks. Provide multiple suitable time sources, monitor offset and source selection, and define the behavior of isolated networks. Chrony supports ordinary NTP sources as well as authenticated Network Time Security where the environment permits it. Precision Time Protocol may be required for specialized measurement, but that requirement should be separated from the cluster's basic need for consistent civil time.

An alert should identify a condition that calls for action. A single corrected memory event may be retained as evidence; a rising rate across one DIMM may justify draining the node. One missed scrape may reflect collector maintenance; a missing host plus lost BMC reachability and switch link suggests a broader failure. Alerts should carry node identity, rack, current scheduler state, recent change, and a runbook entry. Maintenance windows and known drains need suppression rules so operators can distinguish planned silence from a new incident.

Capacity monitoring looks beyond percent busy. Track nodes available by hardware class, drained and down node-hours, repeated faults, repair time, image and firmware age, spare parts, provisioning duration, BMC reachability, storage headroom, monitoring cardinality, and control-plane saturation. Scheduler measures from Chapter 7 then connect infrastructure availability with queue delay and completed work.

Failure, repair, and return to service

The first response to an uncertain node is containment. Stop placing new jobs by draining or quarantining the node, record a reason and incident identifier, and decide what to do with running work. A hard hardware fault may require immediate termination. A correctable warning may allow a checkpoint or natural completion. The decision belongs to policy and observed risk; leaving a suspect node available while diagnosis begins exposes additional jobs to the same failure.

Preserve evidence before power cycling or reimaging. Record scheduler state and affected jobs, BMC event and sensor logs, kernel and service journals, crash data, fabric counters, storage-client errors, current image digest, firmware and driver versions, configuration drift, and recent maintenance. This bundle establishes a timeline and makes repeated faults recognizable. It also prevents a successful reboot from erasing the question of why the node failed.

Diagnosis should identify the layer rather than reward the first action that makes the node answer. Compare the fault with peers in the same rack, image, firmware set, and hardware class. A single node with memory machine checks suggests a local component. Several nodes behind one switch suggest a shared path. Failures beginning after an image promotion suggest software or compatibility. Repeated accelerator resets under one application may require an application reproducer before replacing hardware.

The corrective action can be configuration convergence, service restart, reboot, firmware change, reimage, cable work, component replacement, or retirement. Record replaced serial numbers and update inventory. A motherboard or BMC replacement may change UUIDs, MAC addresses, certificates, firmware defaults, and Secure Boot keys. Restoring the hostname alone does not complete the transfer of node identity.

Return to service follows the acceptance gate in Figure 8.1. The node should boot through the normal provisioner, reach desired state without unexplained drift, authenticate with its own credentials, mount intended storage, join every required fabric at expected width and speed, expose the advertised CPU, memory, NUMA, and accelerator topology, pass hardware diagnostics, and run representative scheduler-launched work. A repair that addressed an intermittent thermal or memory problem also needs a burn-in long enough to exercise the suspected condition.

Slurm can mark nodes drained, down, or available and can run a health-check program. Registration and a successful health-check script are useful signals, but automatic ReturnToService policy should match the evidence the site requires. A daemon restart after an unexpected reboot may justify automatic recovery on a homogeneous, stateless node with comprehensive checks. The same rule can be unsafe for a node with a replaced accelerator, stale mount, or unresolved machine check. Automation should advance lifecycle state only as far as its tests support.

Control-plane continuity and backup

High availability keeps a service operating through a component failure. Backup preserves state that can be restored after loss or corruption. They solve different problems. A primary and backup scheduler controller may protect scheduling decisions, yet both can read the same damaged configuration. A replicated database can faithfully replicate an accidental deletion. Every critical service needs both a continuity design and a recoverable, tested copy of its state.

List the state required to rebuild administration: inventory, DNS and DHCP data, provisioner node records and profiles, image definitions and immutable artifacts, configuration source, secrets and certificate authorities, scheduler configuration and state, accounting database, identity policy, monitoring rules and dashboards, BMC configuration exports where supported, network and storage configuration, runbooks, and vendor or warranty records. Classify which items are reproducible from source and which contain unique data.

Slurm can use a backup slurmctld and persist controller state in StateSaveLocation. SlurmDBD also supports a backup daemon through DbdBackupHost; the primary and backup daemons must reach the same underlying database. Database continuity, coherent backup, and restore remain separate responsibilities. Version compatibility matters because a newer daemon can migrate state or schema in ways an older daemon cannot read. Before an upgrade, preserve configuration, state, database, binaries or packages, and the documented restore order.

Provisioning and identity services deserve the same scrutiny. Redundant DHCP responders without coordinated leases can conflict. Two image servers that serve different content under one mutable URL defeat reproducibility. An identity replica is useful only if DNS, certificates, time, and clients can reach it during the failure being tested. Restore exercises should begin from a stated loss scenario and end with a newly provisioned node and scheduler-launched validation job, not merely a successful database import.

Administration acceptance tests

A cluster administration design is ready when its recovery claims have been demonstrated. Keep the tests small enough to run after routine change and broad enough to cross the boundaries on which jobs depend.

1. Reconcile identity. Compare asset, rack, BMC, interface, DNS, provisioner, monitoring, and scheduler records for sampled nodes. Introduce a controlled mismatch and confirm that it is detected.
2. Exercise power control. Query inventory and sensors, perform graceful and forced power actions on a test node, and verify authorization and audit records through the supported BMC API.
3. Traverse the console chain. Capture UEFI, GRUB, kernel, and userspace login output through the remote serial path. Confirm that a failure at each stage can be distinguished.
4. Recover without the host network. Disable or misconfigure the workload interface on a test node and prove that BMC, console, power, and provisioning recovery remain available.
5. Provision from bare state. Start with an uninstalled or erased node, resolve its identity, deliver the intended image, enroll unique credentials, and reach desired state without manual repair.
6. Test an unknown node. Present an unregistered client to the boot network and confirm that it receives no production secrets or image entitlement.
7. Measure a boot cohort. Reboot a bounded group and record DHCP, boot, image, registry, network, and control-service load. Use the result to set safe batch size.
8. Detect drift. Change a managed file and package on a test node. Confirm detection, review output, convergence, and protection of secret values in diffs.
9. Roll forward and back. Promote a kernel, driver, firmware, or node image through a canary, run workload tests, and exercise the documented rollback with artifacts available.
10. Revoke access. Disable a test identity and expire a credential. Verify login, scheduler, privilege, API, and cached-access behavior across the stated interval.
11. Test zone boundaries. Attempt prohibited paths from external, compute, storage, telemetry, and BMC networks. Confirm default-deny enforcement and logging.
12. Correlate an incident. Trigger a controlled node fault and assemble BMC, kernel, fabric, storage, scheduler, and change evidence on one timeline.
13. Drain and return. Remove a node from placement, repair or reimage it, run the complete acceptance suite, and require an explicit recorded decision before resuming it.
14. Fail a controller. Exercise scheduler, provisioner, monitoring, and identity continuity separately. Observe which existing functions continue and which new operations stop.
15. Restore unique state. Recover a selected control service from backup into an isolated environment and use it to complete an end-to-end operation such as provisioning and validating a node.

The result is an administrative system that can explain every node's identity, state, and permission to run work. Chapter 9 moves from the machines to the service built on them: how projects enter the cluster, receive support and allocations, manage software and data, and complete useful work under an operating cadence.

References and further reading

1. DMTF: Redfish standards and schemas.
2. DMTF: Redfish Specification 1.21.
3. OpenBMC: Redfish, KVM, and console services in bmcweb.
4. OpenBMC: Network security considerations.
5. NSA and CISA: Harden Baseboard Management Controllers.
6. CISA: Mitigating risk from Internet-exposed management interfaces.
7. Linux kernel documentation: Serial console.
8. GNU GRUB manual: Serial terminal.
9. systemd: Console and serial getty generation.
10. iPXE: Chainloading and UEFI network boot.
11. iPXE: Downloading and booting an image.
12. Warewulf 4.6 user guide.
13. Warewulf: Per-node and runtime overlays.
14. Red Hat Enterprise Linux 10: Automated Kickstart installation workflow.
15. Red Hat Enterprise Linux 10: Image-mode updates and rollback.
16. cloud-init: Boot stages and configuration ownership.
17. Ansible: Check mode and diff mode.
18. Ansible Core: Idempotent playbooks and validation.
19. SSSD: Centralized identity, authentication, authorization, and caching.
20. FreeIPA documentation.
21. OpenBSD manual: OpenSSH user and host certificates.
22. SchedMD: Adopting SSH sessions into Slurm jobs.
23. SchedMD: Slurm authentication and key rotation.
24. NIST SP 800-193: Platform Firmware Resiliency Guidelines.
25. fwupd: UEFI capsule update behavior.
26. SLSA 1.2: Build provenance.
27. Prometheus: Monitoring Linux host metrics with Node Exporter.
28. OpenTelemetry: Collector.
29. Performance Co-Pilot project.
30. NVIDIA Data Center GPU Manager documentation.
31. AMD System Management Interface documentation.
32. chrony: Time sources and Network Time Security.
33. Linux kernel documentation: kdump crash dumps.
34. Linux kernel documentation: pstore and ramoops persistent crash logs.
35. Linux kernel documentation: Netconsole.
36. SchedMD: Node drain, reboot, resume, and control operations.
37. SchedMD: Health checks, return to service, and controller configuration.
38. SchedMD: Primary and backup SlurmDBD configuration.
39. SchedMD: Controller high availability and persisted state.
40. SchedMD: Upgrade preparation, state compatibility, and backups.