====== Guide to SLURM Resource Management and Job Submission ====== This guide provides an advanced overview of the **SLURM (Simple Linux Utility for Resource Management)** workload manager and the underlying physical architecture of the UNITE HPC cluster. It covers theoretical network topologies, inter-node communication, deep hardware utilization mechanics, user workflows, and best practices for parallel execution. ===== 1. Theoretical Overview & Cluster Architecture ===== An HPC cluster is a massive parallel computing instrument built from a collection of individual commodity servers called **Nodes**, interconnected by an ultra-fast, low-latency network fabric. Slurm isolates users from this underlying hardware complexity while ensuring optimal resource distribution. Slurm operates under a master-worker orchestration framework managed by three primary daemons: * ''slurmctld'' (Central Controller Daemon): Runs on the head/login architecture. It orchestrates the global job state, assesses queue priorities, processes resource requests, and enforces scheduling fairness policies. * ''slurmd'' (Compute Node Daemon): Monitored natively on every individual compute node. It accepts operational instructions from the central ''slurmctld'', provisions local hardware threads, isolates local memory namespaces, spawns your process steps, and returns exit codes. * ''slurmdbd'' (Database Daemon): Explicitly records cluster accounting metrics, resource tracking logs, project computing hours allocations, and fair-share consumption records. ==== Resource Hierarchies ==== When submitting a workload to Slurm, it is crucial to understand how resource requests translate into physical hardware allocations: * **Cluster:** The entire collective pool of computing hardware, networking switches, and storage layers. * **Partition:** A logical grouping of specific nodes (e.g., vanilla compute nodes vs. high-performance GPU nodes like the ''a40'' pool). * **Node:** A single distinct physical server chassis possessing localized processors, volatile system memory (RAM), and explicit hardware buses. * **Core/Socket:** The physical computational units inside a CPU. * **Task:** An independent process instance. In distributed execution (like MPI), 1 Task generally maps to 1 unique process loop. * **CPUs-per-task:** The number of hardware threads or CPU cores dedicated to backing a single task (critical for shared-memory multithreading models like OpenMP). ==== Deep Dive: CPU and Memory Architecture (NUMA) ==== Modern compute nodes are not monolithic. A single node typically contains two or more CPU **sockets** (physical chips). Memory (RAM) is physically divided and attached directly to specific sockets. This is known as **NUMA (Non-Uniform Memory Access)**. * **Local Memory Access:** If a program running on CPU Socket 0 requests data stored in the RAM physically attached to Socket 0, the access is nearly instantaneous. * **Remote Memory Access:** If a program running on Socket 0 requests data stored in the RAM attached to Socket 1, the data must travel across an interconnect bus on the motherboard. This introduces latency and bandwidth bottlenecks. **Why this matters for Slurm:** When you request resources, Slurm attempts to bind your tasks to specific cores and local memory banks (CPU affinity/pinning) to avoid crossing NUMA boundaries. Requesting scattered resources loosely can result in your application constantly fetching data across the motherboard, severely degrading performance. --- ===== 2. Advanced Cluster Topology and Inter-Node Communication ===== To write optimal code for an HPC cluster, developers must account for how components talk to one another. Communication speeds and delays (**latency**) change drastically based on where your data sits and where it needs to go. ==== Memory and Communication Layout ==== ^ Execution Scope ^ Communication Medium ^ Relative Bandwidth ^ Latency Complexity ^ | **Intra-Core** (Same CPU) | L1 / L2 / L3 Processor Cache | Ultra-High (Terabytes/sec) | Lowest (Nanoseconds) | | **Intra-Node** (Same Server) | Physical RAM & PCIe Bus / NVLink | Very High (Hundreds of GB/s) | Low | | **Inter-Node** (Across Servers) | InfiniBand Network Fabric / Switches | High (Hundreds of Gbps) | Microseconds (Fabric Transit) | ==== Network Topologies: Fat-Tree Architecture ==== HPC networks like the one powering the UNITE cluster avoid standard corporate network setups (which bottleneck under heavy loads) in favor of a specialized **Fat-Tree Topology**. * **Non-Blocking Fabric:** In a standard IT tree, links get narrower as you go up toward the core switches, causing traffic jams. A **Fat-Tree topology** does the opposite: the network connections multiply and get thicker (more bandwidth) closer to the top core switches. * **Full Bisection Bandwidth:** This ensures that if half the compute nodes on the cluster are actively transmitting data to the other half at the exact same time, the network switches won't bottleneck or drop packets. ==== Deep Dive: InfiniBand & OS Kernel Bypass ==== When your code spans across multiple physical nodes (e.g., an MPI job), it bypasses standard slow Ethernet connections. Ethernet requires the operating system's kernel (TCP/IP stack) to package, verify, and route every single byte, which consumes massive CPU cycles and creates latency. The UNITE cluster utilizes **InfiniBand Architecture (IBA)** and **RDMA (Remote Direct Memory Access)**: * **Zero-Copy Networking:** RDMA allows the network interface card (NIC) on Node A to read data directly out of its local RAM and push it directly into the RAM of Node B. * **Kernel Bypass:** Neither the CPU nor the operating system on Node A or Node B is involved in the transfer. This drops network latency down to single-digit microseconds (<2us), allowing distributed applications to scale linearly. ==== GPU Interconnects (PCIe vs. NVLink) ==== On GPU partitions (like the ''a40'' pool), how GPUs talk to each other is critical. * **PCIe Bus:** If two GPUs must communicate via the standard motherboard PCIe bus, bandwidth is limited (typically 32-64 GB/s) and latency increases because data must pass through the CPU. * **NVLink:** High-end cluster architectures utilize NVIDIA NVLink bridges, allowing GPUs to share memory pools directly at massive speeds (hundreds of GB/s). When writing distributed PyTorch or CuPy scripts across multiple GPUs on the same node, the framework relies on this hardware topology. --- ===== 3. Scheduler Theory: How Slurm Assigns Resources ===== Slurm does not simply process jobs sequentially. It uses complex algorithms to maximize cluster utilization and ensure fairness. * **Fair-Share Scheduling:** The priority of your job is dynamically calculated based on historical usage. If you have been running massive jobs all week, your priority score drops to allow other researchers a chance to run their code. * **Backfilling Algorithm:** Large jobs often have to wait for enough nodes to become free simultaneously. This creates temporary "holes" in the cluster's utilization. Slurm will actively scan the queue for smaller, shorter jobs and "backfill" them into these holes, provided they will finish before the large job's resources are fully gathered. * ''Best Practice:'' **Always accurately estimate your #SBATCH --time limit.** If you request 24 hours for a job that takes 1 hour, Slurm cannot use your job for backfilling. If you request 1 hour, Slurm can easily squeeze your job into an upcoming idle gap, significantly reducing your wait time. --- ===== 4. Crucial Cluster Etiquette: The Shared Environment ===== ^ WARNING: NEVER RUN COMPUTATIONS ON THE LOGIN NODE! ^ | When you connect to the UNITE cluster via SSH, you land directly on the **Login Node**. This node's sole purpose is file management, code editing, and checking job queues. Heavy processes like compilation, heavy Python tasks, or model training will slow down the login node for everyone on the cluster. Running intensive code here may result in administrators forcefully terminating your session. | To execute code, you **must** pass your workloads over to a dedicated compute node using either **Interactive Sessions** or **Batch Job Submissions**. --- ===== 5. Key Slurm Commands Workflow ===== Interact with the scheduler fabric using these primary terminal operations: ^ Command ^ Action Description ^ Typical Use Case ^ | ''sinfo'' | Queries the state of partitions and available physical cluster hardware nodes. | Checking which queues are free or down. | | ''srun'' | Allocates and runs parallel tasks. Can open an active interactive terminal session. | Step-by-step code testing and compilation. | | ''sbatch'' | Enqueues a non-interactive shell script to run autonomously in the background. | Production training and massive compute jobs. | | ''squeue'' | Inspects current active allocations and waiting jobs in the scheduler pipelines. | Checking your queue position or job status. | | ''scancel '' | Immediately terminates a queued or running job matching the specific ID. | Aborting an infinite loop or mistaken setup. | --- ===== 6. Interactive Development vs. Non-Interactive Batch ===== ==== Workflow A: Interactive Sessions (Testing & Compilation) ==== To compile libraries, debug scripts line-by-line, or test packages safely on a real compute node, use an interactive resource request via ''srun''. For a standard multi-purpose compute slice: $ srun --partition=unite --cpus-per-task=4 --time=00:30:00 --pty bash For a dedicated deep learning workspace utilizing NVIDIA graphics hardware: $ srun --partition=a40 --gres=gpu:1 --cpus-per-task=4 --time=00:30:00 --pty bash Once executed, Slurm places your terminal directly onto a secure, isolated compute shell. When your testing is complete, type ''exit'' to release the resources back to the queue pool. ==== Workflow B: Batch Processing (Production Runs) ==== For heavy workloads that take hours or days, write a batch script and submit it using ''sbatch''. Once submitted, you can disconnect your computer safely; Slurm handles execution and records outputs into text files on the storage array. Here is a template structure for a production batch file (''run_job.sh''): #!/bin/bash #SBATCH --job-name=unite_production_job #SBATCH --partition=unite # Target partition queue (e.g., unite, a40) #SBATCH --output=logs_%j.out # Output log text file (%j replaces with unique Job ID) #SBATCH --error=logs_%j.err # Error tracking text file #SBATCH --nodes=1 # Number of distinct physical hardware nodes #SBATCH --ntasks=1 # Number of execution application instances #SBATCH --cpus-per-task=4 # CPU worker core threads assigned to this task #SBATCH --mem=16G # Safe RAM allocation limit request #SBATCH --time=02:00:00 # Maximum safety runtime allowance (HH:MM:SS) # 1. Clean the inherited terminal environment states module purge # 2. Execute target execution logic or launch binaries echo "Starting production workload execution on node: $(hostname)" # Run your commands here... To place this file into the cluster scheduler pipeline, execute: $ sbatch run_job.sh --- ===== 7. Reference Guides and Real-World Examples ===== To see how to apply these Slurm parameters across different compiler systems, runtime environments, and compute nodes, follow our language-specific reference documentation pages: * **Native Applications (C++):** * [[unite_cpp_gcc|Compiling C++ Applications with the GCC Module]] — Step-by-step instructions for running basic ''g++'' compilation steps via interactive nodes. * [[unite_cpp_mpi|Compiling and Running Distributed C++ MPI Applications]] — How to request multi-node processing (''--nodes=2'', ''--ntasks-per-node=4'') and handle cross-node communication via InfiniBand fabrics. * **GPU-Accelerated Data Science & ML (Python):** * [[unite_python_cupy|Accelerated Mathematical Computations with CuPy]] — Leveraging GPU hardware tensors on the **unite** partition utilizing tracking hooks like ''--gres=gpu:1''. * [[unite_python_torch|Deep Learning Model Frameworks with PyTorch]] — High-performance neural processing allocations specifically built to execute on the advanced **a40** hardware partition. --- ===== 8. Troubleshooting Common Queue States ===== If you run ''squeue -u $USER'' and see your job sitting in a ''PD'' (Pending) state, look closely at the ''NODELIST(REASON)'' column: * **(Resources):** The cluster is fully utilized right now. Slurm will launch your job automatically as soon as another user's time allocation expires. * **(Priority):** Higher priority workloads (based on user fair-share rules) are currently waiting ahead of you in line. * **(PartitionTimeLimit):** Your requested ''#SBATCH --time'' configuration parameter exceeds the maximum execution ceiling value allowed for that specific partition. * **(InvalidQOS):** You requested a resource combination (like asking for an A40 GPU inside a non-GPU queue) that violates partition definition policies.