Guide to SLURM Resource Management and Job Submission

This guide provides an advanced overview of the SLURM (Simple Linux Utility for Resource Management) workload manager and the underlying physical architecture of the UNITE HPC cluster. It covers theoretical network topologies, inter-node communication, deep hardware utilization mechanics, user workflows, and best practices for parallel execution.

An HPC cluster is a massive parallel computing instrument built from a collection of individual commodity servers called Nodes, interconnected by an ultra-fast, low-latency network fabric. Slurm isolates users from this underlying hardware complexity while ensuring optimal resource distribution.

Slurm operates under a master-worker orchestration framework managed by three primary daemons:

• slurmctld (Central Controller Daemon): Runs on the head/login architecture. It orchestrates the global job state, assesses queue priorities, processes resource requests, and enforces scheduling fairness policies.
• slurmd (Compute Node Daemon): Monitored natively on every individual compute node. It accepts operational instructions from the central slurmctld, provisions local hardware threads, isolates local memory namespaces, spawns your process steps, and returns exit codes.
• slurmdbd (Database Daemon): Explicitly records cluster accounting metrics, resource tracking logs, project computing hours allocations, and fair-share consumption records.

When submitting a workload to Slurm, it is crucial to understand how resource requests translate into physical hardware allocations:

• Cluster: The entire collective pool of computing hardware, networking switches, and storage layers.
• Partition: A logical grouping of specific nodes (e.g., vanilla compute nodes vs. high-performance GPU nodes like the a40 pool).
• Node: A single distinct physical server chassis possessing localized processors, volatile system memory (RAM), and explicit hardware buses.
• Core/Socket: The physical computational units inside a CPU.
• Task: An independent process instance. In distributed execution (like MPI), 1 Task generally maps to 1 unique process loop.
• CPUs-per-task: The number of hardware threads or CPU cores dedicated to backing a single task (critical for shared-memory multithreading models like OpenMP).

Modern compute nodes are not monolithic. A single node typically contains two or more CPU sockets (physical chips). Memory (RAM) is physically divided and attached directly to specific sockets. This is known as NUMA (Non-Uniform Memory Access).

• Local Memory Access: If a program running on CPU Socket 0 requests data stored in the RAM physically attached to Socket 0, the access is nearly instantaneous.
• Remote Memory Access: If a program running on Socket 0 requests data stored in the RAM attached to Socket 1, the data must travel across an interconnect bus on the motherboard. This introduces latency and bandwidth bottlenecks.

Why this matters for Slurm: When you request resources, Slurm attempts to bind your tasks to specific cores and local memory banks (CPU affinity/pinning) to avoid crossing NUMA boundaries. Requesting scattered resources loosely can result in your application constantly fetching data across the motherboard, severely degrading performance.

—

To write optimal code for an HPC cluster, developers must account for how components talk to one another. Communication speeds and delays (latency) change drastically based on where your data sits and where it needs to go.

HPC networks like the one powering the UNITE cluster avoid standard corporate network setups (which bottleneck under heavy loads) in favor of a specialized Fat-Tree Topology.

• Non-Blocking Fabric: In a standard IT tree, links get narrower as you go up toward the core switches, causing traffic jams. A Fat-Tree topology does the opposite: the network connections multiply and get thicker (more bandwidth) closer to the top core switches.
• Full Bisection Bandwidth: This ensures that if half the compute nodes on the cluster are actively transmitting data to the other half at the exact same time, the network switches won't bottleneck or drop packets.

When your code spans across multiple physical nodes (e.g., an MPI job), it bypasses standard slow Ethernet connections. Ethernet requires the operating system's kernel (TCP/IP stack) to package, verify, and route every single byte, which consumes massive CPU cycles and creates latency.

The UNITE cluster utilizes InfiniBand Architecture (IBA) and RDMA (Remote Direct Memory Access):

• Zero-Copy Networking: RDMA allows the network interface card (NIC) on Node A to read data directly out of its local RAM and push it directly into the RAM of Node B.
• Kernel Bypass: Neither the CPU nor the operating system on Node A or Node B is involved in the transfer. This drops network latency down to single-digit microseconds (<2us), allowing distributed applications to scale linearly.

On GPU partitions (like the a40 pool), how GPUs talk to each other is critical.

• PCIe Bus: If two GPUs must communicate via the standard motherboard PCIe bus, bandwidth is limited (typically 32-64 GB/s) and latency increases because data must pass through the CPU.
• NVLink: High-end cluster architectures utilize NVIDIA NVLink bridges, allowing GPUs to share memory pools directly at massive speeds (hundreds of GB/s). When writing distributed PyTorch or CuPy scripts across multiple GPUs on the same node, the framework relies on this hardware topology.

—

Slurm does not simply process jobs sequentially. It uses complex algorithms to maximize cluster utilization and ensure fairness.

• Fair-Share Scheduling: The priority of your job is dynamically calculated based on historical usage. If you have been running massive jobs all week, your priority score drops to allow other researchers a chance to run their code.
• Backfilling Algorithm: Large jobs often have to wait for enough nodes to become free simultaneously. This creates temporary “holes” in the cluster's utilization. Slurm will actively scan the queue for smaller, shorter jobs and “backfill” them into these holes, provided they will finish before the large job's resources are fully gathered.
  • Best Practice: Always accurately estimate your #SBATCH –time limit. If you request 24 hours for a job that takes 1 hour, Slurm cannot use your job for backfilling. If you request 1 hour, Slurm can easily squeeze your job into an upcoming idle gap, significantly reducing your wait time.

—

To execute code, you must pass your workloads over to a dedicated compute node using either Interactive Sessions or Batch Job Submissions.

—

Interact with the scheduler fabric using these primary terminal operations:

—

To compile libraries, debug scripts line-by-line, or test packages safely on a real compute node, use an interactive resource request via srun.

For a standard multi-purpose compute slice:

$ srun --partition=unite --cpus-per-task=4 --time=00:30:00 --pty bash

For a dedicated deep learning workspace utilizing NVIDIA graphics hardware:

$ srun --partition=a40 --gres=gpu:1 --cpus-per-task=4 --time=00:30:00 --pty bash

Once executed, Slurm places your terminal directly onto a secure, isolated compute shell. When your testing is complete, type exit to release the resources back to the queue pool.

For heavy workloads that take hours or days, write a batch script and submit it using sbatch. Once submitted, you can disconnect your computer safely; Slurm handles execution and records outputs into text files on the storage array.

Here is a template structure for a production batch file (run_job.sh):

run_job.sh

#!/bin/bash
#SBATCH --job-name=unite_production_job
#SBATCH --partition=unite           # Target partition queue (e.g., unite, a40)
#SBATCH --output=logs_%j.out        # Output log text file (%j replaces with unique Job ID)
#SBATCH --error=logs_%j.err         # Error tracking text file
#SBATCH --nodes=1                   # Number of distinct physical hardware nodes
#SBATCH --ntasks=1                  # Number of execution application instances
#SBATCH --cpus-per-task=4           # CPU worker core threads assigned to this task
#SBATCH --mem=16G                   # Safe RAM allocation limit request
#SBATCH --time=02:00:00             # Maximum safety runtime allowance (HH:MM:SS)
 
# 1. Clean the inherited terminal environment states
module purge
 
# 2. Execute target execution logic or launch binaries
echo "Starting production workload execution on node: $(hostname)"
# Run your commands here...

To place this file into the cluster scheduler pipeline, execute:

$ sbatch run_job.sh

—

To see how to apply these Slurm parameters across different compiler systems, runtime environments, and compute nodes, follow our language-specific reference documentation pages:

• Native Applications (C++):
  • Compiling C++ Applications with the GCC Module — Step-by-step instructions for running basic g++ compilation steps via interactive nodes.
  • Compiling and Running Distributed C++ MPI Applications — How to request multi-node processing (–nodes=2, –ntasks-per-node=4) and handle cross-node communication via InfiniBand fabrics.
• GPU-Accelerated Data Science & ML (Python):
  • Accelerated Mathematical Computations with CuPy — Leveraging GPU hardware tensors on the unite partition utilizing tracking hooks like –gres=gpu:1.
  • Deep Learning Model Frameworks with PyTorch — High-performance neural processing allocations specifically built to execute on the advanced a40 hardware partition.

—

If you run squeue -u $USER and see your job sitting in a PD (Pending) state, look closely at the NODELIST(REASON) column:

• (Resources): The cluster is fully utilized right now. Slurm will launch your job automatically as soon as another user's time allocation expires.
• (Priority): Higher priority workloads (based on user fair-share rules) are currently waiting ahead of you in line.
• (PartitionTimeLimit): Your requested #SBATCH –time configuration parameter exceeds the maximum execution ceiling value allowed for that specific partition.
• (InvalidQOS): You requested a resource combination (like asking for an A40 GPU inside a non-GPU queue) that violates partition definition policies.