Differences

This shows you the differences between two versions of the page.

--- examples [2025/12/29 21:36] – dimitar
+++ examples [2026/06/13 15:24] (current) – nshegunov
@@ Line 1: / Line 1: @@
-====Description====
+====== Guide to SLURM Resource Management and Job Submission ======
-This page provides examples on how to use the cluster. There are language specific examples for **C/C++**, and **Python**, which showcase how you can compile and run applications which are written in those languages on the cluster. Additionally, there are examples for how to leverage the different resources of the cluster. These examples are written in **C++**, but the concepts apply to a program written in any language.
+This guide provides an advanced overview of the **SLURM (Simple Linux Utility for Resource Management)** workload manager and the underlying physical architecture of the UNITE HPC cluster. It covers theoretical network topologies, inter-node communication, deep hardware utilization mechanics, user workflows, and best practices for parallel execution.
-----
+===== 1. Theoretical Overview & Cluster Architecture =====
-====Simple C/C++ program====
+An HPC cluster is a massive parallel computing instrument built from a collection of individual commodity servers called **Nodes**, interconnected by an ultra-fast, low-latency network fabric. Slurm isolates users from this underlying hardware complexity while ensuring optimal resource distribution.
-The following is a simple **C/C++** program which performs element-wise addition of 2 vectors. It does **not** use any dependent libraries:
-<code C>
+Slurm operates under a master-worker orchestration framework managed by three primary daemons:
-#include <stdio.h>
-#include <stdlib.h>
-#include <time.h>
-#include <sys/time.h>
-/*
+  * ''slurmctld'' (Central Controller Daemon): Runs on the head/login architecture. It orchestrates the global job state, assesses queue priorities, processes resource requests, and enforces scheduling fairness policies.
- * Perform element-wise addition of two vectors
+  * ''slurmd'' (Compute Node Daemon): Monitored natively on every individual compute node. It accepts operational instructions from the central ''slurmctld'', provisions local hardware threads, isolates local memory namespaces, spawns your process steps, and returns exit codes.
- *
+  * ''slurmdbd'' (Database Daemon): Explicitly records cluster accounting metrics, resource tracking logs, project computing hours allocations, and fair-share consumption records.
- * Parameters:
- *   a: First input vector
- *   b: Second input vector
- *   result: Output vector (a + b)
- *   size: Number of elements in vectors
- */
-void vector_addition(const double *a, const double *b, double *result, size_t size) {
-    for (size_t i = 0; i < size; i++) {
-        result[i] = a[i] + b[i];
-    }
-}
-int main() {
+==== Resource Hierarchies ====
-    const size_t size = 10000000;
+When submitting a workload to Slurm, it is crucial to understand how resource requests translate into physical hardware allocations:
-    printf("========================================\n");
+  * **Cluster:** The entire collective pool of computing hardware, networking switches, and storage layers.
-    printf("Vector Addition Example in C\n");
+  * **Partition:** A logical grouping of specific nodes (e.g., vanilla compute nodes vs. high-performance GPU nodes like the ''a40'' pool).
-    printf("========================================\n");
+  * **Node:** A single distinct physical server chassis possessing localized processors, volatile system memory (RAM), and explicit hardware buses.
-    printf("Vector size: %zu elements\n", size);
+  * **Core/Socket:** The physical computational units inside a CPU.
+  * **Task:** An independent process instance. In distributed execution (like MPI), 1 Task generally maps to 1 unique process loop.
+  * **CPUs-per-task:** The number of hardware threads or CPU cores dedicated to backing a single task (critical for shared-memory multithreading models like OpenMP).
-    printf("\nAllocating memory...\n");
+==== Deep Dive: CPU and Memory Architecture (NUMA) ====
-    double *vector_a = (double *)malloc(size * sizeof(double));
+Modern compute nodes are not monolithic. A single node typically contains two or more CPU **sockets** (physical chips). Memory (RAM) is physically divided and attached directly to specific sockets. This is known as **NUMA (Non-Uniform Memory Access)**.
-    double *vector_b = (double *)malloc(size * sizeof(double));
-    double *result = (double *)malloc(size * sizeof(double));
-    if (vector_a == NULL || vector_b == NULL || result == NULL) {
+  * **Local Memory Access:** If a program running on CPU Socket 0 requests data stored in the RAM physically attached to Socket 0, the access is nearly instantaneous.
-        fprintf(stderr, "Error: Memory allocation failed!\n");
+  * **Remote Memory Access:** If a program running on Socket 0 requests data stored in the RAM attached to Socket 1, the data must travel across an interconnect bus on the motherboard. This introduces latency and bandwidth bottlenecks.
-        return 1;
-    }
-    printf("Initializing vectors...\n");
+**Why this matters for Slurm:** When you request resources, Slurm attempts to bind your tasks to specific cores and local memory banks (CPU affinity/pinning) to avoid crossing NUMA boundaries. Requesting scattered resources loosely can result in your application constantly fetching data across the motherboard, severely degrading performance.
-    srand(time(NULL));
-    for (size_t i = 0; i < size; i++) {
-        vector_a[i] = (double)rand() / RAND_MAX;
-        vector_b[i] = (double)rand() / RAND_MAX;
-    }
-    printf("Performing vector addition...\n");
+---
-    vector_addition(vector_a, vector_b, result, size);
-    printf("First 5 elements of result:\n");
+===== 2. Advanced Cluster Topology and Inter-Node Communication =====
-    for (int i = 0; i < 5; i++) {
-        printf("  result[%d] = %.6f\n", i, result[i]);
-    }
-    free(vector_a);
+To write optimal code for an HPC cluster, developers must account for how components talk to one another. Communication speeds and delays (**latency**) change drastically based on where your data sits and where it needs to go.
-    free(vector_b);
-    free(result);
-    return 0;
+==== Memory and Communication Layout ====
-}
+^ Execution Scope ^ Communication Medium ^ Relative Bandwidth ^ Latency Complexity ^
-</code>
+| **Intra-Core** (Same CPU) | L1 / L2 / L3 Processor Cache | Ultra-High (Terabytes/sec) | Lowest (Nanoseconds) |
+| **Intra-Node** (Same Server) | Physical RAM & PCIe Bus / NVLink | Very High (Hundreds of GB/s) | Low |
+| **Inter-Node** (Across Servers) | InfiniBand Network Fabric / Switches | High (Hundreds of Gbps) | Microseconds (Fabric Transit) |
-The following is the respective batch script for compiling and running the program. You can see the output of the program in the generated //vector_sum_%j.out// file.
+==== Network Topologies: Fat-Tree Architecture ====
+HPC networks like the one powering the UNITE cluster avoid standard corporate network setups (which bottleneck under heavy loads) in favor of a specialized **Fat-Tree Topology**.
-<code bash>
+  * **Non-Blocking Fabric:** In a standard IT tree, links get narrower as you go up toward the core switches, causing traffic jams. A **Fat-Tree topology** does the opposite: the network connections multiply and get thicker (more bandwidth) closer to the top core switches.
-#!/bin/bash
+  * **Full Bisection Bandwidth:** This ensures that if half the compute nodes on the cluster are actively transmitting data to the other half at the exact same time, the network switches won't bottleneck or drop packets.
-#SBATCH --job-name=vector_sum
-#SBATCH --output=vector_sum_%j.out
-#SBATCH --error=vector_sum_%j.err
-#SBATCH --nodes=1
-#SBATCH --ntasks=1
-#SBATCH --cpus-per-task=1
-#SBATCH --time=00:10:00
-#SBATCH --partition=unite
-echo "========================================="
+==== Deep Dive: InfiniBand & OS Kernel Bypass ====
-echo "SLURM Job Information"
+When your code spans across multiple physical nodes (e.g., an MPI job), it bypasses standard slow Ethernet connections. Ethernet requires the operating system's kernel (TCP/IP stack) to package, verify, and route every single byte, which consumes massive CPU cycles and creates latency.
-echo "========================================="
-echo "Job ID: $SLURM_JOB_ID"
-echo "Node: $SLURM_NODELIST"
-echo "Starting at: $(date)"
-echo ""
-# Load necessary modules
+The UNITE cluster utilizes **InfiniBand Architecture (IBA)** and **RDMA (Remote Direct Memory Access)**:
-module load gcc
-# Compile the program
+  * **Zero-Copy Networking:** RDMA allows the network interface card (NIC) on Node A to read data directly out of its local RAM and push it directly into the RAM of Node B.
-echo "Compiling vector_sum.c..."
+  * **Kernel Bypass:** Neither the CPU nor the operating system on Node A or Node B is involved in the transfer. This drops network latency down to single-digit microseconds (<2us), allowing distributed applications to scale linearly.
-gcc -O3 -march=native -o vector_sum vector_sum.c -lm
-if [ $? -ne 0 ]; then
+==== GPU Interconnects (PCIe vs. NVLink) ====
-    echo "Error: Compilation failed!"
+On GPU partitions (like the ''a40'' pool), how GPUs talk to each other is critical.
-    exit 1
-fi
-echo "Compilation successful!"
+  * **PCIe Bus:** If two GPUs must communicate via the standard motherboard PCIe bus, bandwidth is limited (typically 32-64 GB/s) and latency increases because data must pass through the CPU.
-echo ""
+  * **NVLink:** High-end cluster architectures utilize NVIDIA NVLink bridges, allowing GPUs to share memory pools directly at massive speeds (hundreds of GB/s). When writing distributed PyTorch or CuPy scripts across multiple GPUs on the same node, the framework relies on this hardware topology.
-echo "Running vector_sum..."
+---
-./vector_sum
-echo ""
+===== 3. Scheduler Theory: How Slurm Assigns Resources =====
-echo "Job finished at: $(date)"
-</code>
-----
-====Simple Python program====
+Slurm does not simply process jobs sequentially. It uses complex algorithms to maximize cluster utilization and ensure fairness.
-The following is a simple **Python** program which performs element-wise addition of 2 vectors. It does **not** use any dependent libraries:
-<code Python>
+  * **Fair-Share Scheduling:** The priority of your job is dynamically calculated based on historical usage. If you have been running massive jobs all week, your priority score drops to allow other researchers a chance to run their code.
-#!/usr/bin/env python3
+  * **Backfilling Algorithm:** Large jobs often have to wait for enough nodes to become free simultaneously. This creates temporary "holes" in the cluster's utilization. Slurm will actively scan the queue for smaller, shorter jobs and "backfill" them into these holes, provided they will finish before the large job's resources are fully gathered.
-import random
+    * ''Best Practice:'' **Always accurately estimate your #SBATCH --time limit.** If you request 24 hours for a job that takes 1 hour, Slurm cannot use your job for backfilling. If you request 1 hour, Slurm can easily squeeze your job into an upcoming idle gap, significantly reducing your wait time.
-import time
-def vector_addition(a, b):
+---
-    """
-    Perform element-wise addition of two vectors
-    Parameters:
+===== 4. Crucial Cluster Etiquette: The Shared Environment =====
-        a: First input vector (list)
-        b: Second input vector (list)
-    Returns:
+^ WARNING: NEVER RUN COMPUTATIONS ON THE LOGIN NODE! ^
-        result: Output vector (a + b)
+| When you connect to the UNITE cluster via SSH, you land directly on the **Login Node**. This node's sole purpose is file management, code editing, and checking job queues. Heavy processes like compilation, heavy Python tasks, or model training will slow down the login node for everyone on the cluster. Running intensive code here may result in administrators forcefully terminating your session. |
-    """
-    return [a[i] + b[i] for i in range(len(a))]
+To execute code, you **must** pass your workloads over to a dedicated compute node using either **Interactive Sessions** or **Batch Job Submissions**.
-def main():
+---
-    size = 10000000
-    print("=" * 40)
+===== 5. Key Slurm Commands Workflow =====
-    print("Vector Addition Example in Python")
-    print("=" * 40)
-    print(f"Vector size: {size:,} elements")
-    print("\nAllocating and initializing vectors...")
+Interact with the scheduler fabric using these primary terminal operations:
-    random.seed(time.time())
-    vector_a = [random.random() for _ in range(size)]
+^ Command ^ Action Description ^ Typical Use Case ^
-    vector_b = [random.random() for _ in range(size)]
+| ''sinfo'' | Queries the state of partitions and available physical cluster hardware nodes. | Checking which queues are free or down. |
+| ''srun'' | Allocates and runs parallel tasks. Can open an active interactive terminal session. | Step-by-step code testing and compilation. |
+| ''sbatch'' | Enqueues a non-interactive shell script to run autonomously in the background. | Production training and massive compute jobs. |
+| ''squeue'' | Inspects current active allocations and waiting jobs in the scheduler pipelines. | Checking your queue position or job status. |
+| ''scancel <ID>'' | Immediately terminates a queued or running job matching the specific ID. | Aborting an infinite loop or mistaken setup. |
-    print("Performing vector addition...")
+---
-    result = vector_addition(vector_a, vector_b)
-    print("\nFirst 5 elements of result:")
+===== 6. Interactive Development vs. Non-Interactive Batch =====
-    for i in range(5):
-        print(f"  result[{i}] = {result[i]:.6f}")
+==== Workflow A: Interactive Sessions (Testing & Compilation) ====
+To compile libraries, debug scripts line-by-line, or test packages safely on a real compute node, use an interactive resource request via ''srun''.
-if __name__ == "__main__":
+For a standard multi-purpose compute slice:
-    main()
+<code>
+$ srun --partition=unite --cpus-per-task=4 --time=00:30:00 --pty bash
 </code>
-The following is the respective batch script for running the program. You can see the output of the program in the generated //vector_sum_%j.out// file.
+For a dedicated deep learning workspace utilizing NVIDIA graphics hardware:
+<code>
-<code bash>
+$ srun --partition=a40 --gres=gpu:1 --cpus-per-task=4 --time=00:30:00 --pty bash
-#!/bin/bash
-#SBATCH --job-name=vector_sum
-#SBATCH --output=vector_sum_%j.out
-#SBATCH --error=vector_sum_%j.err
-#SBATCH --nodes=1
-#SBATCH --ntasks=1
-#SBATCH --cpus-per-task=1
-#SBATCH --time=00:10:00
-#SBATCH --partition=unite
-echo "========================================="
-echo "SLURM Job Information"
-echo "========================================="
-echo "Job ID: $SLURM_JOB_ID"
-echo "Node: $SLURM_NODELIST"
-echo "Starting at: $(date)"
-echo ""
-echo "Python version:"
-python3 --version
-echo ""
-echo "Python executable location:"
-which python3
-echo ""
-echo "========================================="
-echo "Running vector_sum.py"
-echo "========================================="
-echo ""
-python3 vector_sum.py
-echo ""
-echo "========================================="
-echo "Job finished at: $(date)"
-echo "========================================="
-</code>
-----
-====Python program with dependencies====
-The following is a simple **Python** program which computes the sum of 2 vectors 3 times using **NumPy**.
-<code Python>
-#!/usr/bin/env python3
-import numpy as np
-import time
-def vector_addition(size=10000000):
-    print(f"Initializing vectors of size {size:,}...")
-    vector_a = np.random.rand(size)
-    vector_b = np.random.rand(size)
-    print("Performing vector addition...")
-    result = vector_a + vector_b
-    return result
-def main():
-    print("=" * 60)
-    print("Vector Addition Example using NumPy")
-    print("=" * 60)
-    sizes = [1000000, 10000000, 50000000]
-    for size in sizes:
-        result = vector_addition(size)
-        print(f"\nVector size: {size:,} elements")
-        print(f"First 5 elements of result: {result[:5]}")
-        print("-" * 60)
-if __name__ == "__main__":
-    main()
 </code>
+Once executed, Slurm places your terminal directly onto a secure, isolated compute shell. When your testing is complete, type ''exit'' to release the resources back to the queue pool.
-The following is the respective batch script for compiling and running the program. You can see the output of the program in the generated //vector_sum_numpy_%j.out// file. The script showcases 3 different ways for managing **Python** dependencies based on your use case. This is controlled through  the **PYTHON_ENV_METHOD** variable defined in the script. Please read the comments in the script for the configuration of the environment which you need to do on the login node. The dependency in the current example is **NumPy** but the approach for dependency management is generic.
+==== Workflow B: Batch Processing (Production Runs) ====
+For heavy workloads that take hours or days, write a batch script and submit it using ''sbatch''. Once submitted, you can disconnect your computer safely; Slurm handles execution and records outputs into text files on the storage array.
-<code bash>
+Here is a template structure for a production batch file (''run_job.sh''):
+<file bash run_job.sh>
 #!/bin/bash
-#SBATCH --job-name=vector_sum_numpy
+#SBATCH --job-name=unite_production_job
-#SBATCH --output=vector_sum_numpy_%j.out
+#SBATCH --partition=unite           # Target partition queue (e.g., unite, a40)
-#SBATCH --error=vector_sum_numpy_%j.err
+#SBATCH --output=logs_%j.out        # Output log text file (%j replaces with unique Job ID)
-#SBATCH --nodes=1
+#SBATCH --error=logs_%j.err         # Error tracking text file
-#SBATCH --ntasks=1
+#SBATCH --nodes=1                   # Number of distinct physical hardware nodes
-#SBATCH --cpus-per-task=1
+#SBATCH --ntasks=1                  # Number of execution application instances
-#SBATCH --time=00:15:00
+#SBATCH --cpus-per-task=4           # CPU worker core threads assigned to this task
-#SBATCH --partition=unite
+#SBATCH --mem=16G                   # Safe RAM allocation limit request
+#SBATCH --time=02:00:00             # Maximum safety runtime allowance (HH:MM:SS)
-################################################################################
+# 1. Clean the inherited terminal environment states
-# CONFIGURATION: Choose your Python environment method
+module purge
-################################################################################
-# Options: "venv", "conda", or "module"
-PYTHON_ENV_METHOD="venv"
-VENV_PATH="$HOME/venvs/numpy_env"
+# 2. Execute target execution logic or launch binaries
-CONDA_ENV_NAME="numpy_env"
+echo "Starting production workload execution on node: $(hostname)"
-CONDA_MODULE="anaconda3"
+# Run your commands here...
-PYTHON_MODULE="python/3.13"
+</file>
-NUMPY_MODULE="python/3.13/numpy/2.2.2"
-################################################################################
+To place this file into the cluster scheduler pipeline, execute:
-# Setup Instructions (run once on login node before first job submission)
+<code>
-################################################################################
+$ sbatch run_job.sh
-# For venv:
-#   python3 -m venv $HOME/venvs/numpy_env
-#   source $HOME/venvs/numpy_env/bin/activate
-#   pip install numpy
-#   deactivate
-#
-# For conda:
-#   module load anaconda3
-#   conda create -n numpy_env python=3.9 numpy
-#   conda deactivate
-#
-# For module:
-#   Check available modules: module avail python
-#   You need to load both python and numpy modules in the script. The numpy module needs to be compatible with the python module.
-#   Then you need to modify PYTHON_MODULE and NUMPY_MODULE variables above accordingly.
-################################################################################
-echo "========================================="
-echo "SLURM Job Information"
-echo "========================================="
-echo "Job ID: $SLURM_JOB_ID"
-echo "Node: $SLURM_NODELIST"
-echo "Starting at: $(date)"
-echo ""
-echo "Python environment method: $PYTHON_ENV_METHOD"
-echo ""
-if [ "$PYTHON_ENV_METHOD" = "venv" ]; then
-    echo "Activating Python virtual environment..."
-    if [ -f "$VENV_PATH/bin/activate" ]; then
-        source "$VENV_PATH/bin/activate"
-        echo "Virtual environment activated: $VENV_PATH"
-    else
-        echo "ERROR: Virtual environment not found at $VENV_PATH"
-        echo "Please create it first (see setup instructions in script)"
-        exit 1
-    fi
-elif [ "$PYTHON_ENV_METHOD" = "conda" ]; then
-    echo "Activating Conda environment..."
-    module load "$CONDA_MODULE"
-    source activate "$CONDA_ENV_NAME"
-    echo "Conda environment activated: $CONDA_ENV_NAME"
-elif [ "$PYTHON_ENV_METHOD" = "module" ]; then
-    echo "Loading environment modules..."
-    module load "$PYTHON_MODULE"
-    module load "$NUMPY_MODULE"
-    echo "Modules loaded: $PYTHON_MODULE, $NUMPY_MODULE"
-else
-    echo "ERROR: Invalid PYTHON_ENV_METHOD='$PYTHON_ENV_METHOD'"
-    echo "Valid options: venv, conda, module"
-    exit 1
-fi
-# Verify Python and NumPy
-echo ""
-echo "Python3 version:"
-python3 --version
-echo ""
-echo "NumPy version:"
-python3 -c "import numpy; print(f'NumPy {numpy.__version__}')"
-echo ""
-echo "Python executable location:"
-which python3
-echo ""
-echo "========================================="
-echo "Running vector_sum_numpy.py"
-echo "========================================="
-echo ""
-python3 vector_sum_numpy.py
-echo ""
-echo "Cleaning up environment..."
-if [ "$PYTHON_ENV_METHOD" = "venv" ]; then
-    deactivate
-    echo "Virtual environment deactivated"
-elif [ "$PYTHON_ENV_METHOD" = "conda" ]; then
-    conda deactivate
-    echo "Conda environment deactivated"
-elif [ "$PYTHON_ENV_METHOD" = "module" ]; then
-    # Modules are automatically unloaded when job ends
-    echo "Modules will be unloaded when job completes"
-fi
-echo ""
-echo "========================================="
-echo "Job finished at: $(date)"
-echo "========================================="
 </code>
-----
+---
-====C/C++ program with dependencies====
+===== 7. Reference Guides and Real-World Examples =====
-The following is a simple **C/C++** program which compresses and decompresses a string using **zLib**.
-<code C++>
+To see how to apply these Slurm parameters across different compiler systems, runtime environments, and compute nodes, follow our language-specific reference documentation pages:
-#include <stdio.h>
-#include <string.h>
-#include <zlib.h>
-#include <stdlib.h>
-#define CHUNK 16384
+  * **Native Applications (C++):**
+    * [[unite_cpp_gcc|Compiling C++ Applications with the GCC Module]] — Step-by-step instructions for running basic ''g++'' compilation steps via interactive nodes.
+    * [[unite_cpp_mpi|Compiling and Running Distributed C++ MPI Applications]] — How to request multi-node processing (''--nodes=2'', ''--ntasks-per-node=4'') and handle cross-node communication via InfiniBand fabrics.
+  * **GPU-Accelerated Data Science & ML (Python):**
+    * [[unite_python_cupy|Accelerated Mathematical Computations with CuPy]] — Leveraging GPU hardware tensors on the **unite** partition utilizing tracking hooks like ''--gres=gpu:1''.
+    * [[unite_python_torch|Deep Learning Model Frameworks with PyTorch]] — High-performance neural processing allocations specifically built to execute on the advanced **a40** hardware partition.
-int main() {
+---
-    const char *original = "Hello, this is a test string for zlib compression! "
-                          "We'll compress this text and then decompress it to verify it works.";
-    printf("Original string: %s\n", original);
+===== 8. Troubleshooting Common Queue States =====
-    printf("Original length: %lu bytes\n\n", strlen(original));
-    // Compression
+If you run ''squeue -u $USER'' and see your job sitting in a ''PD'' (Pending) state, look closely at the ''NODELIST(REASON)'' column:
-    uLong source_len = strlen(original) + 1;
-    uLong compressed_len = compressBound(source_len);
-    unsigned char *compressed = (unsigned char *)malloc(compressed_len);
-    if (compress(compressed, &compressed_len, (unsigned char *)original, source_len) != Z_OK) {
-        fprintf(stderr, "Compression failed!\n");
-        free(compressed);
-        return 1;
-    }
-    printf("Compressed length: %lu bytes\n", compressed_len);
-    printf("Compression ratio: %.2f%%\n\n", 100.0 * (1.0 - (double)compressed_len / source_len));
-    // Decompression
-    uLong decompressed_len = source_len;
-    unsigned char *decompressed = (unsigned char *)malloc(decompressed_len);
-    if (uncompress(decompressed, &decompressed_len, compressed, compressed_len) != Z_OK) {
-        fprintf(stderr, "Decompression failed!\n");
-        free(compressed);
-        free(decompressed);
-        return 1;
-    }
-    printf("Decompressed string: %s\n", decompressed);
-    printf("Decompressed length: %lu bytes\n\n", decompressed_len);
-    if (strcmp(original, (char *)decompressed) == 0) {
-        printf("SUCCESS: Original and decompressed strings match!\n");
-    } else {
-        printf("ERROR: Strings don't match!\n");
-    }
-    free(compressed);
-    free(decompressed);
-    return 0;
-}
-</code>
-The following is the respective batch script for compiling and running the program. You can see the output of the program in the generated //zlib_compress_%j.out// file. **C/C++** dependencies are generally compiled from source. If this is the case you can use the **Simple C/C++ program** example. The batch script showcases loading the dependent library on the system and linking against it. The cluster uses modules for managing the installed dependencies. Make sure to use compatible compiler and library.
-<code bash>
-#!/bin/bash
-#SBATCH --job-name=zlib_compress
-#SBATCH --output=zlib_compress_%j.out
-#SBATCH --error=zlib_compress_%j.err
-#SBATCH --time=00:05:00
-#SBATCH --nodes=1
-#SBATCH --ntasks=1
-#SBATCH --cpus-per-task=1
-#SBATCH --partition=unite
-module load gcc/latest
-module load zlib/1/latest-gcc
-echo "Loaded modules:"
-module list
-echo ""
-echo "Compiling zlib_compress.c..."
-gcc -o zlib_compress zlib_compress.c -lz
-if [ $? -eq 0 ]; then
-    echo "Compilation successful!"
-    echo ""
-    echo "Running the program:"
-    echo "===================="
-    ./zlib_compress
-else
-    echo "Compilation failed!"
-    exit 1
-fi
-</code>
-----
-====C++ program which uses MPI====
-The following is an example **C/C++** application which uses **MPI** to perform element-wise addition of two vectors. Each **MPI** task computes the addition of its local region and then sends it back to the leader. Using **MPI** with **Python** is similar assuming that you know how to manage **Python** dependencies on the cluster which is described in the previous section. What is important here is to understand how to manage the resources of the system.
-<code C++>
-#include <stdio.h>
-#include <stdlib.h>
-#include <mpi.h>
-#define VECTOR_SIZE 100000
-int main(int argc, char** argv) {
-    int rank, size;
-    int i;
-    MPI_Init(&argc, &argv);
-    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
-    MPI_Comm_size(MPI_COMM_WORLD, &size);
-    int local_size = VECTOR_SIZE / size;
-    int *local_a = (int*)malloc(local_size * sizeof(int));
-    int *local_b = (int*)malloc(local_size * sizeof(int));
-    int *local_c = (int*)malloc(local_size * sizeof(int));
-    int *a = NULL;
-    int *b = NULL;
-    int *c = NULL;
-    if (rank == 0) {
-        a = (int*)malloc(VECTOR_SIZE * sizeof(int));
-        b = (int*)malloc(VECTOR_SIZE * sizeof(int));
-        c = (int*)malloc(VECTOR_SIZE * sizeof(int));
-        for (i = 0; i < VECTOR_SIZE; i++) {
-            a[i] = i + 1;
-        }
-        for (i = 0; i < VECTOR_SIZE; i++) {
-            b[i] = (i + 1) * 2;
-        }
-    }
-    MPI_Scatter(a, local_size, MPI_INT, local_a, local_size, MPI_INT, 0, MPI_COMM_WORLD);
-    MPI_Scatter(b, local_size, MPI_INT, local_b, local_size, MPI_INT, 0, MPI_COMM_WORLD);
-    printf("Process %d: Adding %d elements\n", rank, local_size);
-    for (i = 0; i < local_size; i++) {
-        local_c[i] = local_a[i] + local_b[i];
-    }
-    MPI_Gather(local_c, local_size, MPI_INT, c, local_size, MPI_INT, 0, MPI_COMM_WORLD);
-    if (rank == 0) {
-        printf("\nFirst 5 elements of (A + B): ");
-        for (i = 0; i < 5; i++) {
-            printf("%d ", c[i]);
-        }
-        printf("\n");
-        free(a);
-        free(b);
-        free(c);
-    }
-    free(local_a);
-    free(local_b);
-    free(local_c);
-    MPI_Finalize();
-    return 0;
-}
-</code>
-The following is the respective batch script for compiling and running the program. You can see the output of the program in the generated //vector_sum_mpi_%j.out// file. The //ntasks// parameter of the batch script specifies the number of **MPI** tasks to be started. This is how you can leverage the resources of the system in order to increase the work done in parallel by your application. The **MPI** tasks are not guaranteed to be executed on different nodes in the cluster, they can also be on a separate physical cores. What is guaranteed is that every task will have the resources to execute in parallel.
-<code bash>
-#!/bin/bash
-#SBATCH --job-name=vector_sum_mpi
-#SBATCH --output=vector_mpi_%j.out
-#SBATCH --error=vector_mpi_%j.err
-#SBATCH --ntasks=4
-#SBATCH --time=00:05:00
-#SBATCH --partition=unite
-module load mpi/latest
-echo "Compiling vector_sum_mpi.c..."
-mpicc -o vector_sum_mpi vector_sum_mpi.c
-if [ $? -ne 0 ]; then
-    echo "Compilation failed!"
-    exit 1
-fi
-echo "Compilation successful!"
-echo "Running with $SLURM_NTASKS MPI processes..."
-echo "----------------------------------------"
-mpirun -np $SLURM_NTASKS ./vector_sum_mpi
-echo "----------------------------------------"
-echo "Job completed!"
-echo "----------------------------------------"
-</code>
-----
-====C++ program which uses multiple threads====
-The following is a simple **C++** program which computes the sum of 2 vectors. It uses multiple **threads**. Each **thread** computes the sum for its respective region.
-<code C++>
-#include <iostream>
-#include <vector>
-#include <thread>
-#define VECTOR_SIZE 100000
-void vector_add_worker(int thread_id, int start_idx, int end_idx,
-                       const int* a, const int* b, int* c) {
-    int elements = end_idx - start_idx;
-    std::cout << "Thread " << thread_id << ": Adding " << elements
-              << " elements" << std::endl;
-    for (int i = start_idx; i < end_idx; i++) {
-        c[i] = a[i] + b[i];
-    }
-}
-int main(int argc, char** argv) {
-    if (argc != 2) {
-        std::cerr << "Usage: " << argv[0] << " <number_of_threads>" << std::endl;
-        return 1;
-    }
-    int num_threads = std::atoi(argv[1]);
-    if (num_threads <= 0) {
-        std::cerr << "Error: Number of threads must be positive" << std::endl;
-        return 1;
-    }
-    std::cout << "Using " << num_threads << " threads" << std::endl;
-    std::vector<int> a(VECTOR_SIZE);
-    std::vector<int> b(VECTOR_SIZE);
-    std::vector<int> c(VECTOR_SIZE);
-    for (int i = 0; i < VECTOR_SIZE; i++) {
-        a[i] = i + 1;
-        b[i] = (i + 1) * 2;
-    }
-    int elements_per_thread = VECTOR_SIZE / num_threads;
-    std::vector<std::thread> threads;
-    for (unsigned int t = 0; t < num_threads; t++) {
-        int start_idx = t * elements_per_thread;
-        int end_idx = (t == num_threads - 1) ? VECTOR_SIZE : (t + 1) * elements_per_thread;
-        threads.emplace_back(vector_add_worker, t, start_idx, end_idx,
-                           a.data(), b.data(), c.data());
-    }
-    for (auto& thread : threads) {
-        thread.join();
-    }
-    std::cout << "\nFirst 5 elements of (A + B): ";
-    for (int i = 0; i < 5; i++) {
-        std::cout << c[i] << " ";
-    }
-    std::cout << std::endl;
-    return 0;
-}
-</code>
-The following is the respective batch script for compiling and running the program. You can see the output of the program in the generated //vector_sum_threads_%j.out// file. The //cpus-per-task// parameter of the batch script specifies the number of cores to be allocated for each task (**MPI** process). You can combine the use of **MPI** tasks and **threads** in order to start one process per node. Then each node can use multiple **threads** locally to do work in parallel, while the **threads** share the context of the process.
-<code bash>
-#!/bin/bash
-#SBATCH --job-name=vector_sum_threads
-#SBATCH --output=vector_sum_threads_%j.out
-#SBATCH --error=vector_sum_threads_%j.err
-#SBATCH --nodes=1
-#SBATCH --ntasks=1
-#SBATCH --cpus-per-task=8
-#SBATCH --time=00:05:00
-#SBATCH --partition=unite
-echo "Job started at: $(date)"
-echo "Running on node: $(hostname)"
-echo "Number of CPUs allocated: $SLURM_CPUS_PER_TASK"
-echo "----------------------------------------"
-module load gcc
-echo "Compiling vector_sum_threads.cpp..."
-g++ -std=c++11 -pthread -O3 vector_sum_threads.cpp -o vector_sum_threads
-if [ $? -eq 0 ]; then
-    echo "Compilation successful!"
-    echo "----------------------------------------"
-    echo "Running vector_sum_threads with $SLURM_CPUS_PER_TASK threads..."
-    ./vector_sum_threads $SLURM_CPUS_PER_TASK
-    echo "----------------------------------------"
-    echo "Job finished at: $(date)"
-else
-    echo "Compilation failed!"
-    exit 1
-fi
-</code>
-----
-====C++ program which uses GPU====
-The following is an example **Cuda** application which uses **Nvidia GPU** to perform element-wise addition of two vectors. Using **Cuda** with **Python** is similar assuming that you know how to manage **Python** dependencies on the cluster which is described in a previous section. What is important here is to understand how to manage the resources of the system.
-<code C++>
-#include <stdio.h>
-#include <stdlib.h>
-#include <cuda_runtime.h>
-#include <sys/time.h>
-#define CUDA_CHECK(call) \
-    do { \
-        cudaError_t error = call; \
-        if (error != cudaSuccess) { \
-            fprintf(stderr, "CUDA Error: %s:%d, %s\n", __FILE__, __LINE__, \
-                    cudaGetErrorString(error)); \
-            exit(EXIT_FAILURE); \
-        } \
-    } while(0)
-/*
- * CUDA kernel for vector addition
- * Each thread computes one element of the result vector
- *
- * Parameters:
- *   a: First input vector
- *   b: Second input vector
- *   c: Output vector (result)
- *   n: Number of elements
- */
-__global__ void vectorAddKernel(const float *a, const float *b, float *c, int n) {
-    // Calculate global thread ID
-    int idx = blockIdx.x * blockDim.x + threadIdx.x;
-    // Check if thread is within bounds
-    if (idx < n) {
-        c[idx] = a[idx] + b[idx];
-    }
-}
-int main() {
-    const int N = 50'000'000;
-    const size_t bytes = N * sizeof(float);
-    printf("========================================\n");
-    printf("CUDA Vector Addition Example\n");
-    printf("========================================\n");
-    printf("Vector size: %d elements\n", N);
-    int deviceId;
-    cudaDeviceProp props;
-    CUDA_CHECK(cudaGetDevice(&deviceId));
-    CUDA_CHECK(cudaGetDeviceProperties(&props, deviceId));
-    printf("\nGPU Information:\n");
-    printf("  Device: %s\n", props.name);
-    printf("  Compute Capability: %d.%d\n", props.major, props.minor);
-    printf("  Total Global Memory: %.2f GB\n",
-           props.totalGlobalMem / (1024.0 * 1024.0 * 1024.0));
-    printf("  Multiprocessors: %d\n", props.multiProcessorCount);
-    printf("  Max Threads per Block: %d\n", props.maxThreadsPerBlock);
-    printf("  Warp Size: %d\n", props.warpSize);
-    printf("\nAllocating host memory...\n");
-    float *h_a = (float *)malloc(bytes);
-    float *h_b = (float *)malloc(bytes);
-    float *h_c_gpu = (float *)malloc(bytes);
-    float *h_c_cpu = (float *)malloc(bytes);
-    if (!h_a || !h_b || !h_c_gpu || !h_c_cpu) {
-        fprintf(stderr, "Error: Host memory allocation failed!\n");
-        return 1;
-    }
-    for (int i = 0; i < N; i++) {
-        h_a[i] = (float)rand() / RAND_MAX;
-        h_b[i] = (float)rand() / RAND_MAX;
-    }
-    float *d_a, *d_b, *d_c;
-    CUDA_CHECK(cudaMalloc(&d_a, bytes));
-    CUDA_CHECK(cudaMalloc(&d_b, bytes));
-    CUDA_CHECK(cudaMalloc(&d_c, bytes));
-    double transfer_start = getTime();
-    CUDA_CHECK(cudaMemcpy(d_a, h_a, bytes, cudaMemcpyHostToDevice));
-    CUDA_CHECK(cudaMemcpy(d_b, h_b, bytes, cudaMemcpyHostToDevice));
-    int threadsPerBlock = 256;
-    int blocksPerGrid = (N + threadsPerBlock - 1) / threadsPerBlock;
-    printf("\nKernel Configuration:\n");
-    printf("  Threads per block: %d\n", threadsPerBlock);
-    printf("  Blocks per grid: %d\n", blocksPerGrid);
-    printf("  Total threads: %d\n", blocksPerGrid * threadsPerBlock);
-    vectorAddKernel<<<blocksPerGrid, threadsPerBlock>>>(d_a, d_b, d_c, N);
-    CUDA_CHECK(cudaMemcpy(h_c_gpu, d_c, bytes, cudaMemcpyDeviceToHost));
-    printf("\nFirst 5 elements of result:\n");
-    for (int i = 0; i < 5; i++) {
-        printf("  c[%d] = %.6f\n", i, h_c_gpu[i]);
-    }
-    CUDA_CHECK(cudaFree(d_a));
-    CUDA_CHECK(cudaFree(d_b));
-    CUDA_CHECK(cudaFree(d_c));
-    CUDA_CHECK(cudaEventDestroy(start));
-    CUDA_CHECK(cudaEventDestroy(stop));
-    free(h_a);
-    free(h_b);
-    free(h_c_gpu);
-    free(h_c_cpu);
-    return 0;
-}
-</code>
-The following is the respective batch script for compiling and running the program. You can see the output of the program in the generated //vector_sum_cuda_%j.out// file. The //gres// parameter of the batch script specifies the number of **GPUs** to be allocated in total. You can combine the use of **MPI** tasks and **GPUs** in order to start one process per node. Then each node can use multiple **GPUs** locally to do work in parallel.
-<code bash>
-#!/bin/bash
-#SBATCH --job-name=vector_sum_cuda
-#SBATCH --output=vector_sum_cuda_%j.out
-#SBATCH --error=vector_sum_cuda_%j.err
-#SBATCH --nodes=1
-#SBATCH --ntasks=1
-#SBATCH --cpus-per-task=1
-#SBATCH --gres=gpu:1
-#SBATCH --time=00:10:00
-#SBATCH --partition=unite
-echo "========================================="
-echo "SLURM Job Information"
-echo "========================================="
-echo "Job ID: $SLURM_JOB_ID"
-echo "Node: $SLURM_NODELIST"
-echo "GPU(s): $SLURM_GPUS_ON_NODE"
-echo "Starting at: $(date)"
-echo ""
-module load nvidia/cuda/12-latest
-echo "Compiling vector_sum_cuda_cuda.cu..."
-nvcc -O3 -o vector_sum_cuda vector_sum_cuda.cu
-if [ $? -ne 0 ]; then
-    echo "Error: Compilation failed!"
-    exit 1
-fi
-echo "Compilation successful!"
-echo ""
-echo "Running vector_sum_cuda..."
-./vector_sum_cuda
-echo ""
-echo "Job finished at: $(date)"
-</code>
-----
+  * **(Resources):** The cluster is fully utilized right now. Slurm will launch your job automatically as soon as another user's time allocation expires.
+  * **(Priority):** Higher priority workloads (based on user fair-share rules) are currently waiting ahead of you in line.
+  * **(PartitionTimeLimit):** Your requested ''#SBATCH --time'' configuration parameter exceeds the maximum execution ceiling value allowed for that specific partition.
+  * **(InvalidQOS):** You requested a resource combination (like asking for an A40 GPU inside a non-GPU queue) that violates partition definition policies.