Differences

This shows you the differences between two versions of the page.

--- examples [2025/12/29 20:57] – [MPI] dimitar
+++ examples [2026/06/13 15:24] (current) – nshegunov
@@ Line 1: / Line 1: @@
-====MPI4PI====
+====== Guide to SLURM Resource Management and Job Submission ======
-TOD
-----
+This guide provides an advanced overview of the **SLURM (Simple Linux Utility for Resource Management)** workload manager and the underlying physical architecture of the UNITE HPC cluster. It covers theoretical network topologies, inter-node communication, deep hardware utilization mechanics, user workflows, and best practices for parallel execution.
-==== PyTorch====
-Consider the following simple python test script( "pytorch_test.py"):
-<code python>
-import torch
-def test_pytorch():
+===== 1. Theoretical Overview & Cluster Architecture =====
-    print("PyTorch version:", torch.__version__)
-    print("CUDA available:", torch.cuda.is_available())
-    if torch.cuda.is_available():
-        print("CUDA device:", torch.cuda.get_device_name(0))
-        device = torch.device("cuda")
-    else:
-        device = torch.device("cpu")
-    # Simple tensor operation
-    x = torch.tensor([1.0, 2.0, 3.0], device=device)
-    y = torch.tensor([4.0, 5.0, 6.0], device=device)
-    z = x + y
-    print("Tensor operation result:", z)
-test_pytorch()
+An HPC cluster is a massive parallel computing instrument built from a collection of individual commodity servers called **Nodes**, interconnected by an ultra-fast, low-latency network fabric. Slurm isolates users from this underlying hardware complexity while ensuring optimal resource distribution.
-</code>
-To test it on the unite cluster you can use the folling sbatch scrpit to run it:
+Slurm operates under a master-worker orchestration framework managed by three primary daemons:
-<code bash>
-#!/bin/bash
-#SBATCH --job-name=pytorch_test
-#SBATCH --output=pytorch_test.out
-#SBATCH --error=pytorch_test.err
-#SBATCH --time=00:10:00
-#SBATCH --partition=a40
-#SBATCH --gres=gpu:1
-#SBATCH --mem=4G
-#SBATCH --cpus-per-task=2
-# Load necessary modules (modify based on your system)
+  * ''slurmctld'' (Central Controller Daemon): Runs on the head/login architecture. It orchestrates the global job state, assesses queue priorities, processes resource requests, and enforces scheduling fairness policies.
-module load python/pytorch-2.5.1-llvm-cuda-12.3-python-3.13.1-llvm
+  * ''slurmd'' (Compute Node Daemon): Monitored natively on every individual compute node. It accepts operational instructions from the central ''slurmctld'', provisions local hardware threads, isolates local memory namespaces, spawns your process steps, and returns exit codes.
+  * ''slurmdbd'' (Database Daemon): Explicitly records cluster accounting metrics, resource tracking logs, project computing hours allocations, and fair-share consumption records.
-# Activate your virtual environment if needed
+==== Resource Hierarchies ====
-# source ~/your_env/bin/activate
+When submitting a workload to Slurm, it is crucial to understand how resource requests translate into physical hardware allocations:
-# Run the PyTorch script
+  * **Cluster:** The entire collective pool of computing hardware, networking switches, and storage layers.
-python3.13 pytorch_test.py
+  * **Partition:** A logical grouping of specific nodes (e.g., vanilla compute nodes vs. high-performance GPU nodes like the ''a40'' pool).
+  * **Node:** A single distinct physical server chassis possessing localized processors, volatile system memory (RAM), and explicit hardware buses.
+  * **Core/Socket:** The physical computational units inside a CPU.
+  * **Task:** An independent process instance. In distributed execution (like MPI), 1 Task generally maps to 1 unique process loop.
+  * **CPUs-per-task:** The number of hardware threads or CPU cores dedicated to backing a single task (critical for shared-memory multithreading models like OpenMP).
-</code>
+==== Deep Dive: CPU and Memory Architecture (NUMA) ====
-----
+Modern compute nodes are not monolithic. A single node typically contains two or more CPU **sockets** (physical chips). Memory (RAM) is physically divided and attached directly to specific sockets. This is known as **NUMA (Non-Uniform Memory Access)**.
-====Pandas====
-Consider the following simple python test script( “pandas_test.py”):
-<code python>
-import pandas as pd
-import numpy as np
-# Create a simple DataFrame
+  * **Local Memory Access:** If a program running on CPU Socket 0 requests data stored in the RAM physically attached to Socket 0, the access is nearly instantaneous.
-data = {
+  * **Remote Memory Access:** If a program running on Socket 0 requests data stored in the RAM attached to Socket 1, the data must travel across an interconnect bus on the motherboard. This introduces latency and bandwidth bottlenecks.
-    'A': [1, 2, 3, 4],
-    'B': [5, 6, 7, 8],
-    'C': [9, 10, 11, 12]
-}
-df = pd.DataFrame(data)
-print("Original DataFrame:")
-print(df)
-# Test basic operations
+**Why this matters for Slurm:** When you request resources, Slurm attempts to bind your tasks to specific cores and local memory banks (CPU affinity/pinning) to avoid crossing NUMA boundaries. Requesting scattered resources loosely can result in your application constantly fetching data across the motherboard, severely degrading performance.
-print("\nSum of each column:")
-print(df.sum())
-print("\nMean of each column:")
+---
-print(df.mean())
-# Adding a new column
+===== 2. Advanced Cluster Topology and Inter-Node Communication =====
-df['D'] = df['A'] + df['B']
-print("\nDataFrame after adding new column D (A + B):")
-print(df)
-# Filtering rows
+To write optimal code for an HPC cluster, developers must account for how components talk to one another. Communication speeds and delays (**latency**) change drastically based on where your data sits and where it needs to go.
-filtered_df = df[df['A'] > 2]
-print("\nFiltered DataFrame (A > 2):")
-print(filtered_df)
-# Check if NaN values exist
+==== Memory and Communication Layout ====
-print("\nCheck for NaN values:")
+^ Execution Scope ^ Communication Medium ^ Relative Bandwidth ^ Latency Complexity ^
-print(df.isna().sum())
+| **Intra-Core** (Same CPU) | L1 / L2 / L3 Processor Cache | Ultra-High (Terabytes/sec) | Lowest (Nanoseconds) |
-</code>
+| **Intra-Node** (Same Server) | Physical RAM & PCIe Bus / NVLink | Very High (Hundreds of GB/s) | Low |
+| **Inter-Node** (Across Servers) | InfiniBand Network Fabric / Switches | High (Hundreds of Gbps) | Microseconds (Fabric Transit) |
-You can use the following snatch script to run it:
+==== Network Topologies: Fat-Tree Architecture ====
-<code bash>
+HPC networks like the one powering the UNITE cluster avoid standard corporate network setups (which bottleneck under heavy loads) in favor of a specialized **Fat-Tree Topology**.
-#!/bin/bash
-#SBATCH --job-name=pytorch_test
-#SBATCH --output=pytorch_test.out
-#SBATCH --error=pytorch_test.err
-#SBATCH --time=00:10:00
-#SBATCH --partition=a40
-#SBATCH --gres=gpu:1
-#SBATCH --mem=4G
-#SBATCH --cpus-per-task=2
-# Load necessary modules (modify based on your system)
+  * **Non-Blocking Fabric:** In a standard IT tree, links get narrower as you go up toward the core switches, causing traffic jams. A **Fat-Tree topology** does the opposite: the network connections multiply and get thicker (more bandwidth) closer to the top core switches.
-module load python/3.13.1-llvm
+  * **Full Bisection Bandwidth:** This ensures that if half the compute nodes on the cluster are actively transmitting data to the other half at the exact same time, the network switches won't bottleneck or drop packets.
-module load python/3.13/pandas/2.2.3
-# Activate your virtual environment if needed
+==== Deep Dive: InfiniBand & OS Kernel Bypass ====
-# source ~/your_env/bin/activate
+When your code spans across multiple physical nodes (e.g., an MPI job), it bypasses standard slow Ethernet connections. Ethernet requires the operating system's kernel (TCP/IP stack) to package, verify, and route every single byte, which consumes massive CPU cycles and creates latency.
-# Run the PyTorch script
+The UNITE cluster utilizes **InfiniBand Architecture (IBA)** and **RDMA (Remote Direct Memory Access)**:
-python3.13 pandas_test.py
-</code>
-----
-====Simple C/C++ program====
-The following is a simple **C/C++** program which performs element-wise addition of 2 vectors. It does **not** use any dependent libraries:
-<code C>
+  * **Zero-Copy Networking:** RDMA allows the network interface card (NIC) on Node A to read data directly out of its local RAM and push it directly into the RAM of Node B.
-#include <stdio.h>
+  * **Kernel Bypass:** Neither the CPU nor the operating system on Node A or Node B is involved in the transfer. This drops network latency down to single-digit microseconds (<2us), allowing distributed applications to scale linearly.
-#include <stdlib.h>
-#include <time.h>
-#include <sys/time.h>
-/*
+==== GPU Interconnects (PCIe vs. NVLink) ====
- * Perform element-wise addition of two vectors
+On GPU partitions (like the ''a40'' pool), how GPUs talk to each other is critical.
- *
- * Parameters:
- *   a: First input vector
- *   b: Second input vector
- *   result: Output vector (a + b)
- *   size: Number of elements in vectors
- */
-void vector_addition(const double *a, const double *b, double *result, size_t size) {
-    for (size_t i = 0; i < size; i++) {
-        result[i] = a[i] + b[i];
-    }
-}
-int main() {
+  * **PCIe Bus:** If two GPUs must communicate via the standard motherboard PCIe bus, bandwidth is limited (typically 32-64 GB/s) and latency increases because data must pass through the CPU.
-    const size_t size = 10000000;
+  * **NVLink:** High-end cluster architectures utilize NVIDIA NVLink bridges, allowing GPUs to share memory pools directly at massive speeds (hundreds of GB/s). When writing distributed PyTorch or CuPy scripts across multiple GPUs on the same node, the framework relies on this hardware topology.
-    printf("========================================\n");
+---
-    printf("Vector Addition Example in C\n");
-    printf("========================================\n");
-    printf("Vector size: %zu elements\n", size);
-    printf("\nAllocating memory...\n");
+===== 3. Scheduler Theory: How Slurm Assigns Resources =====
-    double *vector_a = (double *)malloc(size * sizeof(double));
-    double *vector_b = (double *)malloc(size * sizeof(double));
-    double *result = (double *)malloc(size * sizeof(double));
-    if (vector_a == NULL || vector_b == NULL || result == NULL) {
+Slurm does not simply process jobs sequentially. It uses complex algorithms to maximize cluster utilization and ensure fairness.
-        fprintf(stderr, "Error: Memory allocation failed!\n");
-        return 1;
-    }
-    printf("Initializing vectors...\n");
+  * **Fair-Share Scheduling:** The priority of your job is dynamically calculated based on historical usage. If you have been running massive jobs all week, your priority score drops to allow other researchers a chance to run their code.
-    srand(time(NULL));
+  * **Backfilling Algorithm:** Large jobs often have to wait for enough nodes to become free simultaneously. This creates temporary "holes" in the cluster's utilization. Slurm will actively scan the queue for smaller, shorter jobs and "backfill" them into these holes, provided they will finish before the large job's resources are fully gathered.
-    for (size_t i = 0; i < size; i++) {
+    * ''Best Practice:'' **Always accurately estimate your #SBATCH --time limit.** If you request 24 hours for a job that takes 1 hour, Slurm cannot use your job for backfilling. If you request 1 hour, Slurm can easily squeeze your job into an upcoming idle gap, significantly reducing your wait time.
-        vector_a[i] = (double)rand() / RAND_MAX;
-        vector_b[i] = (double)rand() / RAND_MAX;
-    }
-    printf("Performing vector addition...\n");
+---
-    vector_addition(vector_a, vector_b, result, size);
-    printf("First 5 elements of result:\n");
+===== 4. Crucial Cluster Etiquette: The Shared Environment =====
-    for (int i = 0; i < 5; i++) {
-        printf("  result[%d] = %.6f\n", i, result[i]);
-    }
-    free(vector_a);
+^ WARNING: NEVER RUN COMPUTATIONS ON THE LOGIN NODE! ^
-    free(vector_b);
+| When you connect to the UNITE cluster via SSH, you land directly on the **Login Node**. This node's sole purpose is file management, code editing, and checking job queues. Heavy processes like compilation, heavy Python tasks, or model training will slow down the login node for everyone on the cluster. Running intensive code here may result in administrators forcefully terminating your session. |
-    free(result);
-    return 0;
+To execute code, you **must** pass your workloads over to a dedicated compute node using either **Interactive Sessions** or **Batch Job Submissions**.
-}
-</code>
-The following is the respective batch script for compiling and running the program. You can see the output of the program in the generated //vector_sum_%j.out// file.
+---
-<code bash>
+===== 5. Key Slurm Commands Workflow =====
-#!/bin/bash
-#SBATCH --job-name=vector_sum
-#SBATCH --output=vector_sum_%j.out
-#SBATCH --error=vector_sum_%j.err
-#SBATCH --nodes=1
-#SBATCH --ntasks=1
-#SBATCH --cpus-per-task=1
-#SBATCH --time=00:10:00
-#SBATCH --partition=unite
-echo "========================================="
+Interact with the scheduler fabric using these primary terminal operations:
-echo "SLURM Job Information"
-echo "========================================="
-echo "Job ID: $SLURM_JOB_ID"
-echo "Node: $SLURM_NODELIST"
-echo "Starting at: $(date)"
-echo ""
-# Load necessary modules
+^ Command ^ Action Description ^ Typical Use Case ^
-module load gcc
+| ''sinfo'' | Queries the state of partitions and available physical cluster hardware nodes. | Checking which queues are free or down. |
+| ''srun'' | Allocates and runs parallel tasks. Can open an active interactive terminal session. | Step-by-step code testing and compilation. |
+| ''sbatch'' | Enqueues a non-interactive shell script to run autonomously in the background. | Production training and massive compute jobs. |
+| ''squeue'' | Inspects current active allocations and waiting jobs in the scheduler pipelines. | Checking your queue position or job status. |
+| ''scancel <ID>'' | Immediately terminates a queued or running job matching the specific ID. | Aborting an infinite loop or mistaken setup. |
-# Compile the program
+---
-echo "Compiling vector_sum.c..."
-gcc -O3 -march=native -o vector_sum vector_sum.c -lm
-if [ $? -ne 0 ]; then
+===== 6. Interactive Development vs. Non-Interactive Batch =====
-    echo "Error: Compilation failed!"
-    exit 1
-fi
-echo "Compilation successful!"
+==== Workflow A: Interactive Sessions (Testing & Compilation) ====
-echo ""
+To compile libraries, debug scripts line-by-line, or test packages safely on a real compute node, use an interactive resource request via ''srun''.
-echo "Running vector_sum..."
+For a standard multi-purpose compute slice:
-./vector_sum
+<code>
+$ srun --partition=unite --cpus-per-task=4 --time=00:30:00 --pty bash
-echo ""
-echo "Job finished at: $(date)"
 </code>
-----
-====Simple Python program====
-The following is a simple **Python** program which performs element-wise addition of 2 vectors. It does **not** use any dependent libraries:
-<code Python>
-#!/usr/bin/env python3
-import random
-import time
-def vector_addition(a, b):
-    """
-    Perform element-wise addition of two vectors
-    Parameters:
-        a: First input vector (list)
-        b: Second input vector (list)
-    Returns:
-        result: Output vector (a + b)
-    """
-    return [a[i] + b[i] for i in range(len(a))]
-def main():
-    size = 10000000
-    print("=" * 40)
-    print("Vector Addition Example in Python")
-    print("=" * 40)
-    print(f"Vector size: {size:,} elements")
-    print("\nAllocating and initializing vectors...")
-    random.seed(time.time())
-    vector_a = [random.random() for _ in range(size)]
-    vector_b = [random.random() for _ in range(size)]
-    print("Performing vector addition...")
-    result = vector_addition(vector_a, vector_b)
-    print("\nFirst 5 elements of result:")
-    for i in range(5):
-        print(f"  result[{i}] = {result[i]:.6f}")
-if __name__ == "__main__":
+For a dedicated deep learning workspace utilizing NVIDIA graphics hardware:
-    main()
+<code>
+$ srun --partition=a40 --gres=gpu:1 --cpus-per-task=4 --time=00:30:00 --pty bash
 </code>
+Once executed, Slurm places your terminal directly onto a secure, isolated compute shell. When your testing is complete, type ''exit'' to release the resources back to the queue pool.
-The following is the respective batch script for running the program. You can see the output of the program in the generated //vector_sum_%j.out// file.
+==== Workflow B: Batch Processing (Production Runs) ====
+For heavy workloads that take hours or days, write a batch script and submit it using ''sbatch''. Once submitted, you can disconnect your computer safely; Slurm handles execution and records outputs into text files on the storage array.
-<code bash>
+Here is a template structure for a production batch file (''run_job.sh''):
+<file bash run_job.sh>
 #!/bin/bash
-#SBATCH --job-name=vector_sum
+#SBATCH --job-name=unite_production_job
-#SBATCH --output=vector_sum_%j.out
+#SBATCH --partition=unite           # Target partition queue (e.g., unite, a40)
-#SBATCH --error=vector_sum_%j.err
+#SBATCH --output=logs_%j.out        # Output log text file (%j replaces with unique Job ID)
-#SBATCH --nodes=1
+#SBATCH --error=logs_%j.err         # Error tracking text file
-#SBATCH --ntasks=1
+#SBATCH --nodes=1                   # Number of distinct physical hardware nodes
-#SBATCH --cpus-per-task=1
+#SBATCH --ntasks=1                  # Number of execution application instances
-#SBATCH --time=00:10:00
+#SBATCH --cpus-per-task=4           # CPU worker core threads assigned to this task
-#SBATCH --partition=unite
+#SBATCH --mem=16G                   # Safe RAM allocation limit request
+#SBATCH --time=02:00:00             # Maximum safety runtime allowance (HH:MM:SS)
-echo "========================================="
+# 1. Clean the inherited terminal environment states
-echo "SLURM Job Information"
+module purge
-echo "========================================="
-echo "Job ID: $SLURM_JOB_ID"
-echo "Node: $SLURM_NODELIST"
-echo "Starting at: $(date)"
-echo ""
-echo "Python version:"
+# 2. Execute target execution logic or launch binaries
-python3 --version
+echo "Starting production workload execution on node: $(hostname)"
-echo ""
+# Run your commands here...
+</file>
-echo "Python executable location:"
+To place this file into the cluster scheduler pipeline, execute:
-which python3
+<code>
-echo ""
+$ sbatch run_job.sh
-echo "========================================="
-echo "Running vector_sum.py"
-echo "========================================="
-echo ""
-python3 vector_sum.py
-echo ""
-echo "========================================="
-echo "Job finished at: $(date)"
-echo "========================================="
 </code>
-----
+---
-====Python program with dependencies====
+===== 7. Reference Guides and Real-World Examples =====
-The following is a simple **Python** program which computes the sum of 2 vectors 3 times using **NumPy**.
-<code Python>
+To see how to apply these Slurm parameters across different compiler systems, runtime environments, and compute nodes, follow our language-specific reference documentation pages:
-#!/usr/bin/env python3
-import numpy as np
-import time
-def vector_addition(size=10000000):
+  * **Native Applications (C++):**
-    print(f"Initializing vectors of size {size:,}...")
+    * [[unite_cpp_gcc|Compiling C++ Applications with the GCC Module]] — Step-by-step instructions for running basic ''g++'' compilation steps via interactive nodes.
+    * [[unite_cpp_mpi|Compiling and Running Distributed C++ MPI Applications]] — How to request multi-node processing (''--nodes=2'', ''--ntasks-per-node=4'') and handle cross-node communication via InfiniBand fabrics.
+  * **GPU-Accelerated Data Science & ML (Python):**
+    * [[unite_python_cupy|Accelerated Mathematical Computations with CuPy]] — Leveraging GPU hardware tensors on the **unite** partition utilizing tracking hooks like ''--gres=gpu:1''.
+    * [[unite_python_torch|Deep Learning Model Frameworks with PyTorch]] — High-performance neural processing allocations specifically built to execute on the advanced **a40** hardware partition.
-    vector_a = np.random.rand(size)
+---
-    vector_b = np.random.rand(size)
-    print("Performing vector addition...")
+===== 8. Troubleshooting Common Queue States =====
-    result = vector_a + vector_b
-    return result
+If you run ''squeue -u $USER'' and see your job sitting in a ''PD'' (Pending) state, look closely at the ''NODELIST(REASON)'' column:
-def main():
-    print("=" * 60)
-    print("Vector Addition Example using NumPy")
-    print("=" * 60)
-    sizes = [1000000, 10000000, 50000000]
-    for size in sizes:
-        result = vector_addition(size)
-        print(f"\nVector size: {size:,} elements")
-        print(f"First 5 elements of result: {result[:5]}")
-        print("-" * 60)
-if __name__ == "__main__":
-    main()
-</code>
-The following is the respective batch script for compiling and running the program. You can see the output of the program in the generated //vector_sum_numpy_%j.out// file. The script showcases 3 different ways for managing **Python** dependencies based on your use case. This is controlled through  the **PYTHON_ENV_METHOD** variable defined in the script. Please read the comments in the script for the configuration of the environment which you need to do on the login node. The dependency in the current example is **NumPy** but the approach for dependency management is generic.
-<code bash>
-#!/bin/bash
-#SBATCH --job-name=vector_sum_numpy
-#SBATCH --output=vector_sum_numpy_%j.out
-#SBATCH --error=vector_sum_numpy_%j.err
-#SBATCH --nodes=1
-#SBATCH --ntasks=1
-#SBATCH --cpus-per-task=1
-#SBATCH --time=00:15:00
-#SBATCH --partition=unite
-################################################################################
-# CONFIGURATION: Choose your Python environment method
-################################################################################
-# Options: "venv", "conda", or "module"
-PYTHON_ENV_METHOD="venv"
-VENV_PATH="$HOME/venvs/numpy_env"
-CONDA_ENV_NAME="numpy_env"
-CONDA_MODULE="anaconda3"
-PYTHON_MODULE="python/3.13"
-NUMPY_MODULE="python/3.13/numpy/2.2.2"
-################################################################################
-# Setup Instructions (run once on login node before first job submission)
-################################################################################
-# For venv:
-#   python3 -m venv $HOME/venvs/numpy_env
-#   source $HOME/venvs/numpy_env/bin/activate
-#   pip install numpy
-#   deactivate
-#
-# For conda:
-#   module load anaconda3
-#   conda create -n numpy_env python=3.9 numpy
-#   conda deactivate
-#
-# For module:
-#   Check available modules: module avail python
-#   You need to load both python and numpy modules in the script. The numpy module needs to be compatible with the python module.
-#   Then you need to modify PYTHON_MODULE and NUMPY_MODULE variables above accordingly.
-################################################################################
-echo "========================================="
-echo "SLURM Job Information"
-echo "========================================="
-echo "Job ID: $SLURM_JOB_ID"
-echo "Node: $SLURM_NODELIST"
-echo "Starting at: $(date)"
-echo ""
-echo "Python environment method: $PYTHON_ENV_METHOD"
-echo ""
-if [ "$PYTHON_ENV_METHOD" = "venv" ]; then
-    echo "Activating Python virtual environment..."
-    if [ -f "$VENV_PATH/bin/activate" ]; then
-        source "$VENV_PATH/bin/activate"
-        echo "Virtual environment activated: $VENV_PATH"
-    else
-        echo "ERROR: Virtual environment not found at $VENV_PATH"
-        echo "Please create it first (see setup instructions in script)"
-        exit 1
-    fi
-elif [ "$PYTHON_ENV_METHOD" = "conda" ]; then
-    echo "Activating Conda environment..."
-    module load "$CONDA_MODULE"
-    source activate "$CONDA_ENV_NAME"
-    echo "Conda environment activated: $CONDA_ENV_NAME"
-elif [ "$PYTHON_ENV_METHOD" = "module" ]; then
-    echo "Loading environment modules..."
-    module load "$PYTHON_MODULE"
-    module load "$NUMPY_MODULE"
-    echo "Modules loaded: $PYTHON_MODULE, $NUMPY_MODULE"
-else
-    echo "ERROR: Invalid PYTHON_ENV_METHOD='$PYTHON_ENV_METHOD'"
-    echo "Valid options: venv, conda, module"
-    exit 1
-fi
-# Verify Python and NumPy
-echo ""
-echo "Python3 version:"
-python3 --version
-echo ""
-echo "NumPy version:"
-python3 -c "import numpy; print(f'NumPy {numpy.__version__}')"
-echo ""
-echo "Python executable location:"
-which python3
-echo ""
-echo "========================================="
-echo "Running vector_sum_numpy.py"
-echo "========================================="
-echo ""
-python3 vector_sum_numpy.py
-echo ""
-echo "Cleaning up environment..."
-if [ "$PYTHON_ENV_METHOD" = "venv" ]; then
-    deactivate
-    echo "Virtual environment deactivated"
-elif [ "$PYTHON_ENV_METHOD" = "conda" ]; then
-    conda deactivate
-    echo "Conda environment deactivated"
-elif [ "$PYTHON_ENV_METHOD" = "module" ]; then
-    # Modules are automatically unloaded when job ends
-    echo "Modules will be unloaded when job completes"
-fi
-echo ""
-echo "========================================="
-echo "Job finished at: $(date)"
-echo "========================================="
-</code>
-----
-====C/C++ program with dependencies====
-The following is a simple **C/C++** program which compresses and decompresses a string using **zLib**.
-<code C++>
-#include <stdio.h>
-#include <string.h>
-#include <zlib.h>
-#include <stdlib.h>
-#define CHUNK 16384
-int main() {
-    const char *original = "Hello, this is a test string for zlib compression! "
-                          "We'll compress this text and then decompress it to verify it works.";
-    printf("Original string: %s\n", original);
-    printf("Original length: %lu bytes\n\n", strlen(original));
-    // Compression
-    uLong source_len = strlen(original) + 1;
-    uLong compressed_len = compressBound(source_len);
-    unsigned char *compressed = (unsigned char *)malloc(compressed_len);
-    if (compress(compressed, &compressed_len, (unsigned char *)original, source_len) != Z_OK) {
-        fprintf(stderr, "Compression failed!\n");
-        free(compressed);
-        return 1;
-    }
-    printf("Compressed length: %lu bytes\n", compressed_len);
-    printf("Compression ratio: %.2f%%\n\n", 100.0 * (1.0 - (double)compressed_len / source_len));
-    // Decompression
-    uLong decompressed_len = source_len;
-    unsigned char *decompressed = (unsigned char *)malloc(decompressed_len);
-    if (uncompress(decompressed, &decompressed_len, compressed, compressed_len) != Z_OK) {
-        fprintf(stderr, "Decompression failed!\n");
-        free(compressed);
-        free(decompressed);
-        return 1;
-    }
-    printf("Decompressed string: %s\n", decompressed);
-    printf("Decompressed length: %lu bytes\n\n", decompressed_len);
-    if (strcmp(original, (char *)decompressed) == 0) {
-        printf("SUCCESS: Original and decompressed strings match!\n");
-    } else {
-        printf("ERROR: Strings don't match!\n");
-    }
-    free(compressed);
-    free(decompressed);
-    return 0;
-}
-</code>
-The following is the respective batch script for compiling and running the program. You can see the output of the program in the generated //zlib_compress_%j.out// file. **C/C++** dependencies are generally compiled from source. If this is the case you can use the **Simple C/C++ program** example. The batch script showcases loading the dependent library on the system and linking against it. The cluster uses modules for managing the installed dependencies. Make sure to use compatible compiler and library.
-<code bash>
-#!/bin/bash
-#SBATCH --job-name=zlib_compress
-#SBATCH --output=zlib_compress_%j.out
-#SBATCH --error=zlib_compress_%j.err
-#SBATCH --time=00:05:00
-#SBATCH --nodes=1
-#SBATCH --ntasks=1
-#SBATCH --cpus-per-task=1
-#SBATCH --partition=unite
-module load gcc/latest
-module load zlib/1/latest-gcc
-echo "Loaded modules:"
-module list
-echo ""
-echo "Compiling zlib_compress.c..."
-gcc -o zlib_compress zlib_compress.c -lz
-if [ $? -eq 0 ]; then
-    echo "Compilation successful!"
-    echo ""
-    echo "Running the program:"
-    echo "===================="
-    ./zlib_compress
-else
-    echo "Compilation failed!"
-    exit 1
-fi
-</code>
-----
-====MPI====
-The following is an example **C/C++** application which uses **MPI** to perform element-wise addition of two vectors. Each **MPI** task computes the addition of its local region and then sends it back to the leader. Using **MPI** with **Python** is similar assuming that you know how to manage **Python** dependencies on the cluster which is described in the previous section. What is important here is to understand how to manage the resources of the system.
-<code C++>
-#include <stdio.h>
-#include <stdlib.h>
-#include <mpi.h>
-#define VECTOR_SIZE 100000
-int main(int argc, char** argv) {
-    int rank, size;
-    int i;
-    MPI_Init(&argc, &argv);
-    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
-    MPI_Comm_size(MPI_COMM_WORLD, &size);
-    int local_size = VECTOR_SIZE / size;
-    int *local_a = (int*)malloc(local_size * sizeof(int));
-    int *local_b = (int*)malloc(local_size * sizeof(int));
-    int *local_c = (int*)malloc(local_size * sizeof(int));
-    int *a = NULL;
-    int *b = NULL;
-    int *c = NULL;
-    if (rank == 0) {
-        a = (int*)malloc(VECTOR_SIZE * sizeof(int));
-        b = (int*)malloc(VECTOR_SIZE * sizeof(int));
-        c = (int*)malloc(VECTOR_SIZE * sizeof(int));
-        for (i = 0; i < VECTOR_SIZE; i++) {
-            a[i] = i + 1;
-        }
-        for (i = 0; i < VECTOR_SIZE; i++) {
-            b[i] = (i + 1) * 2;
-        }
-    }
-    MPI_Scatter(a, local_size, MPI_INT, local_a, local_size, MPI_INT, 0, MPI_COMM_WORLD);
-    MPI_Scatter(b, local_size, MPI_INT, local_b, local_size, MPI_INT, 0, MPI_COMM_WORLD);
-    printf("Process %d: Adding %d elements\n", rank, local_size);
-    for (i = 0; i < local_size; i++) {
-        local_c[i] = local_a[i] + local_b[i];
-    }
-    MPI_Gather(local_c, local_size, MPI_INT, c, local_size, MPI_INT, 0, MPI_COMM_WORLD);
-    if (rank == 0) {
-        printf("\nFirst 5 elements of (A + B): ");
-        for (i = 0; i < 5; i++) {
-            printf("%d ", c[i]);
-        }
-        printf("\n");
-        free(a);
-        free(b);
-        free(c);
-    }
-    free(local_a);
-    free(local_b);
-    free(local_c);
-    MPI_Finalize();
-    return 0;
-}
-</code>
-The following is the respective batch script for compiling and running the program. You can see the output of the program in the generated //vector_sum_mpi_%j.out// file. The //ntasks// parameter of the batch script specifies the number of **MPI** tasks to be started. This is how you can leverage the resources of the system in order to increase the work done in parallel by your application. The **MPI** tasks are not guaranteed to be executed on different nodes in the cluster, they can also be on a separate physical cores. What is guaranteed is that every task will have the resources to execute in parallel.
-<code bash>
-#!/bin/bash
-#SBATCH --job-name=vector_sum_mpi
-#SBATCH --output=vector_mpi_%j.out
-#SBATCH --error=vector_mpi_%j.err
-#SBATCH --ntasks=4
-#SBATCH --time=00:05:00
-#SBATCH --partition=unite
-module load mpi/latest
-echo "Compiling vector_sum_mpi.c..."
-mpicc -o vector_sum_mpi vector_sum_mpi.c
-if [ $? -ne 0 ]; then
-    echo "Compilation failed!"
-    exit 1
-fi
-echo "Compilation successful!"
-echo "Running with $SLURM_NTASKS MPI processes..."
-echo "----------------------------------------"
-mpirun -np $SLURM_NTASKS ./vector_sum_mpi
-echo "----------------------------------------"
-echo "Job completed!"
-echo "----------------------------------------"
-</code>
-----
+  * **(Resources):** The cluster is fully utilized right now. Slurm will launch your job automatically as soon as another user's time allocation expires.
+  * **(Priority):** Higher priority workloads (based on user fair-share rules) are currently waiting ahead of you in line.
+  * **(PartitionTimeLimit):** Your requested ''#SBATCH --time'' configuration parameter exceeds the maximum execution ceiling value allowed for that specific partition.
+  * **(InvalidQOS):** You requested a resource combination (like asking for an A40 GPU inside a non-GPU queue) that violates partition definition policies.