====== Slurm Commands Cheat Sheet ======

===== 1. Cluster Status (sinfo) =====

''sinfo'' displays partitions and node states.

^ Command ^ Description ^
| <code>sinfo</code> | Basic view of partitions and nodes |
| <code>sinfo -N</code> | Show individual node names |
| <code>sinfo -p gpu</code> | Filter by specific partition (e.g., GPU) |
| <code>sinfo -o "%P %a %l %D %t %N"</code> | Custom: Partition, Availability, Time, Nodes, State, Names |

**Use cases:**
  * Check available partitions
  * View total node count  
  * Monitor node states: <code>idle, alloc, mix, down, drain</code>

===== 2. Job Submission & Management =====

==== Submit & Monitor ====
<code bash>
sbatch my_job.slurm      # Submit batch script
squeue                   # View all jobs in queue
squeue -u $USER          # Show only your jobs
squeue -j 123456         # Specific job details
squeue -o "%.18i %.9P %.20j %.8u %.2t %.10M %.6D %R"  # Custom format
</code>

==== Cancel Jobs ====
<code bash>
scancel 123456           # Cancel specific job
scancel -u $USER         # Cancel all your jobs
scancel 123456_17        # Cancel array task 17
</code>

==== Job Control ====
<code bash>
scontrol show job 123456     # Detailed job information
scontrol show node node001   # Detailed node information
scontrol hold 123456         # Hold/pause job
scontrol release 123456      # Release held job
</code>

===== 3. Interactive Work =====

==== Allocate Resources ====
<code bash>
salloc -p cpu -N 1 -n 1 -c 8 --time=01:00:00
</code>

**Then run inside allocation:**
<code bash>
srun hostname
srun python script.py
</code>

**Run interactive steps:**
<code bash>
srun --partition=short --ntasks=4 --gres=gpu:1  --time=02:00:00 --pty bash
</code>

===== 4. Reporting & Statistics =====

==== Job Accounting ====
<code bash>
sacct -j 123456                                              # Job history
sacct -j 123456 --format=JobID,JobName,Partition,AllocCPUS,State,Elapsed,MaxRSS
sacct -u $USER --starttime today                             # Your jobs today
</code>

==== Live Statistics ====
<code bash>
sstat -j 123456.batch                                        # Running job stats
sstat -j 123456.batch --format=AveCPU,MaxRSS,MaxVMSize       # Specific metrics
seff 123456                                                  # Efficiency summary
</code>

===== 5. Job Lifecycle States =====

^ State ^ Meaning ^
| **PENDING** | Waiting in queue |
| **RUNNING** | Currently executing |
| **COMPLETED** | Finished successfully |
| **FAILED** | Error occurred |
| **CANCELLED** | Manually stopped |
| **TIMEOUT** | Time limit exceeded |
| **OUT_OF_MEMORY** | RAM exhausted |
| **NODE_FAIL** | Node hardware failure |

==== Common PENDING Reasons ====
  * **Resources** – No free resources available
  * **Priority** – Lower queue priority
  * **QOSMax...** – Quality of Service limits
  * **AssocGrp...** – Account/group limits
  * **ReqNodeNotAvail** – Requested node unavailable
  * **Dependency** – Waiting on another job

===== 6. Resource Specification =====

**Most common errors come from incorrect resource requests.**

==== Key Parameters ====
^ Parameter ^ Description ^
| <code>--nodes (-N)</code> | Number of nodes requested |
| <code>--ntasks (-n)</code> | Total tasks/processes (MPI ranks) |
| <code>--cpus-per-task (-c)</code> | CPU cores per task (OpenMP threads) |
| <code>--mem</code> | Total RAM per node |
| <code>--mem-per-cpu</code> | RAM per CPU core |
| <code>--time</code> | Max walltime (HH:MM:SS) |
| <code>--partition (-p)</code> | Target partition/queue |

==== Pro Tips ====
<code bash>
# Pure MPI (1 core per rank)
--nodes=2 --ntasks-per-node=32 --cpus-per-task=1

# Hybrid MPI+OpenMP  
--nodes=2 --ntasks-per-node=4 --cpus-per-task=8

# OpenMP
--nodes=1 --ntasks=1 --cpus-per-task=16
</code>

**Golden Rule:** <code>--ntasks-per-node × --cpus-per-task</code>  ** should match node core count for optimal packing.**