SemiAnalysis

Slurm

Slurm is an open-source job scheduler and the de facto standard for HPC for over 20 years. Used on over 60% of TOP500 supercomputers and over 50% of AI training clusters.

Access

  • ✓
    Head node provisioned and accessible via a simple SSH command
  • ✓
    Standard Slurm commands functional: sinfo, squeue, scontrol, salloc, sbatch, srunSlurm man pages
  • ✓
    Shared filesystem configured with default home directory and reasonable quota
  • ✓
    Essential packages available: python, git, curl, wget, apt, vim, nano
  • ✓
    sudo access available on head node for package installation
  • ✓
    Easy to add new users and groups via CLI or console
  • ✓
    Easy to enforce RBAC on users and groups on cluster and storage
  • ✓
    Integration with external IDPs (Okta, Google, Microsoft, GitHub via OIDC/OAuth 2.0)
  • ✓
    Integration with sacct for job tracking and resource utilization by usersacct docs
  • ✓
    Passwordless SSH connectivity between nodes enabled

Configuration

  • ✓
    GPUDirect RDMA enabled on worker nodes via dma_buf (nvidia-open kernel module). The out-of-tree nvidia_peermem module is deprecated and does not pass on its own.GPUDirect RDMA docs
  • ✓
    Topology-aware scheduling configured via Slurm topology.yaml/topology.conf (topology/block or topology/tree) with rack/block/NVLink topology awarenessSlurm topology.yaml
  • ✓
    NVCC compiler installed and accessible
  • ✓
    HPC-X or equivalent MPI implementation installed without hunting in /optNVIDIA HPC-X
  • ✓
    NCCL properly installed, configured, and up to dateNCCL docs
  • ✓
    Lmod installed and configuredLmod docs
  • ✓
    Default CPUs per task, memory per CPU, and other settings are logical

Containers

  • ✓
    Support for the Pyxis pluginNVIDIA Pyxis
  • ✓
    Support for EnrootNVIDIA Enroot
  • ✓
    Docker available on worker nodes with modern NVIDIA Container ToolkitNVIDIA Container Toolkit
  • ✓
    Singularity and Apptainer container supportApptainer docs

Networking and Collectives

  • ✓
    NCCL auto-configuration: NCCL_MIN_NCHANNELS, NCCL_PROTO, NCCL_ALGO NOT in /etc/nccl.confNCCL environment variables
  • ✓
    RoCEv2 configuration: NCCL_IB_GID_INDEX=3 in /etc/nccl.confNCCL_IB_GID_INDEX
  • ✓
    High bandwidth NICs/HCAs named correctly (mlx5_0, mlx5_1, etc.)

Monitoring and Health Checks

  • ✓
    DCGM background health checks enabled and plugged into Slurm HealthCheckProgramDCGM docs
  • ✓
    Prolog and Epilog scripts lightweight (<30s to get on a node via srun)Slurm Prolog/Epilog
  • ✓
    SHARP support for enhanced NCCL performanceNVIDIA SHARP docs
  • ✓
    ncu (Nsight Compute) usable on compute nodes with hardware counters accessible. Needs NVreg_RestrictProfilingToAdminUsers=0 on the host, otherwise ERR_NVGPUCTRPERM. Confirm by profiling a trivial kernel under srun, not just the binary. A full GPU rental should include full profiling access.Nsight Compute CLI
  • ✓
    Non-root Nsight Compute access and fixing ERR_NVGPUCTRPERM.Nsight Compute profiling setup
  • ✓
    perf top / perf stat usable on compute nodes for CPU-side profiling without root: perf_event_paranoid <= 1 and kptr_restrict = 0. Catches host-side data-loading and launch-overhead bottlenecks.perf security docs
  • ✓
    Dashboard includes Slurm job accounting data via sacctSlurm Accounting
  • ✓
    Automatic remediation systems for failed nodes

Performance Testing

  • ✓
    Compute performs as expected (GEMMs, MAMF, bandwidth, etc.)MAMF finder
  • ✓
    Storage performs as expected (fio, etc.)fio
  • ✓
    Network performs as expected (nccl-tests or rccl-tests)nccl-tests

General Expectations

  • ✓
    Console, dashboard, CLI and/or API available to manage resources
  • ✓
    24x7 support availability
  • ✓
    Process for security fixes and upgrades exists, proactive notifications are clear
  • ✓
    Integration with comprehensive monitoring and alerting systems
  • ✓
    Integration with active and passive health check systems

All expectations