Access
- ✓Kubeconfig simple download or properly configured login nodeKubeconfig docs
- ✓RBAC options with remote SSO provider integrationK8s RBAC docs
- ✓Helm access available without custom external authenticationHelm docs
Configuration
- ✓GPU Operator installed and configuredGPU Operator docs
- ✓Network Operator installed and configuredNetwork Operator docs
- ✓MPI Operator deployed or easy to installMPI Operator
- ✓CSI provider with ReadWriteMany supportK8s PV Access Modes
- ✓Default StorageClass configured and functional, PVCs provision without hanging
- ✓Host path storage option available for high performance and caching
- ✓Ingress/egress to cluster properly configured
- ✓MetalLB or external load balancer available, public IPs assignableMetalLB docs
Profiling Access
- ✓
ncu(Nsight Compute) usable in GPU pods with hardware counters accessible. NeedsNVreg_RestrictProfilingToAdminUsers=0on the host andSYS_ADMINon the pod, otherwiseERR_NVGPUCTRPERM. A full GPU rental should include full profiling access, bare metal or virtualized.FixingERR_NVGPUCTRPERM - ✓Non-root Nsight Compute access and fixing
ERR_NVGPUCTRPERM.Nsight Compute profiling setup - ✓
perf top/perf statusable in pods for CPU-side profiling. Host needsperf_event_paranoid<= 1 andkptr_restrict= 0.perf security docs
Other
- ✓GPU Operator up to date for drivers, container toolkit, including latest security patchesGPU Operator docs
- ✓Network Operator up to date for InfiniBand or Spectrum-X RoCENetwork Operator docs
- ✓Documentation for Broadcom or Pollara RoCE NICs passthrough
- ✓Experience with MPI Operator and PyTorchJob from KubeflowMPI Operator
- ✓Experience with JobSet, Volcano, Kueue or other OSS training frameworksKueue docs
- ✓Experience with llm-d, SGLang OME, or other OSS inference frameworksllm-d
- ✓ACS and other BIOS settings monitored on underlying hosts
- ✓kube-prometheus-stack,
dcgmifor monitoringdmesgfor ECCs, XIDs, and similar errorskube-prometheus-stack
Performance Testing
- ✓Compute performs as expected (GEMMs, MAMF, bandwidth, etc.)MAMF finder
- ✓Storage performs as expected (
fio, etc.)fio - ✓Network performs as expected (nccl-tests or rccl-tests)nccl-tests
General Expectations
- ✓Console, dashboard, CLI and/or API available to manage resources
- ✓24x7 support availability
- ✓Process for security fixes and upgrades exists, proactive notifications are clear
- ✓Integration with comprehensive monitoring and alerting systems
- ✓Integration with active and passive health check systems