Kubernetes

Kubernetes is the industry-standard container orchestration software. It is the de facto standard for inference and growing in popularity for training.

Access

  • Kubeconfig simple download or properly configured login nodeKubeconfig docs
  • RBAC options with remote SSO provider integrationK8s RBAC docs
  • Helm access available without custom external authenticationHelm docs

Configuration

  • GPU Operator installed and configuredGPU Operator docs
  • Network Operator installed and configuredNetwork Operator docs
  • MPI Operator deployed or easy to installMPI Operator
  • CSI provider with ReadWriteMany supportK8s PV Access Modes
  • Default StorageClass configured and functional, PVCs provision without hanging
  • Host path storage option available for high performance and caching
  • Ingress/egress to cluster properly configured
  • MetalLB or external load balancer available, public IPs assignableMetalLB docs

Profiling Access

  • ncu (Nsight Compute) usable in GPU pods with hardware counters accessible. Needs NVreg_RestrictProfilingToAdminUsers=0 on the host and SYS_ADMIN on the pod, otherwise ERR_NVGPUCTRPERM. A full GPU rental should include full profiling access, bare metal or virtualized.Fixing ERR_NVGPUCTRPERM
  • Non-root Nsight Compute access and fixing ERR_NVGPUCTRPERM.Nsight Compute profiling setup
  • perf top / perf stat usable in pods for CPU-side profiling. Host needs perf_event_paranoid <= 1 and kptr_restrict = 0.perf security docs

Other

  • GPU Operator up to date for drivers, container toolkit, including latest security patchesGPU Operator docs
  • Network Operator up to date for InfiniBand or Spectrum-X RoCENetwork Operator docs
  • Documentation for Broadcom or Pollara RoCE NICs passthrough
  • Experience with MPI Operator and PyTorchJob from KubeflowMPI Operator
  • Experience with JobSet, Volcano, Kueue or other OSS training frameworksKueue docs
  • Experience with llm-d, SGLang OME, or other OSS inference frameworksllm-d
  • ACS and other BIOS settings monitored on underlying hosts
  • kube-prometheus-stack, dcgmi for monitoring dmesg for ECCs, XIDs, and similar errorskube-prometheus-stack

Performance Testing

  • Compute performs as expected (GEMMs, MAMF, bandwidth, etc.)MAMF finder
  • Storage performs as expected (fio, etc.)fio
  • Network performs as expected (nccl-tests or rccl-tests)nccl-tests

General Expectations

  • Console, dashboard, CLI and/or API available to manage resources
  • 24x7 support availability
  • Process for security fixes and upgrades exists, proactive notifications are clear
  • Integration with comprehensive monitoring and alerting systems
  • Integration with active and passive health check systems

All expectations