Cluster Overview
- ✓Number of nodes, pods, workloads running
- ✓Current node health status
- ✓Resource utilization (CPU, memory, disk, network) at cluster/node/namespace/pod levels
- ✓Trends and summaries
- ✓Workload stability (restarts, pending pods, scaling events)
- ✓Control plane health (API server latency, etcd performance, controller manager and scheduler metrics)
Resource Management Integration
- ✓Slurm integration with job stats by user, type, etc.Slurm Accounting
- ✓Kubernetes integration with kube-state-metrics, node-exporter, dcgm-exporter, cAdvisordcgm-exporter
- ✓Resource quotas with limits and usage of GPUs for users and groups
- ✓Real-time GPU node availability status
- ✓Scheduler resource contention monitoring
Monitoring Stack
- ✓kube-prometheus-stack for Kubernetes metrics (node state, resource usage, control plane, workload performance)kube-prometheus-stack
- ✓NVIDIA DCGM integration for comprehensive GPU monitoring or AMD equivalentDCGM docs
- ✓KV cache usage for horizontal pod autoscaling with serving (
gpu_cache_usage_perc) - ✓Integrated alert management and notification systems
Hardware Monitoring
- ✓Node-level power draw with real-time power consumption monitoring
- ✓Fan speed monitoring for cooling system performance and status
- ✓Temperature monitoring (CPU, RAM, NIC, transceiver, other critical components)
- ✓PCIe AER rates for Advanced Error Reporting on PCIe bus health
- ✓
dmesglogs monitoring via Promtail for system-level message capturePromtail docs - ✓GB200 NVL72 specialized monitoring requirements
Performance Monitoring
- ✓
ncu(Nsight Compute) on PATH so non-root users can profile GPU kernels. Binary check only — counter access is the next item.Nsight Compute CLI - ✓GPU hardware counters accessible without
sudo:NVreg_RestrictProfilingToAdminUsers=0set and the driver reloaded, otherwiseERR_NVGPUCTRPERMat runtime even withncuon PATH. Confirm with a live profile of a trivial kernel, not just the binary.FixingERR_NVGPUCTRPERM - ✓Setup reference and worked examples. Containers also need
SYS_ADMIN(or --privileged) for hardware counters. A full GPU rental should include full profiling access, bare metal, VM, or container.Nsight Compute profiling setup - ✓GPU MODE Lecture 44: profiling GPU kernels with Nsight.GPU MODE Lecture 44: NVIDIA Profiling
- ✓
perf top/perf statusable for CPU-side profiling without root:perf_event_paranoid<= 1 andkptr_restrict= 0 for system-wide and per-PID access. Catches host-side bottlenecks in data loading, preprocessing, and kernel launch overhead.perf_event_paranoid/ perf security
Network Monitoring
- ✓NVLink/XGMI throughput for inter-GPU communication bandwidth
- ✓PCIe throughput for host CPU to GPU data transfer rates
- ✓InfiniBand/RoCEv2 throughput for high-speed network performance between nodesRDMA perftest
Resource Management
- ✓User and group quotas
- ✓Node availability
- ✓Scheduler job history
General Expectations
- ✓Console, dashboard, CLI and/or API available to manage resources
- ✓24x7 support availability
- ✓Process for security fixes and upgrades exists, proactive notifications are clear
- ✓Integration with active and passive health check systems