Benchmarking is a core part of HPC operations — whether you're validating a new cluster, justifying a procurement, or hunting down a performance regression. But running benchmarks well on heterogeneous hardware introduces challenges that the documentation doesn't always prepare you for. This presentation shares our experience running the NVIDIA HPC benchmark container on A100 GPUs and submitting results to the Top500 and Green500 lists, covering the practical details of tuning HPL parameters, navigating the submission process, and the surprises we encountered along the way. We also attempted runs across mixed A100 and H100 nodes, which raised questions about how GPU generational differences affect Linpack scaling, interconnect saturation, and whether heterogeneous submissions are even meaningful. Beyond Linpack, we'll discuss our use of the OSU Micro-Benchmarks (OMB) suite as a cluster diagnostics tool, using pairwise node communication tests to identify fabric bottlenecks, misconfigured adapters, and inconsistent latency across the cluster, and more.
