HACCing the Universe

Simulations of large-scale structure formation that can simultaneously encompass a representative volume of the universe and resolve the dark matter halos that host galaxies are required for both planning and analyzing current and future astronomical surveys of galaxies across the sky. In order to harness the power of modern supercomputing systems for running such simulations we have developed the Hardware/Hybrid Cosmology Code (HACC) to address the issues of massive concurrency and heterogeneity. HACC uses n-body methods and splits the calculation of the gravitational force into a long-range component that is highly portable and a short-range component that is tuned to specific compute node architectures. We have developed and used variants of HACC for x86, IBM Cell (LANL/Roadrunner), IBM Blue Gene (ANL/Mira), and GPGPU (ORNL/Titan) systems. This talk will focus on how our experiences with various memory hierarchies and potential performance bottlenecks has influenced our iterations of code design in order to achieve better load-balancing and higher performance.