Beyond Petascale with the HipGISAXS Software Suite

We have developed HipGISAXS, a software suite to analyze GISAXS and SAXS data for structural characterization of materials at the nano scale using X-rays. The software has been developed as a massively-parallel system capable of harnessing the raw computational power offered by clusters and supercomputers built using graphics processors (GPUs), Intel Phi co-processors, or commodity multi-core CPUs. Currently the forward GISAXS simulation is a major component of HipGISAXS, which simulates the X-ray scattering process based on the Distorted Wave Born Approximation (DWBS) theory, for any given nano structures and morphologies with a set of experimental configurations. These simulations are compute-intensive, and have a high degree of parallelism available, making them well-suited for fine-grained parallel computations on highly parallel many core processors like GPUs. Furthermore, a large number of such simulations can be carried out simultaneously for various experimental input parameters. HipGISAXS also includes a Reverse Monte Carlo based modeling tool for SAXS data. With HipGISAXS we have demonstrated a sustained compute performance of over 1 Petaflop on 8000 GPU nodes of the Titan supercomputer at ORNL, and have shown it to be highly scalable.