Decoding Inverse Imaging Problems in Materials with Distributed Deep Learning

Materials physics abounds with challenging inverse problems, from inverse materials design to reconstruction of material properties from imaging/scattering data. One of the oldest inverse problems in materials is the reconstruction of the local atomic scattering potential from convergent beam electron diffraction (CBED). In this talk, we will present results on a potential reconstruction approach based on deep neural networks (DNN). In particular, we used data parallelism to distribute the training of a DNN over 12,000 GPUs on the Summit supercomputer. By efficient utilization of Summit's burst buffer and half-precision arithmetics, the data processing rates of the DNN reached upwards of 100 GB/s, allowing for the processing of local CBED patterns from 60,000+ crystal structures in the matter of minutes. We will present challenges encountered in distributed training at these scales, such as I/O bottlenecks and DNN convergence and how they can be mitigated. Finally, initial results on our DNN-based reconstruction of local atomic potentials of strontium irridate superlattices will be presented.