Performing Hartree-Fock many-body physics calculations with large language models

In the last few years, large language models (LLMs) have exhibited an unprecedented ability to perform complex linguistic and reasoning tasks. To date, evaluation of scientific and mathematical reasoning ability in LLMs has been limited to simplistic pedagogical tasks, often algorithmically extracted. However, there are no existing evaluations of the ability of LLMs to understand and solve multi-step graduate-level calculations within physics, skills that are critical prerequisites to performing advanced physics research. To close this gap, we produce a novel dataset of multi-step Hartree-Fock many-body physics calculations through annotation of a well-defined class of condensed matter theory papers. The dataset distills each paper into a sequence of prompt-answer pairs. We first evaluated the ability of LLMs to extract relevant information from research papers and aid in the construction of the prompts. We then evaluated their ability to perform the sequence of many-body physics calculations and investigated the effects of few-shot prompting. The success and challenges of our investigation are pertinent for both physics and machine learning research.