Offline Reinforcement Learning for Rotation and Density Profile Control in Tokamaks

We report preliminary experimental results from the DIII-D tokamak demonstrating real-time control of rotation and density profiles using a fully data-driven reinforcement learning (RL) pipeline. Rotation control is advantageous for increasing understanding of MHD stability and energy transport whereas, density profile control supports high-performance plasmas while avoiding density-limit disruptions. A major challenge in developing RL-based controllers is the lack of accurate simulation environments for training. To address this, we use a probabilistic recurrent neural network trained on historical Zipfit reconstructions including rotation, density, temperature, pressure, and safety factor (q-profile), to model plasma dynamics. Using an ensemble of such bootstrapped networks, we capture the state transition and uncertainty in predictions, creating a probabilistic simulator which can be used for training robust RL policies. The RL agents control profiles by actuating neutral beam power, neutral beam torque, electron cyclotron heating, and gas puffing. These policies are then deployed on the DIII-D Plasma Control System, where they operate in real time using RTCakeNN-inferred profiles. Our approach enables robust policy learning and effective sim-to-real transfer from Zipfit based training to live RTCakeNN profiles. Results from initial shots show successful control of rotation profiles, demonstrating the promise of RL for adaptive and intelligent tokamak control. Work supported by US DOE under DE-FC02-04ER54698, DE-SC0024544 and DE-SC0015480.