The Limitations of Model-Based Experimental Design and Parameter Estimation in Sloppy Systems

Models of complex systems often involve a large number of unknown parameters to be estimated from data. In practice, these models are often sloppy, i.e., have parameter combinations with an exponential hierarchy of sensitivities as measured by eigenvalues of the Fisher Information Matrix. It has been argued that the extreme insenstivity of the model to changes in some parameters is what enables models to be predictive, analogous to the concept of irrelevance and effective theories in renormalizable systems. However, the unidentifiability of the irrelevant, sloppy parameters is often seen as a bottleneck for predictive modeling. It has been suggested that Optimal Experimental Design (OED) can be used to mitigate the effects of sloppiness and accurately estimate all of a model's parameters. We study models of two complex biological processes and show that when fit to "optimally designed experiments", they can have well-identified parameters, but that the models' predictive power is surprisingly greatly diminished. I interpret these results in the context of generalization error in machine learning and relevance and irrelevance for renormalizable systems.