Alleles versus genotypes: Genetic interactions and the dynamics of selection in sexual populations

Physical interactions between amino-acids are essential for protein structure and activity, while protein-protein interactions and regulatory interactions are central to cellular function. As a consequence of these interactions, the combined effect of two mutations can differ from the sum of the individual effects of the mutations. This phenomenon of genetic interaction is known as epistasis. However, the importance of epistasis and its effects on evolutionary dynamics are poorly understood, especially in sexual populations where recombination breaks up existing combinations of alleles to produce new ones. Here, we present a computational model of selection dynamics involving many epistatic loci in a recombining population. We demonstrate that a large number of polymorphic interacting loci can, despite frequent recombination, exhibit cooperative behavior that locks alleles into favorable genotypes leading to a population consisting of a set of competing clones. As the recombination rate exceeds a certain critical value this ``genotype selection'' phase disappears in an abrupt transition giving way to ``allele selection'' - the phase where different loci are only weakly correlated as expected in sexually reproducing populations. Clustering of interacting sets of genes on a chromosome leads to the emergence of an intermediate regime, where localized blocks of cooperating alleles lock into genetic modules. Large populations attain highest fitness at a recombination rate just below critical, suggesting that natural selection might tune recombination rates to balance the beneficial aspect of exploration of genotype space with the breaking up of synergistic allele combinations.