Dissecting the genetic architecture of stress tolerance in crops is critical to understand and improve adaptation. In temperate climates, early planting of chilling-tolerant varieties could provide longer growing seasons and drought escape, but chilling tolerance (75,000 data points from ∼16,000 plots) in multi-environment field trials in Kansas under natural chilling stress (sown 30-45 days early) and normal growing conditions. Joint linkage mapping with early-planted field phenotypes revealed an oligogenic architecture, with 5-10 chilling tolerance loci explaining 20-41% of variation. Surprisingly, several of the major chilling tolerance loci co-localize precisely with the classical grain tannin (Tan1 and Tan2) and dwarfing genes (Dw1 and Dw3) that were under strong directional selection in the US during the 20th century. These findings suggest that chilling sensitivity was inadvertently selected due to coinheritance with desired nontannin and dwarfing alleles. The characterization of genetic architecture with NAM reveals why past chilling tolerance breeding was stymied and provides a path for genomics-enabled breeding of chilling tolerance.