Early planted sorghum usually experiences cooler day/night temperatures, which may result in delayed growth, floral initiation, and infertile pollen, limiting productivity in high altitudes and temperate regions. Genetic variability for cold tolerance in sorghum has been evaluated by characterizing germination, emergence, vigor, and seedling growth under sub-optimal temperatures. However, the compounded effect of early season cold on plant growth and development and subsequent variability in potential grain yield losses has not been evaluated. Agro-morphological and physiological responses of sorghum grown under early-, mid-, and standard planting dates in West Texas were characterized from seed-to-seed. A set of diverse lines and hybrids with two major sources of tolerance, and previously selected for seedling cold tolerance were used. These were evaluated with a standard commercial hybrid known for its seedling cold tolerance and some cold susceptible breeding lines as checks. Variabilities in assessed parameters at seedling, early vegetative, and maturity stages were observed across planting dates for genotypes and sources of cold tolerance. Panicle initiation was delayed, and panicle size reduced, resulting in decreased grain yields under early and mid-planting dates. Coupled with final germination percent, panicle width and area were significant unique predictors of yield under early and mid-planting dates. Significant variability in performance was observed not only between cold tolerant and susceptible checks, but noticeably between sources of cold tolerance, with the Ethiopian highland sources having lesser yield penalties than their Chinese counterparts. Thus, screening for cold tolerance should not be limited to early seedling characterization but should also consider agronomic traits that may affect yield penalties depending on the sources of tolerance.