Large environmental fluctuations often cause mass extinctions, extirpating species and transforming communities. While the effects on community structure are evident in the fossil record, demographic consequences for populations of individual species are harder to evaluate because fossils reveal relative but not absolute abundances. However, genomic analyses of living species that have survived a mass extinction event offer the potential for understanding the demographic effects of such environmental fluctuations on extant species. Here we show how environmental variation since the late Pliocene has shaped demographic changes in extant corals of the genus Orbicella, major extant reef builders in the Caribbean that today are currently endangered. We use genomic approaches to provide estimates of previously unknown current and past population sizes and infer demographic dynamics over the last 3 million years. Our results suggest populations of all three Orbicella declined around 2-1 million years ago (Ma), coincident with the extinction of at least 50% of Caribbean coral species. The estimated changes in population size are consistent across the three species despite their ecological differences. Subsequently, two shallow-water specialists expanded their population size at least two fold, likely as a result of coping with the empty niche left by the disappearance of their sister competitor species O. nancyi (the organ-pipe Orbicella). Our study suggests that populations of modern Orbicella species are capable of rebounding from reductions in population size under suitable conditions and that the effective population size of many modern corals provides rich standing genetic variation for corals to adapt to climate change. For conservation genetics our study suggest the urgent need to evaluate contemporary genetic variation under appropriate demographic evolutionary models.