In Peierls-distorted materials, photoexcitation leads to a strongly coupled transient response between structural and electronic degrees of freedom, always measured independently of each other. Here we use transient reflectivity in the extreme ultraviolet to quantify both responses in photoexcited bismuth in a single measurement. With the help of first-principles calculations based on density-functional theory (DFT) and time-dependent DFT, the real-space atomic motion and the temperature of both electrons and holes as a function of time are captured simultaneously, retrieving an anticorrelation between the A1g phonon dynamics and carrier temperature. The results reveal a coherent, bi-directional energy exchange between carriers and phonons, which is a dynamical counterpart of the static Peierls-Jones distortion, providing first-time validation of previous theoretical predictions.