This is the data repository of the training and test data sets for EmbryoNet. The data is structured in multiple packages. EmbryoNet_Models (DOI 10.48606/31) contains the trained neural networks, the other packages are imaging data. All data are brightfield timelapse images of one or multiple embryos recorded in multiwell plates in either the Acquifer Imaging Machine or the Keyence BZ-X810 microscope. The microscope type is included in the name of the archive, e.g. BMP_Acquifer.zip. Training data images are accompanied by json-files with the classification from human annotators, while test data sets also have the jsons of EmbryoNet's classification. The dataset EmbryoNet_Image-data: Stickleback 1 (DOI 10.48606/32) contains training data for the Stickleback version of EmbryoNet, and EmbryoNet_Test-data: Stickleback (DOI 10.48606/33) contains the evaluation data. EmbryoNet_Training-data: Medaka (DOI 10.48606/35) and EmbryoNet_Test-data: Medaka (DOI 10.48606/34) contain the respective data for Medaka. The other packages are zebrafish images. The two archives named EmbryoNet_Test-data 1&2 (DOI: 10.48606/29 & 10.48606/30) are the zebrafish test data sets. The zebrafish training data sets are named after the signaling molecule: EmbryoNet_training-data: BMP (DOI 10.48606/18), EmbryoNet_training-data: Retinoic acid (DOI 10.48606/20), EmbryoNet_training-data: Wnt (DOI 10.48606/21), EmbryoNet_training-data: FGF (DOI 10.48606/22), EmbryoNet_training-data: Nodal (DOI 10.48606/23), EmbryoNet_training-data: Shh (DOI 10.48606/25) and EmbryoNet_training-data: PCP (DOI 10.48606/26). EmbryoNet_training-data: WT (DOI 10.48606/16) contains the training data of untreated embryos. The datasets EmbryoNet_Training-data: Severities - Keyence (DOI 10.48606/28) and EmbryoNet_Training-data: Severities - Acquifer (DOI 10.48606/27) contain the training and evaluation data of the Severities experiments with different inhibitor concentrations. Inside a zip file the data is arranged in experiment folders, named in the format DATE_Molecule_concentration, e.g. 201222_FGF_10uM. Inside these experiment folders the data is organized after multiwell plate or microscope positions, A001-D006 for the Acquifer data and XY01-XY24 for the Keyence data.