This dataset contains the raw and anonymized (i.e., defaced) structural and functional magnetic resonance imaging (s/fMRI) data from 24 deaf signers and a control group of 24 hearing non-signers, as well as the stimulus materials used during the experiment, all associated with the manuscript entitled "Modality-Independent Core Brain Network for Language as Proved by Sign Language" by the same team of authors. Data are organized in Brain Imaging Data Structure (BIDS) format.