Extreme events in turbulent flows are in general linked with sudden changes in the dynamics of the system. Despite the increasing interest in modelling these events, their predictability remains largely unexplored. Here we focus on the case of sudden turbulence/chaos decay in pipe flow and in a reduced-order model of shear flows (Moehlis, Faisst and Eckhardt,2004). We study the predictability of decay events with massive ensembles of simulations. For this purpose we introduce a new pseudo-spectral GPU code to perform fast simulations of turbulent pipe flow and describe a methodology to characterize predictability of decay events using the results of the ensembles of simulations. We report the decrease of predictability as one goes back in time with respect to a decay event, and show that this decrease does not need to be monotonic.