State-of-the-art radial velocity (RV) exoplanet searches are limited by the effects of stellar magnetic activity. Magnetically active spots, plage, and network regions each have different impacts on the observed spectral lines and therefore on the apparent stellar RV. Differentiating the relative coverage, or filling factors, of these active regions is thus necessary to differentiate between activity-driven RV signatures and Doppler shifts due to planetary orbits. In this work, we develop a technique to estimate feature-specific magnetic filling factors on stellar targets using only spectroscopic and photometric observations. We demonstrate linear and neural network implementations of our technique using observations from the solar telescope at HARPS-N, the HK Project at the Mt. Wilson Observatory, and the Total Irradiance Monitor onboard SORCE. We then compare the results of each technique to direct observations by the Solar Dynamics Observatory. Both implementations yield filling factor estimates that are highly correlated with the observed values. Modeling the solar RVs using these filling factors reproduces the expected contributions of the suppression of convective blueshift and rotational imbalance due to brightness inhomogeneities. Both implementations of this technique reduce the overall activity-driven rms RVs from 1.64 to 1.02m/s, corresponding to a 1.28m/s reduction in the rms variation. The technique provides an additional 0.41m/s reduction in the rms variation compared to traditional activity indicators.