We demonstrate that the modulated surface photovoltage spectroscopy (modulated SPS) technique can be applied to investigate interface states in the bandgap, i.e. interface passivation, of crystalline silicon coated with a downshift layer such as hydrogenated aluminum nitride with embedded terbium ions by suppressing straylight with a cut-off filter. Different hydrogen contents influence the surface photovoltage spectra at photon energies below the bandgap of crystalline silicon. Modulated SPS reveals that at higher hydrogen content there is a lower signal and, thus, a lower density of surface defect states. Our experiments show that modulated SPS can become a powerful tool for characterizing defect states at interfaces which cannot be easily studied by other methods.