In the present work, we studied the interface of terbium doped aluminum oxynitride (Tb-doped AlNxOy:H) deposited under different hydrogen flows with p-type doped crystalline silicon by applying transient surface photovoltage spectroscopy. We observed strong accumulation with concomitant passivation of boron acceptors in the crystalline silicon and defect generation near the interface. With increasing hydrogen flows, the net negative charge in the Tb-doped AlNxOy:H layer decreased, surface photovoltage signals related to defects increased, surface photovoltage transients decayed faster, and the slowest relaxation of charge carriers separated in space changed from trap limited to hopping transport via an exponential distribution of trap states in energy.