The thermal conductivity of polycrystalline aluminum nitride (AlN) films with inhomogeneous structures is experimentally and theoretically investigated. The influence of the grain morphology and size evolution along the cross plane direction of the films is studied by thickness-dependent 3m measurements on AlN monolayers. For AlN/AlN multilayer samples, the impact of oxygen-related defects localized at the interface between two AlN layers, is also analyzed. When the total thickness of these multilayers is downsized from 1107 nm to 270 nm, their measured effective thermal conductivity reduces by 47%, which is smaller than the corresponding reduction of 58% for monolayers. In multilayers, this decrease is due to the additive contributions of the thermal resistances arising from the AlN and AlN/AlN interfaces. The experimental data are interpreted through an analytical model developed for nanocrystalline films with inhomogeneous structures. It is shown that the size effects on the phonon mean free paths and the intrinsic thermal resistance resulting from the inhomogeneous microstructure predominate as the film thickness increases, whilst the contribution of the interface thermal resistance strengthens when the thickness is scaled down.