The effective thermal conductivity of nanocrystalline films of AlN with inhomogeneous microstructure is investigated experimentally and theoretically. This is done by measuring the thermal conductivity of the samples with the 3-omega method and characterizing their microstructure by means of electron microscopy. The relative effect of the microstructure and the interface thermal resistance on the thermal conductivity is quantified through an analytical model. Thermal measurements showed that when the thickness of an AlN film is reduced from 1460 to 270 nm, its effective thermal conductivity decreases from 8.21 to 3.12 WYm-1?K-1, which is two orders of magnitude smaller than its bulk counterpart value. It is shown that both the size effects of the phonon mean free paths and the intrinsic thermal resistance resulting from the inhomogeneous microstructure predominate for thicker films, while the contribution of the interface thermal resistance strengthens as the film thickness is scaled down. The obtained results demonstrate that the structural inhomogeneity in polycrystalline AlN films can be efficiently used to tune their cross- plane thermal conductivity. In addition, thermal conductivity measurements of epitaxially grown InP layers on silicon using Raman spectroscopy are reported.