The In xAl 1-xN/GaN system is found to show compositional instability at the lattice-matched composition (x = 0.18) in epitaxial layers grown by metal organic chemical vapor deposition. The breakdown in compositional homogeneity is triggered by threading dislocations with a screw component propagating from the GaN underlayer, which tend to open up into V-grooves at a certain thickness of the In xAl 1-xN layer. The V-grooves coalesce at ∼200 nm and are filled with material that exhibits a significant drop in indium content and a broad luminescence peak. Transmission electron microscopy suggests that the structural breakdown is due to heterogeneous nucleation and growth at the facets of the V-grooves. © 2012 American Institute of Physics.