We study the geometric phase that is acquired when the second-order mode of an optical beam undergoes a cyclic transformation. We find that a geometric phase appears when the initial and intermediate modes have different quantities of orbital angular momentum. The phase is similar to the one measured previously for transformations of first-order modes. However, we find no accumulated geometric phase when the initial and intermediate modes have zero orbital angular momentum.