Records of clastic sediment from lakes that receive glacial meltwater provide a means for inferring past glacial fluctuations. However, increased clastic sedimentation has been interpreted to indicate both glacial advance and recession. Here we examine the timing of clastic sediment deposition in Challpacocha, a glacially fed lake located downvalley from Quelccaya Ice Cap (QIC), Peru, in comparison with known ice cap margin fluctuations. Using a comparison with a nearby non-glacial lake, Yanacocha, we document that a record of clastic sediment flux from Challpacocha reflects meltwater-derived sediment. We then compare the clastic sediment record from Challpacocha with the Qori Kalis glacier moraine record. We find that organic-rich sediment was deposited primarily during the late Holocene advance of QIC (before ~CE 1490). In contrast, clastic sediment was deposited during QIC recession (~CE 1490-1970). Peaks in clastic sediment flux are associated with the onset of ice retreat. We suggest that this pattern of clastic sediment deposition reflects changing meltwater production and/or glacial erosion during ice advance and retreat and supports models of increased clastic sedimentation during ice retreat.