Payments for ecosystem services are becoming popular components in strategies to conserve ecosystems and biodiversity, but their effectiveness remains poorly documented. Here we present counterfactual-based evidence on the conservation outcomes of the pilot stage of Peru's National Forest Conservation Program (NFCP). The NFCP provides direct payments to indigenous communities in the Amazon, conditional on avoided deforestation and the adoption of sustainable production systems. Using a spatially explicit quasi-experimental evaluation design, we show that the payment scheme has achieved only small conservation impacts, in terms of avoided deforestation. Counter-intuitively, these materialized largely on land not enrolled for conservation, due to spillover effects. Conservation effects on contracted land were negligible because communities were not chosen according to high deforestation threats, and they self-enrolled low-pressure forest areas for conservation. Occasional non-sanctioned contract incompliance contributed to these outcomes. We highlight implications for the design and implementation of up-scaled national conservation programs. Methodologically, we demonstrate the important role of choosing the appropriate spatial scale in evaluating area-based conservation measures.