Cacao (Theobroma cacao) cultivation maintaining a high proportion of shade trees in a diverse composition (agroforestry) is currently being viewed as a sustainable land use practice. Our research hypothesis was that cacao agroforests (AF) can support relatively high tree diversity, as compared to surrounding primary and/or secondary forests. The objective of this study was to assess the impact of forest conversion on tree communities by comparing tree composition, community characteristics (richness and diversity) and spatial structure (density, canopy height, basal area) among primary forest, secondary forest, and cacao AF. In total, we collected data from 30 25 × 25 m plots on three land use systems (20 in cacao AF, five in secondary, and five in primary forests) in San Alejandro, Peruvian Amazon. All trees with DBH ≥ 10 cm were counted, identified to species, and their height and DBH were recorded. Our results support the hypothesis that cacao AF present a relatively high tree species richness and diversity, although they are no substitute for natural habitats. We identified most common species used for shading cacao. Tree species composition similarity was highest between cacao AF and secondary forest. Vegetation structure (density, height, DBH) was significantly lower compared to primary and secondary forest. Species richness and diversity were found to be highest in the primary forest, but cacao AF and secondary forests were fairly comparable. The tree species cultivated in cacao AF are very different from those found in primary forest, so we question whether the relatively high tree diversity and richness is able to support much of the diversity of original flora and fauna.