Crops are the main source of toxic cadmium for humans due to uptake from naturally or anthropogenically polluted soils. Chronic Cd ingestion causes kidney, liver, and skeletal damage along with an increased risk of cancer. Cacao is known to accumulate Cd and may therefore be potentially harmful to human health. Consequently, cocoa production on intensely polluted soils should be avoided. Cocoa products from South America in particular often exceed the limits for Cd, but the factors that drive Cd uptake are as yet poorly studied. In this study, we measured Cd concentrations in defatted cocoa powder from unfermented seeds of 40 different trees on 20 farms in the Huánuco Region, Peru, and associated the Cd levels with the farms’ soil, field management, and nearby vegetation diversity. The mean Cd concentration found in cocoa of the study region was 2.46 mg kg −1 with a range of 0.2–12.56 mg kg −1 . The maximum content measured was an order of magnitude higher than the allowed limit of 1.5 mg kg −1 and was the highest reported so far in the literature. Soil Cd content was the most relevant driver of Cd concentration in cacao. In addition, fertilizer use caused significantly higher Cd concentration in cocoa. Higher biodiversity of herbs was positively correlated with Cd contents in cocoa. The study shows that, apart from the known correlation of soil conditions with Cd accumulation in cacao seeds, changes in fertilization and plant composition may be promising measures to counteract Cd contamination in regions with high soil Cd content.