The loss in forest area due to human occupancy is not the only threat to the remaining biodiversity: forest fragments are susceptible to additional human impact. Our aim was to investigate the effect of human impact on tree community features (species composition and abundance, and structural descriptors) and check if there was a decrease in the number of slender trees, an increase in the amount of large trees, and also a reduction in the number of tree species that occur in 20 fragments of Atlantic montane semideciduous forest in southeastern Brazil. We produced digital maps of each forest fragment using Landsat 7 satellite images and processed the maps to obtain morphometric variables. We used investigative questionnaires and field observations to survey the history of human impact. We then converted the information into scores given to the extent, severity, and duration of each impact, including proportional border area, fire, trails, coppicing, logging, and cattle, and converted these scores into categorical levels. We used linear models to assess the effect of impacts on tree species abundance distribution and stand structural descriptors. Part of the variation in floristic patterns was significantly correlated to the impacts of fire, logging, and proportional border area. Structural descriptors were influenced by cattle and outer roads. Our results provided, for the first time, strong evidence that tree species occurrence and abundance, and forest structure of Atlantic seasonal forest fragments respond differently to various modes of disturbance by humans.