This article presents an intelligent system using deep learning algorithms and the transfer learning approach to detect oil palm units in multispectral photographs taken with unmanned aerial vehicles. Two main contributions come from this piece of research. First, a dataset for oil palm units detection is carefully produced and made available online. Although being tailored to the palm detection problem, the latter has general validity and can be used for any classification application. Second, we designed and evaluated a state-of-the-art detection system, which uses a convolutional neural network to extract meaningful features, and a classifier trained with the images from the proposed dataset. Results show outstanding effectiveness with an accuracy peak of 99.5% and a precision of 99.8%. Using different images for validation taken from different altitudes the model reached an accuracy of 97.5% and a precision of 98.3%. Hence, the proposed approach is highly applicable in the field of precision agriculture.