Potato is economically important in Peru, which is the first potato producer in Latin America, however, the quality of native potatoes need to be improved to increment their consumption. An automatic classification process to detect potato defects is important within the entire production chain to guarantee the high quality of the product. In the present research, a Convolutional Neural Network is used to detect defects in the Huayro potato surface. This is an Andean potato originally from Peru and is special because it has very marked eyes that can complicate the differentiation from pests that leaves holes in the potato. An adaptive learning was proposed in the work, where the principal idea is to evaluate continuously the learning of the neural network to adapt the training process (in this case the training data) to increment the learning performance. The detection results were around 88.2% of F1 score, providing a good performance of the algorithm.