Coffee is one of the most important agricultural products and consumed beverages in the world. Then, adequate control of the diseases is necessary to guarantee its production. Coffee rust is a relevant coffee disease, which is caused by the fungus hemileia vastatrix. Recently, deep learning techniques have been used to identify coffee diseases and the severity of each disease. In this paper, we propose a new interpretable transfer learning method to estimate the severity of coffee rust called CoffeeSE. The proposed method consists of four stages: Leaf segmentation, patch sampling, patch-based classification, and quantification/interpretation analysis. On the classification stage, a Brazilian dataset is used to transfer by fine-tuning new weights to a pre-trained classifier. So, this new classifier is tested in Peruvian coffee leaves infected with coffee rust. Our approach shows acceptable quantification results according to an expert agronomist. In addition, an interpretability module of the patch-classifier is proposed to provide a visual and textual explanation of the most relevant pixels used in the classification process.