Ear recognition has gained attention in recent years. The possibility of being captured from a distance, contactless, without the cooperation of the subject and not be affected by facial expressions makes ear recognition a captivating choice for surveillance and security applications, and even more in the current COVID-19 pandemic context where modalities like face recognition fail due to mouth and facial covering masks usage. Applying any deep learning (DL) algorithm usually demands a large amount of training data and appropriate network architectures, therefore we introduce a large-scale database and explore fine-tuning pre-trained convolutional neural networks (CNNs) looking for a robust representation of ear images taken under uncontrolled conditions. Taking advantage of the face recognition field, we built an ear dataset based on the VGGFace dataset and use the Mask-RCNN for ear detection. Besides, adapting the VGGFace model to the ear domain leads to a better performance than using a model trained for general image recognition. Experiments on the UERC dataset have shown that fine-tuning from a face recognition model and using a larger dataset leads to a significant improvement of around 9% compared to state-of-the-art methods on the ear recognition field. In addition, we have explored score-level fusion by combining matching scores of the fine-tuning models which leads to an improvement of around 4% more. Open-set and close-set experiments have been performed and evaluated using Rank-1 and Rank-5 recognition rate metrics.