Automatic recognition using ear images is an active area of interest within the biometrics community. Human ears are a stable and reliable source of information since they are not affected by facial expressions, do not suffer extreme change over time, are less prone to injuries, and are fully visible in mask-wearing scenarios. In addition, ear images can be passively captured from a distance, making it convenient when implementing surveillance and security applications. At the same time, deep learning-based methods have proven to be powerful techniques for unconstrained recognition. However, to truly benefit from deep learning techniques, it is necessary to count on a large-size variable set of samples to train and test networks. In this work, we built a new dataset using the VGGFace dataset, fine-tuned pre-train deep models, analyzed their sensitivity to different covariates in data, and explored the score-level fusion technique to improve overall recognition performance. Open-set and close-set experiments were performed using the proposed dataset and the challenging UERC dataset. Results show a significant improvement of around 9% when using a pre-trained face model over a general image recognition model; in addition, we achieve 4% better performance when fusing scores from both models.