Deaf people use systems of communication based on sign language and finger spelling. Finger spelling is a system where each letter of the alphabet is represented by a unique and discrete movement of the hand. RGB and depth images can be used to characterize hand shapes corresponding to letters of the alphabet. There exists an advantage of depth sensors, as Kinect, over color cameras for finger spelling recognition: depth images provide 3D information of the hand. In this paper, we propose a model for finger spelling recognition based on depth information using kernel descriptors, consisting of four stages. The performance of this approach is evaluated on a dataset of real images of the American Sign Language finger spelling. Different experiments were performed using a combination of both descriptors over depth information. Our approach obtains 92.92% of mean accuracy with 50% of samples for training, outperforming other state-of-the-art methods.