Worldwide, chronic pain has established as one of the foremost medical issues due to its 35% of comorbidity with depression and many other psychological problems. Traditionally, self-report (VAS scale) or physicist inspection (OPI scale) perform the pain assessment; nonetheless, both methods do not usually coincide [14]. Regarding self-assessment, several patients are not able to complete it objectively, like young children or patients with limited expression abilities. The lack of objectivity in the metrics draws the main problem of the clinical analysis of pain. In response, various efforts have tried concerning the inclusion of objective metrics, among which stand out the Prkachin and Solomon Pain Intensity (PSPI) metric defined by face appearance [5]. This work presents an in-depth learning approach to pain recognition considering deep facial representations and sequence analysis. Contrasting current state-of-the-art deep learning techniques, we correct rigid deformations caught since registration. A preprocessing stage is applied, which includes facial frontalization to untangle facial representations from non-affine transformations, perspective deformations, and outside noises passed since registration. After dealing with unbalanced data, we fine-tune a CNN from a pre-trained model to extract facial features, and then a multilayer RNN exploits temporal relation between video frames. As a result, we overcome state-of-the-art in terms of average accuracy at frames level (80.44%) and sequence level (84.54%) in the UNBC-McMaster Shoulder Pain Expression Archive Database.