Every year more than 5.4 million new cases of skin cancer are reported in the US. Melanoma is the most lethal type with only 5% occurrence rate, but accounts for over 75% of all skin cancer deaths. Non-melanoma skin cancer, especially basal cell carcinoma (BCC) is the most commonly occurring and often curable type that affects more than 3 million people and causes about 2000 deaths in the US annually. The current diagnosis involves visual inspection, followed by biopsy of the lesions. The major drawbacks of this practice include difficulty in border detection causing incomplete treatment and, the inability to distinguish between clinically similar lesions. Melanoma is often mistaken for the benign lesion pigmented seborrheic keratosis (pSK), making it extremely important to differentiate benign and malignant lesions. In this work, a novel feature extraction algorithm based on phasors was performed on the Fluorescence Lifetime Imaging (FLIM) images of the skin to reliably distinguish between benign and malignant lesions. This approach, unlike the standard FLIM data processing method that requires time-deconvolution of the instrument response from the measured time-resolved fluorescence signal, is computationally much simpler and provides a unique set of features for classification. Subsequently, FLIM derived features were selected using a double step cross validation approach that assesses the reliability and the performance of the resultant trained classifier. Promising FLIM-based classification performance was attained for detecting benign from malignant pigmented (sensitivity: ~80%, specificity: 79%, overall accuracy: ~79%) and non-pigmented (sensitivity: ~88%, specificity: 83%, overall accuracy: ~87%) lesions.