We aimed at developing fast and accurate predictive algorithms to quantify trans-fat, conjugated linoleic acid (CLA), and other fatty acids using portable and handheld infrared sensors for butter and margarine products. Butter (n = 21) and margarine (n = 15) samples were collected from local grocery stores in Lima, Peru. Their fatty acid content was determined by gas chromatography fatty acid methyl ester analysis (GC-FAME). Infrared spectra were collected using portable (five-reflections) and handheld (single-reflection) infrared (FTIR-ATR) spectrometers and a palm-sized Near-Infrared (FT-NIR) sensor. None of the margarine samples, except those made with partially hydrodenated oils (PHOs), contained trans-fat, and the trans-fat levels in the butters ranged from 0.24 to 0.62 g trans-fat/serving. Partial least squares regression models showed strong correlation (RPre ≥0.91), low standard error of prediction (SEP≤2.62), and high predictive performance based on the ratio of prediction to deviation (RPD: 1.4–15.1) and Ratio of Error Range (RER: 5.7–56.9). Quantification and classification models obtained with the five-reflections FTIR-ATR system exhibited best performances in terms of RPre, SEP, RPD, and RER, followed by the single-reflection FTIR-ATR and FT-NIR systems, respectively. Portable and handheld devices therefore can provide real-time and in situ results to the fat/oil and dairy industry and regulatory agencies for actionable decisions.