A modular architecture for real-time feature-based tracking is presented. This architecture takes advantage of temporal and spatial information contained in a video stream, combining robust classifiers with motion estimation to achieve real-time performance. The relationship among features is exploited to obtain a robust detection and a stable tracking. The effectiveness of this architecture is demonstrated in a face tracking system using eyes and lips as features. A pre-processing stage based on skin color segmentation, density maps and low intensity characteristic of facial features reduces the number of image regions that are candidates for eyes and lips. Support Vector Machines are then used in the classification process, whereas a combination of Kalman filters and template matching is used for tracking.