Automatic object detection and tracking has been widely applied in the video surveillance systems for homeland security and data fusion in the remote sensing and airborne imagery. The typical applications include human motion analysis, vehicle detection, and architectural building detection. Here we conduct object detection and tracking under planar constraints for interesting objects. Planar surface abounds in man-made environment. It provides much useful information for image understanding and then can be adopted to improve the performance of object detection and tracking. The experiments on real data show that object detection and tracking could be successfully implemented by incorporating planar information of interesting objects. © 2013 SPIE.