Video background modeling is an important preprocessing stage for various applications and principal component pursuit (PCP) is among the state-of-the-art algorithms for this task. One of the main drawbacks of PCP is its sensitivity to jitter and camera movement. This problem has only been partially solved by a few methods devised for jitter or small transformations. However, such methods cannot handle the case of moving or panning cameras. We present a novel, fully incremental PCP algorithm, named incPCP-PTI, that is able to cope with panning scenarios and jitter by continuously aligning the low-rank component to the current reference frame of the camera. To the best of our knowledge, incPCP-PTI is the first low rank plus additive incremental matrix method capable of handling these scenarios. Results on synthetic videos and CDNET2014 videos show that incPCP-PTI is able to maintain a good performance in the detection of moving objects even when panning and jitter are present in a video.