Standard state estimation approaches do not provide guaranteed confidence regions for finite data amounts. For some applications, in particular safety critical ones, this can be of interest. In this paper, we suggest a method to construct a moving horizon state estimator (MHE) able to provide confidence regions (CR) of the state estimate with exact probability under mild assumptions on the noise. To do so a novel algorithm called sign perturbed sums (SPS), as presented by B.Cs. Csaji et al. in [4], is combined with a MHE approach. The paper develops the idea for single input single output (SISO) linear time invariant (LTI) state-space systems and uses simulations and a comparison to results obtained in the standard way to confirm the potential of the method.