The present paper describes an investigation on parametric resonance in head seas in which a new third-order coupled mathematical model is considered. The restored modes of heave, roll and pitch are contemplated. The discussion is illustrated for the case of a transom stern fishing vessel at different speeds. It is pointed out that numerical simulations employing the new model are successfully compared to experimental results previously obtained for the vessel. Considering that analyticity is an important tool when handling complex stability issues, some theoretical dynamic characteristics of the equations are discussed. By means of the analysis of the coupled linear variational equation derived from an extended third-order model, the appearance of super-harmonics and increased rigidity proportional to wave amplitude squared due to third-order terms is demonstrated. In the present paper, an important tool is explored, that is the analysis of the limits of stability obtained from the new model. Limits of stability are a well-known and practical way of looking into the problem of parametric resonance. New limits of stability are derived and compared to the more conventional Strut diagram. Dynamic characteristics associated with the new limits of stability are discussed. The influence of different parameters is investigated, including vessel speed, damping and tuning. Consistent and revealing results are obtained through the analysis of the new limits of stability for different speeds and damping. © 2007 Elsevier Ltd. All rights reserved.