The present paper describes an investigation in which a complete third order coupled mathematical model is contemplated by considering the restoring modes of heave, roll and pitch. Numerical simulations employing this new model are compared to experimental results corresponding to excessive motions of a transom stern fishing vessel in head seas. It is shown that this enhanced model matches its results with the experiments more closely than a second order model. Considering that analyticity is an important tool when handling complex stability issues, the theoretical dynamic characteristics of the equations are discussed. It is shown that the new model, due to the introduction of the third order terms, entails qualitative differences when compared to the more commonly used second order model. The variational equation of the roll motion will not be in the form of a Mathieu equation. In fact, it is shown in the paper that the associated time dependent equation falls into the category of a Hill equation. Additionally, a hardening effect is analytically derived, related to the third order coupling and wave effect. © 2005 Taylor & Francis Group.