Background: The effort-reward imbalance (ERI) scale items are answered in a two-step process, but the justification is questioned for the formulation of summary measure by combining information rated in two steps. Purpose: To examine whether the basic prerequisites of the ERI scales are empirically satisfied and to seek ways to improve the rating procedure. Methods: A polytomous item response theory (IRT) model was applied to the responses of 20,256 workers who completed the ERI scales. To determine the most appropriate statistical justification, three alternative scoring algorithms were compared with regard to the test properties revealed by the IRT analyses and efficiencies of screening performance and criterion validity against depressive symptomatology. Results: The rated raw-score units did not reflect the hypothesized order of lowest stress levels to highest stress levels. Exchanging or collapsing the lowest two categories of a Likert scaled item, where data of different quality are combined, solved this problem, thereby making the test content more appropriate. The modified rating improved the efficiencies of screening performance and the correlation of the stress summary measures against health criterion, i.e., depression. Conclusion: An avoidable measurement error exists in the current ERI scales. Modifying the rating procedure can improve the measurement accuracy. Copyright © Taylor & Francis Group, LLC.