Appropriate consideration of contextual factors is essential for ensuring internal and external validity of randomized and non-randomized evaluations. Contextual factors may confound the association between delivery of the intervention and its potential health impact. They may also modify the effect of the intervention or programme, thus affecting the generalizability of results. This is particularly true for large-scale health programmes, for which impact may vary substantially from one context to another. Understanding the nature and role of contextual factors may improve the validity of study results, as well as help predict programme impact across sites. This paper describes the experience acquired in measuring and accounting for contextual factors in the Multi-Country Evaluation of the IMCI (Integrated Management of Childhood Illness) strategy in five countries: Bangladesh, Brazil, Peru, Uganda and Tanzania. Two main types of contextual factors were identified. Implementation-related factors include the characteristics of the health systems where IMCI was implemented, such as utilization rates, basic skills of health workers, and availability of drugs, supervision and referral. Impact-related factors include baseline levels and patterns of child mortality and nutritional status, which affect the scope for programme impact. We describe the strategies used in the IMCI evaluation in order to obtain data on relevant contextual factors and to incorporate them in the analyses. Two case studies - from Tanzania and Peru - show how appropriate consideration of contextual factors may help explain apparently conflicting evaluation results. © The Author 2005. Published by Oxford University Press in association with The London School of Hygiene and Tropical Medicine. All rights reserved.