Objective Given the potential therapeutic effect of vascular disease control timing to reduce dementia risk, we investigated the age-related influences of vascular risk factor burden on brain structure throughout the lifespan. Methods We studied participants from the community-based prospective Framingham Heart Study. Overall vascular risk factor burden was calculated according to the Framingham Stroke Risk Profile, a validated algorithm that predicts stroke risk. Brain volume was estimated by MRI. We used cross-sectional data to examine how the strength of association between vascular risk factor burden and brain volume changed across each age decade from age 45–54 years through to 85–94 years (N = 2,887). Second, we leveraged up to 40 years of longitudinal data to determine how the strength of association between vascular risk factor burden and brain volume changed when vascular risk factors were examined at progressively earlier ages (N = 7,868). Results In both cross-sectional and longitudinal analyses, higher vascular risk factor burden was associated with lower brain volume across each age decade. In the cross-sectional analysis, the strength of this association decreased with each decade of advancing age (p for trend < 0.0001). In longitudinal analysis, the strength of association between vascular risk factor burden and brain volume was stronger when vascular risk factors were measured at younger ages. For example, vascular risk factor burden was most strongly associated with lower brain volume in later life when vascular risk factors were measured at age 45 years. Conclusion Vascular risk factors at younger ages appear to have detrimental effects on current and future brain volume.