People with disabilities (PWD) face several challenges accessing medical services. However, the extent to which architectural and transportation barriers impede access to healthcare is unknown. In Peru, despite laws requiring that buildings be accessible for PWD, no report confirms that medical facilities comply with such regulations. Thus, we aim to provide an association between these barriers and access to medical facilities. Data from a Peruvian disability survey were analyzed. Participants were 18 years of age and older people who reported having a physical disability. Accessibility was defined by reported struggles accessing medical facilities (health or rehabilitation centers). Absence of ramps, handrails, elevators, adapted bathrooms, and information counters in medical facilities were reported as architectural barriers. The transportation barriers analyzed included struggles using buses or trains. Poisson regression models with robust variance were used to estimate prevalence ratios (PR) and to control for confounding variables. 20,663 participants were included, their mean age was 66.5 years and 57.5% were females. Architectural and transportation barriers reported were 40% and 61%, respectively. All barriers reported were more prevalent in rural compared to urban areas (p < 0.001). Inadequacy of ramps, handrails, and adapted elevators and bathrooms were associated with limited use of rehabilitation centers (p < 0.001) but not of health centers (p > 0.05). Architectural and transportation barriers represent a hindrance to seeking treatment at rehabilitation centers. Actions to improve this situation are needed.