Respiratory failure in COVID-19 is characterized by widespread disruption of the lung’s alveolar gas exchange interface. To elucidate determinants of alveolar lung damage, we performed epithelial and immune cell profiling in lungs from 24 COVID-19 autopsies and 43 uninfected organ donors ages 18–92 years. We found marked loss of type 2 alveolar epithelial (T2AE) cells and increased perialveolar lymphocyte cytotoxicity in all fatal COVID-19 cases, even at early stages before typical patterns of acute lung injury are histologically apparent. In lungs from uninfected organ donors, there was also progressive loss of T2AE cells with increasing age, which may increase susceptibility to COVID-19–mediated lung damage in older individuals. In the fatal COVID-19 cases, macrophage infiltration differed according to the histopathological pattern of lung injury. In cases with acute lung injury, we found accumulation of CD4+ macrophages that expressed distinctly high levels of T cell activation and costimulation genes and strongly correlated with increased extent of alveolar epithelial cell depletion and CD8+ T cell cytotoxicity. Together, our results show that T2AE cell deficiency may underlie age-related COVID-19 risk and initiate alveolar dysfunction shortly after infection, and we define immune cell mediators that may contribute to alveolar injury in distinct pathological stages of fatal COVID-19.