We use joint observations by the Swift X-ray Telescope (XRT) and the Fermi Large Area Telescope (LAT) of gamma-ray burst (GRB) afterglows to investigate the nature of the long-lived high-energy emission observed by Fermi LAT. Joint broadband spectral modeling of XRT and LAT data reveals that LAT nondetections of bright X-ray afterglows are consistent with a cooling break in the inferred electron synchrotron spectrum below the LAT and/or XRT energy ranges. Such a break is sufficient to suppress the high-energy emission so as to be below the LAT detection threshold. By contrast, LAT-detected bursts are best fit by a synchrotron spectrum with a cooling break that lies either between or above the XRT and LAT energy ranges. We speculate that the primary difference between GRBs with LAT afterglow detections and the nondetected population may be in the type of circumstellar environment in which these bursts occur, with late-time LAT detections preferentially selecting GRBs that occur in low wind-like circumburst density profiles. Furthermore, we find no evidence of high-energy emission in the LAT-detected population significantly in excess of the flux expected from the electron synchrotron spectrum fit to the observed X-ray emission. The lack of excess emission at high energies could be due to a shocked external medium in which the energy density in the magnetic field is stronger than or comparable to that of the relativistic electrons behind the shock, precluding the production of a dominant synchrotron self-Compton (SSC) component in the LAT energy range. Alternatively, the peak of the SSC emission could be beyond the 0.1-100 GeV energy range considered for this analysis.