ATLAS measurements of two-particle correlations are presented for s=5.02 and 13 TeV pp collisions and for sNN=5.02 TeV p+Pb collisions at the LHC. The correlation functions are measured as a function of relative azimuthal angle Δφ, and pseudorapidity separation Δη, using charged particles detected within the pseudorapidity interval |η|<2.5. Azimuthal modulation in the long-range component of the correlation function, with |Δη|>2, is studied using a template fitting procedure to remove a "back-to-back" contribution to the correlation function that primarily arises from hard-scattering processes. In addition to the elliptic, cos(2Δφ), modulation observed in a previous measurement, the pp correlation functions exhibit significant cos(3Δφ) and cos(4Δφ) modulation. The Fourier coefficients vn,n associated with the cos(nΔφ) modulation of the correlation functions for n=2-4 are measured as a function of charged-particle multiplicity and charged-particle transverse momentum. The Fourier coefficients are observed to be compatible with cos(nφ) modulation of per-event single-particle azimuthal angle distributions. The single-particle Fourier coefficients vn are measured as a function of charged-particle multiplicity, and charged-particle transverse momentum for n=2-4. The integrated luminosities used in this analysis are, 64 nb-1 for the s=13 TeV pp data, 170 nb-1 for the s=5.02 TeV pp data, and 28 nb-1 for the sNN=5.02 TeV p+Pb data.