Previously published data from our group and others demonstrate that human milk oligosaccharide (HMOs), as well as milk and infant fecal microbial profiles, vary by geography. However, little is known about the geographical variation of other milk-borne factors, such as lactose and protein, as well as the associations among these factors and microbial community structures in milk and infant feces. Here, we characterized and contrasted concentrations of milk-borne lactose, protein, and HMOs, and examined their associations with milk and infant fecal microbiomes in samples collected in 11 geographically diverse sites. Although geographical site was strongly associated with milk and infant fecal microbiomes, both sample types assorted into a smaller number of community state types based on shared microbial profiles. Similar to HMOs, concentrations of lactose and protein also varied by geography. Concentrations of HMOs, lactose, and protein were associated with differences in the microbial community structures of milk and infant feces and in the abundance of specific taxa. Taken together, these data suggest that the composition of human milk, even when produced by relatively healthy women, differs based on geographical boundaries and that concentrations of HMOs, lactose, and protein in milk are related to variation in milk and infant fecal microbial communities.