As single-junction silicon solar cells approach their theoretical limits, tandems provide the primary path to higher efficiencies. CdTe alloys can be tuned with magnesium (CdMgTe) or zinc (CdZnTe) for ideal tandem pairing with silicon. A II-VI/Si tandem holds the greatest promise for inexpensive, high-efficiency top cells that can be quickly deployed in the market using existing polycrystalline CdTe manufacturing lines combined with mature silicon production lines. Currently, all high efficiency polycrystalline CdTe cells require a chloride-based passivation process to passivate grain boundaries and bulk defects. This research examines the rich chemistry and physics that has historically limited performance when extending Cl treatments to polycrystalline 1.7-eV CdMgTe and CdZnTe absorbers. A combination of transmittance, quantum efficiency, photoluminescence, transmission electron microscopy, and energy-dispersive X-ray spectroscopy clearly reveals that during passivation, Mg segregates and out-diffuses, initially at the grain boundaries but eventually throughout the bulk. CdZnTe exhibits similar Zn segregation behavior; however, the onset and progression is localized to the back of the device. After passivation, CdMgTe and CdZnTe can render a layer that is reduced to predominantly CdTe electro-optical behavior. Contact instabilities caused by inter-diffusion between the layers create additional complications. The results outline critical issues and paths for these materials to be successfully implemented in Si-based tandems and other applications.