Charge ordering is prone to occur in crystalline materials with mixed-valence ions. It is presumably accompanied by a structural phase transition, with possible exceptions in compounds that already present more than one inequivalent site for the mixed-valence ions in the charge-disordered phase. In this work, we investigate the representative case of the homometallic Co ludwigite Co22+Co3+O2BO3 (Pbam space group) with four distinct Co crystallographic sites [M1-M4] surrounded by oxygen octahedra. The mixed-valent character of the Co ions up to at least T=873 K is verified through x-ray absorption near-edge structure (XANES) experiments. Single crystal x-ray diffraction (XRD) and neutron powder diffraction (NPD) confirm that the Co ions at the M4 site are much smaller than the others at low temperatures, consistent with a Co3+ oxidation state at M4 and Co2+ at the remaining sites. The size difference between the Co ions in the M4 and M2 sites is continuously reduced upon warming above ≈370 K, indicating a gradual charge redistribution within the M4-M2-M4 (424) ladder in the average structure. Minor structural anomalies with no space group modification are observed near 475 and 495 K, where sharp phase transitions were previously revealed by calorimetry and electrical resistivity data. An increasing structural disorder, beyond a conventional thermal effect, is noted above ≈370 K, manifested by an anomalous increment of XRD Debye-Waller factors and broadened vibrational modes observed by Raman scattering. The local Co-O distance distribution, revealed by Co K-edge extended x-ray absorption fine structure (EXAFS) data and analyzed with an evolutionary algorithm method, is similar to that inferred from the XRD crystal structure below ≈370 K. At higher temperatures, the local Co-O distance distribution remains similar to that found at low temperatures, at variance with the average crystal structure obtained with XRD. We conclude that the oxidation states Co2+ and Co3+ are instantaneously well defined in a local atomic level at all temperatures, however the thermal energy promotes local defects in the charge-ordered configuration of the 424 ladders upon warming. These defects coalesce into a phase-segregated state within a narrow temperature interval (475<T<495 K). Finally, a transition at ≈500 K revealed by differential scanning calorimetry (DSC) in the iron ludwigite Fe3O2BO3 is discussed.