Anisotropic potentials in Janus colloids provide additional freedom to control particle aggregation into structures of different sizes and morphologies. In this work, we perform Brownian dynamics simulations of a dilute suspension of magnetic spherical Janus colloids with their magnetic dipole moments shifted radially towards the surface of the particle in order to gain valuable microstructural insight. Properties such as the mean cluster size, orientational ordering, and nucleation and growth are examined dynamically. Differences in the structure of clusters and in the aggregation process are observed depending on the dipolar shift (s)-the ratio between the displacement of the dipole and the particle radius- A nd the dipolar coupling constant (λ)-the ratio between the magnetic dipole-dipole and Brownian forces. Using these two dimensionless quantities, a structural "phase" diagram is constructed. Each phase corresponds to unique nucleation and growth behavior and orientational ordering of dipoles inside clusters. At small λ, the particles aggregate and disaggregate resulting in short-lived clusters at small s, while at high s the particles aggregate in permanent triplets (long-lived clusters). At high λ, the critical nuclei formed during the nucleation process are triplets and quadruplets with unique orientational ordering. These small clusters then serve as building blocks to form larger structures, such as single-chain, loop-like, island-like, worm-like, and antiparallel-double-chain clusters. This study shows that dipolar shifts in colloids can serve as a control parameter in applications where unique size, morphology, and aggregation kinetics of clusters are required.