The self-assembly of colloidal magnetic Janus particles with a laterally displaced (or shifted), permanent dipole in a quasi-two-dimensional system is studied using Brownian dynamics simulations. The rate of formation of clusters and their structures are quantified for several values of dipolar shift from the particle center, which is nondimensionalized using the particle's radius so that it takes values ranging from 0 to 1, and examined under different magnetic interaction strengths relative to Brownian motion. For dipolar shifts close to 0, chain-like structures are formed, which grow at long times following a power law, while particles of shift higher than 0.2 generally aggregate in ring-like clusters that experience limited growth. In the case of shifts between 0.4 and 0.5, the particles tend to aggregate in clusters of 3 to 6, while for all shifts higher than 0.6 clusters rarely contain more than 3 particles due to the antiparallel dipole orientations that are most stable at those shifts. The strength of the magnetic interactions hastens the rate at which clusters are formed; however, the effect it has on cluster size is lessened by increases in the shift of the dipoles. These results contribute to better understand the dynamics of magnetic Janus particles and can help the synthesis of functionalized materials for specific applications such as drug delivery.