A search for dark matter (DM) particles produced in association with a hadronically decaying vector boson is performed using pp collision data at a centre-of-mass energy of s=13 TeV corresponding to an integrated luminosity of 36.1 fb−1, recorded by the ATLAS detector at the Large Hadron Collider. This analysis improves on previous searches for processes with hadronic decays of W and Z bosons in association with large missing transverse momentum (mono-W/Z searches) due to the larger dataset and further optimization of the event selection and signal region definitions. In addition to the mono-W/Z search, the as yet unexplored hypothesis of a new vector boson Z′ produced in association with dark matter is considered (mono-Z′ search). No significant excess over the Standard Model prediction is observed. The results of the mono-W/Z search are interpreted in terms of limits on invisible Higgs boson decays into dark matter particles, constraints on the parameter space of the simplified vector-mediator model and generic upper limits on the visible cross sections for W/Z+DM production. The results of the mono-Z′ search are shown in the framework of several simplified-model scenarios involving DM production in association with the Z′ boson.[Figure not available: see fulltext.].