The production of a Z boson and a photon in association with a high-mass dijet system is studied using 20.2 fb−1 of proton-proton collision data at a centre-of-mass energy of s=8 TeV recorded with the ATLAS detector in 2012 at the Large Hadron Collider. Final states with a photon and a Z boson decaying into a pair of either electrons, muons, or neutrinos are analysed. Electroweak and total pp → Zγjj cross-sections are extracted in two fiducial regions with different sensitivities to electroweak production processes. Quartic couplings of vector bosons are studied in regions of phase space with an enhanced contribution from pure electroweak production, sensitive to vector-boson scattering processes VV → Zγ. No deviations from Standard Model predictions are observed and constraints are placed on anomalous couplings parameterized by higher-dimensional operators using effective field theory.[Figure not available: see fulltext.].