Four-dimensional seismic (4DS) contains spatial information that provides insights into the location, shape, and movement of fluids (oil, gas, water). It helps engineers to adjust reservoir simulation models and increase their capability of providing reliable production forecasts. Recent probabilistic approaches consider hundreds of numerical simulation model scenarios, which require automated methods to evaluate this large number of numerical models based on observed 4D seismic data. Comparing spatial information of seismic and numerical simulation data is difficult as, usually, these data are converted to maps with different properties. We propose a novel approach to compare 4D seismic data and simulation models using a three-way deep neural network that is trained using a reference image (4D seismic data) with two simulation model candidates. It learns to find the simulation models that best characterize the reference. Our method is underpinned by more than a thousand pairs of simulation models and reference maps evaluated by human specialists for training. For testing, we compare the inter-rater agreement among different specialist groups and generate a reliable test set considering examples in which there was agreement among at least two specialists. We observed that the group with best-trained specialists agree more in their answers and have a considerably higher inter-rater agreement than the less trained groups. When we evaluate our method with the answers from this specialized group, we observe that the simulation model chosen by our method is the one agreed by the specialists in almost 90% of the cases. We also discuss the impact of different noise levels in the input and show that our method outperforms other approaches in the literature if noise is present both in the training and test sets.