In this paper, we address the evaluation of algorithms for partial shape retrieval using a large-scale simulated benchmark of partial views which are used as queries. Since the scanning of real objects is a time-consuming task, we create a simulation that generates a set of views from a target model and at different levels of complexity (amount of missing data). In total, our benchmark contains 7,200 partial views. Furthermore, we propose the use of weighted effectiveness measures based on the complexity of a query. With these characteristics, we aim at jointly evaluating the effectiveness, efficiency and robustness of existing algorithms. As a result of our evaluation, we found that a combination of methods provides the best effectiveness, mainly due to the complementary information that they deliver. The obtained results open new questions regarding the difficulty of the partial shape retrieval problem. As a consequence, potential future directions are also identified.