Today's complex software systems consist of several components that interact in complex ways to provide services to users. In doing so, these systems go through continuous assessment of their context and configure themselves accordingly to keep user satisfaction high. A popular approach to design adaptive software systems is to perform variability modelling, for instance adopting a feature-based approach. Features describe key components and characteristics of a system, which can take different values and be combined in different ways to obtain a system behavior that can best satisfy the needs of different users, who may use the software in different contexts. These design-time models should be complemented by rules that help in deciding when to switch from one valid system configuration to a different one to fit changing user needs or preferences. Eliciting information necessary to build suitable feature models, as well as rules for dynamic reconfigurations that cover relevant scenarios is not an easy task when considering dynamic adaptation in presence of high variability in user profiles. We are experiencing this issue in a project which aims at developing dynamically personalisable software, and specifically a dynamically configurable feedback gathering tool. In this vision paper we propose to use crowdsourcing to elicit knowledge about reconfiguration requirements for dynamically adaptive systems. The proposed approach rests on a two-stage process, which involves the contribution from the crowd of potential system users, as well as from domain experts.