Under the smart city paradigm, cities are changing at a rapid pace. In this context, it is necessary to develop tools that allow service providers to perform rapid deployments of novel solutions that can be validated by citizens. In this sense, the OrganiCity experimentation-as-a-service platform brings about a unique solution to experiment with new urban services in a co-creative way, among all the involved stakeholders. On top of this, it is also necessary to ensure that users are engaged in the experimentation process, so as to guarantee that the resulting services actually fulfill their needs. In this work, we present the engagement monitoring framework that has been developed within the OrganiCity platform. This framework permits the tailored definition of metrics according to the experiment characteristics and provides valuable information about how citizens react to service modifications and incentivization campaigns.