The project was implemented by the Barcelona City Council together with five delivery partners. Because of the project’s nature as both a public policy intervention and evaluation, almost all partners were involved in evaluation to some extent:
- The Catalan Institute of Public Policy Evaluation (Ivalua) conducted:
- the quantitative impact evaluation;
- the quantitative economic evaluation.
- The Young Foundation carried out the ethnographic study.
- The Institute of Governance and Public Policies of the Autonomous University of Barcelona (IGOP) conducted:
- the analysis of the deployment and effects of the community participation policy;
- the study on the implementation process and governance of the project.
- The Institute of Environmental Science and Technology of the Autonomous University of Barcelona (ICTA) was involved in evaluating the determinants of subjective wellbeing among participants.
- The Institut Internacional per l'Acció Noviolenta (Novact) was in charge of carrying out the evaluation of the implementation of the Real Economy Currency (REC).
The project partners represented different professions and areas of expertise, which provided access to diverse knowledge and enabled synergies.
“I think that the mix between university or researchers from the sociological field et cetera mixed with civil servants, with people in the field, collaborating, it was very interesting because the knowledge that we have developed is very different than if we have put the focus only in one part – the academic part or the policy part.”
(Source: B-MINCOME project hearing)
In particular, B-MINCOME benefited from strong expertise in research methodologies. The project’s access to extensive quantitative expertise allowed it to function both as a public policy intervention and an experiment, enabling strong causal inferences in relation to the intervention’s effectiveness and impact. The B-MINCOME evaluation also represented a complex effort to involve actors beyond project partners (see more under ‘Horizontal issues’).
Co-creation and close cooperation between all project partners were key for the evaluation. The dual nature of the project made it particularly important to find a common design and mutual understanding from the start. Since both public policies and experiments face important limitations, the partners needed to be aware of and account for them during design and implementation. As the evaluators noticed, the partners had to compromise on various ideas (such as the need for eligibility criteria, which were mandatory in the implementation of public policy and inevitably influenced the shape of the respondent sample for the experiment) to get to their end goal and to be at least comfortable with what they were trying to achieve.
“So, from the very start of the project, there were these challenges and they had to be solved through co-creation […] Because, of course, we will only be able to evaluate whatever the City Council will design. And to evaluate in a rigorous way, it should be co-created such that they took into consideration the evaluation point of view and we took into consideration what they wanted to do. We didn’t want to answer questions that were not being asked in the first place." (Source: B-MINCOME project hearing)
B-MINCOME secured substantial resources within the project budget for evaluation, although evaluators noted during the hearing that resources still posed some challenges. The counterfactual experimental design chosen as one of the approaches involved a large research effort. Over 1,400 households were surveyed repeatedly (approximately 1.000 treatment households and over 400 control households, in three waves). While the counterfactual evaluation is often more costly than other approaches, in B-MINCOME it was further supplemented with an ethnographic study and other qualitative evaluations pertaining to different elements of the intervention. Additional funds were also mobilised for integrating all research results. These came from the municipal budget independently of UIA financing.