The challenge of evaluating an innovative housing project

When you walk through Mataró, a city of 127,000 inhabitants 34 km north-east of Barcelona at the Costa del Maresme, you might not realize it at first sight, but the city has more than 2.800 private empty flats. At the same time an increasing number of the inhabitants are overburdened by the rental housing cost or they have little access to the private rental market, in particular younger people. Since 2015, rental prices in Mataró have been growing by 27 %. The expenses for rent can be significantly over a 1/3 of the monthly income. Moreover, there is a dramatic shortage of affordable housing in the city. The city’s social housing sector is totally incommensurate to demand. It comprises just 265 units while 1400 households are currently on the waiting list.

The ‘Yes, we rent!’ project of the city of Mataró aims to increase the number of affordable, privately rented flats in the city and improve their accessibility to households with difficulties in the private rental housing market. Key to this is a cooperative-based housing approach, which incentivises owners of empty flats to rehabilitate their flats and rent it below market price through the cooperative.

General objective to evaluate/monitor the ‘Yes, we rent!’ project / your motivation to evaluate the project.[Darstellung des Ziel des Monitorings und was damit erreicht werden soll/ warum das Projekt evaluiert wird]

In order to evaluate the successes and impact of the project the city of Mataró developed with the support of TecnoCampus Mataró-Maresme and the Institute on Government and Public Policy (IGOP) of the Autonomous University of Barcelona an evaluation scheme, for which implementation the two partners are in charge. The monitoring is done with a view to evaluate:

• Housing market changes: the changes in the provision and accessibility as well as the quality of affordable rental flats in Mataró.
• Institutional changes: the institutional and governance changes in the city administration for the provision of affordable rental housing.
• Organisational changes: the economical and institutional capability of a cooperative to take on the function of an organiser and mediator between owners of vacant flats and household looking for affordable rental accommodation.
• Personal changes: the effects on the lifes of the cooperative members who can enjoy affordable housing and the willingness of owners to let their vacant flats below market price.

Based on the evaluation results the city of Mataró intends to take decisions on the continuity of the project and its possible scalability. The evaluation aims also at the replication of the project in other cities in Catalonia.

To be able to evaluate the above mentioned “changes”, TecnoCampus and the Institute on Government and Public Policy have developed for each evaluation area research questions and their corresponding hypotheses as well as qualitative and quantitative indicators to be able to analyse the effect and the variables.

The “changes” are primarily measured:

§  “Personal change” through owner and tenant surveys as well as secondary data analysis from tax and census/ population register and from local health authorities;

§  “Institutional change” through questionnaires, focus groups with tenants, municipal staff and owners and interviews i.e. with the technical office of the ‘Yes, we rent! Project, other involved technicians from the city administration and the city council;

§  “Organisational change” through questionnaires, focus groups with tenants, municipal staff and owners, interviews with members of the driving group of the cooperative and observations taking part in meetings of the cooperative and working sessions of the cooperative with the municipality and the technical support team;

§  “Housing market change” through secondary data analysis from the tax and census/ population register as well as from two major real state agencies.

For a detailed overview about the evaluation scheme (in Spanish) download this document.

While TecnoCampus and IGOP implemented the evaluation scheme different challenges arose.

GDPR restrictions

The legal requirements of the General Data Protection Regulation (GDPR) make it difficult to TecnoCampus and IGOP to receive access to the city administration data (and other sources) to be able to monitor the project as planned in the evaluation scheme. This required the two partners to clarify all binding steps of the data protection regulation and, based on the results, to work out together with the city council a legal framework and a procedure to obtain legal and periodically the data of the different city administration units and other sources. Also with other institutions legal documents had to be prepared to get data access for the monitoring of the project.

This circumstance required time and familiarisation with a new matter on all sides. To facilitate this, the partners hired external support to prepare the legal frameworks. As the project continued, relevant data could be retrieved first at a later stage of the project.

Vague or changing project objectives

Innovative projects cannot be planned in all details from the beginning and evolve. This is also the case with the 'Yes, we rent! project. As the project has evolved and gained focus over the time, also objectives, outcomes and potential impacts became clearer and even changed. This demanded the adaptation of the evaluation scheme and data frame during the ongoing monitoring process. Thus, the project could not be monitored the same way from the beginning.

COVID 19

Covid-19  – as everywhere – had its impact on the monitoring of the project. Due to the COVID-19 restrictions interviews and direct observations could not be carried out as planned. Focus group discussions had to be postponed. As solution interviews were shifted online and the observations concentrated on online-meetings of the project partners and the ‘driving group’ of the cooperative. This shift from qualitative evaluation fieldwork to “online” fieldwork demanded its time.

Lack of input or collection of data by project participants

For the monitoring, data had to be collected and provided by project participants during the implementation of the project. But some project participants did not comply with this to the extent required filling in monitoring documents or collecting relevant data as requested in the evaluation scheme. The reasons may have been that the importance for the data collection was not seen. To counteract this TecnoCampus and IGOP had to encourage partners to fill in questionnaires and monitoring forms, explaining the necessity and meaningfulness of the evaluation of the project. In support they set up a data register and follow-up system.

Also some administrative units, not directly involved in the project, did not comply with the data provision as expected. To gain their commitment, extra meetings were organised to explain the importance of the required administrative data for the evaluation of the project. Also examples of legal frameworks from other projects were presented to gain their support.

Based on the experience gained so far, the following recommendations for monitoring similar projects are made:

§  Develop and agree on legal framework and general procedure on data transfer before the project starts. For this, get together with the necessary institutions early on and get professional support if needed.

§  When implementing new approaches be open and flexible to adapt your monitoring and evaluation scheme, even if it demands extra workload. So check your monitoring scheme once a while whether it still meets the adapted objectives and can measure relevant impacts of the project.

§  In the beginning explain well to institutions from which you need data why certain data is important and needs to be collected and provided by them. Consider how these institutions can also benefit from the evaluation.

§  When being responsible for the evaluation, engage in non-participatory observation so as to maintain one’s status as neutral observer and to gain the trust of all participants. Do not take sites in discussions or get directly involved to enable frank dialogue during interviews and observations.

§  Measuring impacts is often only possible after a certain period of time has elapsed since the project was implemented. Thus, consider monitoring and evaluation activities even after a longer period of time after the pilot project has been completed.