Evaluating integrated approaches

Evaluation has long been an obligation for EU funded projects, not only to comply with conventional accountability procedures of public expenditure but also to improve policies and programme performance. The UIA Permanent Secretariat together with Ecorys developed a significant knowledge base on this topic, available online.

 We focus here on a less explored aspect of evaluation: how to measure the added value of integrated territorial development, and more specifically:

• What methodologies have cities used to measure the added value of multi-sectoral approaches and, to what extent was it possible to assess the interrelation between indicators from different realms (social, environmental, cultural, economic)? 
• How did being a UIA project encourage comprehensive and joint monitoring and evaluation of results and to what extent did the involvement of different stakeholders/citizens influence the project and does this come through in added value?
• Did the project have a method for measuring the territorial added value of the project? 

The way to structure and undertake analysis for an evaluation is summarised in the chapter on ‘Evaluation Approaches’ on the UIA web site. Designing the intervention logic of the project, choosing reasonable baseline and target indicators, and collecting data, all these aspects are reported and discussed through real-life examples stemming from UIA cities.

The particularity of the Integrated Territorial Development approach lies in its multifaceted character, combining four principles, as illustrated in the graphic below.

Challenges for evaluating so many dimensions may appear off-putting and over-complex, but many UIA projects undertook complex evaluations often with the support of academic partners.

Academic or research organisations were often one of the partners of the UIA projects consortiums, playing a significant role in designing, conducting and publishing the evaluation process. And while most such processes were not undertaken with the specific aim to capture the integrated nature of the project, they successfully managed to assess the impact each project had in the areas that match the four principles that inform the integrated territorial approach. For example, the Plan Einstein in Utrecht conducted a theory-based evaluation, led by an academic who moved from the University of Oxford to Roehampton University. The evaluation aimed to understand how their place-based and cross-sectoral approach contributed to rethinking asylum seeker reception. Part of their final evaluation included a survey of local residents in Overvecht - the site of the first reception centre - to see how attitudes to the reception centre had changed over time. This was a good example of how place-based aspects can be introduced into an evaluation design. 

By contrast, OASIS in Paris involved three of its project partners to conduct sectoral evaluations, instead of assigning the task to one sole entity. Thus, the Laboratory for Interdisciplinary Evaluation of Public Policies of the Sciences Po (LIEPP) assessed the behavioural aspects of children's wellbeing in the newly redesigned schoolyards, while the Interdisciplinary Laboratory of Energies of Tomorrow (LIED), associated with ESIEE Paris, as well as Météo-France, evaluated the climate impact of the redesign. These sectoral evaluations revealed the integrated approach of OASIS, combining place-based intervention and a participative approach in order to achieve cross-sectoral impact in both rethinking education policies and contributing to the climate resiliency plan.

In some other UIA projects, academic partners were also involved in training future evaluators. For example, in the case of USE-IT in Birmingham, both Birmingham University and the Birmingham Voluntary Service Council developed courses for training community researchers. The researchers later evaluated skills and the process of integration of the migrants in the regional public organisations.

Change can be assessed in various ways, each carrying its own set of values and indicators. From generating policy and governance improvements, behavioural or cultural norms changes, and different business models to an entire new spatial reconfiguration, all these aspects represent different facets of a project’s impact. 

UIA projects chose to generally focus on commissioning evaluations related to their core thematic area. For example, Vilawatt in Viladecans, as a project focusing on energy transition, initially chose to develop three separate evaluations related to its thematic focus: a social and governance impact report, an economic impact report and an energy impact report. It was one of the project partners, BCNecologia, the Urban Ecology Agency of Barcelona, that suggested merging all three evaluations into one, so that the monitored indicators will allow a better understanding of the cross-sectoral impact. 

Similarly, Air Heritage in Portici, acknowledged that it needed to complement its original evaluation methodology focused on the quality of air to include new datasets related to behavioural and cultural change on air pollution. For this, the project underwent extensive qualitative research engaging university students, assessing their behaviours before and after certain project activities in order to document changes in perception about air pollution.

In other cases, such as of USE-IT! in Birmingham (see Box 6.1) the evaluation wanted to capture from the beginning how the project contributed to a cultural change on the relationships between stakeholders. Or, as in the case of Home Silk Road in Lyon Metropole, where its approach to evaluation purposely aimed to capture the added value of its co-design practices. Specifically, it assessed how the partnership logic contributed to enriching the project implementation and to fostering its innovative approach toward existing housing and cultural policies. As a result, the project managed to document the interrelation of combining different intervention logic and needs. On the one hand, the GIE La Ville Autrement which needed a site to build a new emergency shelter, and on the other hand, the cultural and social innovation laboratory CCO, with a history of half a century in Villeurbanne, which was looking for a new location and a bigger concert hall. Another notable mention is that the project's stakeholders were involved in outlining the evaluation methodology, and co-designing the evaluation questions, criteria and indicators. This approach facilitated that the elements that are captured through evaluation have a high relevance for stakeholders and can better serve their diverse outlooks for making the case on the connection between their activities and larger policy objectives related to equitable access to housing and cultural revitalisation policies.

In fact, this was the later realisation of the Vilawatt team, that the implementation of their project enabled a fast advancement of their own internal model of innovation that allows a strong mindset change in the public administration of Viladecans to be geared towards partnership, experimentation and problem solving.

Relevant datasets are needed in order to be able to capture the impact of a measure on a territorial scale. Several UIA projects showed that they needed to invest in the data collection technology itself and in thorough methodologies in order to determine baseline values. As such, projects such as CitiCap in Lahti (see Box 5.2) employed several innovation patents owned by private entities which were partners in the projects in order to develop the technology that could measure a personal carbon footprint. In addition, the university partners worked closely over a period of six months in order to determine what is an average value that could then be used as a reference point for the future actions in the project. Similarly, Prato Urban Jungle in Prato installed over 30 sensors, to complement the only two existing ones. The sensors were able to capture before/after data related to the impact of the forestation interventions, but also map mobility patterns. As such, these new datasets and especially the methodology of how data is collected will be used in the future to support the city reach its climate neutrality goals.