Edit 12 October 2022
by UIA Permanent Secretariat, AEIDL & Eutropian
Integrated development in action!
Report homepage

Evaluating integrated approaches


This chapter describes how evaluation can render insights into projects in relation to integrated territorial development. It looks specifically at how the added-value of place-based, partnership and cross-sectoral approaches was assessed. The information in this chapter is intended to be complementary to the previous UIA study which looked at to monitoring and evaluation. The section is structured around a set of takeaways to consider how to better integrate the evaluation of the integrated territorial development within the overall evaluation logic of projects.

Lahti’s CitiCap APP showing individual data that can be used for trading CO2 allowances under the project’s voluntary Personal Carbon Trading scheme
Lahti’s CitiCap APP showing individual data that can be used for trading CO2 allowances under the project’s voluntary Personal Carbon Trading scheme. 


Definition and interpretations

Evaluation has long been an obligation for EU funded projects, not only to comply with conventional accountability procedures of public expenditure but also to improve policies and programme performance. The UIA Permanent Secretariat together with Ecorys developed a significant knowledge base on this topic, available online.

 We focus here on a less explored aspect of evaluation: how to measure the added value of integrated territorial development, and more specifically:

  • What methodologies have cities used to measure the added value of multi-sectoral approaches and, to what extent was it possible to assess the interrelation between indicators from different realms (social, environmental, cultural, economic)? 
  • How did being a UIA project encourage comprehensive and joint monitoring and evaluation of results and to what extent did the involvement of different stakeholders/citizens influence the project and does this come through in added value?
  • Did the project have a method for measuring the territorial added value of the project? 

The way to structure and undertake analysis for an evaluation is summarised in the chapter on ‘Evaluation Approaches’ on the UIA web site. Designing the intervention logic of the project, choosing reasonable baseline and target indicators, and collecting data, all these aspects are reported and discussed through real-life examples stemming from UIA cities.

The particularity of the Integrated Territorial Development approach lies in its multifaceted character, combining four principles, as illustrated in the graphic below.


Figure 5.1  Combining the Integrated Territorial Development principles in the Evaluation exercise
Combining the Integrated Territorial Development principles in the Evaluation exercise

Challenges for evaluating so many dimensions may appear off-putting and over-complex, but many UIA projects undertook complex evaluations often with the support of academic partners.

Analysis of the case studies and key takeaways

Academic or research organisations were often one of the partners of the UIA projects consortiums, playing a significant role in designing, conducting and publishing the evaluation process. And while most such processes were not undertaken with the specific aim to capture the integrated nature of the project, they successfully managed to assess the impact each project had in the areas that match the four principles that inform the integrated territorial approach. For example, the Plan Einstein in Utrecht conducted a theory-based evaluation, led by an academic who moved from the University of Oxford to Roehampton University. The evaluation aimed to understand how their place-based and cross-sectoral approach contributed to rethinking asylum seeker reception. Part of their final evaluation included a survey of local residents in Overvecht - the site of the first reception centre - to see how attitudes to the reception centre had changed over time. This was a good example of how place-based aspects can be introduced into an evaluation design. 

By contrast, OASIS in Paris involved three of its project partners to conduct sectoral evaluations, instead of assigning the task to one sole entity. Thus, the Laboratory for Interdisciplinary Evaluation of Public Policies of the Sciences Po (LIEPP) assessed the behavioural aspects of children's wellbeing in the newly redesigned schoolyards, while the Interdisciplinary Laboratory of Energies of Tomorrow (LIED), associated with ESIEE Paris, as well as Météo-France, evaluated the climate impact of the redesign. These sectoral evaluations revealed the integrated approach of OASIS, combining place-based intervention and a participative approach in order to achieve cross-sectoral impact in both rethinking education policies and contributing to the climate resiliency plan.

In some other UIA projects, academic partners were also involved in training future evaluators. For example, in the case of USE-IT in Birmingham, both Birmingham University and the Birmingham Voluntary Service Council developed courses for training community researchers. The researchers later evaluated skills and the process of integration of the migrants in the regional public organisations.

Key takeaway 1

Ensure that your evaluators engage collaboratively in the evaluation and avoid being an external voice. In order to do so, it is useful to consider involving a research organisation as a project partner, so that the evaluation is an ongoing process that also benefits from an insider perspective.  Use participative methodologies to elicit the views of stakeholders and citizens and remember that participating in an evaluation is part of the learning curve for all partners.

Change can be assessed in various ways, each carrying its own set of values and indicators. From generating policy and governance improvements, behavioural or cultural norms changes, and different business models to an entire new spatial reconfiguration, all these aspects represent different facets of a project’s impact. 

UIA projects chose to generally focus on commissioning evaluations related to their core thematic area. For example, Vilawatt in Viladecans, as a project focusing on energy transition, initially chose to develop three separate evaluations related to its thematic focus: a social and governance impact report, an economic impact report and an energy impact report. It was one of the project partners, BCNecologia, the Urban Ecology Agency of Barcelona, that suggested merging all three evaluations into one, so that the monitored indicators will allow a better understanding of the cross-sectoral impact. 

Similarly, Air Heritage in Portici, acknowledged that it needed to complement its original evaluation methodology focused on the quality of air to include new datasets related to behavioural and cultural change on air pollution. For this, the project underwent extensive qualitative research engaging university students, assessing their behaviours before and after certain project activities in order to document changes in perception about air pollution.

In other cases, such as of USE-IT! in Birmingham (see Box 6.1) the evaluation wanted to capture from the beginning how the project contributed to a cultural change on the relationships between stakeholders. Or, as in the case of Home Silk Road in Lyon Metropole, where its approach to evaluation purposely aimed to capture the added value of its co-design practices. Specifically, it assessed how the partnership logic contributed to enriching the project implementation and to fostering its innovative approach toward existing housing and cultural policies. As a result, the project managed to document the interrelation of combining different intervention logic and needs. On the one hand, the GIE La Ville Autrement which needed a site to build a new emergency shelter, and on the other hand, the cultural and social innovation laboratory CCO, with a history of half a century in Villeurbanne, which was looking for a new location and a bigger concert hall. Another notable mention is that the project's stakeholders were involved in outlining the evaluation methodology, and co-designing the evaluation questions, criteria and indicators. This approach facilitated that the elements that are captured through evaluation have a high relevance for stakeholders and can better serve their diverse outlooks for making the case on the connection between their activities and larger policy objectives related to equitable access to housing and cultural revitalisation policies.

In fact, this was the later realisation of the Vilawatt team, that the implementation of their project enabled a fast advancement of their own internal model of innovation that allows a strong mindset change in the public administration of Viladecans to be geared towards partnership, experimentation and problem solving.

Focus on evaluating system change outcomes

USE-IT! in Birmingham tested an alternative approach to prior urban regeneration interventions, one that is focused on recognizing and enabling the assets of local communities to be connected to the anchor institutions of the area such as hospitals, and universities. As the project was trying to make the case why its approach is different to the previous urban regeneration efforts of the last 40 years, it also paid special attention in evaluating the place-based systemic changes it generated.

Thus, in addition to evaluating the impact of each work package, USE-IT brought evidence on how it enabled a cultural change on the relationships that can exist between anchor organisations and members of a localised deprived community.

“A place is complex and multifaceted. Although, by working through an ecosystem approach we were able to prove that all partners were valuable for the local economy, even the social enterprises have been recognised to be taken into account in regeneration projects” said USE-IT municipality representative.

For this, the project collaborated with a regional think-tank - the Centre for Local Economic Strategies (CLES). The evaluation process was grouped into two phases: a retrospective process of impact evaluation of each work package, and a final longitudinal study of the impact of activity in the final year of the programme. By the end of this process, there was documented evidence on how USE-IT!’s model for recruitment of foreign-born medical professionals for the local NHS branch provided a policy and cultural change of an anchor institution's approach to employment. Nowadays, this practice is being scaled up across the Black Country and recognised throughout the UK. Moreover, Birmingham City Council now aims to replicate USE-Its! model of bridging the place-based assets at micro level (targeted community) with the regional level (anchor organisations) for other inclusive urban development programmes.


Key takeaway 2

Recognize the diversity of change-making processes and their impact. Consider not only the impact in your thematic area of focus, but also how these changes are significant in your current socio-economic and cultural context. Go beyond measuring sectoral results, and try to evaluate the processes e.g. collaboration, empowerment, decision making.

Relevant datasets are needed in order to be able to capture the impact of a measure on a territorial scale. Several UIA projects showed that they needed to invest in the data collection technology itself and in thorough methodologies in order to determine baseline values. As such, projects such as CitiCap in Lahti (see Box 5.2) employed several innovation patents owned by private entities which were partners in the projects in order to develop the technology that could measure a personal carbon footprint. In addition, the university partners worked closely over a period of six months in order to determine what is an average value that could then be used as a reference point for the future actions in the project. Similarly, Prato Urban Jungle in Prato installed over 30 sensors, to complement the only two existing ones. The sensors were able to capture before/after data related to the impact of the forestation interventions, but also map mobility patterns. As such, these new datasets and especially the methodology of how data is collected will be used in the future to support the city reach its climate neutrality goals.

Invest in creating a baseline to facilitate measuring added value

CitiCap in Lahti, Finland, focused on how to enable the city’s transition toward sustainable mobility by formulating the city’s first Sustainable Urban Mobility Plan (SUMP) informed by two innovative elements: a personal carbon trading (PCT) pilot scheme and a bicycle highway based on co-designed and technology-driven elements. Given the fact that a PCT scheme had not been tested before at the city level in the recent decade in Europe, the project invested significant attention in developing a baseline for the data that would later be evaluated. 

To do so, it used a variety of research methods including postal and online surveys, interviews, participation diaries and workshops with city residents. The gathered data was directly connected to mobility patterns and behaviours of Lahti citizens who opted for using the CITICAP app, but also with key socio-economic indicators such as income, mobility impairments and household mobility needs.

Consequently, the baseline informed the average value proposed by the PCT pilot scheme to be observed by each participating citizen (17- 21 kg CO2eq), but also allowed the overall evaluation of the impact of a PCT scheme for behaviour change toward sustainable mobility options. As a result, the evaluation results captured the cross-sectoral implications of a PCT scheme, both in terms of environmental impact, but also of social norms around what would be a fair trading scheme, as the CitiCap Lahti University of Technology (LUT) representative explains: 

“We knew that there are people who are already below this target line, and some people who have, let's say, 100 kilos a week of CO2 emission. So we were discussing [n.b. in  citizen workshops] who should be the ones to reduce and what would be a fair way? Obviously, there were a lot of different opinions. But on average, people felt that their situation in life should have an impact. And because in this baseline we also asked some background questions from users, we were also able to calculate how much on average your emissions have increased, if you, for example, have one or two children, or if you are living far away from the city centre. So we were able to calculate how many additional emissions allowances you should get from these different aspects of life”.   (Representative of Lahti University of technology)

The results of the evaluation were also fully documented and transferred into academic knowledge, with the core team of CITICAP from the municipality and the Lahti University of Technology (LUT) publishing several peer-reviewed papers in English.

Key takeaway 3

Try to measure the added value of place-based approaches and look for new forms of data to facilitate this. It is noteworthy to consider simultaneously what technologies to employ in order to get place-based data, but also the capabilities of your research team and for what purpose you need the data.