Expert article
Edit 20 March 2022
by Constantinos Cartalis

Evaluation - An indispensable component of innovation projects

Evaluation - an indispensable part of innovation projects
Evaluation - an indispensable part of innovation projects
Evaluation is an indispensable component of any project, especially innovation ones. It provides an understanding of the reliability of the applied methodologies, processes, plans, etc. as far as the achievement of the anticipated outcomes. It also supports the early recognition of inconsistencies and errors, a fact that is highly supportive of rectification actions. In this Web Article, a part of the evaluation process of the Climate Shelters project of the City of Barcelona is provided and discussed accordingly. The evaluation part refers to the air temperature field in the schoolyards following the implementation of blue, green, and grey interventions.

Introduction

The Climate Shelters (GBG_A2CC) project of the City of Barcelona aims at the development of a network of Climate Shelters by converting schoolyards by means of blue, green, and grey interventions. The project supports the adaptation of Barcelona to climate change in terms of extreme temperatures and heatwaves.

The project also supports the implementation of the Sustainable Development Goals (SDG) of Barcelona as presented in the document “Barcelona: Sustainable Future (Barcelona Sustainable Future – Seventeen social, economic and environmental objectives; https://ajuntament.barcelona.cat/agenda2030/ca) namely:

SDG 13.1- By 2030, 100% of the population will be less than 300 meters from a climate shelter and with a water garden in every district.

SDG 13.3 – In the decade from 2020 to 2030, Barcelona will have effective tools for improving the education, awareness-raising, and human and institutional capacity for the mitigation, adaptation, impact reduction, and early warning of climate change.

Barcelona Sustainable future


The project is supported by the Urban Innovation Actions (UIA) program of the European Commission (https://uia-initiative.eu/en/uia-cities/barcelona-call3). It includes a solid monitoring and evaluation process in order to assess if, and to which extent, the implemented interventions fulfilled the objectives of the project.

Why the evaluation process is important?

Evaluation is a continuous function to assess progress in achiev­ing expected results, to define bottlenecks in implementation, and to highlight whether there are any unintended effects from the project and its activities.

Evaluation facilitates learning from past successes and challenges and those encountered during implementation. It links to the processes of planning and monitoring. Planning helps to design the road map towards the objectives of the project while monitoring supports the collection of data in support of the assessment of changes.

The elements of an evaluation process – which if developed together with all key stakeholders will encourage participation,  increase ownership of the results, and  better inform and influence decision-making and public policies design - are:

(a) tools to organize intended results, i.e. measurable changes;

(b) the monitoring  plan, which contains a description of the processes to gather the relevant data on the set indicators as well as of the required tools. In particular, the monitoring plan systematically organizes the collection of specific data for analysis to support real-time, evidence-based decision-making;

(c) the various processes and methods for monitoring (such as regular input and output data gathering and review, participatory monitoring, process monitoring, accuracy definition, and error analysis) and for evaluation (including impact evaluation, environmental assessment, statistical analysis, etc.);

and, (d) the information system which is an organized repository of data to assist evaluation and analysis. In this Web Article, the evaluation process is described and the results regarding changes in the outdoor (schoolyard) air temperature field are presented for two case studies: Font d’en Fargas school and Poblenou school.

Results reflect positive changes due to the implementation of green, blue, and grey interventions at the schoolyards of the project. It should be mentioned that the evaluation process was also performed for the indoor air temperature fields, yet the results for the indoor air temperature fields will be discussed in a future publication.

The evaluation process of the Climate Shelters project

In the framework of the Climate Shelters project, the City of Barcelona designed and performed a well-structured evaluation process aiming to assess, among other parameters, the outdoor air temperature field at the schoolyards of the schools converted to Climate Shelters. The evaluation was designed and performed by the Barcelona Institute for Global Health (ISGlobal) and the Public Health Agency of Barcelona (ASPB), while  it also focused on other aspects as well, such as : (a) the indoor air temperature field (b) the developed dynamics in the playground in terms of the diversification of play possibilities and socializing patterns, (c) health, well-being and types and levels of physical activity of the users, and (d) the attention levels in 5th grade children (to be published by ISGlobal in the near future). 

Smart Citizen Kits (SCKs) were installed (Figure 1) outdoors at all schools participating in the Climate Shelters project; continuous measurements were made for the period May 2020 to September 2021. The scope of the measurements was to capture the initial (pre-interventions) and final (post-interventions) air temperature fields in the schoolyards. Control schools (i.e. no interventions implemented) were also monitored for comparison purposes, following the same approach.

Table 1 provides the type of measurements performed and the associated parameters.

Figure 1. The Smart Citizen Kit (Source: ISGlobal).
Figure 1. The Smart Citizen Kit (Source: ISGlobal).

Table 1. Measurements and related parameters of the monitoring and evaluation plan (Source: ISGlobal).

Measurement

Parameter

Average temperature in the schoolyards

Average temperature (℃)

Maximum temperature in the schoolyards

Average temperature (℃) of the 90th percentile

Temperature anomalies in the schoolyards

Kriging maps of temperature (℃) anomalies

The evaluation methodology

The evaluation methodology was based on the statistical analysis of the measurements acquired by all SCKs for the entire monitoring period. In particular, kriging maps for the temperature anomalies were prepared. Kriging is the most commonly used geo-statistical approach for spatial interpolation. Kriging techniques rely on a spatial model between observations to predict attribute values at unsampled locations. The objective is to understand the relationships between observations separated by different lag distances.

Evaluation results - before and after the interventions

Case study 1. The Font d’en Fargas school

The monitoring process supported the acquisition of measurements of air temperature at several spots of the schoolyard (Figure 2a). Such measurements allow the definition of the air temperature field (Figure 2b), prior to the implementation of the blue, green, and grey interventions. The comparative examination of Figures 2a and 2b shows that the schoolyard exhibits higher air temperatures in all areas where greenery is limited and the surface is covered by impermeable construction materials.

Monitoring spots at the schoolyard (yellow dots)
Figure 2. (a) Monitoring spots at the schoolyard (yellow dots) (Source: ISGlobal).
Figure 2. (b) Spatial depiction of the air temperature field in the schoolyard, prior to the project’s interventions. (Source: ISGlobal).
Figure 2. (b) Spatial depiction of the air temperature field in the schoolyard, prior to the project’s interventions. (Source: ISGlobal).


Figure 2. (a) Monitoring spots at the schoolyard (yellow dots) (b) Spatial depiction of the air temperature field in the schoolyard, prior to the project’s interventions. (Source: ISGlobal).

In Figure 3, the plan for the blue, green, and grey interventions for the conversion of the schoolyard of Font d’en Fargas school to Climate Shelter is presented.

Figure 3. Plan of green, blue and grey interventions at the Font d’en Fargas school (Source: CEB).
Figure 3. Plan of green, blue and grey interventions at the Font d’en Fargas school (Source: CEB).

As seen in Figure 3, considerable attention is oriented towards the placement of greenery in those areas in the schoolyard showing high air temperatures. In particular, areas 5 and 6 of Figure 3 correspond to the areas shown with the blue arrow in Figure 2b, while area 7 corresponds to the one noted with the red arrow in the same Figure. Figure 4 shows the intervention that was implemented at point 7 (nature-based solution as well as change of the impervious surface with bare soil).

Figure 4. Green intervention in the schoolyard of the Font d’en Fargas school (Source: Communication Department, City of Barcelona).
Figure 4. Green intervention in the schoolyard of the Font d’en Fargas school (Source: Communication Department, City of Barcelona).

Figure 5 shows the plan of interventions (image a) as well as the air temperature fields in the schoolyard before (image b) and after (image c) of the interventions. The impact of the interventions to air temperature in the schoolyard is shown in Figure 5 (images to the left and right refer to the state of the schoolyard before and after the interventions, respectively). As seen, the air temperature anomaly has considerably decreased (by roughly 2-3 degrees Celsius) in all areas denoted with the arrows in Figure 2b.

Figure 5. (a) Plan of interventions for the Font d’en Fargas school (source: Borràs et al., unpublished data).
Figure 5. (a) Plan of interventions for the Font d’en Fargas school (source: Borràs et al., unpublished data).
Figure 5. (b) The air temperature fields before the interventions (source: Borràs et al., unpublished data).
Figure 5. (b) The air temperature fields before the interventions (source: Borràs et al., unpublished data).
Figure 5. (c) The air temperature fields after the interventions (source: Borràs et al., unpublished data).
Figure 5. (c) The air temperature fields after the interventions (source: Borràs et al., unpublished data).

Figure 5. At the top (image a), the plan of interventions for the Font d’en Fargas school is seen. The air temperature fields before and after the interventions are provided in images b and c respectively (source: Borràs et al., unpublished data).

Case study 2. The Poblenou school

The plan for green and grey interventions at the Poblenou school is provided in Figure 6. The green intervention implemented in area 2 is shown in Figure 7; it consists of the construction of a school garden as well as of a shading pergola.

Figure 6. Plan of green and grey interventions at the Poblenou school (Source: CEB).
Figure 6. Plan of green and grey interventions at the Poblenou school (Source: CEB).
Figure 7. Green intervention in the schoolyard of the Poblenou school (Source: Communication Department, City of Barcelona).
Figure 7. Green intervention in the schoolyard of the Poblenou school (Source: Communication Department, City of Barcelona).

Figure 8 shows the plan of interventions (image a) as well as the air temperature fields in the schoolyard before (image b) and after (image c) of the interventions. A considerable decrease of the temperature field is observed in the schoolyard, as a result of the intervention presented in Figure 6 (point 2) and Figure 7. The improvement of the temperature anomaly accounts to roughly 2-3 degrees Celsius. However, an issue for consideration is the increase of the temperature anomaly observed at the area where the grey intervention took place.

Figure 8. (a) Plan of interventions for the Poblenou school (source: Borràs et al., unpublished data).
Figure 8. (a) Plan of interventions for the Poblenou school (source: Borràs et al., unpublished data).
Figure 8. (b) Air temperature fields before the interventions (source: Borràs et al., unpublished data).
Figure 8. (b) Air temperature fields before  the interventions (source: Borràs et al., unpublished data).
Figure 8. (c) Air temperature fields  after the interventions (source: Borràs et al., unpublished data).
Figure 8. (c) Air temperature fields  after the interventions (source: Borràs et al., unpublished data).

Figure 8. At the top (image a), the plan of interventions for the Poblenou school. is seen. The air temperature fields before and after the interventions are provided in images b and c respectively (source: Borràs et al., unpublished data).

Some concluding points

Evaluation constitutes an important process that needs to be an indispensable part of any project, not to mention an innovation project that tests novel hypotheses, methodologies, or approaches.

Usually, projects fail to define and/or sustain a thorough evaluation process due to the lack of data, measurements, and observations or in the event that the needed know-how is missing.

The Climate Shelters project has given strong attention to an evaluation process that is supported by a continuous monitoring process of various environmental and climate parameters in the schoolyards and in classrooms, as well as health, well-being, types and levels of physical activity of users and attention levels among 5th grade children.

The benefits of a well-designed evaluation strategy are evident, a fact that indicates the strong need the evaluation process to be a main pillar of the next phase of the Climate Shelters project, namely its upscaling in the framewotk of the "Let's transform our schoolyards" program that operates under the principle that the schoolyard also acts as a classroom. In this way, errors, if any, in the selection of the blue, green, and grey infrastructures as well as non-effective implementations will be identified, adapted and/or fixed accordingly. 

Share

Other news from this project

A selection of actions in support of an adaptation plan to climate change

A blueprint of actions at the city scale for a successful adaptation plan to climate change - Recommendations from the Barcelona GBG_AS2C project

This Web article presents a blueprint of actions for cities to draft their own adaptation plan to climate change, also exploiting their local knowledg...

GBG_AS2C - Blue, Green & Grey_Adapting Schools to Climate Change

Climate adaptation

Barcelona - Spain

Climate adaptation
Design of the school yard - Mare Nostrum school

Upscaling with a vision - The school yard as a school

In this Web Article, the upscaling plan of the Climate Shelters project of the City of Barcelona is presented. The plan reflects a vision, namely to c...

GBG_AS2C - Blue, Green & Grey_Adapting Schools to Climate Change

Climate adaptation

Barcelona - Spain

Climate adaptation
Graphical representation of the Climate Shelters project

Journal No. 3 The “final mile” of the Climate Shelters project: communication, evaluation and replication

The Journal (the 3rd for the Climate Shelters project) aims to provide an overview of the progress of the Climate Shelters project, with emphasis give...

GBG_AS2C - Blue, Green & Grey_Adapting Schools to Climate Change

Climate adaptation

Barcelona - Spain

Sketch performance indicators

Indicators as tools for Adaptation to Climate Change at the city scale - The Climate Shelters project of the City of Barcelona

In this Web Article, the approach of the City of Barcelona towards the development of Performance Indicators as tools for Adaptation to Climate Change...

GBG_AS2C - Blue, Green & Grey_Adapting Schools to Climate Change

Climate adaptation

Barcelona - Spain

Climate adaptation
The Climate Shelters project

The Climate Shelters project - Climate innovation to beat the heat

In his 2nd Zoom-in, Constantinos Cartalis, UIA Expert for the Climate Shelters (Adapting Schools to Climate Change through green, blue and grey Soluti...

GBG_AS2C - Blue, Green & Grey_Adapting Schools to Climate Change

Climate adaptation

Barcelona - Spain

Climate adaptation
Temperarure

Changing the lens in urban design to cope with the impacts of climate change to health - The Barcelona approach

Protecting citizens at the neighbourhood scale in the event of heat waves is at the core of the Climate Shelters project of the City of Barcelona. Rea...

GBG_AS2C - Blue, Green & Grey_Adapting Schools to Climate Change

Climate adaptation

Barcelona - Spain

Climate adaptation
Data collecting for environmental purpose

Do the changes made in climate shelter schools improve pupils’ health?

Three institutions taking part in the project as scientific partners are taking samples and collecting data before, during and after the interventions...

GBG_AS2C - Blue, Green & Grey_Adapting Schools to Climate Change

Climate adaptation

Barcelona - Spain

Climate adaptation
School sketch

Cultivating a School Culture for Climate Action - The Barcelona example

In a world that’s becoming increasingly complex, there is a need to prepare especially young people for a future that will require good knowledge, an ...

GBG_AS2C - Blue, Green & Grey_Adapting Schools to Climate Change

Climate adaptation

Barcelona - Spain

Climate Shelters Journal 4: update on Barcelona's project

Climate Shelters Journal 2: update on Barcelona's project

In this second Journal, Constanitnos Cartalis guides us through the progress and the achievements of the “Climate Shelters” project; he describes the ...

GBG_AS2C - Blue, Green & Grey_Adapting Schools to Climate Change

Climate adaptation

Barcelona - Spain

Climate adaptation
The 11 schools in the Barcelona project are now “climate shelters”

The 11 schools in the Barcelona project are now “climate shelters”

Work has finished on turning the 11 schools into climate shelters, with the aim of helping the city to become more resilient to the climate emergency....

GBG_AS2C - Blue, Green & Grey_Adapting Schools to Climate Change

Climate adaptation

Barcelona - Spain

Climate Shelter Schools evaluate their transformations

Climate Shelter Schools evaluate their transformations

The eleven schools taking part in the “Adapting Schools through Green, Blue and Grey” project, jointly funded by the European UIA programme, have rece...

GBG_AS2C - Blue, Green & Grey_Adapting Schools to Climate Change

Climate adaptation

Barcelona - Spain

Climate Shelter

Barcelona: a city in climate action

Image of the Can Fabras school yard following the conversion of the school to Climate Shelter ...

GBG_AS2C - Blue, Green & Grey_Adapting Schools to Climate Change

Climate adaptation

Barcelona - Spain

Climate adaptation
Image of Barcelona

Science in the City

Image of Barcelona by Kaspars Upmanis on Unsplash...

GBG_AS2C - Blue, Green & Grey_Adapting Schools to Climate Change

Climate adaptation

Barcelona - Spain

Climate adaptation
Co2 reduction
Greener Europe
Heat islands
Nature based solutions
Climate shelters in schools

Adapting Barcelona to Climate Change: A multi-criteria approach for the selection of schools to be converted to Climate Shelters - Zoom In

This zoom in focuses on a challenging feature of the Barcelona “Climate Shelters” UIA project, namely the methodology and criteria for the selection o...

GBG_AS2C - Blue, Green & Grey_Adapting Schools to Climate Change

Climate adaptation

Barcelona - Spain

Climate adaptation
Co2 reduction
Eco-district
Europe closer to citizens
Greener Europe
Heat islands
Nature based solutions

Barcelona’s Climate Shelters project The challenge of communicating unwelcome messages on climate change

To connect with people and communicate climate change effectively, one should convey more than climate facts to people. Instead, there is a need to un...

GBG_AS2C - Blue, Green & Grey_Adapting Schools to Climate Change

Climate adaptation

Barcelona - Spain

Climate adaptation
Europe closer to citizens
Greener Europe
Heat islands
Nature based solutions
GBG_AS2C journal: Climate shelters in Barcelona

GBG_AS2C journal: Climate shelters in Barcelona

In this journal, UIA Expert Constantinos Cartalis provides a contextualization of the Climate Shelters project within the policy context of the Europe...

GBG_AS2C - Blue, Green & Grey_Adapting Schools to Climate Change

Climate adaptation

Barcelona - Spain

Logo of the project showing a graphic synthesis a school yard with green and blue interventions

Barcelona’s pilot project to beat the heat

It is only 10 am but air temperature in Barcelona is already high; another heat wave has hit the city, causing the further enhancement of the heat isl...

GBG_AS2C - Blue, Green & Grey_Adapting Schools to Climate Change

Climate adaptation

Barcelona - Spain

Climate adaptation
Co2 reduction
Greener Europe
Heat islands
Nature based solutions
OASIS project visits GBG_AS2C in Barcelona

The team behind Oasis project in Paris visits Barcelona

Like GBG_AS2C, Paris also has its own climate refuge project in schools. There, they are known as ‘climate oases’, but their purpose is the same: turn...

GBG_AS2C - Blue, Green & Grey_Adapting Schools to Climate Change

Climate adaptation

Barcelona - Spain