Evaluation governance

The analysed UIA M&E case studies show the importance of designating a person, team, partner or subcontractor responsible for overseeing the evaluation. The case studies had either a designated evaluation focal point (e.g. CURANT, Steam City, BRIDGE, CALICO, U-RLP, CoRE) or a number of partners tasked with various aspects of the evaluation from the project initiation (e.g. B-MINCOME, OASIS). In Curing the Limbo, the evaluation was coordinated by the quality assurance team comprised of representatives of all partners. The Utrecht U-RLP project benefitted not only from the involvement of its lead evaluator, but also the presence of the academic Advisory Board. While in Vienna CoRE, the subcontractor leading the evaluation effort set up a think tank for this purpose.

The presence of a designated evaluator, evaluation team or partner adds weight to this component in the overall project scheme. As seen in the case studies, a committed evaluator can guarantee that evaluation receives enough attention in complex project environments. This can, for example, take the form of giving impetus to the development of a common approach (e.g. in U-RLP, Steam City), making sure that enough data is available from the start (e.g. in B-MINCOME, BRIDGE) or that results are communicated to the partnership and discussed (e.g. in CURANT or CALICO).

The managers or implementers will naturally focus on carrying out the activities, so the evaluation will likely not be their top priority. A designated evaluator will be there specifically to set up good research conditions and learn about whether, how and why the project worked to make that knowledge available to others.

This is not to say that evaluators’ work is or should be fully separate from that of project managers and implementing partners. On the contrary, the experiences of analysed M&E case studies, as discussed in other lessons, suggest that cooperation in evaluation pays off. Yet, some projects saw benefits in separating the function of project execution from that of the evaluation (e.g. B-MINCOME, CoRE, CURANT).

'We rightly decided […] that we need somebody to – while we implement the activities – have an eye on how we are performing, where we have to adapt […] we need a kind of outside view too to tell us if we’re still in line or if we need to turn at some point and go another way.” (Source: CoRE representative)

The M&E case studies reveal an integrative potential in having a dedicated evaluation partner. This is particularly the case when the evaluation adopts a more participatory approach whereby implementation partners are also involved in evaluation. In the initial phases, an evaluator can assist in bringing together the multitude of perspectives and ideas of what the project tries to accomplish, contributing to the creation of a common vision (e.g. in CURANT, Curing the Limbo, U-RLP, Steam City, CALICO).

During the later stages, a designated evaluator collects data from all of the project partners, enabling them to capture a holistic view of the project, adding a fresh outsider perspective, one that can surely be valuable for managers throughout the project.

In complex and multidimensional evaluations, where more than one partner is involved in the evaluation, an ‘evaluation integrator’ may be necessary to pull together all of the evaluation’s results. In Barcelona B-MINCOME, the integration of results was undertaken after project completion. It was considered important enough to receive additional financing from the municipality beyond UIA support.

Since Athens Curing the Limbo did not have a clearly designated evaluator from the start, its experience is illustrative of the important role an evaluation focal point can play. The project was implemented by several partners with strong expertise and individual understanding of evaluation.

It took time for the Athens Curing the Limbo partners to arrive at a shared interpretation of the project’s goals and how their achievement could realistically be evaluated. The role of recreating a common framework was undertaken by the quality assurance team composed of representatives of all partners. This was a new task assigned to this team during the project’s implementation.

It is hard to deny the value of a leader in any type of endeavour. This proves true in evaluation efforts in large-scale innovative urban interventions. It comes as no surprise that this role is particularly pronounced when evaluation is distributed across a larger number of project partners, when the integrative process proves crucial for generating a final understanding of the whole experience.

In the analysed UIA M&E case studies, the evaluators were closely involved from the start, whether they were brought into the partnership as partners or subcontractors (see Lesson#1). This early involvement was positively perceived both by partners implementing activities and evaluators themselves.

Engaging evaluators as early as the project development or inception phase creates an opportunity to come up with a better intervention design, which in turn translates to the project’s evaluability. For example, the analysed case studies show that evaluators can help partners to reach, if not common then, better understanding of their objectives, especially when interventions integrate multiple partners from various sectors (e.g. CURANT, Steam City, BRIDGE, U-RLP). While evaluators may feel that it is not their role to bring everyone exactly to the same page (e.g. in CURANT), they can certainly add value by bringing ideas closer, if involved from the start. Based on their expertise, evaluators can provide a different perspective on how much and what exactly is achievable within a given project, as well as what seems to fall beyond the project’s reach, setting realistic expectations for results and impact.

Evaluators can further elicit better understanding of causal pathways between specific actions, results and further outcomes. By supporting partners in improving the project design, evaluators help to increase the evaluability. After all, it is only possible to evaluate the project’s effectiveness or impact when they are well-defined. We describe the role of evaluators in finding a common ground in terms of project objectives when discussing theory-based approaches and the theory of change, specifically in the Evaluation approaches section (see Lesson #5), as this approach seems particularly well-suited for eliciting more clarity as to the project’s aspirations.

The evaluator’s early participation can further help to increase the project’s evaluability by facilitating the establishment of an appropriate M&E framework (e.g. Steam City, U-RLP). This means supporting partners in, for example: (1) developing project indicators, especially those related to higher-level outcomes which can be both time-consuming and challenging (e.g. experiences of CURANT, Steam City, OASIS); (2) defining the necessary data and its sources (e.g. B-MINCOME, BRIDGE); (3) setting up processes to ensure consistent data collection (e.g. Curing the Limbo). The evaluator’s support in ensuring the collection of relevant and quality data is important considering that data unavailability can be a challenging obstacle in evaluation (e.g. experiences of BRIDGE).

Finally, early engagement with evaluators, especially from project partners, creates an important opportunity for better alignment of the evaluation design with project activities and outcomes (see Lesson #2 in Evaluation approaches), as well as managing partners’ expectations in terms of evaluation results.

Bringing evaluation partners into the project early on affords the evaluators comprehensive insight into the work done from start to finish. This level of knowledge can be hard to recreate for someone that becomes involved at later stages, especially in complex and innovative interventions such as UIA’s.

As the UIA M&E case studies show, evaluators’ early and consistent involvement positively influences the perception of the evaluator by other partners. This seems particularly important given that the evaluator is someone who, at the end of the day, expresses a judgement about the conducted work. Other partners are likely to treat the evaluator as someone who, essentially, knows what he or she is talking about when they express opinions about the project (e.g. Steam City, CoRE).

"It is important to feel this kind of view from the beginning to the end, and not just [that evaluation] gets done by somebody who makes an evaluation, but who is not so much involved in the main ideas you want to reach with this project.” (Source: CoRE representative)

During project implementation, as part of a consortium or through other forms of close cooperation, the evaluation partner is more likely to receive:

• better access to information through established relations with other partners and easier access to target groups (e.g. the experiences of BRIDGE in accessing schools where assistance from partners was necessary);
• resultant deeper knowledge of the project and possibly better understanding of its dynamics;
• direct access to communication and coordination procedures (e.g. participation in meetings) which allow for feeding evaluation insights into the project itself (e.g. in BRIDGE, CURANT, U-RLP, CALICO).

With better access to data, designated evaluators are able to produce more robust results. They can witness how the project unfolds in practice with all of the challenges and obstacles. The insider perspective provides evaluators with a good sense of what is feasible in terms of suggesting changes and improvements. All of these factors can contribute to easier uptake of evaluation results by other project partners.

Most of the analysed UIA M&E case studies engaged universities (e.g. CURANT, B-MINCOME, CALICO, OASIS, BRIDGE, U-RLP). Some also engaged consultancies with a specific focus (B-MINCOME, CoRE, BRIDGE) or NGOs (B-MINCOME, Curing the Limbo) in their evaluation efforts. Their experiences show that the participation of such evaluators provided access to additional in-depth thematic expertise, an advanced research skillset and prior relevant experience.

The availability of know-how translates to better chances of selecting a more fitting evaluation approach suitable for complex innovative projects, taking into account ethical, political or practical constraints. In some UIA interventions, in addition to M&E plans, evaluators prepared groundwork documents or manuals which outlined the theoretical understanding of the adopted evaluation approach and a rationale for its selection. For examples of those, you can consult the ‘Groundwork for evaluation and literature study’ developed by CeMIS for Antwerp CURANT or the ‘Groundwork for evaluation and state-of-play’ developed by the evaluators from Vrije Universiteit Brussel for Brussels CALICO. So, with the multiplicity of approaches, paradigms and research methods available, an experienced evaluator can work as a guide to choose an approach sensitive enough to answer evaluation questions.

With theory-based approaches, evaluators with subject matter expertise were able to ground the projects’ theories of change in established science or, in fact, verify/validate them against current science. This was the case for the evaluations of Antwerp CURANT, Rotterdam BRIDGE and Utrecht U-RLP. The need for subject matter expertise was also underlined by the evaluators of Brussels CALICO who conducted the evaluation within the action research paradigm. The power of an action researcher to substantially steer the project in a specific direction should, in their view, be grounded in adequate substantive knowledge.

Experienced evaluators can also steer the partners through the difficulties of evaluation implementation, explaining the significance of various elements or steps of the research process. For example, in Rotterdam BRIDGE one of the project representatives admitted that the evaluator’s role became more central over time, e.g. in capturing the project’s aspirations.

In Utrecht U-RLP, the evaluator devoted a lot of effort to explaining the procedure and significance of recreating the theory of change to the partners. This process also unveiled that, in guiding the partners, an evaluator can also function as an educator. As such, in addition to subject matter and research expertise, the evaluator should be able to rely on a strong set of soft skills, including an ability to communicate effectively. This set of soft skills can certainly help with the evaluator’s main tasks, such as collecting the data from other partners and stakeholders and communicating results.

The availability of advanced research expertise enabled the implementation of tailored evaluation designs, making use of a wide array of data collection and analysis methods. Familiar with the long-standing debates about the advantages and disadvantages of data collection methods, experienced evaluators help offset the shortcomings of one with the strengths of another. Indeed, all analysed UIA M&E case studies proposed mixed-method research designs. Where experimental approaches did not prove feasible for practical, ethical or political reasons (i.e. CURANT, U-RLP, BRIDGE), evaluators were able to propose alternative non-experimental approaches.  

The availability of experienced evaluators enables the implementation of complex and multi-dimensional evaluations, aspiring to glimpse at the possible impact. For example, the Barcelona B-MINCOME project benefited from the extensive research experience of a number of partners. As a result, it is a rare example of a public policy intervention also designed from the start as a social experiment, involving various additional types of research. The combined expertise allowed partners to conduct evaluations of different types of impact (social and economic) at various levels (individual, community, institutional) including through applying a counterfactual approach. In addition to quantitative data collection and analysis methods, qualitative methods were also employed, including an ethnographic study.

The Utrecht U-RLP project benefited not only from the knowledge and experience of its academic evaluator from the University College London, but that of the Advisory Board involving distinguished scholars and chaired by a representative of the University of Oxford. The Advisory Board met three times to discuss the methodology, interim findings and draft final report. With its multidisciplinary in-depth academic expertise and independence, it was able to provide subject matter expertise, but also challenge the team by questioning some of its assumptions or exposing biases. To have such support can be particularly helpful for evaluators who are brought into the partnership early on and work closely with other partners, becoming strongly involved in the project. The Advisory Board was in a position to provide guidance to the researchers in dynamic evaluation circumstances, when initial plans were not feasible due to changes external to the project. It was also able to add nuance to the findings.

Similarly, in Antwerp CURANT, the evaluator CeMIS was also able to identify potential biases in project partners’ approaches and communicate these findings early enough to make changes.

Evaluators with a long-standing research background in working with various stakeholder groups can draw from their experience to employ techniques, facilitating the work with more vulnerable populations such as young children. For example, in Paris OASIS, in order to talk to children as young as five years old the evaluators used puppets to conduct the survey.  

The challenges of research among young children in Paris OASIS underlined the value of complementary expertise at the service of evaluation. As the partner responsible for evaluating social impact was fine-tuning its data collection instruments to children’s capacities, striving for simplicity, it also consulted with partners responsible for climatic impact to make sure that the instruments remained true to the scientific nature of the project (see the Data collection and Horizontal issues sections). In fact, evaluations such as those of Paris  OASIS, Barcelona B-MINCOME or Rotterdam BRIDGE, which aim to capture different kinds of impacts (e.g. social, economic or environmental), need to ensure different kinds of expertise, either combined in the capacities of one evaluator or seeking more than one evaluation partner.

Alongside their knowledge and skills, academic partners also contribute their reputation for scientific rigour which can add credibility to the final evaluation results, especially when the evaluated interventions are innovative and pioneering, and therefore risky. This can create a stronger argument for any follow-up interventions or upscaling. 

As evidenced by the analysed UIA M&E case studies, including evaluators in the partnership creates important opportunities (see Lesson #2), but the proximity to the project activities can also present challenges worth considering at the outset.

As in most research with human subjects, evaluators face the challenge of building trust with their target group (see the Data collection and Horizontal issues sections). UIA M&E case studies were no exception, with the issue being indicated e.g. in Antwerp CURANT, Brussels CALICO and Barcelona B-MINCOME. The fact that some of the studied evaluations involved vulnerable populations (e.g. refugees or children), entailed sharing sensitive information or required substantial participation of respondents made the challenge more visible. As a practical response to this challenge, the evaluators in some projects increased their presence during project activities to make sure that they were visible and familiar to their target group (e.g. CURANT, B-MINCOME). This was particularly the case in the first months of the projects.

In very open and participative evaluation models, such as in Athens Curing the Limbo or Brussels CALICO, by using the action research paradigm the partners can wear a ‘double hat’ of implementers and evaluators. This could pose challenges, such as doubts about their objectivity or, in relation to other stakeholders, doubts about the chances of gathering honest feedback if not enough trust was established. One of the ways to address these limitations is to develop a true learning culture within the partnership and foster open communication. Another possible solution is to make use of various data collection methods, including some guaranteeing anonymity of respondents.    

From our observations, there is no right answer when it comes to positioning the evaluator within the partnership. Even in a single project, in some context and for some specific research activities more integration may work, while in others independence will be of more importance. Either way, weigh your options and choose the best set-up for your circumstances.

The recommendation to involve the evaluator in the project partnership early on is explored in Lesson #2 and is mirrored here in the recommendation to involve partners in the evaluation. The experiences in the analysed UIA case studies show that it is beneficial to include partners in the evaluation, both at the design and implementation stages in various ways. Most importantly, project partners are the ones who determine what they want to achieve and thus what evaluation needs to ask about and measure.

Partners offer distinctive substantive expertise which can help to set up appropriate indicators, baseline and realistic target values, determine information sources (e.g. stakeholders who should be involved in the evaluation), etc.

They can be involved in the evaluation in a number of ways, e.g. as providers of information about the project and its context (as in all analysed case studies), as gatekeepers to other respondents (which was important e.g. in BRIDGE), as data collectors (e.g. through monitoring activities), etc. In Antwerp CURANT, the evaluators developed the approach in consultations with the partners but left the actual decision about adopting it to the implementing partners.

In projects such as Athens Curing the Limbo and Brussels CALICO, which chose action research as its paradigm, the partners essentially conduct the evaluation.

Involvement of partners at each stage strengthens their ownership of the evaluation exercise, builds relations with the evaluator and creates foundations for greater willingness to accept evaluation results (e.g. Steam City, CURANT).

The review of UIA M&E case studies showed that in all of them partners worked in a very collaborative, open and learning-oriented manner, building a shared culture which would facilitate both the implementation and evaluation of bold and innovative endeavours.

While sometimes this collaborative, creative and learning spirit was additionally stimulated by plans to continue or upscale the innovative solution later on (e.g. in CALICO, BRIDGE, B-MINCOME), the reviewed case studies seemed to also have been influenced by the culture of the UIA itself, a culture very much appreciated in the projects. This shows that the donor also can have an important role to play in fostering a certain type of project mindset.

UIA project partners shared an awareness that implementing innovative solutions would be venturing into the unknown. The learning-oriented mindset was expressed in shared attentiveness to activities’ performance and the dynamic changes of needs, as well as in a willingness to experiment with new solutions and observe their effectiveness.

This kind of culture permeated the projects to the extent that sometimes partners would simply see it as natural. This is despite the fact that the UIA projects are partially implemented from European funds and led by public authorities, which operate in demanding and frequently rigid legal and policy frameworks.

In practical terms collaboration between partners was facilitated by various opportunities to meet. These were emphasised throughout hearings with project representatives. Steering committees representing all partners as part of projects’ overall governance created a regular platform for exchanges of information and feedback. Those exchanges were supplemented by various other platforms, such as lower-level meetings, workshops and more one-on-one relations, which were particularly intensive in the initial stages of the project and evaluation development. In some projects, even more opportunities were created for the evaluators to participate in cooperation and exchange of information. For example, in Brussels CALICO, the evaluator also participated in a communication committee, strategic committee, community care committee, governance committee, other committees and ad hoc working groups. Importantly, these meeting opportunities created the conditions for reflection on what worked, but also what did not.

Approaches adopted by projects such as Athens Curing the Limbo or Brussels CALICO, based on the action research paradigm, proved particularly conducive to continuous learning. The action research paradigm almost ‘institutionalises’ the learning mindset as its very premise is continuous weaving together of research and action. It embeds feedback and learning loops in monitoring, evaluation and project management. The action researcher, in turn, does not function as an external observer and assessor, but rather as a participant in various activities, such as committee meetings, providing input and advocating for various actions.

The very hands-on, participatory approach to M&E adopted by Vienna CoRE also allowed the project to quickly adjust. The activities were regularly monitored through SWOT workshops and SWOT reports, which provided an immediate insight into their performance, including unmet needs. The results of the SWOT analysis gave rise to immediate modifications and adaptations, such as the introduction of childcare to encourage participation of women or the creation of ‘women’s café’ where they could cook and do crafts while discussing sensitive topics such as family law and divorce, which – as the analysis showed – they were interested in.

In other case studies, the fact that evaluations were planned in a number of phases with reports produced at several points in time allowed the evaluators to feed their interim results into the partnership while the projects were still ongoing (e.g. CURANT, BRIDGE, U-RLP) and created opportunities to introduce changes.

In Antwerp CURANT, changes were made to the activities offered when evaluation findings showed a mismatch with the refugees’ needs or expectations. For example, the project introduced shorter, tailor-made educational trajectories for young refugees who were eager to start working as soon as possible. When the evaluation results indicated that the refugees were overburdened with activities or appointments, their trajectories were adapted and made less intense. In Antwerp CURANT, the implementation of changes may have been facilitated by the fact that the evaluation results also often overlapped with the observations of partners themselves.

To complete the picture and very much in line with the famous quote from Eleanor Roosevelt that freedom makes a huge requirement of every human being, it needs to be said that maintaining a cooperative and learning culture can also be a challenge. To keep this attitude running goes against a very human desire to return to the comfort zone, as noted by representatives of Athens Curing the Limbo.

The effort required to maintain such an attitude and readiness to challenge and be challenged should be acknowledged and taken into account when assigning resources to innovative projects in general and their evaluations specifically.

The experiences of UIA M&E case studies confirm that the effort required for an evaluation corresponds with the scale of the evaluated intervention. The large-scale interventions that UIA funded predictably demanded a lot of resources. Even more so considering the projects’ pilot and innovative character. Such projects by definition serve evaluation purposes and seek to maximise learning. Consequently, their evaluation efforts required significant resources.

The resources that should be ensured are both financial and human, and will depend on the scope and ambition of the evaluation. If the evaluation aspires to go beyond immediate results to glimpse at the early signs of impact, as in UIA M&E case studies, it will require more elaborate designs. This usually means a wider and repeated data collection and analysis, which generates higher costs. Here, in particular, experimental designs such as that in Barcelona B-MINCOME are known to require significant financial effort related to repeated large-scale quantitative data collection. If the evaluation additionally aims to provide feedback about the process, the ‘how’ and ‘why’ the project works or does not work, it will demand a stronger qualitative research component and deeper engagement with stakeholders, time-intensive for researchers. When further coupled with a limited timeframe, it may create a need for more people to be involved.

Engagement with some groups of stakeholders, especially vulnerable target groups (e.g. refugees or children) will likely require additional resources. It may be necessary to ensure translation and interpretation to carry out research activities (e.g. B-MINCOME, CURANT, U-RLP) or to develop innovative interview techniques (e.g. OASIS). More effort will likely need to be devoted to building trust with respondents and securing their participation.     

The people involved in evaluation need to display an appropriate level of expertise, both in terms of research and specific project subject matter (see Lesson #3 under Evaluation governance). The value of that expertise should be reflected in the resources allocated for research.

Further still, each of the people engaged in the evaluation should be allocated enough time for their work, given the difficulty involved in evaluating innovation. The planning should – to the extent possible – create a financial buffer to enable dynamic responses to changes in the project and for the evaluation to be able to account for the unexpected.

Apart from the actual working hours, evaluations should be assigned enough time for implementation, including foreseeing enough time for development of the approach and design (see Lesson #3 under Evaluation approaches). The discussions held with representatives of UIA M&E case studies suggest that this may demand several months and even as long as half a year (e.g. U-RLP, B-MINCOME). In Antwerp CURANT, the ‘Groundwork for evaluation and literature study’ developed by CeMIS was published in May 2017, while the official project initiation took place in November 2016. A similar amount of time was necessary for the evaluators from Vrije Universiteit Brussel to develop the ‘Groundwork for evaluation and state-of-play’ for Brussels CALICO.

Our review shows that planning evaluation in time, preferably in a number of phases, is crucial for effective use of its findings. Learning loops are important and should be incorporated in the project structure (see Lesson #6 under Evaluation governance). Providing adequate space and opportunity during the project’s lifecycle to reflect on what is happening in implementation helps to react if something goes wrong or if anything unexpected happens, as well as to sufficiently capture the processual dimension of the project and innovation itself.

Additionally, as UIA funding ends at a time when project impact is rarely capturable, it would be a good practice to incorporate into the projects elements of planning for evaluation after the funding ends. This could take the form of developing tools for further M&E for further use by project partners and target groups (see the development of tools for future evaluation in CALICO).