Antwerp CURANT

The evaluation of CURANT was conducted by the Centre for Migration and Intercultural Studies (CeMIS) of the University of Antwerp. CeMIS was present from the project’s outset as part of the partnership, overseeing the project evaluation component. At the same time, it also tried to maintain a level of independence within the partnership, balancing its insider and outsider position for the benefit of the evaluation. This balancing act was not always easy, and this will be further explored in the context of specific challenges that presented themselves.

While CeMIS was the evaluation focal point, other partners responsible for implementation of services were also involved in evaluation throughout the project. In particular, they were consulted in the process of developing the evaluation approach through six group interviews, with participation from project designers and coordinators, social and education workers and psychotherapists. Once CeMIS finalised the approach based on partner input, including the project’s theory of change, the material was presented to the delivery partners who decided on further evaluation steps.

Delivery partners also provided data for the evaluation and otherwise supported the research process (by facilitating access to respondents for example, especially among young refugees).

Overall, the evaluation benefited from a cooperative spirit within the partnership. Various platforms were created for the evaluation partner and delivery partners to meet and exchange feedback to improve the project.

Complementary to the group interviews that it organised with delivery partners, CeMIS also participated in major preparatory meetings during the initial stages of the project and presented evaluation results at different points throughout its duration. This was possible, as the evaluation was structured into four phases:

• Phase one: Preparatory and start-up phase, which aimed to develop the programme theory and a baseline measurement.
• Phases two and three: Two rounds of fieldwork, which aimed to gather data through in-depth interviews and focus groups, proceeding to data analysis and evaluation (two evaluation reports were produced).
• Phase four: The final evaluation and policy-oriented final report.

Want to learn more about the evaluation results? Check out the ‘Policy recommendations’ report and the first and second evaluation reports.

The flexibility to alter activities testifies to the fact that the project embraced a learning mindset beyond the evaluation itself. The results obtained by CeMIS often converged with observations made by delivery partners, which evaluators perceived as logical considering the high level of interaction within the evaluation. While observations sometimes overlapped, CeMIS was perceived as being more objective, possibly adding weight to evaluation results.

In addition to bringing academic objectivity, thanks to its involvement in consultations with all partners and continuous focus on examining the intervention, CeMIS enjoyed a holistic view of the project. As such, this allowed it to play an integrative role in answering questions as to whether, how and why the project worked.

CeMIS’s integrative function may have been strengthened by the choice of a theory-based evaluation approach which entailed the development of a shared theory of change. Theory development also tested the limits of CeMIS’s role in this respect. The process revealed some differences in how partners viewed specific aspects of the project, and CeMIS reflected on whether it wanted to be the one to bring the partners to a common denominator.     

In conducting the evaluation, CeMIS had to balance its position within the project. As an insider, it had stronger interactions with project partners, which allowed for easier access to information, sharing of results and feeding into the project. As an outsider not implementing the activities, it had to establish enough presence to build a necessary relationship of trust with young research participants. At the same time, too strong a presence could have blurred the line between CeMIS and delivery partners in the eyes of young respondents, especially refugees. This, in turn, could lead respondents to answer questions in a way they believed to be desirable and, therefore, safe.

Another challenge faced by the evaluators was the fact that they were not able to measure the results of the whole project. The evaluation needed to finish six months before the end of the project to allow the evaluation report to be written, presented and disseminated before the end of the implementation period (when financing for the project stopped).

As an evaluation partner, CeMIS was responsible for developing the evaluation approach and designing the methodology. Following discussions between the evaluator and the delivery partners, the former proposed a theory-driven approach.

The approach was considered suitable for new and complex interventions such as CURANT, in which causal mechanisms are not yet firmly established, and allowed for the integration of various stakeholder perspectives. It also provided evaluators with an opportunity to assess whether the intervention worked or not, as well as how and why. In other words, in addition to evaluating outcomes, the approach was conducive to analysing the transformation process between the start of the intervention and its outcomes.

A counterfactual approach was also initially considered as an option. In the project’s case, the control group would have had to be composed of refugees who had applied for the project but, after the screening procedure, had been declared ineligible. This approach was eventually rejected, since during the design of the methodology, it was not clear whether the evaluators would have access to such a group of young refugees. Furthermore, after the completion of the selection interviews, the number of young refugees declared as ineligible was low. 

The approach worked around a ‘change model’ (or a ‘theory of change’) developed in consultation with delivery partners during the initial phase. It was laid out in a document, ‘Groundwork for evaluation and literature study’. The change model reflected the partners’ assumptions on the actions required to support the social and structural integration of young migrants and why these actions would address the problem. It was, therefore, a stakeholder-driven change model which CeMIS additionally connected to academic literature. While the development of a change model can be a challenging process, it was not perceived as particularly problematic within the project, which accepted the model proposed by CeMIS.

The evaluation tried to establish whether CURANT lived up to the stakeholders’ expectations about communal living and individualised case management. There were six specific questions, as listed in the second evaluation report, that the evaluators attempted to answer:

1. Did the CURANT communal living setup facilitate regular, informal, meaningful and spontaneous contact between refugees and Dutch-speaking locals?
2. Did CURANT engender diversification in the social networks of refugees and Dutch-speaking locals?
3. How did CURANT’s communal living setup contribute to refugee integration?
4. What are the major strengths and pitfalls of CURANT’s case management approach?
5. What was CURANT’s outcome, in terms of refugees’ participation in education and the labour market?
6. What are the major limitations to CURANT’s approach?

CURANT can be considered a good practice example when it comes to theory-driven evaluation. The approach was reflected in the whole evaluation process, including the definition of outcomes (i.e. improved language skills, competency for independent living, diversification of social networks), the formulation of research questions and the development of research tools.

The evaluators used a mixed-methods approach, meaning they combined qualitative and quantitative research. Choosing different methods strengthened the research, as the shortcomings of one method could be compensated for with the advantages of another. 

The main focus was, however, placed on qualitative longitudinal data collection, with the same beneficiaries interviewed at different points in time to capture changes. Three rounds of interviews were conducted at three specific points in the beneficiaries’ trajectories in the project; at the start, after one year and at the end. In total, 48 in-depth interviews were conducted with 19 buddies, and 42 in-depth interviews were conducted with 24 refugees.

As the evaluators of CURANT noted, it was difficult to reach refugees, so a lot of effort was put into securing their participation. The first step involved building trust between refugees and researchers. This trust was important to encourage sharing, but also to increase the willingness of the refugees to participate in the research in the first place. With their packed schedules, additional interviews with CeMIS researchers became less of a priority for the youngsters.

The researchers could count on the support of social workers and interpreters, and the refugees were interviewed at home, which offered a safe environment.

Qualitative data was complemented with data obtained through quantitative research. A baseline survey was conducted with 58 buddies and 65 refugees. The questions asked referred to descriptive statistics, such as age, sex, country of birth and socioeconomic status, but also to the identification of the respondent’s position/situation at the beginning of the project (e.g. a refugee’s place of residence before entering CURANT). The initial survey was followed by the final survey. In the latter, some of the questions from the baseline questionnaire were repeated and other, more evaluative questions, were added. In total, 29 buddies and 31 refugees filled in both the baseline and the final survey.

In conducting the survey, the evaluators faced a challenge related to the sensitivity of questions. The first version of the baseline questionnaire covered topics on refugees’ mental well-being and resilience, using:

• Hopkins Symptom Checklist-37 for refugee adolescents (Bean et al., 2004);
• Resilience Scale for Adolescents (Hjemdal et al., 2003).

It became clear that youngsters found it difficult to answer trauma symptom- and family-related questions, so these were eventually omitted. In the evaluators’ view, such questions could force the youngsters to relive their trauma. In the final questionnaire, CeMIS focused on CURANT and whether it improved – from the perspectives and experiences of the youngsters – their well-being, social and language skills, knowledge of the Belgian society, etc.

Another challenges in implementing the survey was language-related. The survey was translated into languages understandable for the refugees (Arabic, Pashtu, Dari, Kurdish, Tigrinya, Somali). However, during its implementation, the respondents still struggled to understand the questions. To facilitate the process, an interpreter was provided to explain the questions. However, he struggled with translation of some common research terms, such as the ‘strongly disagree-strongly agree’ Likert Scale. He tried to explain this scale to respondents using a graphic notation ‘--  -  +/-  +  ++’. 

The reliance on qualitative data helped CeMIS to develop an in-depth understanding of the impact of activities, while also allowing for flexibility. The complementary use of quantitative data enabled the easier identification of tendencies, which could be better understood by looking at the qualitative data collected. Additionally, the data collected from the beneficiaries was complemented by project registration data from the Antwerp Public Centre for Social Welfare and Vormingplus regio Antwerpen, allowing for better statistics on refugee and buddy profiles.

The process of designing the evaluation approach had a participatory element when it came to the inclusion of the other project partners. The participatory approach did not trickle down to the beneficiaries themselves, however. A possible improvement for designing a similar evaluation exercise might include refugees settled in the country for several years as part of the design team (integrating a bottom-up approach as opposed to top-bottom). Their role would be to advise the executive partners and evaluators on the needed activities and the development of the intervention. For example, they could provide advice on questions to avoid in order to prevent further traumatisation of the new refugees, how to better phrase questions, how to better approach the beneficiaries, etc.

The theory-driven approach developed and adopted by CURANT has high transferability potential. It tests well on complex, untried interventions tackling sensitive social issues. It can offer additional results in comparison with other approaches, as it allows the evaluators not only to assess if the planned intervention worked, but also to explain how and why it worked. Being a theory-driven approach, it might slow down the reflexivity and adaptability circle, however. As a result, changing or adapting the format of the activities based on evaluation results might be a slower process in comparison with needs-based participatory and developmental approaches. The heavy reliance on longitudinal qualitative research means that the methodology is best suited for interventions where the number of participants is lower.

Some of the more specific lessons that can be drawn from CURANT’s experience include that:

• Evaluators should be involved in the project from the application writing stage onwards. In CURANT, it was important for the researchers to attend the partners’ discussions on the shortcomings of contemporary policies and how the project could provide innovative solutions. This input fuelled the groundwork for evaluation of the project.
• Even if delivery partners and evaluators may sometimes overlap in their conclusions about the project’s impact and processes, having a more independent partner to conduct the evaluation allows the research to benefit from a more holistic view of the project and the potential for additional perspectives. The fact that such an evaluator represents an established academic institution offers higher objectivity and a more comprehensive research effort, increasing the strength and persuasiveness of results.
• The advantages and disadvantages, benefits and risks related to the insider or outsider position of the evaluator in relation to the partnership should be analysed and understood. Various positionings can be used to the evaluation’s advantage. Either way, the researchers must be comfortable with their participation in the evaluation.
• It is beneficial to reflect on the role of the evaluator in the project from the outset. When the evaluation approach is based on a change model, tensions may become evident between different visions (in terms of project outcomes, for example) and these will require negotiation. It is therefore important to establish whether the evaluator is expected and willing to be the one to get partners onto the same page.
• It is very important to build trust between research participants and researchers. This can prove particularly demanding in work with vulnerable participants, who may come with traumatic experiences which they are reluctant to share, or for whom the idea of research might be difficult to understand. These and other factors may translate into higher mistrust towards the researchers. The researchers’ increased presence in the projects, at least at the beginning, helps to establish good relations and encourage sharing, as does the creation of a safe space for participation by choosing an appropriate venue and ensuring the required support is in place. 
• Continuity and stability must be ensured in the implementation and evaluation of activities. This is facilitated when the same Project Manager is present throughout the entire implementation of the project. In interventions implemented by public authorities and where pressures may arise for specific results, the Project Manager can function as a shield between the evaluators and sources of political pressure.