Brussels CALICO

Responsibility for the monitoring and evaluation of CALICO was given to expert academic institutions – research groups for the Belgian Ageing Studies and Cosmopolis from the Vrije Universiteit Brussel (VUB). In terms of project management, monitoring and evaluation were structured as a separate work package. However, as the project was co-created by partners who had cooperated on previous initiatives, monitoring and evaluation were designed in a cooperative manner. Partners wanted to be involved from the beginning and expressed the need for usefulness of insights and experiences stemming from CALICO for their broader actions outside of the project itself.

Group interviews with project partners were organised at an early stage of implementation, providing the opportunity for each to share their motives, perceived challenges, and expectations for the results of the monitoring and evaluation. Consequently, research was structured to address stakeholders’ information needs, and expectations were consensually managed. During the interviews, for example, the need for attention to both ‘factual’, number-based indicators, as well as the gathering of personal stories through adequate, qualitative research techniques was stressed.

The interviews also led to a common understanding among project partners that social impact will take longer to occur than the funding lifetime of the project. Outcomes over the next five years should therefore be taken into account, and some may prove unexpected and difficult to grasp. In cooperation with stakeholders, the initial monitoring and evaluation plan (outlined in the application form) was refined and detailed. Importantly, a thorough list of research questions, covering a broad span of topics and assigned to monitoring and evaluation types, target groups and data collection methods, was created.

Because CALICO uses a participatory social action research methodology, feedback and learning loops are intrinsically tied into monitoring and evaluation and project management. The action researcher does not function as an external observer and assessor, rather as an active participant in various activities (such as committee meetings), providing input and advocating for various actions and groups – especially future inhabitants of the housing.

CALICO employed a participatory model in both project implementation and monitoring and evaluation activities. In terms of monitoring and evaluation, this meant adopting a participatory social action methodology through several guiding principles:

• The inclusion of participants (stakeholders and target group representatives) alongside researchers in the design, conducting and assessment of research results;
• A focus on delivering practical solutions relevant to project objectives – in this case, improving the living conditions of inhabitants;
• Openness to diverse audiences and representatives from groups outside of the ‘usual suspects’ and attention paid to the role of power relationships amongst groups and actors;
• Governance of the monitoring and evaluation process using deliberation and sociocracy (consent-based decision making);
• Appreciative inquiry, which means a focus on strengths and opportunities instead of problems and pitfalls.

This approach was useful in the case of CALICO, particularly as it is aimed at addressing vulnerable groups’ needs. The participatory approach strengthens the agency and empowerment of participants, which is line with the overall goals of the project.

By design, research was focused on the results and impact of the project, as well as its processes. Thus, tangible effects were considered and their measurement planned, but means to capture what happened during project implementation –  i.e. how those effects were obtained –  were also planned for. Such a design is especially important in innovative projects.

The CALICO team formulated an extensive list of research questions related to impact evaluation and process evaluation, divided into four categories:

• Development and realisations;
• Positive impact;
• Sustainability and future;
• Gender equity and inclusion of older people.

You can find the list of all research questions in the report, which was prepared at the initial stage of project implementation.

CALICO decided to use both quantitative and qualitative research methods for data collection:

• Standardised questionnaire surveys addressed to all residents of the co-housing schemes, designed as a longitudinal study conducted at two points near the beginning and end of the project. The questionnaire consisted of questions on loneliness, subjective wellbeing, quality of life and housing. Standardised questions and scales widely used by institutions (academic researchers, the World Health Organization) were used to provide data comparability.
• Data from project monitoring, including the number of participants in various sessions and meetings, and databases created by project partners with information on residents.
• The analysis of documents created during the project, such as financial documentation, implementation reports and other supplementary material.
• Focus group interviews with representatives of the project partners, conducted in two series at the beginning of the project and near its end.
• Individual interviews with representatives of key stakeholder groups (CALICO residents, residents of neighbouring real estate complexes, community professionals and policymakers).
• Participation of researchers in the various committees and working groups set up in the project.

Overall, a carefully conceptualised monitoring and evaluation framework, based on mixed-methods research, was considered by the project team as the best guarantee for providing quality information and useful insights.

At the time of preparing this case study, CALICO is in the project implementation stage. As a result, not all the planned research activities have been conducted. Focus group interviews with project partners were held at an early stage with the objective of identifying the motivations, expectations and specific needs of each organisation in relation to the evaluation activities. These discussions provided important input for refinement of the monitoring and evaluation design. The researchers also participated in meetings of project committees from the outset, engaging in the co-creation of the project in line with the action research assumptions – not only collecting data, but also contributing to activities and sharing their expertise.

The project also presents an inspiring case wherein research benefits from project partners’ specific capacities. Although not foreseen at the stage of application, representatives of important stakeholders were included in the research (i.e. neighbours living in close proximity to the future CALICO buildings). To this end, students of urban and/or sociology studies at ULB (Université Libre de Bruxelles) were invited to act as interviewers. Importantly, not only were they involved as individuals conducting interviews, but they also engaged more deeply with preparation beforehand. After conducting interviews, a platform for critical reflection was also created; students could provide their insights through an innovative workshop-based method of roleplaying, in which they represented different groups of people. Thus, not only was good quality data gathered, but also valuable input for project implementation and communication.

To some degree, CALICO was negatively impacted by the COVID-19 pandemic, which hindered important project actions and data collection rounds. Although one of the project partners was able to conduct activities aimed at setting up the community, the other two partners experienced interruptions and barriers to organising meetings. Despite these challenges, the selection of future residents was only slightly delayed and was finalised by June 2020. Moreover, the research team were able to collect data despite lockdown measures, and the first rounds of questionnaire surveys and individual interviews were held. As most interactions had to be held online, however, rather than in person, it was more challenging to build trust and relationships with residents.

One significant challenge to data collection which the CALICO team were aware of from the beginning was the limited opportunity to collect data about the impact of the project on its target groups. Since a major action of CALICO was to build the housing – a time-consuming process prone to delays – the risk of not being able to conduct the second wave survey (post-intervention) within the planned timeframe was clearly visible. As the project has not ended at the time of writing this case study, it remains to be seen if residents will have moved to the housing when the second wave survey is conducted.

While this may be a drawback in terms of grasping the initial impact of the project, it has to be stressed that only a study conducted after a considerable amount of time could provide information on long-term effects. CALICO addressed this problem by planning to provide research tools for project partners, which will be used after the period of UIA funding has ended. Impact is also measured and evaluated by other, qualitative research techniques. In fact, even if the second wave survey is conducted before the residents move in, the effects of their participation in community planning and design could be grasped.

The research scheme is summarised in the table below:

Participation was a prevalent theme in CALICO, as most project actions were based on the ideas of co-creation, co-construction, co-production and co-design by stakeholders and future residents of the housing. The monitoring and evaluation design stemmed from the participatory action research approach, which means involving stakeholders and target groups in the conceptualisation, methodology development, implementation of the research, data collection and assessment. In fact, such an approach goes a little further – affirming the influence of the researchers on the subject of research and research participants (in this case, the CALICO project and future residents of the constructed housing). Reciprocity of the relations between actors (researchers, stakeholders and target groups) is one of the key premises of the participatory social action methodologies. In CALICO, this materialised in holding regular meetings on different levels and organising various project committees in which the researcher is not just a passive observer, but an active participant.

Among the multiple topics included in the research questions were gender equity and the inclusion of older people. Since improving actions related to these issues was one of the goals of CALICO, the questions addressed how equity and inclusion were achieved, and to what extent they influenced other impacts and outcomes. Attention was also paid to identifying success factors and obstacles experienced by project stakeholders while working towards equality and inclusion.

It is also important to note that CALICO was a pilot project aimed at testing solutions for housing governance models, addressing vulnerable groups’ needs and community-based care. One of the monitoring and evaluation objectives was to gather knowledge about the project’s processes, providing insight into CLT management, evidence and recommendations for public policy. This impact was planned for, and political advocacy was initiated early, instead of waiting until the project had ended.

CALICO is a strong case for employing participatory social action research in innovative urban projects. It also exemplifies how careful planning of monitoring and evaluation activities benefits implementation, as well as capturing multilevel effects and impacts. The approach used in CALICO is particularly transferrable to projects related to social issues and vulnerable target groups. In this case, monitoring and evaluation activities can strengthen inclusion and empowerment. The highlights from our case study are as follows:

• Monitoring and evaluation benefit from access to expert institutions and engaged personnel (in this case, academia) with domain-specific and methodological knowledge (CLTs and participatory action research, respectively).
• As project outcomes – here, for example, the improvement of residents’ quality of life through the provision of housing – are difficult to observe during implementation, the feasibility of gathering meaningful measurements of impact might seem questionable. This challenge can be addressed by preparing tools for project partners to assess impact after the period of funding.
• It is important to consider not only impact and outcomes, but also the processes involved in monitoring and evaluation. As one of the project team put it, in the case of CALICO “bricks are one thing, but community is what it’s about”. CALICO was not just construction work or physical housing space – it was also building interpersonal relations and creating the foundations for neighbourhood cooperation and co-governance. This dimension is represented by a number of activities in the project, and the whole process is closely monitored.
• The more engaged the researchers are in active participation in project activities, the more challenging it may be for them to stay objective in their assessment. It may be relatively easier for the evaluation team to act only as observers with an external point of view towards the project.
• While active participation of the researcher is potentially beneficial for project actions and outcomes, as the person can share expertise, voice stakeholders’ views, and drive change, it may be demanding for them to stay objective.