Vienna CORE

The project was implemented by the City of Vienna as lead partner, together with the support of four other partners: the Vienna Social Fund, the Vienna Employment Promotion Fund, the Vienna Business Agency and the Board of Education for Vienna – EuropeanOffice.

During the implementation of the project, three levels of management were foreseen:

• Operational support with the day-to-day management and technical coordination of the project;
• Coordination and joint decision making: A steering group, made up of representatives from the urban authority and all delivery partners, which met on a regular basis;
• Central coordination and consultation: A body was set up in order to address all strategic issues. The strategic consultation body was made up of refugees, representatives from refugee initiatives, associated stakeholders, NGOs and civil society initiatives, and was meeting regularly throughout implementation of the project.

An interesting aspect of the evaluation exercise was the fluidity of the roles of the stakeholders. The asylum seekers and refugees were subjects of the research exercise, but at the same time co-designers of the methodology. On the other hand, project partners were implementing partners, but research subjects at the same time as well. Such fluidity of roles was made functional through strong collaboration and coordination ties between project partners, the external consultant and the refugees, as well as due to the existence of a strong Steering Committee.

Due to the participatory design of the monitoring and evaluation, a learning culture (and thus learning loops) was embedded in the monitoring and evaluation activity. The monitoring and evaluation methodology allowed the project partners to quickly adapt and change the interventions or adapt their format, depending on the results of the evaluation. Initially the info modules were given in the format of lectures, for example, in order to reach multiple beneficiaries at a time. When the monitoring and evaluation results showed that, after a time, the difficulties faced by beneficiaries started to become more personalised, the format was changed. When the analysis showed that women were unable to attend language training due to childcare responsibilities, the project started to offer childcare at the centre, in order to allow them to participate. The topics of the entrepreneurship classes were defined depending on the interests of the beneficiaries, while also taking into account labour market demand. This reflexivity and adaptability had its limitations, however. Offering support with finding a job was one of these limitations, as it went beyond the scope of the project.

The partners intensely reflected upon the evaluation approach. A theory-driven evaluation was considered, but the idea was abandoned, as it was believed that it would impose limitations. A longitudinal study was considered as well, but this approach was not adopted either, since the project team preferred to focus on the activities instead.

After careful consideration, it was decided to design and implement a participatory research methodology. This choice was a good approach for the project, as it allowed for the flexibility needed to feed the results of the evaluation into the activities and constantly adapt them in order to best respond to the needs of the beneficiaries. 

A think tank was set up, including a dedicated work package, with the role of monitoring and analysing the project activities. The monitoring and evaluation activities were managed by an external contractor, Urban Innovation Vienna (UIV), which was supported by the project management and partners. The monitoring and evaluation design was proposed by UIV after consultations with the project partners, and was based on the special nature of the activities. It was proposed as a realistic design to implement, taking into consideration the high number of beneficiaries and wide array of activities. It was also meant to be constructed in such a way as to look more at the process, rather than the outputs.

The core team of the think tank was composed of external consultants, representatives of the partners, and also of refugees who had settled in Vienna for years and already spoke German. They participated in the co-creation of the evaluation design. The role of the refugees was to transpose the participatory and bottom-up approach of the project to the evaluation itself. In this way, they brought the perspective of a beneficiary to the design and provided advice regarding what questions should not be asked to avoid harming the asylum seekers, as well as how some questions should be phrased (language issues, connotations, etc.). The selection of the refugees was done through an assessment centre and, besides their qualifications and experience, the gender criterion was also taken into consideration.

The think tank was – among other things – responsible for the preparation of 3 SWOT reports and the evaluation report. A SWOT analysis and workshop was done for every single activity; the results were used to change the focus of the activities or make minor adjustments in order to best address the changing needs of the beneficiaries. The data collection (through SWOT workshops and interviews) was done by UIV, with the support of the project partners and the refugees, who prepared the list of interviewees and the list of questions. Besides beneficiaries, project partners and practitioners (social workers, activities’ providers) were interviewed as well. The external contractor also conducted observations of different activities and had its own viewpoint of what was happening.

This data was complemented by data collected by the social workers and those responsible for implementing activities (trainers, workshop facilitators, etc.), which was collected continuously throughout implementation of the project.

Since the project was needs-based, the research questions of the evaluation exercise were very much centred around them:

• What are the needs of the asylum seekers/refugees?
• Do we meet their needs?
• Can we do better?

In order to collect the necessary data, the project team used a mixed-methods approach, including qualitative (i.e. interviews, feedback groups, journaling, observations, etc.) and quantitative (i.e. survey) methods. This broad array of methods was used in order to address the needs of every beneficiary sub-group. Written feedback worked for the doctors that they supported, but they needed a different method for the women who could not read or write (written and oral methods, structured and unstructured depending on the activity).

The interviews were conducted at the CORE centre, to create a safety net for the beneficiaries and to provide support through interpreters and social workers, if needed.

While interviews, observations and surveys are methods commonly encountered in other projects financed by the UIA initiative, journaling is a less frequently used method. In the case of this project, it was used by social workers in order to make sure that the information gathered during daily social situations was used as well. This was possible thanks to the structure of the CORE centre, which, due to its size, would host the relevant stakeholders and activities (including interviews), thus increasing the interaction between the beneficiaries themselves, as well as between the beneficiaries and the stakeholders. Journaling was not only used by the social workers, but also by the activity providers (such as workshop facilitators) who had to learn how to recognise valuable data for the evaluation. Initially, they started sharing the relevant observations in a more informal way, but with time it became a more formalised practice during the project meetings.

The ad-hoc groups of beneficiaries proved to be an unexpected source of information. These were groups that gathered in the CORE centre rooms in order to practice a common activity they were interested in (woodwork, knitting etc.). The informal talks between the participants (who usually had the same language background) touched upon sensitive issues that would not be captured through other methods of data collection. This new information could then be brought immediately to the attention of the social worker or relevant stakeholder, as they were all hosted in the same premises, and used to make minor modifications to the intervention, so that it better benefited the target group (e.g. introducing a new workshop on a specific topic of interest).

The micro-perspective of the monitoring and evaluation activities was complemented by the macro-perspective related to international best practices. Besides the interviews and workshops, the think tank broadened the project’s perspective through international benchmarking. The project partners identified 11 benchmark cities that had experience in the field of integrating refugees and whose integration policy was relevant for Vienna. Interviews were also conducted with national and international experts, with a focus on volunteering, as well as mentoring and the buddy system.

Over the three years of implementation, the project faced several challenges. Some were due to the changing political and public support for the project, others to ethical or everyday implementation considerations.

When the government changed in 2017, moving from a central-left to an almost far right coalition, the project partners had issues with implementation of the apprenticeship intervention because the legal opportunity for asylum seekers to undertake an apprenticeship was abolished. After consultations between the partners, it was decided that this intervention would be replaced. The initial favourable public opinion towards asylum seekers/refugees also changed.

When it comes to ethical considerations, the project adopted a ‘privacy by design’ approach. If personal data was not needed (name, age, gender) then it was not required (or it was presented in a coded format). If sensitive questions were intended to be asked just because it would be good to have the information, but not necessary, then it was preferred to drop such questions.

In order to ensure the privacy of the attendees and encourage their participation, the indicator for information modules for refugees (or other low-threshold activities) consisted of the number of participations, not the number of participants (no data collected on the participants).

The researchers also faced initial mistrust on the behalf of the beneficiaries with regards to sharing their honest opinion during the interviews, surveys and feedback opportunities, as they were afraid that this would harm their asylum procedure.

Another possible challenge was the fact that centralised data on refugees is not available in Austria. There is data on asylum seekers, but once they are granted asylum or subsidiary protection they are on the same footing as everyone else residing in Vienna. Without this data, hard-to-reach groups, such as women with a low education and no knowledge of German, are difficult to reach. However, through the ad-hoc groups hosted at the CORE centre, the project team has managed to reach out to many of them.

The CORE project is a good example of a needs-based participatory approach, both at the level of activities and at the level of evaluation. Its approach seems particularly transferable to bottom-up projects, where one of the goals is the empowerment of the vulnerable group it is trying to reach.

At governance level, the good results of such an approach depend on:

• The existence of a strong steering group, where all partners participate at the same level, irrespective of their share of the activities/budget;
• Building links with the local government through ongoing exchange and updates;
• A strong network of partners for the areas that they manage (entrepreneurship, education system, labour market etc.).

Some other valuable lessons learnt are presented below:

• The evaluation methodology should be aligned with project goals: At the design stage of the project, it is important to have a clear objective and choose an evaluation methodology that is beneficial to the project. If the goal is to have a needs-based project, like in the case of CORE, then a participatory research method might be a good choice.
• There should be space for flexibility in data collection: At the design stage of the project, researchers may not be fully aware of all important sources of information that might appear during the implementation of the project. Once new sources of data are identified (such as the informal conversations between the beneficiaries during ad-hoc workshops) it is important to address them and decide which is the best method to collect and use the new information, as well as how to integrate it into the evaluation design.
• Involvement of the beneficiaries in the design of the evaluation can bring positive results, in addition to adding to their empowerment: When working with vulnerable groups, involving representatives of the beneficiaries in the evaluation can ensure an extra layer of protection for the wellbeing of the group, by indicating limits to avoid (re)traumatisation.
• It may be beneficial to consider creating a physical safe space for the target group in which to conduct monitoring and evaluation activities, and where representatives of the target group can be supported by social workers and interpreters at a moment’s notice, if necessary.
• At the structural level, having the project team and activities hosted in the same ‘hub’, the  Centre, has created the opportunity for increased social interaction and support, as well as better data collection (e.g. journaling).
• Since staff turned out to be such a valuable source of information through their observations, it might be considered beneficial to train and prepare them in how to recognise valuable information and how to react when specific issues arise.
• The challenge of the evaluation exercise was to isolate impact. According to one of the team members, a possible way to mitigate this might be to break it down into single activities and assess the development of the beneficiaries from before they joined the activity to after (i.e. if before joining the project the beneficiary did not know how the school system worked in Vienna, afterwards, he/she had an understanding).