VoxPop project has delivered the impact assessment framework

A framework to assess the impact of the project outputs was developed, including planning and monitoring phases as follows:

1. Baseline Setting and Monitoring and Availability Framework: Conceptual analysis of the mobility ecosystem of the city including its entities, the city itself and the society. Definition of the baseline to compare the project outputs.
2. Monitoring and Availability: Plan to monitor and assess the impact of the project outputs.
3. Display: Visualization of the project results to increase transparency of public services.

Baseline Setting and Monitoring and Availability Framework

First an extensive literature review was done to identify benchmark mechanisms for monitoring and assessing project implementations, concluding in the following:

• Two assessment levels: one level for the impact assessment (technical, social and economic in the short, medium and long term) and a second level for assessing the process itself and understand the reasons for its success or failure.
• Principles to consider: equilibrium (positive and negative impact to be assessed); relevancy and non-redundancy (select the relevant indicators only and avoid redundancies); exhaustive (all project aspects should be measured); availability and measurability (the indicators should be available and measurable by the system); evolutive (indicators should be measurable in time).
• A multi-layer framework of metrics is adequate (2-3 levels), including various dimensions (transport, society, quality of life, economy, energy and the environment).
• Aggregated index facilitates to monitor the project implementation. The definition of the weights to calculate this metric are key.
• Normalize metrics facilitate comparability of indicators (within the project or between projects).

Second, the project partners were interviewed, and workshops were held to identify ambitions and expectations. In these, the partners highlighted the capability or VoxPop to develop new decision support tools using quantitative models and including social impacts. Additionally, a pool of external experts has validated the framework.

Finally, the VoxPop Monitoring and Availability Framework developed included two areas: reflexive monitoring to analyse the evolution of the urban mobility context during the project implementation, assessing its alignment with the project long-term ambition; and impact assessment to quantify the impacts of the project at different levels. In both cases the project defined indicators, dimensions (accessibility, inclusivity, pollution… for reflexive monitoring and entities, value proposition, business models… for the impact assessment), domains (society, economy and environment for the reflexive monitoring and institutions, citizens, market… for the impact assessment) and ending in the aggregated index, calculated as shown in the below tables.

Monitoring and Display

A plan based on the above framework was defined to monitor the project execution and impact assessment, which included indicators and metrics description, data sources and stakeholders involvement. The data was collected in three phases: ex-ante (to define the baseline), ex-durante (to do an intermediate evaluation) and ex-post (to do the final evaluation).

The plan includes the methodology for data analysis in terms of quantitative and qualitative indicators computation and analysis, monitoring report to and by project partners and revision of project objectives to decide if corrective measures are needed.

The impact assessment was implemented in a quantitative way, this delivering values for each indicator and allowing to calculate score for each dimension and domain, while the reflexive monitoring was implemented in a qualitative way at indicator level. The results are from 2022 (intermediate [I]) and 2023 (final [F]).

A significant increase has been achieved between the intermediate and the final assessment, especially for the citizens engagement (from 2.1 to 6.7) and the market introduction (from 1.2 to 3.7). This is translated into a good increase of the aggregated index, which is 3,165 [I] and 5.4 [F] in a scale between 0 and 10 in comparison to the baseline which was 0,3. Detailed score (Sc) per indicator, dimension and domain are provided in the tables above.

With regards to the dimensions, 5 of them did not change between the intermediate and the final assessment, while 3 improved significantly (new services, co-creation and business model), 2 improved fairly (challenges and opportunities and results dissemination) and 1 got a little worse (policy design).

Respectively, for the indicators, 11 of them did not change between the intermediate and the final assessment, while 4 improved significantly (New services for niche markets, Citizens engaged in co-creation activities, Business models developed and Scientific and technical production), 1 improved fairly (Challenges and opportunities identified) and 1 got a little worse (Design of public policies).

’

With regards to the reflexive monitoring, very small variations are observed in all domains, positive for the societal and environmental ones, and negative for the economic one, mostly for the bad perception in the costs indicator. There is an increase in the perception of the cost of mobility services and a reduction in the use of public transport services, mostly due to the impact of the COVID-19 crisis and which it seems it has not been recovered. However, it is important to highlight that these results are difficult to measure in the short term and especially to isolate the impact of the VoxPop project from other factors on them. Still, it can be observed that between the intermediate and final assessments there was an overall increase in the indicators (revenues, job creation, cost of mobility services and geographic coverage).