Expert article
Edit 30 March 2024
by Josep Maria Salanova Grau

VoxPop project has delivered the impact assessment framework

VoxPop impact assessment framework
VoxPop impact assessment framework
The project has ended and delivered its outputs. This web article is dedicated to the impact assessment of the project outputs.

People, Processes & Technology towards the digital transformation of the urban mobility system of Lisbon (VoxPop) is the project that will foster digitalization and collaboration between private and public mobility actors in the Portuguese capital. VoxPop is funded by the Urban Innovative Actions (UIA) initiative under the Digital transition topic. The project has ended and delivered its outputs. This web article is dedicated to the impact assessment of the project outputs.

Share

A framework to assess the impact of the project outputs was developed, including planning and monitoring phases as follows:

  1. Baseline Setting and Monitoring and Availability Framework: Conceptual analysis of the mobility ecosystem of the city including its entities, the city itself and the society. Definition of the baseline to compare the project outputs.
  2. Monitoring and Availability: Plan to monitor and assess the impact of the project outputs.
  3. Display: Visualization of the project results to increase transparency of public services.

Baseline Setting and Monitoring and Availability Framework

First an extensive literature review was done to identify benchmark mechanisms for monitoring and assessing project implementations, concluding in the following:

  • Two assessment levels: one level for the impact assessment (technical, social and economic in the short, medium and long term) and a second level for assessing the process itself and understand the reasons for its success or failure.
  • Principles to consider: equilibrium (positive and negative impact to be assessed); relevancy and non-redundancy (select the relevant indicators only and avoid redundancies); exhaustive (all project aspects should be measured); availability and measurability (the indicators should be available and measurable by the system); evolutive (indicators should be measurable in time).
  • A multi-layer framework of metrics is adequate (2-3 levels), including various dimensions (transport, society, quality of life, economy, energy and the environment).
  • Aggregated index facilitates to monitor the project implementation. The definition of the weights to calculate this metric are key.
  • Normalize metrics facilitate comparability of indicators (within the project or between projects).

Second, the project partners were interviewed, and workshops were held to identify ambitions and expectations. In these, the partners highlighted the capability or VoxPop to develop new decision support tools using quantitative models and including social impacts. Additionally, a pool of external experts has validated the framework.

Finally, the VoxPop Monitoring and Availability Framework developed included two areas: reflexive monitoring to analyse the evolution of the urban mobility context during the project implementation, assessing its alignment with the project long-term ambition; and impact assessment to quantify the impacts of the project at different levels. In both cases the project defined indicators, dimensions (accessibility, inclusivity, pollution… for reflexive monitoring and entities, value proposition, business models… for the impact assessment), domains (society, economy and environment for the reflexive monitoring and institutions, citizens, market… for the impact assessment) and ending in the aggregated index, calculated as shown in the below tables.

Reflexive monitoring

Domain

Weight

Dimension

Weight

Indicators

Weight

           

Society

55%

Accessibility

30%

Cost of mobility services

50%

Geographic coverage

50%

Attarctivity

20%

Public transport attractiveness

100%

Inclusiveness

35%

Awareness about VRUs

100%

Well-being and comfort

15%

Management of public space

20%

Quality of public space

33%

Level of service

47%

Economy

20%

Revenues

20%

Revenues

100%

Costs

50%

Costs

100%

Job creation

30%

Job creation

100%

Environment

25%

Air pollution

100%

GHG emissions

60%

Air quality

40%


Impact assessment

Domain

W

Dimension

W

Indicators

W

           

Institutions

30%

Entities

20%

Companies willing to share datasets

100%

Capacity Building

50%

Capacity Building for innovation and analytics

100%

Partnerships

30%

Companies sharing datasets

35%

Volume of shared dadasets

35%

Interoperability of systems

20%

Interoperabilty of European systems

10%

Citizens' participation

35%

Co-creation

100%

Citizens engaged in co-creation activities

100%

Decision making

20%

Challenges and opportunities

35%

Challenges and opportunities identified

100%

Policy design

30%

Design of public policies

100%

Value proposition

20%

Value proposition for added value services

100%

Business model

15%

Business models developed

100%

Market introduction

15%

New services

40%

New services

40%

New services for niche markets

60%

Results dissemination

10%

Communication and dissemination

70%

Scientific and technical production

30%

Penetration of services

50%

Use of new services

40%

Use of new services for VRUs

60%

                     

 

Monitoring and Display

A plan based on the above framework was defined to monitor the project execution and impact assessment, which included indicators and metrics description, data sources and stakeholders involvement. The data was collected in three phases: ex-ante (to define the baseline), ex-durante (to do an intermediate evaluation) and ex-post (to do the final evaluation).

The plan includes the methodology for data analysis in terms of quantitative and qualitative indicators computation and analysis, monitoring report to and by project partners and revision of project objectives to decide if corrective measures are needed.

The impact assessment was implemented in a quantitative way, this delivering values for each indicator and allowing to calculate score for each dimension and domain, while the reflexive monitoring was implemented in a qualitative way at indicator level. The results are from 2022 (intermediate [I]) and 2023 (final [F]).

Impact assessment [I]

 

Domain

Sc

Dimension

Sc

Indicators

Sc

 

           

 

Institutions

3.7

Entities

5

Companies willing to share datasets

5

 

Capacity Building

3.7

Capacity Building for innovation and analytics

3.7

 

Partnerships

2.8

Companies sharing datasets

3

 

Volume of shared dadasets

5

 

Interoperability of systems

0

 

Interoperabilty of European systems

0

 

Citizens’ participation

2.1

Co-creation

2.1

Citizens engaged in co-creation activities

2.1

 

Decision making

5.7

Challenges and opportunities

7.7

Challenges and opportunities identified

7.7

 

Policy design

5.7

Design of public policies

5.7

 

Value proposition

5.0

Value proposition for added value services

5

 

Business model

2.0

Business models developed

2

 

Market introduction

1.2

New services

0.4

New services

1.1

 

New services for niche markets

0

 

Results dissemination

7.6

Communication and dissemination

10

 

Scientific and technical production

2

 

Penetration of services

0.5

Use of new services

1.3

 

Use of new services for VRUs

0

 

               

 

Impact assessment [F]

 

Domain

Sc

Dimension

Sc

Indicators

Sc

 

           

 

Institutions

3.7

Entities

5

Companies willing to share datasets

5

 

Capacity Building

3.6

Capacity Building for innovation and analytics

3.6

 

Partnerships

2.8

Companies sharing datasets

3

 

Volume of shared dadasets

5

 

Interoperability of systems

0

 

Interoperabilty of European systems

0

 

Citizens’ participation

6.9

Co-creation

6.9

Citizens engaged in co-creation activities

6.9

 

Decision making

6.7

Challenges and opportunities

10

Challenges and opportunities identified

10

 

Policy design

4.3

Design of public policies

4.3

 

Value proposition

5

Value proposition for added value services

5

 

Business model

6

Business models developed

6

 

Market introduction

3.7

New services

6.4

New services

1.1

 

New services for niche markets

10

 

Results dissemination

8.6

Communication and dissemination

10

 

Scientific and technical production

5.2

 

Penetration of services

0.5

Use of new services

1.3

 

Use of new services for VRUs

0

 

               

 

A significant increase has been achieved between the intermediate and the final assessment, especially for the citizens engagement (from 2.1 to 6.7) and the market introduction (from 1.2 to 3.7). This is translated into a good increase of the aggregated index, which is 3,165 [I] and 5.4 [F] in a scale between 0 and 10 in comparison to the baseline which was 0,3. Detailed score (Sc) per indicator, dimension and domain are provided in the tables above.

With regards to the dimensions, 5 of them did not change between the intermediate and the final assessment, while 3 improved significantly (new services, co-creation and business model), 2 improved fairly (challenges and opportunities and results dissemination) and 1 got a little worse (policy design).

Respectively, for the indicators, 11 of them did not change between the intermediate and the final assessment, while 4 improved significantly (New services for niche markets, Citizens engaged in co-creation activities, Business models developed and Scientific and technical production), 1 improved fairly (Challenges and opportunities identified) and 1 got a little worse (Design of public policies).

With regards to the reflexive monitoring, very small variations are observed in all domains, positive for the societal and environmental ones, and negative for the economic one, mostly for the bad perception in the costs indicator. There is an increase in the perception of the cost of mobility services and a reduction in the use of public transport services, mostly due to the impact of the COVID-19 crisis and which it seems it has not been recovered. However, it is important to highlight that these results are difficult to measure in the short term and especially to isolate the impact of the VoxPop project from other factors on them. Still, it can be observed that between the intermediate and final assessments there was an overall increase in the indicators (revenues, job creation, cost of mobility services and geographic coverage).

 

Reflexive monitoring

Domain

W

Dimension

W

Indicators

W

Sc I

Sc F

BL

           

 

 

 

Society

55%

Accessibility

30%

Cost of mobility services

50%

G

VG

A

Geographic coverage

50%

A

VG

 

Attarctivity

20%

Public transport attractiveness

100%

W*

F

G

Inclusiveness

35%

Awareness about VRUs

100%

A

A

A

Well-being and comfort

15%

Management of public space

20%

A

A

 

Quality of public space

33%

A

A

 

Level of service

47%

G

G

G

Economy

20%

Revenues

20%

Revenues

100%

G

VG

 

Costs

50%

Costs

100%

G

B

 

Job creation

30%

Job creation

100%

G

VG

 

Environment

25%

Air pollution

100%

GHG emissions

60%

G-VG

G-VG

G

Air quality

40%

G-VG

G

G

                   

 

 

 

Other news from this project

Zoom-in about the VoxPop open call for innovators

This video presents the second zoom-in of the VoxPop project. ...

VoxPop - People, Processes & Technology towards the digital transformation of the urban mobility system of Lisbon

Digital transition

Lisbon - Portugal

Digital transition
Lisbon inhabitant's profile

VoxPop project has delivered the Lisbon Urban Mobility Observatory

People, Processes & Technology towards the digital transformation of the urban mobility system of Lisbon (VoxPop) is the project that will foster digi...

VoxPop - People, Processes & Technology towards the digital transformation of the urban mobility system of Lisbon

Digital transition

Lisbon - Portugal

Digital transition
The Vox Pop Project Journal No 3

The Vox Pop Project Journal No 3

The Vox Pop Project - Journal No 3 ...

VoxPop - People, Processes & Technology towards the digital transformation of the urban mobility system of Lisbon

Digital transition

Lisbon - Portugal

Digital transition
VoxPop open call for urban mobility innovation

VoxPop open call for urban mobility innovation

People, Processes & Technology towards the digital transformation of the urban mobility system of Lisbon (VoxPop) is the project that will foster digi...

VoxPop - People, Processes & Technology towards the digital transformation of the urban mobility system of Lisbon

Digital transition

Lisbon - Portugal

Digital transition
Panoramic of Lisbon

The Vox Pop Project Journal No 2

People, Processes & Technology towards the digital transformation of the urban mobility system of Lisbon (VoxPop) is the new project that will foster ...

VoxPop - People, Processes & Technology towards the digital transformation of the urban mobility system of Lisbon

Digital transition

Lisbon - Portugal

Digital transition
Mobility observatory

VoxPop is delivering the first outputs

VoxPop - People, Processes & Technology towards the digital transformation of the urban mobility system of Lisbon

Digital transition

Lisbon - Portugal

Digital transition
lisobn

Trends on EU Urban mobility data sharing platforms: the digital transition of the cities’ mobility system

In the last ten years, efforts have been made by the European Commission and the European Parliament to create EU policies at a local level. In this f...

VoxPop - People, Processes & Technology towards the digital transformation of the urban mobility system of Lisbon

Digital transition

Lisbon - Portugal

Traveling with mobility impairments in Lisbon

Traveling with mobility impairments in Lisbon

Presentation of the status of the mobility of people with impairments in Lisbon and the VoxPop navigation support tool for vulnerable city users. ...

VoxPop - People, Processes & Technology towards the digital transformation of the urban mobility system of Lisbon

Digital transition

Lisbon - Portugal

Mobility
Open Call for European Innovaters

VoxPop open call for proposals has been published

VoxPop - People, Processes & Technology towards the digital transformation of the urban mobility system of Lisbon

Digital transition

Lisbon - Portugal

Digital transition
The role of data in regulating urban mobility

The role of data in regulating urban mobility

VoxPop - People, Processes & Technology towards the digital transformation of the urban mobility system of Lisbon...

VoxPop - People, Processes & Technology towards the digital transformation of the urban mobility system of Lisbon

Digital transition

Lisbon - Portugal

Digital transition
Desired paths without bike lane using bike-sharing system (BBS) - GIRA in October 2021 12th & 14th.

VoxPop - Advanced analytics to gather actionable insights to the city of Lisbon.

VoxPop - People, Processes and Technology towards the digital transformation of the urban mobility system of Lisbon – was created to facilitate the di...

VoxPop - People, Processes & Technology towards the digital transformation of the urban mobility system of Lisbon

Digital transition

Lisbon - Portugal

Digital transition
Bike-sharing system (BBS) user journeys May 2021, 5th

VoxPop has laid the foundations to roll out the digitalization of the mobility landscape of Lisbon

The Innovators Alliance has laid the foundations for the definition of the data sharing framework by identifying the barriers, and started identifying...

VoxPop - People, Processes & Technology towards the digital transformation of the urban mobility system of Lisbon

Digital transition

Lisbon - Portugal

Digital transition
Placing users at the core of mobility products development: the case of VoxPop in Lisbon

Placing users at the core of mobility products development: the case of VoxPop in Lisbon

When the City of Lisbon started the conception of its UIA initiative VoxPop this was framed as an undertaking aiming to foster the sharing of data and...

VoxPop - People, Processes & Technology towards the digital transformation of the urban mobility system of Lisbon

Digital transition

Lisbon - Portugal

Digital transition
Active mobility and public transport in Lisbon

The VoxPop Project Journal No 1

This is the first edition of the VoxPop journal and aims at introducing the project, the expectations of the partners and the most relevant challenges...

VoxPop - People, Processes & Technology towards the digital transformation of the urban mobility system of Lisbon

Digital transition

Lisbon - Portugal

Digital transition
Bicycles in Avenida Guerra Junqueiro, Lisbon

VoxPop started rolling out the proposed solutions to facilitate/promote digitalization of mobility in Lisbon

People, Processes & Technology towards the digital transformation of the urban mobility system of Lisbon (VoxPop) is the new project that will foster ...

VoxPop - People, Processes & Technology towards the digital transformation of the urban mobility system of Lisbon

Digital transition

Lisbon - Portugal

Digital transition
VoxPop has developed its first value proposition, a secure bike parking solution as an enabler of cycling in the City of Lisbon

VoxPop has developed its first value proposition, a secure bike parking solution as an enabler of cycling in the City of Lisbon

Covid-19 led to major changes in the way people move. In Lisbon, as in many other cities worldwide, bicycles took over the streets once the first lock...

VoxPop - People, Processes & Technology towards the digital transformation of the urban mobility system of Lisbon

Digital transition

Lisbon - Portugal

Digital transition
Lisbon

The digitalization of mobility in Lisbon

People, Processes & Technology towards the digital transformation of the urban mobility system of Lisbon (VoxPop) is the new project that will foster ...

VoxPop - People, Processes & Technology towards the digital transformation of the urban mobility system of Lisbon

Digital transition

Lisbon - Portugal

Digital transition