PRIORITIZATION OF URBAN MOBILITY INFRASTRUCTURE PROJECTS BASED ON URBAN CONFIGURATION AND ON MULTIDIMENSIONAL SCALING

ABSTRACT

Purpose:

This article aims to propose an alternative methodology of analysis, assessment, and prioritization of urban mobility master projects capable of supporting investment decision-making, considering financial, economic, and space variables.

Originality/value:

There are no clear criteria on project proposal selection, what corroborates the critics about the selection process and prioritization of urban mobility projects in the Ministry of Cities. Intrinsically economic issues that are leveled in recent methodologies of investment and project analyses are variables considered hard to be converted into financial benefits to make up the financial evaluation.

Design/methodology/approach:

The methodology proposed will draw on the multidimensional scaling as a multivariate assessment tool in order to evaluate and prioritize the projects using their economic variables, the spatial variables of the urban planning, and the financial models of each project. The economic variables arise from the economic analysis available in the projects, as well as the projects’ financing models. The spatial variables acquired from the social logic of space theory or spatial syntax allow the assessment of cities’ integration capacity before project implementation and after their development, making it possible to identify improvements in urban mobility conditions.

Findings:

The use of multidimensional scaling by applying economic, financial, and spatial variables has proved to be effective in promoting decision-making, whether by indicating the best project or identifying how close and/or distant they are from the objectives of a given public policy.

KEYWORDS:
Space syntax; Projects prioritization; Infrastructure; Urban mobility; Multidimensional scaling

RESUMO

Objetivo:

Este artigo tem como objetivo propor uma metodologia alternativa de análise, avaliação e priorização de projetos de mobilidade urbana capaz de auxiliar a tomada de decisão de investimento, considerando variáveis financeiras, econômicas e espaciais.

Originalidade/valor:

A ausência de critérios claros para selecionar propostas de projetos corrobora as críticas ao processo de seleção e priorização de projetos de mobilidade urbana do Ministério das Cidades. Metodologias recentes de investimentos e análises de projeto nivelam questões intrinsecamente econômicas difíceis de ser convertidas em benefícios financeiros para efetuar a avaliação financeira.

Design/metodologia/abordagem:

A metodologia utilizada é o escalonamento multidimensional, como ferramenta de avaliação multivariada para analisar e priorizar projetos usando as variáveis econômicas e espaciais do planejamento urbano e os modelos financeiros de cada projeto. As variáveis econômicas resultam de análises econômicas disponíveis nos projetos, bem como dos modelos de financiamento dos projetos. As variáveis espaciais obtidas da lógica social da teoria espacial ou sintaxe espacial permitem avaliar a capacidade de integração das cidades antes da implantação dos projetos e após o desenvolvimento deles, possibilitando identificar melhoria nas condições de mobilidade urbana.

Resultados:

O uso do escalonamento multidimensional com a aplicação de variáveis financeiras, econômicas e espaciais se provou eficaz para promover a tomada de decisão, seja pela indicação do melhor projeto ou pela identificação da proximidade e/ou distância até o atingimento dos objetivos de uma determinada política pública.

PALAVRAS-CHAVE:
Sintaxe espacial; Priorização de projetos; Infraestrutura; Mobilidade urbana; Escalonamento multidimensional

1

INTRODUCTION

The methods of an economic feasibility analysis of urban mobility projects are widely criticized nationwide and internationally.

The Brazilian Federal Court of Accounts (Tribunal de Contas da União – TCU) (2015) carried out an operational audit with the purpose of assessing governance in mobility public policies concerning the fiscal year of 2014, according to the Operational Audit Report. These audit findings pointed that:

• The goals and indicators used by the federal government cannot assess and measure the progress and scope of the policy objectives.

• The objectives and guidelines defined and declared by the national policy have not been clearly considered as a selection criterion for the urban mobility intervention proposals submitted to the federal government by states and municipalities.

• The cooperative efforts among the government spheres are insufficient for the adequate implementation of the urban mobility public policy.

• The federal government actions are not aligned in order to prioritize non-motorized means of transportation over motorized ones, as well as public transportation services over individual motorized transportation.

Flyvbjerg (2013) signals that ex-ante estimates of project costs and benefits, in other words, cost/benefit analyses (CBA) and social and environmental impact assessments, are, in several instances, typically different from actual ex-post costs and benefits. Studies by Flyvbjerg, Bruzelius, and Rothengatter (2003), Flyvbjerg, Holm, and Buhl (2002, 2005), and Flyvbjerg (2009) indicate how costs and benefits estimates of large projects are biased and inaccurate, therefore, unreliable.

Despite criticisms of the method, the lack of alternatives to economic assessment leads researchers to improve the analysis criteria. That is the reason for the interest in studying and improving the economic assessment methodology for the prioritization of urban mobility infrastructure projects.

Confronted by that perspective, Muldoon-Smith et al. (2015) indicate that the urban economics traditional studies have not considered the importance of urban configuration to construct their analyses, suggesting the configurational analysis incorporation into their projects’ decision-making. Space syntax is a theory developed by Hillier and Hanson (1984). It allows not only to identify structural problems but also to verify occupation trends, making it possible to contextualize the physical space by its occupation (Bandeira, 2005).

Intrinsically economic issues indicated in the current investment and project analysis methodologies are difficult variables to convert into financial benefits in order to compose financial assessments. In this sense, the quest for a methodology capable of assessing variables of distinct dimensions without loss of information during the process of providing pecuniary value corroborates the importance of this study. Moreover, to correlate distinct dimensions through a link between them requires a methodology that consists of multidimensional instruments of multivariate analysis capable of finding dissimilarities between projects in order to categorize them.

2

MATERIALS AND METHODS

In this paper, economic assessment variables that are already considered in current projects are combined with variables indicating space changes through space syntaxes and variables indicating the origins of financial resources for the project execution, following the project’s financing model.

The internal rate of return (IRR) and the cost/benefit ratio (C/B) are both economic variables that had been mentioned in the mobility Plans projects, and they will repeatedly be used. For the composition of the financial variables group, the type of sources of financial resources for the project’s execution will be identified in order to check whether the project will cause fiscal impacts on city finances or not, verifying the existence or not of financial sustainability for the projects.

The QGis software was used for the space syntax utilization, allowing to carry out the construction of maps capable of linearly representing the network of paths known as axial and segment maps and of generating output data of the variables to be measured integration and average depth. We intend to measure the performance of space variables ex-ante (t-1) and ex-post (t) concerning projects that have been completed or that have been selected in the same governmental program, which, in this case, it was the “PAC 2 Large Cities Mobility”, to identify variables oscillation, which will serve as the project assessment criterion in the space dimension.

In order to promote the analytic treatment of this work, the use of techniques with several independent variables, the multivariate analysis, allows interpreting phenomena concerning several measurements for the observed objects simultaneously (Corrar, Paulo, & Dias, 2007). The multidimensional scaling (MDS) allows the creation of associations even among variables of different dimensional values, enabling it to compare distinct projects to carry out their scaling as a tool of prioritization and investment decisions in urban mobility. The best feature in multidimensional scaling is that it may be used with variables in any scale (Souza, 2010).

Therefore, the MDS allows the creation of a reference for comparison among the projects, known as an ideal point (IP). More than indicating how different the projects are among themselves, it is possible to identify which ones are closer to the characteristics considered ideal for the projects to achieve.

According to the logical design of the model in the discussion, we intend to identify several analysis dimensions for investment decision-making in specific projects (Figure 2.1).

Figure 2.1
MODEL LOGIC DESIGN

The XLSTAT^® data analysis statistic tool used as an extension of Microsoft Excel® allows the variables obtained in the economic-financial analyses and the QGis software output variables to be submitted to MDS. Then, result analysis will enable the selection and prioritization of projects in the sample.

3

LITERATURE REVIEW

Literature review on the analysis and evaluation of urban infrastructure for the purpose of assisting investments in public policies retrieved few tools. Pujadas, Pardo-Bosch, Aguado-Renter, and Aguado (2017) proposed combining multi-criteria decision-making (MCDM) and multi-attribute utility theory (MAUT), incorporating the value function (VF) concept and assigning weights through the analytic hierarchy process (AHP) into an integrated model called the Integrated Value Model for Structural Assessment (Mives), to assess, prioritize and select public investments in the city of Barcelona. Pujadas, Cavalaro, and Aguado (2018) use the MIVES model now to study the conditions of urban pavements in a case applied to the city of Barcelona. Roigé, Pujadas, Cardús, and Aguado (2019) present a Mives MCA methodology to evaluate Barcelona’s water distribution network and propose strategies for its renewal.

3.1

Projects economic feasibility analysis

The economic benefits and necessary investments related to each horizon year are stratified by economic-financial modeling, and it is necessary to build cash flows which will be useful to build economic efficiency indicators (Pereira & Silveira, 2013). Those indicators that involve the capital investment assessment process seek to comply with the following order: 1. cash flow results dimensioning, 2. the economic assessment of those flows, 3. return rate definition, and 4. risk identification (Assaf, 2011).

When the present value of a project’s cash inflows, i.e., when the project’s cash net benefits are discounted from investments and costs throughout this period, it is possible to identify how much the project is capable of generating wealth for its investors (Assaf, 2011; Brigham & Ehrhardt, 2012). This method is known as net present value (NPV). When the interest rate generates a null NPV, causing the inputs to annul the outputs, that rate is referred to as IRR.

The C/B analysis criterion is another tool widely used in transportation projects: it analyzes investment alternatives through monetary quantification of the largest possible number of costs and benefits and uses shadow prices and net benefits (Dalbem, Brandão, & Macedo-Soares, 2010). Some authors appropriately refer to that method as a C/B rate analysis criterion, and De Melo and Setti (2007) consider a project economically viable when the C/B relation is higher than or equal to the unit: the highest the relation, the more attractive the project is.

The C/B ration, or benefit-cost coefficient, is based on the effort of providing pecuniary value to the benefits along with the project, discounting the return on investment rate, that is, whatever financial return which can be expected from the economic benefits identified in the projects. The criterion can either be the CBA or the C/B ratio. Paranaiba (2017) indicates its broad use in several countries: Brazil, United Kingdom, United States of America, Australia, New Zealand, Sweden, and Germany, and the World Bank Group itself is one of the pioneers in the use of the CBA methodology, having helped global dissemination of that project analysis method.

According to Mackie, Nellthorp, and Laird (2005), Mackie, Worsley, and Eliasson (2014), and Mackie and Worsley (2013), the CBA is restricted to those impacts of which the effects can be measured and assessed in financial terms. Many of the observed manuals omit non-financial impacts, increasing subjectivity in the decision-maker’s judgment, without clarifying the weight attributed to the variables. The second limitation is the political priority given to potential impacts, and decision-makers do not seem to know to what extent the investments in transport systems can contribute to increasing productivity, and correct regional unbalances.

3.2

Space syntax

Space syntax can be used as a scientific theory to understand urban space and to identify its capability of contributing to assessing those urban mobility projects. Space syntax is important for the model built in this paper, as within its theoretical, methodological, and tooling framework, it is possible to create a relation between space and society based on the connection between configuration and functionality, which affects urban mobility and related economic issues (Holanda, 2007). In this sense, space syntax contributes to providing topological, geometric, and morphological variables to analyze urban systems. This paper aims at identifying variables for a comparative investigation of the ex-ante and ex-post urban configuration of the projects. The analysis is limited to the urban syntax topological aspects, which provide an understanding of the global impact of the project’s infrastructure interventions, namely global integration and average depth.

The space syntax space analysis technique lets us identify the flow potentiality within a specific urban space (Barros, Medeiros, Silva, & Holanda, 2008). Do Carmo, Raia, and Nogueira (2014) indicate the use of convex space and axial line, with “lines presenting the two key properties of being both very simple and global, so everything we need to know is how much we can see from a point” (Hillier, 2001 apud Medeiros, 2013, p. 150). The analysis of this combination supports the construction of Axial Maps seeking to be the linear representation of the paths network, illustrating the movement generation potential on each (Medeiros, 2013).

For Barros (2006), it is possible to portray vehicle flows and pedestrian flows as well, disclosing the axes capacities for more or less intense flows of those movements in terms of potential. For that, the linear representation must be sketched on a cartographic base, considering the drawing of the smallest possible number of the longest straight lines (Medeiros, 2013), as depicted in Figure 3.2.1(A). Each fraction of the network is identified for the axial map linear representation, as in Figure 3.2.1(B).

Figure 3.2.1
AXIAL MAP CONSTRUCTION

This axial map linear representation holds out a possibility of constructing a connections matrix to measure, quantify and hierarchize different levels of connection between each way and the complexity where it is inserted, subject to space syntactic analysis (Medeiros, 2013). This hierarchization occurs by identifying the axes with a significant flow potential in comparison with the axes with lower flow potential (Paranaiba, 2018).

Integration plays a paramount role in the selection of specific axes within the urban network, once the axes more integrated are the ones more permeable and accessible in the urban space, and from those one can access other axes (Medeiros, 2013). For Ugalde and Rigatti (2007), integration plays an important role in space syntax analysis by making it possible to reach an understanding of depth.

Depth is the most important concept in quantitative analysis. The depth between two adjacent spaces is 1. Thus, the notion of depth is the minimum number of spaces that must be passed through to go from one space to the destination. Relations of depth involve the notion of asymmetry and the measure of relative asymmetry generalized. In the case of real urban systems, RA values are highly influenced by the total number of spaces in a system, and real relative asymmetry (RRA) is introduced to reduce the system size effect. RA is divided by the RA of a diamond-shaped system with the same number of spaces (Kim & Sohn, 2002, p. 414).

When an axial map is systemically observed, the average depth of that system is found by analyzing the depth relation of each axis to reach all axes of the system. The “shallower” a system is, the smaller is the average depth, while greater average depths indicate labyrinth-character systems, with low integration (Medeiros, 2013; Loureiro, 2016).

For a more assertive use of the space syntax technique in studies of the transport area, Barros (2006, 2014) indicates analytic improvement in the use of segment maps, namely because the potential movement is not the same along an axis, something that the axial map is unable to measure. For that purpose, the focus must be on transforming axial map lines into axes segments, considering way crossings or links as a reference, more specifically at the knots, as a current designation in the transports area (Barros, 2006), as shown in Figure 3.2.2.

Figure 3.2.2
MAP OF SEGMENTS CONSTRUCTION

Therefore, applying the syntactic analysis per segment enables us to identify the specific potential movement of each segment. In study regions in which the routes and cycleways of public transportation lines do not occur along the entire axis but only in determining segments, the map of segments allows a more assertive assessment of the specific potential of each segment.

3.3

Urban mobility sources of financing

When the sources of funds to implement urban mobility are defined in the projects, they can be an important source of information to identify how enforceable the project will be. In this sense, it is important that the proposed model is capable of differentiating projects presenting financing proposals to provide financial sustainability. For instance, whether there are infrastructure execution and management performance bonds within the scope of the project to be ranked and prioritized. Paranaiba (2016) characterizes financing sources as follows:

• Loans: They are not necessarily deemed to be financing models, but they are frequently used to compose the origins of funds to finance urban mobility infrastructure projects. Given their relevance, they will be inserted within the proposed analysis context.

• Additional fuel taxation: The mechanism is an additional taxation over private cars’ fuel consumption in areas of heavier traffic, aiming to raise funds for investments in street maintenance. In Brazil, “Cide-Combustíveis” is the Economic Domain Intervention Contribution, governed by Law no. 10.336/2001 (Paranaiba, 2016).

• Urban toll: The urban toll logic is to offset the transportation cost of an additional car, which for owners (marginal cost) is lower than the effective marginal cost of the cause in traffic: time, vehicle cost, the marginal maintenance cost of other car users, pollutant emissions and congestion.

• Land value capture: The main goal in gaining property value is to recover the transport investment capital cost, capturing the land value increment resulting from investments in transports (Medda, 2011). These are similar to property taxes, but the values derived from financial benefits with the local provision of public services are considered: improvement in accessibility, offering conditions for active mobility or a new subway line construction (Olsen & Fearnley, 2014);

• Consortiated urban operations: In this modality, the public sector, when defining zoning in the Master Plan, sets the relation between the buildable area and the plot area, referred to as a basic utilization coefficient (CAB). Based on the utilization coefficient, the possibility of building above the CAB is offered, constituting an additional building potential with consideration, which is the building right granted for consideration. The purpose is to requalify a city area or implement and/or expand urban infrastructures with interventions in city areas where the real estate interest demand is above the limits set by the urbanistic legislation (Companhia de Desenvolvimento Urbano da Região do Porto do Rio de Janeiro, 2014).

Figure 3.3.1 below displays a remissive comparison indicating the advantages and disadvantages of each one of those modalities.

Figure 3.3.1
REMISSIVE COMPARISON OF URBAN MOBILITY FINANCING MODALITIES

Model:	Advantages	Disadvantages
Loans	• Existence of internal and external development banks with resources for urban mobility.	Generation of liability for the contracting party. Project budget is frequently underestimated, generating more expenditure.
Fuel taxation (Cide-Combustíveis)	Resource distribution for all states.	Institutional insecurity after several alterations, which include a reduction to zero. Municipalities do not know when and until when they can count on the resources.
Urban toll	Generation of extra revenue, possible ability to finance urban mobility investments, and costs. Ability to reduce traffic congestion.	Opposition of public opinion about implementing hurdles in several cities around the world. Still not applied in Brazil.
Gain in property value	It does not represent an increase in public expenditure; it just captures property valorization derived from urban mobility improvement. Generation of revenues for several project phases: financing, implementation, operation, and expansion. Strong relation between transport and land use.	In Brazil, its participation in urban property tax collection is not clear yet.
Syndicated urban operations	Financial leverage for financing, without the need for real estate speculation. Revitalization and reordering of specific areas.	Rent-seeking can provoke gentrification. Even with the high capability to capture great volumes of resources in the Certificates of Additional Construction Potential (Certificados de Potential Adicional de Construção – Cepac) financial operation, there is no focus on generating revenues in the urban consortium operations (OUC) economic exploration.

• Source: Elaborated by the authors.

It is possible to identify that, according to Law no. 12,587/2012 of the definition of non-tariff sources of financing, financing forms can be categorized by ranking them among those generating revenues without public expenditure.

This categorization of projects into dichotomous variables (zero for projects generating expenditures and one for those generating revenues) makes it viable to verify the financial sustainability of the projects, regardless of the model adopted. The important thing is, once transformed into output data as values of a variable identifying the financing source, it is possible to differentiate the projects that have the concern of being financed by revenue generation from those which directly or indirectly generate public expenditure.

3.4

Multidimensional scaling

As indicated by Machado, Duarte, and Duarte (2011), the MDS techniques are developed in order to carry out space representations of objects and complex stimulations as to how people make judgments, which represent each object, such as a point in a dimensional space. The MDS seeks to represent measures of proximity between objects in such a way that a visual inspection is possible, incorporating dimensions sufficiently able to represent similarity or dissimilarity between the pairs of objects (Souza, 2010).

The distance measurement usually employed in MDS is the Euclidean distance. According to Esmalifalak, Ajirlou, Behrouz, and Esmalifalak (2015), the Euclidean metrics is a function d: R^M x R^M → R, designating any two vectors (objects, individuals, projects) i = i₁, ..., i_m and j = j₁, ..., j_m and m = 1, ..., M the dimension space, which will indicate the distance between any two vectors. Mathematically, the Euclidean distance between the vector of i and j can be defined as follows:

(1) $d\left(i,j\right)=\sqrt{{\displaystyle \sum _{m=1}^M{\left(i_m-j_m\right)}^2}}$

in which:

M = 1, 2, ..., m;

i_m = value of variable m for vector i;

j_m = value of variable m for vector j.

Cardoso and Scarpel (2010) indicate that MDS produces a matrix of distances d_ij and seeks to find the arrangement of points in space as M-dimensions, in such a way that n points coordinates along the dimensions, producing an Euclidean matrix of distances with elements as close as possible to the elements in the distance matrix. The difference between those two matrixes represents noise, given the inaccuracy in the measurement, and, according to Souza (2010), it is quantifiable by the sum of all errors over the pairs (i,j), defined by Kruskal (1964) as stress level.

The stress level represents the model’s trustworthiness, a measure similar to the correlation coefficient, quantifying how bad the difference between the proximity measures and the corresponding distances would be (Bevilacqua, 2004). The stress model is available in order to measure the stress level, which, according to Equation 2:

(2) $Stress=\frac{{\displaystyle {\sum }_{ij}{\left(\delta \left(ij\right)-d_{ij}\right)}^2}}{{\displaystyle \sum d_{ij}^2}}$

in which:

δ(ij) = dissimilarity between the i-th and j-th objects;

d_ij = original distance between the i-th and j-th objects.

In Equation 2, as the dissimilarity approaches the original distance between objects i and j, the lower the stress is; and the opposite will demonstrate that the number of dimensions used to estimate dissimilarity is insufficient, although two dimensions facilitate visualization for assessment. However, the number of dimensions also depends on the number of assessed objects, and it is interesting to adopt “a number of objects four times higher than the number of dimensions” (Hair, Black, Babin, Anderson, & Tatham, 2009, p. 496).

In order to define the number of sufficient dimensions, the elbow criterion and the Shepherd diagram methods comply with the tests adopted in the MDS literature. The elbow criterion relates the Kruskal stress to dimensionality. After calculation of the distances between each pair of points and having obtained the stress level between distances and disparities, should the stress be considered high, the points must be moved to minimize stress; should the stress below, the perceptual map may be constructed.

3.5

Ideal point

Perceptual maps represent how distant the objects are from one another, indicating the dissimilarity among them. However, the spatial arrangement does not indicate which is the best point or what the selection and prioritization criterion must be. For Hair et al. (2009), it is possible to reach that goal by including a point representing the perfect combination among attributes, indicating an ideal object that serves as a reference to identify how distant the objects are from the IP (Figure 3.5.1).

Figure 3.5.1
IDEAL POINT ON THE PERCEPTUAL MAP

The IP was constructed based on the interests of a point that represented the expected feature improvements. Observing the map, proximity with the IP will identify the order of preference among the points depicted. In this case, the order would be C, F, D, E, A, B, wherein C has the highest preference and B has the lowest preference. Punctual representation was the method used, which is the easiest method to understand, as it uses the Euclidean distance, which is a straight line to order preference (Hair et al., 2009).

Thus, with the IP insertion, it is possible to scale preference by identifying the points with higher priority than others, providing each point with a given set of features. Another feature that makes the existence of an IP essential in the model developed in this paper is that it represents the set of features that an urban mobility project can represent in relation to the will for solutions in a specific moment.

In this sense, if all projects represented in the perceptual map are distant from the IP, the administrator responsible for the selection can perceive that none of them will meet the expected goals. This avoids the risk of choosing the “least bad project” among a set of bad projects or set projects that are distant from the investment goals.

4

METHODOLOGY TO ANALYZE AND PRIORITIZE URBAN INFRASTRUCTURE PROJECTS

The assessment model we intend to suggest assumes that benefits estimated in CBA analysis may be understood as effects of a cause and consequence process, and the urban configuration can contribute to the formation of traffic jams, number of accidents, assisted population, traveling time, and fuel consumption, among others. Another assumption that substantiates the proposal of this new model is the need for indicating a financial sustainability project: it is important to prioritize projects that do not cause fiscal and fair impacts.

These two new concepts, added to the financial and economic analyses currently employed in Transport and Mobility Plans (Planos Diretores de Transporte e Mobilidade – PDTM, in Portuguese), serve as sources of input data for statistic treatment. The structure of the ranking and prioritization method based on perceptual maps, which indicate the projects relative distance considering the proposed set of variables, has eight phases:

• First phase: selection of PDTM.

• Second phase: financial and economic assessment identification.

• Third phase: maps of segments preparation.

• Fourth phase: financing methods identification.

• Fifth phase: IP construction.

• Sixth phase: input data standardization.

• Seventh phase: MDS application.

• Eighth phase: projects ranking and prioritization.

Data processing aims to foster standardization, avoiding discrepancies and preventing some data from benefiting over others because of the gradient of their input data. Thus, in order to carry out the MDS, the use of standardized variables by the z-score represents that variables have the same influence on Euclidean distances that they have in the variables of the projects.

5

MODEL APPLICATION

In order to carry out a case study, a sample of cities with PDMT submitted to the Ministry of Cities in the PAC2 Large Cities Mobility Program and with all desired variables for the model was selected. The chosen cities were Belém (State of Pará), Brasília (Federal District), and Manaus (State of Amazonas). The city of Cuiabá (State of Mato Grosso) has an axial map and a mobility project but lacks the economic feasibility indicators, and it will also be added. The reason for this inclusion is to observe whether the model is able to differentiate Cuiabá in a negative way, compared to the other three cities, which have all the variables. The goal is to identify whether the MDS results will convey the deficiency in economic feasibility indicators of that city’s project.

For that, the highest values of each variable in the sample were employed to construct the IP. Therefore, there will be a benchmark to identify which is the best project in relation to the others, as well as their dissimilarities.

5.1

Segment maps preparation

All projects in the sample have details of the interventions on the cities’ urban infrastructure. Therefore it is possible, based on a preexisting axial map, to project interventions and construct a map of segments of the road network before the existence of the project and a second map upon its existence, calculating the variation of topological variables (Figure 5.1.1).

Figure 5.1.1
MAPS OF SEGMENTS FOR PROPOSED SITUATIONS IN BELÉM, BRASÍLIA, CUIABÁ, AND MANAUS

In order to compose the input data for the model in this phase, which sought to survey the space syntax variables for those cities, the data are displayed in Figure 5.1.2.

Figure 5.1.2
SYNTHESIS OF INPUT DATA ORIGINATED FROM THE TOPOLOGICAL VARIABLES

City	Average integration variation	Opposite of the average depth variation
Belém	6.31%	+7.50%
Brasília	1.60%	+1.04%
Cuiabá	-0.60%	-1.37%
Manaus	8.03%	+5.38%

• Source: Elaborated by the authors.

Arranging the variables this way (changing the average depth variation sign) will make it easier, upon the MDS performance, for the model to understand that the average depth negative variations are, in fact, indicators of improvement. When the average depth displays positive variations, the model will identify them as mobility setbacks.

5.2

Financing methods

The fourth phase in the model application proposes the projects financing structure survey, whether through fundraising or operation and maintenance. The financing methods of the projects in the sample were gathered through documental research and then ranked (Figure 5.2.1).

Figure 5.2.1
SYNTHESIS OF DATA ORIGINATED FROM THE PROJECTS FINANCING METHODS

City	Revenues other than fares	Alternative revenues	Budgetary subventions	Cross subventions	Dichotomous variable value (1.0)
Belém			X	X	0
Brasília			X	X	0
Cuiabá			X		0
Manaus			X	X	0

• Source: Elaborated by the authors.

Besides the federal funds requested within the PAC2 Large Cities Mobility program, Belém indicates the use of a consortiated trust fund from the Government of the State of Pará. Brasília proposes improvements in fare management to create a consortiated trust and a loan from the Inter-American Development Bank (IDB), besides subvention granting. Cuiabá indicates agreements with the National Department of Transport Infrastructure (Departamento Nacional de Infraestrutura de Transportes – DNIT, in Portuguese) as a source of funding, credit facility from Caixa Econômica Federal (a Federal Bank), and counterparts from the state government. Manaus indicates fare funds. In fact, all projects make recommendations for financing sources to be prepared, except for the Cuiabá plan, which does not mention the form of fundraising to fulfill the investments proposed in the project.

5.3

Ideal point construction

The main contribution of the model is setting the IP as guidance for comparison between projects; also, the definition itself is a great challenge. This is so because public policy guidelines and goals do not clarify “how much” mobility improvement is expected and “how” it is expected, they just evidence “what” is expected. Therefore, two IPs will be constructed, so in a comparative way they can provide the subjectivity present in the objectives of the public policy assessed:

• IP type 1: For this IP, the highest values of each variable of the sample will be employed. Therefore, it will be the benchmark to check what is the best project in relation to the others and their dissimilarities.

• IP type 2: The construction of this IP employs values that meet the subjectivity of the public policy objectives in order to verify the relative distance of the projects from their expectations. According to this criterion, it is possible to eliminate all projects at very long relative distances from the IP and also the ones considerably close to it. The criteria will be:

  • 1) RIR: According to the PPP Portal (2013), transportation concessions orbit around an internal return rate of 11% per year.

  • 2) C/B ratio: In a scenario in which the benefits minimally offset the costs, the adopted value for this variable is 1.

  • 3) Topological variables: The cities included in the sample were assessed by Medeiros (2013) within the same group composed of 18 cities, 17 of which are capitals. In this way, the values of the variables can be the highest variation of the sample variables in relation to the average of the group constructed by Medeiros (2013).

  • 4) Financing: The value assumed will be 1, which is the financial sustainability indication, without direct or indirect costs for the population.

5.4

Input data standardization

The values surveyed to be employed in multidimensional scaling have the following nominal values for IP type 1 (Figure 5.4.1) and IP type 2 (Figure 5.4.2).

Figure 5.4.1
INPUT DATA WITH IP TYPE 1 (NOMINAL VALUES)

Group	Variable/City	Belém	Brasília	Cuiabá	Manaus	PI-1
Financing	Type of model	0	0	0	0	0
Economic	RIR	41%	37.78%	0	21%	41%
	C/B	2.53	3.66	0	1.36	3.66
Topological	Variation integration average	6.31%	1.60%	-0.60%	8.03%	8.03%
	Average depth variation symmetric number	7.50%	1.04%	-1.37%	5.38%	7.50%

• Source: Elaborated by the authors.

Figure 5.4.2
INPUT DATA WITH IP TYPE 2 (NOMINAL VALUES)

Group	Variable/City	Belém	Brasília	Cuiabá	Manaus	PI-2
Financing	Type of model	0	0	0	0	1
Economic	RIR	41%	37.78%	0	21%	11%
	C/B	2.53	3.66	0	1.36	1
Topological	Variation integration average	6.31%	1.60%	-0.60%	8.03%	34.55%
	Average depth variation symmetric number	7.50%	1.04%	-1.37%	5.38%	15.75%

• Source: Elaborated by the authors.

Depending on the Z-score to standardize data, the values to be effectively employed as input data are presented for IP type 1 (Figure 5.4.3) and IP type 2 (Figure 5.4.4).

Figure 5.4.3
INPUT DATA WITH IP TYPE 1 (Z-SCORE)

Group	Variable City	Belém	Brasília	Cuiabá	Manaus	PI-1
Financing	Type of model	0	0	0	0	0
Economic	RIR	0.722	0.541	-1.582	-0.402	0.722
	C/B	0.181	0.901	-1.424	-0.560	0.901
Topological	Variation integration average	0.414	-0.778	-1.335	0.849	0.849
	Average depth variation symmetric number	0.872	-0.742	-1.345	0.342	0.872

• Source: Elaborated by the authors.

Figure 5.4.4
INPUT DATA WITH IP TYPE 2 (Z-SCORE)

Group	Variable/City	Belém	Brasília	Cuiabá	Manaus	PI-2
Financing	Type of model	-0.447	-0.447	-0.447	-0.447	1.789
Economic	RIR	1.081	0.896	-1.271	-0.066	-0.640
	C/B	0.577	1.378	-1.207	-0.247	-0.502
Topological	Variation integration average	-0.259	-0.591	-0.747	-0.137	1.734
	Average depth variation symmetric number	0.277	-0.696	-1.060	-0.042	1.521

• Source: Elaborated by the authors.

These data standardized by the Z-score were employed as input data for multidimensional scaling using the XLSTAT^® statistic tool, an Excel add-in, in the following phase.

5.5

MDS application

In this phase, the starting point was generating proximity matrixes (quadratic matrixes) considering the Euclidean distance among the projects variables to calculate dissimilarity among those projects. Figure 5.5.1 shows the matrix with IP type 1, and Figure 5.5.2 shows the matrix with IP type 2.

Figure 5.5.1
QUADRATIC MATRIX FOR IP TYPE 1

	Belém	Brasília	Cuiabá	Manaus	IP-1
Belém	0	2.140	3.983	1.511	0.842
Brasília	2.140	0	3.254	2.617	2.300
Cuiabá	1.511	2.617	0	3.123	4.517
Manaus	1.511	2.617	3.123	0	1.918
IP-1	0.842	2.300	4.517	1.918	0

• Source: Elaborated by the authors.

Figure 5.5.2
QUADRATIC MATRIX FOR IP TYPE 2

	Belém	Brasília	Cuiabá	Manaus	IP-2
Belém	0	1.317	3.277	1.453	3.827
Brasília	1.317	0	3.396	2.049	4.606
Cuiabá	3.277	3.396	0	1.944	4.325
Manaus	1.453	2.049	1.944	0	3.368
IP-2	3.827	4.606	4.325	3.368	0

• Source: Elaborated by the authors.

The perceptual map of the condition in which the IP type 1 is shown in Figure 5.5.3, and the perceptual map of the condition in which the IP is type 2 is shown in Figure 5.5.4.

Figure 5.5.3
PERCEPTUAL MAP WITH THREE DIMENSIONS (IP-1)

Figure 5.5.4
PERCEPTUAL MAP WITH THREE DIMENSIONS (IP-2)

The perceptual maps analysis made it possible to visualize that Cuiabá remained distant from all projects, as well as from the IP in both situations (IP-1 and IP-2). That was expected to be the outcome from the model, as the Cuiabá project displayed a negative performance in all dimensions. If Cuiabá were close to the IPs or to the other projects in any of the situations, that would be a sufficient reason to reject the model, which did not occur.

Finally, the output data to analyze dissimilarity among points are presented in the comparison among pairs of points, for IP-1 (Figure 5.5.5) and for IP-2 (Figure 5.5.6).

Figure 5.5.5
COMPARATIVE TABLE BETWEEN PAIRS (IP-1)

Pair	Dissimilarity	Distance	Rank (dissimilarity)	Rank (distance)
Belém-IP-1	0.842	0.843	1	1
Belém-Manaus	1.511	1.509	2	2
Manaus-IP-1	1.918	1.912	3	3
Belém-Brasília	2.140	2.138	4	4
Brasília-IP-1	2.300	2.296	5	5
Brasília-Manaus	2.617	2.624	6	6
Cuiabá-Manaus	3.123	3.116	7	7
Brasília-Cuiabá	3.254	3.250	8	8
Belém-Cuiabá	3.983	3.987	9	9
Cuiabá-IP-1	4.517	4.524	10	10

• Source: Elaborated by the authors.

Figure 5.5.6
COMPARATIVE TABLE BETWEEN PAIRS (IP-2)

Pair	Dissimilarity	Distance	Rank (dissimilarity)	Rank (distance)
Belém-Brasília	1.317	1.316	1	1
Belém-Manaus	1.453	1.442	2	2
Cuiabá-Manaus	1.944	1.935	3	3
Brasília-Manaus	2.049	2.054	4	4
Belém-Cuiabá	3.277	3.287	5	5
Manaus-IP-2	3.368	3.366	6	6
Brasília-Cuiabá	3.396	3.394	7	7
Belém-IP-2	3.827	3.831	8	8
Cuiabá-PI-2	4.325	4.327	9	9
Brasília-IP-2	4.606	4.603	10	10

• Source: Elaborated by the authors.

After completing calculations of dissimilarities of pairs among all projects and the IPs, the following section of this paper brings results analysis.

6

MODEL RESULTS ANALYSIS

The main issue is the IP definition. Therefore, we used two ways to set IPs, and the MDS was generated twice to compare the projects’ variables with each type of IP. The IP type 1 was set in order to identify the best project in the sample, and the IP type 2 simulated conditions complying with the objectives of the PAC2 Large Cities Mobility program.

Analyzing scenarios separately:

• Scenario with the IP type 1: In this scenario, Belém was identified as the best project in the sample, as it is the closest to the IP (Figure 6.1).

Figure 6.1
RANKING OF PROJECT DISTANCES IN RELATION TO IP-1

Pair	Dissimilarity	Distance	Rank (dissimilarity)	Rank (distance)
Belém-IP-1	0.843	0.843	1	1
Manaus-IP-1	1.912	1.912	3	3
Brasília-IP-1	2.296	2.296	5	5
Cuiabá-IP-1	4.524	4.524	10	10

• Source: Elaborated by the authors.

Belém has the highest RIR among the projects (41%) and the best average depth variation (7.50% reduction). Although it is not the best in C/B ratio and average integration, it has significant values for these variables. Belém and Cuiabá were extremes in our sample, and should the model not indicate that condition; it should have been rejected, which did not occur.

Manaus is closer to the IP than Brasília and Cuiabá, and the latter is distant from every other project. When we compare the relative distance between projects, Belém and Manaus are very close to one another (dissimilarity = 1.511 and distance = 1.471). It is the shortest distance on the perceptual map, thereby allowing us to classify them both as good projects, as they have the best performances in all variables. Brasília could be qualified as an intermediate project, as it is third place in IP relative distance, but remained distant from Cuiabá (dissimilarity = 3.254 and distance = 3.234).

• Scenario with IP type 2: In this scenario, all projects are distant from the IP if they are subject to a condition of specific objectives (Figure 6.2).

Figure 6.2
RANKING OF PROJECT DISTANCES IN RELATION TO IP-2

Pair	Dissimilarity	Distance	Rank (dissimilarity)	Rank (distance)
Manaus-IP-2	3.368	3.366	6	6
Belém-IP-2	3.827	3.831	8	8
Cuiabá-IP-2	4.325	4.327	9	9
Brasília-IP-2	4.606	4.603	10	10

• Source: Elaborated by the authors.

Under conditions in which the IP values do not depend on the results of the other projects in the sample, we identified that every project is distant from the IP. This corroborates the hypothesis that it is not enough for a project to be better than others: it is necessary to meet guideline expectations and the goals of the public policy under analysis.

It has been perceived that urban mobility projects seek to display benefits (the project’s intentions in their assessments), but when they are confronted with the variables assessing the sustainability of those benefits offered, divergences among variables arise. Brasília showed difficulties in presenting significant variations in topological variables, as it displays a high average depth inherent to local urban configuration. This left Brasília’s dissimilarity from IP type 2 very close to the dissimilarity found in Cuiabá.

Upon verification of proximity among projects compared with dissimilarity between them and the IP, it is possible to imagine that the administrator coordinating the proposal selection process could reject all projects, as none met the proposed goals. However, it is necessary to clearly develop and indicate the goals in order to construct the IP.

7

CONCLUSIONS

The general objective of this paper was to build a methodology for analysis, assessment, and prioritization of urban mobility plan projects capable of assisting investment decision-making, considering financial, economic, and space variables.

It has been verified that concatenating those tools in a single model improved assessment performance and gap filling. For such purpose, the MDS statistic tools managed to consolidate the behavior of those variables, allowing a consistent assessment of samples with four projects in the PAC2 Large Cities Mobility program. Moreover, MDS allows an easily understandable reading due to the projects positioning comparison on a perceptual map, representing their dissimilarities through relative distances as well as in input data of comparative tables between all projects and the IP. It is important to point out that the model does not consider inter-dependent projects.

The use of MDS with the application of economic, financial, and space variables proved to be effective in facilitating decision-making in order to identify how close and/or distant projects are from the objectives of a given public policy. This last question innovates in public administration, as it allows to refuse projects that do not comply with the set of conditions to meet the objectives proposed in the programs, even though they stand out among other projects.

We also verified that MDS could group projects with similar performances, allowing to define groups of projects to be prioritized in urban mobility investment programs, as well as groups that must be discarded from the assessment.

REFERENCES

• Assaf, A., Neto (2011). Curso de administração financeira (2a ed.). São Paulo: Atlas.
• Bandeira, M. I. V. Q. B (2005). Metodologia de priorização da expansão da rede de distribuição de gás natural com base na sintaxe espacial: Uma aplicação no setor residencial em Fortaleza (Dissertação de mestrado, Universidade Federal do Ceará, Fortaleza, CE, Brasil).
• Barros, A. P. B. G (2006). Estudo exploratório da sintaxe espacial como ferramenta de alocação de tráfego (Dissertação de mestrado, Universidade de Brasília, Brasília, DF, Brasil).
• Barros, A. P. B. G. (2014). Diz-me como andas que te direi onde estás: Inserção do aspecto relacional na análise da mobilidade urbana para o pedestre (Tese de Doutorado, Universidade de Brasília, Brasília, DF, Brasil).
• Barros, A. P. B. G., Medeiros, V. A. S., Silva, P. C. M., & Holanda, F. R. B. (2008). Análise de sistemas de transporte urbano por meio da sintaxe espacial. Congresso Luso-Moçambicano de Engenharia, Maputo, Moçambique, 5.
• Bevilacqua, S. (2004). O emprego da multidimensional scaling: Estudo de caso envolvendo seis instituições de ensino superior do nordeste paulista, uma contribuição para a qualidade em serviços. Revista Pesquisa e Desenvolvimento Engenharia da Produção, (3), 43–53.
• Brigham, E. F., & Ehrthardt, M. C. (2012). Administração financeira: Teoria e prática São Paulo: Cengage Learning.
• Cardoso, M. M., Junior, & Scarpel, R. A. (2010). Construção do mapa perceptual dos riscos socioambientais utilizando o escalonamento multidimensional (MDS). Encontro Nacional de Engenharia da Produção, São Carlos, SP, Brasil, 30.
• Companhia de Desenvolvimento Urbano da Região do Porto do Rio de Janeiro (2014). Registro de Operação Urbana Consorciada da Região do Porto do Rio de Janeiro. Código ISIN nº BRMCRJCPA003 São Paulo: BMFBovespa.
• Corrar, L. J., Paulo, E., & Dias, J. M., Filho (2007). Análise multivariada: Para os cursos de Administração, Ciências Contábeis e Economia São Paulo: Atlas.
• Dalbem, M. C., Brandão, L., & Macedo-Soares, T. D. L. A. (2010). Avaliação econômica de projetos de transporte: Melhores práticas e recomendações para o Brasil. Revista de Administração Pública, 44(1), 87–117.
• De Melo, R. A., & Setti, J. R. (2007). Fluxos mínimos de veículos para implantação de faixas adicionais em aclives de rodovias de pista simples. Transportes, 15(1), 16–23.
• Do Carmo, C. L., Raia, A. A., Júnior, & Nogueira, A. D. (2014). Aplicações da sintaxe espacial no planejamento da mobilidade urbana. Ciência & Engenharia, 22(1), 29–38.
• Esmalifalak, H., Ajirlou, A. I., Behrouz, S. P., & Esmalifalak, M. (2015). (Dis)integration levels across global stock markets: A multidimensional scaling and cluster analysis. Expert Systems with Applications, 42(22), 8393–8402.
• Flyvbjerg, B. (2009). Survival of the unfittest: Why the worst infrastructure gets built, and what we can do about it. Oxford Review of Economic Policy, 25(3), 344–367.
• Flyvbjerg, B. (2013). Quality control and due diligence in project management: Getting decisions right by taking the outside view. International Journal of Project Management, 31(5), 760–774.
• Flyvbjerg, B., Bruzelius, N., & Rothengatter, W. (2003). Megaprojects and risk: An anatomy of ambition Cambridge: Cambridge University Press.
• Flyvbjerg, B., Holm, M. S., & Buhl, S. (2002). Underestimating costs in public works projects: Error or lie? Journal of the American Planning Association, 68(3), 279–295.
• Flyvbjerg, B., Holm, M. S., & Buhl, S. (2005). How (in)accurate are demand forecasts in public works projects? The case of transportation. Journal of the American Planning Association, 71(2), 131–146.
• Hair, J. F., Black, W. C., Babin, B. J., Anderson, R. E., & Tatham, R. L. (2009). Análise multivariada de dados Porto Alegre: Bookman.
• Herdeiro, R. F. C. (2012). Escalonamento multidimensional. In L. J. Corrar, E. Paulo, & J. M. Dias, Filho (Orgs.), Análise multivariada São Paulo: Atlas.
• Hillier, B., & Hanson, J. (1984). The social logic of space Cambridge: University Press.
• Holanda, F. R. B (2007). Arquitetura sociológica. Revista Brasileira de Estudos Urbanos e Regionais, 9(1), 115–129.
• Kim, H. K., & Sohn, D. W. (2002). An analysis of the relationship between land use density of office buildings and urban street configuration: Case studies of two areas in Seoul by space syntax analysis. Cities, 19(6), 409–418. doi:10.1016/S0264-2751(02)00071-9
  » https://doi.org/10.1016/S0264-2751(02)00071-9
• Kruskal, J. B. (1964). Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis. Psychometrika, 29(1), 1–27.
• Loureiro, V. (2016). Mapa axial – sintaxe espacial – ferramentas de análise: Workshop introdutório. Brasília: UniCeub.
• Machado, J. T., Duarte, F. B., & Duarte, G. M. (2011). Analysis of stock market indices through multidimensional scaling. Communications in Nonlinear Science and Numerical Simulation, 16(12), 4610–4618.
• Mackie, P., Nellthorp, J., & Laird, J. (2005). Notes on the economic evaluation of transport projects [Transport Notes Series, Nº. TRN 5]. World Bank, Washington, DC.
• Mackie, P., & Worsley, T. (2013). International comparisons of transport appraisal practice: Overview report Leeds, UK: Institute for Transport Studies, University of Leeds.
• Mackie, P., Worsley, T., & Eliasson, J. (2014). Transport appraisal revisited. Research in Transportation Economics, 47, 3–18.
• Medda, F. (2011). Land value finance: Resources for public transport. In R. Sietchiping (Ed.), Innovative land and property taxation (pp. 42–54). Nairobi, Kenya: UN Habitat.
• Medeiros, V. (2013) Urbis Brasiliae: O labirinto das cidades brasileiras Brasília: Editora UnB.
• Muldoon-Smith, K., Greenhalgh, P., Conroy-Dalton, R., Alvanides, S., King, H., & Sparkes, B. (2015). Urban transactions: Investigating the relationship between spatial preference and spatial configuration in the city of Leeds. International Space Syntax Symposium, London, UK, 10.
• Olsen, S., & Fearnley, N. (2014). Policy transfer of public transport funding schemes: The case of Norway. Research in Transportation Economics, 48, 429–433.
• Paranaiba, A. de C. (2016). Os danos sociais e econômicos dos subsídios na mobilidade urbana do Brasil. Mises: Interdisciplinary Journal of Philosophy, Law and Economics, 4(2), 411–417. doi:10.30800/mises.2016.v4.141
  » https://doi.org/10.30800/mises.2016.v4.141
• Paranaiba, A. de C. (2017). Metodologia para priorização de projetos de infraestrutura em mobilidade urbana com base na configuração urbana e no escalonamento multidimensional (Tese de doutorado, Universidade de Brasília, Brasília, DF, Brasil).
• Paranaiba, A. de C. (2018). Praxiologia e sintaxe espacial: Articulação epistemológica em favor da mobilidade urbana como ação humana. Mises: Interdisciplinary Journal of Philosophy, Law and Economics, 6(2), 363–371. doi:10.30800/mises.2018.v6.565.
  » https://doi.org/10.30800/mises.2018.v6.565
• Pereira, A. C., & Silveira, O., Neto (2013). Viabilidade econômica de projetos e aspectos particulares em empreendimentos tipo project finance aplicados a investimentos em infraestrutura de transportes. Revista Holos, 28(6), 203–219.
• Portal PPP (2013). As concessões de transporte público de massa e as vozes das ruas. Retrieved from http://www.pppbrasil.com.br/portal/content/artigo-concess%C3%B5es-de-transporte-p%C3%BAblico-de-massa-e-vozes-das-ruas
  » http://www.pppbrasil.com.br/portal/content/artigo-concess%C3%B5es-de-transporte-p%C3%BAblico-de-massa-e-vozes-das-ruas
• Pujadas, P., Cavalaro, S. H. P., & Aguado, A. (2018). Mives multi-criteria assessment of urban-pavement conditions: Application to a case study in Barcelona. Road Materials and Pavement Design, 20(213), 1–17. doi:10.1080/14680629.2018.1474788
  » https://doi.org/10.1080/14680629.2018.1474788
• Pujadas P., Pardo-Bosch F., Aguado-Renter, A., & Aguado A. (2017). Mives multi-criteria approach for the evaluation, prioritization, and selection of public investment projects. A case study in the city of Barcelona. Land Use Policy, 64, 29–37. doi:10.1016/j.landusepol.2017.02.014
  » https://doi.org/10.1016/j.landusepol.2017.02.014
• Roigé, N., Pujadas P., Cardús, J., & Aguado, A. (2019). Water network renewal strategy: A case study of Aigües De Barcelona. Proceedings of the Institution of Civil Engineers – Water Management, 173(2), 1–27. doi:10.1680/jwama.18.00100
  » https://doi.org/10.1680/jwama.18.00100
• Souza, E. C. (2010). Os métodos biplot e escalonamento multidimensional nos delineamentos experimentais (Tese de doutorado, Universidade de São Paulo, Piracicaba, SP, Brasil).
• Tribunal de Contas da União (2015). Relatório de auditoria operacional: Governança em políticas públicas de mobilidade: TC 020.745/2014-1 Brasília: TCU.
• Ugalde, C. M., & Rigatti, D. (2007). Configuração espacial e desenvolvimento urbano-regional. Anais do Encontro Nacional da Anpur, 12.

Edited by

EDITORIAL BOARD

Editor-in-chief

Gilberto Perez

Associate editor

David Ferreira Lopes Santos

Technical support

Vitória Batista Santos Silva

EDITORIAL PRODUCTION

Publishing coordination

Jéssica Dametta

Language editor

Daniel de Almeida Leão

Layout designer

Emap

Graphic designer

Libro

Publication Dates

• Publication in this collection
  23 July 2021
• Date of issue
  2021

History

• Received
  02 Dec 2019
• Accepted
  17 Nov 2020