The quality of OpenStreetMap in a large metropolis in northeast Brazil: Preliminary assessment of geospatial data for road axes

This paper evaluates the data quality of road axes using the OpenStreetMap (OSM) collaborative mapping platform. OSM was chosen owing to the abundance of data and registered contributors (~ 6 million). We assumed the OSM collaborative data could complement the reference mappings by its quality parameters. We used the cartographic quality indicators of positional accuracy, thematic accuracy, and completeness to validate vector files from OSM. We analyzed the positional accuracy of linear features and we developed the automation of the positional accuracy process. The tool verified the completeness of road axes and thematic accuracy. The positional accuracy of linear features was also used, performed to obtain a range of scales, which reflected the characteristics of mapped areas and varied from 1:22,500 to 1:25,000. The completeness of road axes was 82% of the checked areas. By evaluating the thematic accuracy, we found that the absence of road axes toponymy in editions caused errors in the OSM features (i.e., 58% of road axes without information). As such, we concluded that collaborative data complements the reference cartography by measuring the heterogeneity of information in various regions and filtering the OSM data, despite its being useful for certain analyses.


Introduction
Measuring quality in geospatial data is essential for reference mappings with higher accuracy or thematic purposes. The data quality process is beyond the assessment of geometric accuracy and relates to the completeness, thematic accuracy and, logical consistency. According to Keates (1973), reference mappings are developed by official mapping agencies to represent all features visible in a landscape for various uses (Kent, 2009). This concept has been the only approach associated with the production of geospatial data (Elwood, Goodchild and Sui 2012). However, technological development has changed the collection, processing, analysis, and dissemination of geospatial data. Development of new data sources, establishment of digital process methodologies, and the Internet have allowed the daily production of enormous data volumes. The increase in the volume of geospatial information available in virtual databases is associated with big data. The term "big data" refers to a flow of digital data in different sources, such as numerical modeling, smartphones, and access to the Internet and social networks (Yang et al. 2017).
Technological advances have allowed any individual with a computer or smartphone and internet access to generate geospatial data (Ganapati 2011). This characteristic is associated with web 2.0, which is defined by a phenomenon in which users become fundamental agents in the production and management of data, ceasing to be just viewers and consumers (Cormode and Krishnamurthy 2008). According to Cormode and Krishnamurthy (2008), the term web 2.0 was coined around 2004, and many of the first truly web 2.0 sites began emerging in late 2003 and early 2004.
In this context, Goodchild (2007) introduced the concept of Voluntary Geographic Information (VGI), where individuals with access to the Internet can publish geographical information on a specific location, validate information posted by other users, and produce geospatial data, functions that were previously available only to the specialists. According to Johnson and Sieber (2012), as VGI gained popularity, the governmental bodies of some countries, responsible for the generation of reference mapping, used VGI for the maintenance and update of otherwise costly and time-consuming cartographic bases. In addition, the official mapping agencies generating geospatial data are to embrace rapid technological advances in the geographical space (Maulia 2018).
VGI platforms are used to generate and update geospatial data. For example, since May 2020, over 6.5 million users and 14.7 billion features have been recorded in the OpenStreetMap platform (OSM). OSM was created in 2004 by Steve Coast, who established a web-based map data under an open license to add, visualize, and distribute geographic data. Steve Coast developed OSM in response to many restrictions for obtaining certain data from the European-based Ordnance Survey (Chilton 2011). According to Neis and Zipf (2012) and Bravo (2017), OSM enabled the establishment of a universal database of highways, streets, and paths. As a result, one can share map data without restrictions or copyrights, which substantially increases the platform's information coverage (Perkins 2011).
Several techniques are used to obtain geospatial data, ranging from the field-based topographic and aerial photogrammetric surveys to the web-based collaborative VGI platforms. Efficient data handling and integration help detect issues related to urban growth, establish methodological procedures to assess the data quality, and extract and integrate the growing amounts of geographic data (Brovelli et al. 2019). The instructions on how to assess the quality of geographic metadata are provided by the International Organization for Standardization in ISO 19157 (ISO 2013): • Positional accuracy refers to the accuracy of the feature position (i.e., points, lines, areas) within a spatial reference system. It is usually assessed by comparing the position of features with their counterparts in reference data, which represent the "true" position; • Thematic accuracy refers to the accuracy of classes or thematic tags associated with specific locations or objects placed in geographic space. Classes can be assigned to pixels in a land cover map, whereas tags can be assigned to a vector-encoded entity, such as a highway, river, building, or green area; • Logical consistency refers to the adherence to logical rules for data structure, attribution, and relationships as per the product's specifications; • Completeness refers to the presence or absence of features, their attributes, and relationships compared to the product's specification. It is divided into a) commission, which explains the presence of excess data in a dataset, and b) omission, which explains the absence of data from a dataset.
• Usability (or fitness-for-use) refers to the external quality of a dataset and user needs. All mentioned data quality elements may be combined to describe the overall usability of a specific dataset for a particular use, i.e., its fitness-for-purpose.
The VGI data lacks the standard quality validation process for its products in terms of reference mapping. Therefore, a series of studies have analyzed the quality of these data types, aiming at their integration. The first analysis (Haklay 2010) compared the positional accuracy and completeness of data between the OSM platform and the United Kingdom reference mapping system. The possibility of using the VGI data was verified. The latest studies on the evaluation of the VGI data quality are summarized in Table 1.  Table 1 permits the observation of the characteristics of developed methods, defines knowledge gaps, and establishes a hypothesis for this research. We noticed that collaborative mapping lacks quick data updates on the changes in geographic space and the heterogeneity of information. The integration of reference mapping and VGI data requires frequent checks on the data quality related to the study location, number and characteristics of the contributors, form of data entry, and time of editing.
The topic of data heterogeneity has been covered by Jasim and Al-Hamadani (2020), Frassinelli (2019), andZhou (2018). Because their analyses were carried out in different study areas, it is still unclear whether heterogeneity serves as a quality indicator per ISO 19157 (ISO 2013). Regardless, heterogeneity should be evaluated and presented for the user. So far, Minghini and Frassinelli (2019) suggested the most suitable approach, where they were able to detect the issues associated with VGI data updating. Jasim and Al-Hamadani (2020), Küçük and Anbaroğlu (2019) analyzed geospatial data in the commercial software ArcGIS, which limits the user access by its high license cost. It also complicates the reusing of new tools developed by other users with a focus on application development, the feature that is possible under an open OpenStreetMap license.
Quality control is important when working with geospatial data, especially with VGI, where no specifications for data creation exist. In geoinformation, the ISO principles and guidelines are considered for quality assessment. Recently, the ISO 19113 (ISO 2002) andISO 19114 (ISO 2003) standards have been replaced by ISO 19157 (ISO 2013). The updated standard defines the following data quality elements: completeness, logical consistency, positional accuracy, temporal quality, thematic accuracy, and usability.
In Brazil, the production of reference cartographic data at a national level (a scale of less than 1:25,000) is coordinated by the Brazilian Institute of Geography and Statistics (IBGE) and the Directorate of the Geographic Service (DSG). It is important to point out that we present the acronyms in portuguese language. DSG provides matrix and vector topographic maps, ortho-images, and digital surface models at scales ranging from 1:25,000 to 1:250,000. IBGE supplies the sheets in matrix and raster formats at various scales and provides access to continuous bases in Brazil at the 1:1,000,000 and 1:250,000 scales. Mapping in Brazil at a scale of 1:25,000 provides a total area coverage of 5% (Sluter et al. 2019). Germany, Austria, and Switzerland have dense reference networks for the surveys, while some municipalities in Brazil still lack them for updated mapping (Klein et al. 2017). Camboim et al. (2015) pointed out that mapping Brazil, with its vast territory, is costly and is predominantly financed by governmental agencies. As such, we suggest integrating alternative methods for the quicker and cheaper production of geospatial data in Brazil by incorporating the quality-associated VGI data into reference cartographic bases.
The first evaluation of the geospatial data quality of cartographic products in Brazil dates back to the Decree-Law no. 89.817 "Technical Norms of National Cartography" (Brazil 1984). It established a set of diagnostic criteria for the accuracy and distribution of errors based on a statistical indicator of positional quality, the so-called Cartographic Accuracy Standard (PEC). PEC is divided into three classes (i.e., A, B, and C), where class A is characterized by the highest rigor of evaluation of the cartographic product, and class C has the lowest rigor. A PEC value is established for each class and is associated with a standard error. This standard error corresponds to 60.8% of the PEC value and a 90% probability of the normal distribution. The PEC values are used for both planimetric and altimetric assessments.
In 2011, the Technical Specification for the Acquisition of Vector Geospatial Data -ET-ADGV (DSG 2011) was presented by the Directorate of the Geographic Service (DSG). This specification updated the standards of the Decree-Law no. 89.817 (Brazil 1984), which failed to meet the needs of digital media. A new statistical indicator for ET-ADGV (DSG 2011) was established to revise the Cartographic Accuracy Standard for Digital Cartographic Products (PEC-PCD). This indicator is similar to PEC with an extra class (D). Thus, the cartographic products are classified as A, B, C, and D. PEC-PCD is associated with the scale of certain cartographic products and allows their classification by the maximum error obtained by the discrepancies that draw up a sample of points. It serves as a statistical indicator with a 90% confidence level based on a probabilistic graph of normal distribution. More recent versions of ET-ADGV developed by DSG in 2015 and 2016, the so-called Technical Specification for the Acquisition of Vector Geospatial Data for Defense of the Land Force -ET-ADGV-DefenseFT (DSG 2015; DSG 2016), addressed several aspects of PEC-PCD. In 2015, DSG created the Technical Specification for Quality Control of Geospatial Data -ET-CQDG (DSG 2015), which covers methodologies for assessing the quality of cartographic products similar to ISO 19157 (ISO 2013). ET-CQDG (DSG 2015) allows calculating the Euclidean distances of a sample of homologous points between the reference and evaluated cartographic products. The calculated Euclidean distance permits the classification of the cartographic product on a scale and evaluates it based on two conditions: 1) whether 90% of the Euclidean distance is less than or equal to the PEC-PCD value, and 2) whether the square mean error is less than or equal to the standard error.
Although several studies have verified the quality of OSM, gaps regarding its use still exist, especially given the lack of a standard in the data provision and VGI heterogeneity. Moreover, some VGI methodologies have already been incorporated into the reference mapping. As such, we aim to assess the quality of the OSM data from different sectors of the same study area using various sample spaces by measuring the OSM quality aspects and obtaining the characteristics of data heterogeneity. The study results show that we may successfully incorporate the existing geospatial information from other parts of the world and improve the parameters for integrating this data into the reference mapping.

Methodology
The study area is located in Salvador (BA), Brazil, with a population of approximately 2.9 million people (2017). Salvador (BA) is the largest city in the Northeast Region, with a total area of 693 km2 and a population density of 4,187/km 2 . The reference map used for this study forms part of the cartographic and cadastral system of the municipality (SICAD). Figure 1 shows the subdivision of the municipality of Salvador (BA) by prefectures and neighborhoods. Besides the subdivision by prefectures and neighborhoods, the physical completeness of the road axes was determined by their overlapping with the Salvador (BA) development regions (e.g., central area, coastal expansion, miolo, and Subúrbio). First, we evaluated the positional accuracy of the road axes using the double buffer method by Tveite and Langaas (1999) and adapted by Santos (2015). We followed the Cartographic Accuracy Standard for digital cartographic products in our statistical tests. Merchant (1982) presented a sampling method to select punctual features, where at least 20 control points are needed to evaluate the positional quality. We applied the same methodology using two samples with 20 linear features.
The behavior of point features tends to differ from that of linear features, and more studies are needed on the sampling method for linear features. However, this issue was ignored herein as we established the positional accuracy following Merchant (1982). Other studies have shown that the positional accuracy is the main indicator of the geospatial data quality (Ibrahim, Ramadan and Hefny 2019;Fonte et al. 2017;Roberto 2013;ISO, 2013). Theoretical analyses on the positional accuracy led to the development of QGIS 3.10 using the Python scripts. This new tool automated the assessment of the positional accuracy of linear features according to the PEC-PCD scales and permitted the spatial visualization of the discrepancies from quartile subdivision. In this way, the user can select the scale and data to evaluate and view the reference data in the graphical interface. The results are presented as an attributes table in the primary screen of QGIS 3.10. This script is available at https://github.com/eliasnaim/ AcuraciaPosicional_PEC-PCD.
As stated previously, VGI heterogeneity is one of the main issues to be resolved. It is associated with the number of features represented, different ways to obtain data, and the time of the edits. In Brazil, a preliminary study by Elias et al. (2018) assessed the positional accuracy in different sample sets of point features and revealed a scale variation from 1:20,000 to 1:30,000 in the OSM database. Based on this evidence, the positional accuracy was analyzed using two sample sets in the region of the checks.
Therefore, more than one sample should be used to evaluate positional quality more accurately in a particular region. Santos (2015) found that 90% of the tested lines had an average discrepancy less than or equal to the PEC value (class and scale used to generate the buffer distance), whereas the root mean square of the average discrepancy was less than the PEC-PCD standard error value. If both conditions are satisfied, the cartographic product is classified according to its class and scale. Figure 2 shows an example of the application of the double buffer method.

Figure 2: Double buffer method
Second, we evaluated the completeness of road axes using the method by Haklay (2010), where the dimensions of the road axes were compared to the reference data, and the study area was subdivided into the grids with a spatial resolution of 1 x 1 km. The study by Haklay (2010) pioneered the use of grids to assess the quality of the OSM data, and more studies followed Brovelli et al. (2016), Zhang and Malczewski (2018), Brovelli and Zamboni (2018) and Martini, Kuper and Breunig (2019). Herein, we also subdivided the study area and linear features corresponding to the road axes in OSM and SICAD into the grids with a spatial resolution of 1 x 1 km. The evaluation of the magnitude of discrepancies verified the completeness in the lengths between these two databases in relation to each grid.
We performed the completeness analysis by filtering the vector files missing in the reference data and present on the platform (e.g., bicycle paths, pedestrian paths, bus lanes, tracks, race tracks). Thus, we obtained the results that resembled the related attributes.
Third, we evaluated the thematic accuracy by confirming the tags of the road axes in OSM. The verifications were completed using the name and hierarchy of the road axes (Figure 3).

Figure 3: Compatibility between SICAD and OSM
Other attributes of the road axes in OSM, such as the number of traffic lanes, types of tracks (e.g., single, double, triple), and maximum speeds were not evaluated because they lacked in the SICAD database, making the comparative analysis impossible.
The sampling method used to select the evaluated items for the thematic accuracy corresponded to the parameters established by ISO 2859-1 (ISO 1999) and ET-CQDG (DSG 2015). Although the sampling procedure outlined in ISO 2859-1 (ISO 1999) foresees the sample rejection to a number of incorrect elements, these elements were computed to verify the types of inconsistencies inherent to routes vector files in OSM.
To verify the hierarchy of routes, we harmonized the categories assigned to OSM ( Figure 3) and SICAD following the methodology by Zhang and Malczewski (2018). They assessed the quality of cartography of road axes of Canada in OSM and adapted the official classification of Canadian routes of Digital Mapping Technologies Inc. to OSM. Herein, we aligned the hierarchy of road axes in OSM and SICAD based on the classification provided in Chapter III of the Brazilian Traffic Code, Law 9503/1997. Table 2 shows the compatibility of the OSM and Brazilian CTB road hierarchy classifications. We evaluated the thematic accuracy relative to the quantity (percentage) of correct items in the OSM and SICAD verified road hierarchies. Bicycle paths, pedestrian paths, bus lanes, and race tracks in OSM were left unharmonized and untested in the sample space being incompatible with the road axes in SICAD. We also verified that the names of road axes were compatible between OSM and SICAD. Being mandatory for filling by a collaborating user, all attributes of the road hierarchy in OSM were present. The filling of the names in OMS is optional. As such, empty fields were observed for some categories. It allowed the evaluation of the completeness of this attribute of road axes in said sample space.
The completeness and thematic accuracy analyses were performed for Salvador (BA) using the subdivision of prefectures and neighborhoods. Figure 4 presents the flowchart of the methodological procedures used.

Positional Accuracy
To automatically evaluate the positional accuracy, we used QGIS 3.10; and its user interface design is shown in Figure 5. The average area discrepancies and RMS are outlined in Table 3 on scales of 1:22,500 and 1:25,000, respectively. Figure 6 shows the map with the distribution of linear features in the study area.  Using the calculated average discrepancies, we validated the analysis on the scales applying the established verifications for 90% of the obtained values. The variations in average discrepancies were related to the threshold values in the buffer, where the evaluated scale was characterized. Different results were obtained for the two analyzed samples for class A.
To scrutinize the discrepancies, we analyzed the arrangement of road axes in OSM and the Salvador geodatabase. The heterogeneity of the discrepancies may be related to the period in which the editions were initiated or to the BING image characteristics. Figure 7 demonstrates apparent displacements in the linear features. Figure 8 reveals almost identical linear features of OSM and the Salvador geodatabase, providing spatial evidence of their variations. In addition to the features related to the editing period of the element associated with the parameters of the image used as support for this, however, these errors can also be attributed to the level of detail, with which the linear features were edited in OSM.

Completeness
We evaluated the physical completeness of vector files by characterizing linear features and calculating the discrepancies in the length of linear tracks using the 1 × 1 km grid system.
The results of the completeness analysis were superimposed on the vector files of the development regions of Salvador (BA) (i.e., middle, suburb, coastal expansion, and central) The city subdivisions originated from the IBGE census tracts, which generated traffic zones in the source-destination survey (1995), and the Sustainable Mobility Plan of Salvador (BA) (PLANMOB, 2017). These are shown in figure 9.
The length of track axes after the exclusion of these data was determined to be 3,451,706.92 m. A previous evaluation demonstrated the potential to estimate the number of taxpayers in this municipality. The cartographic base was used as a reference with a temporal variation of 12 years in relation to OSM. The discrepancy of 1,003,516.90 m revealed the completeness of features in this area (Table 4).  The obtained results showed the growth in the representation of road axes in Salvador (BA). Considering the discrepancies and temporal variations in the SICAD database in the study area, the OSM vector files and the completeness methodology are deemed to have the potential to verify expansion aspects in certain localities, especially in the regions with significant changes in a certain period.
The Miolo region presented greater discrepancies compared to the other regions (Figure 9), indicating urban growth associated with the construction of a new subway line to the municipality and the expansion of traffic routes. Despite the outdated mapping of Salvador (BA), the authoritative-type mapping (2006) allowed the identification of different dynamics associated with the contributions, mainly in the completeness assessment. The temporal variation of OSM in relation to the geospatial data of SICAD permitted the recognition of smaller domains within the study area with higher contributions in recent years. Therefore, we were able to compare the initiatives for expansion, growth, and construction.

Thematic Accuracy
The population sample and the number of evaluated elements were calculated according to the ET-CQDG (DSG 2015). Although this specification presents an acceptable limit of errors for a given sample, it was neglected herein. First, we checked the number of items to be evaluated from the sample population. The total number of edited routes in OSM reached 19,082 elements, and the total number of evaluated elements amounted to 315. Table 5 shows the verification of the thematic accuracy of the linear features classification (type) in OSM, using vector files in SICAD for reference. The sample consisted of approximately 32 features (points) in each polygon.  Considering a 90% confidence level for a population of 19,082 and a sample of 315 elements, the differences greater than the sampling error (4.61%) were regarded representative. Correctness it is around 77% by the thematic accuracy. As such, OSM is a reliable tool that can be used in the future. It is possible indicate high variability due 25% correctness amplitude in the neighborhood comparison greater than 4.61% sampling error.
As mentioned earlier, the insertion of the classification (type) in the OSM road axes by a contributing user is mandatory, making such high confidence levels possible.   Out of the 315 evaluated linear features, 116 (37%) were classified as correct in relation to the Salvador geodatabase (Table 6), 17 linear features had incorrect names, and 182 features were nameless. The pathways with similar spellings of the same name were treated as correct. The main sources of errors for the road axes names in OSM were related to the items without information. The absence of names in the evaluated features was encountered in 58% of the analyzed samples. The analysis performed by prefecture and neighborhood suggested certain variability in the obtained values, which ranged from 13% (Subúrbio) to 84% (Barra-Pituba). For example, the prefectures of Barra-Pituba and Centro-Brotas had the highest percentages of correct names equaling 84% and 69%, respectively. In contrast, the prefectures and neighborhoods of Liberdade-São Caetano, Valéria, and Subúrbio showed the lowest percentages of correct names with 16%, 16%, and 13%, respectively. Regarding the incorrect names, we made queries to the vector files to investigate the origin of the errors. We verified that only one feature located in the prefecture and neighborhood of Liberdade-São Caetano, was incorrectly named. In the other 16 road axes with incorrect names, the errors were related to their topological consistency. In general, the majority of the road axes in OSM were named correctly.
Certain breaks in the vector files of the reference axes in the Salvador geodatabase characterized changes in their names and other attributes and differed from the homologous OSM features ( Figure 10). This indicated the discrepancies related to the logical consistency of linear OSM features. The inconsistencies presented in Figure 11 demonstrate the flaws in the topological structure of reference axes. This error was verified for 16 linear features out of the 17 incorrectly named ones. As a result, the primary source of errors associated with the OSM road axes names was related to the absence of this element. In the prefecture and neighborhoods of Liberdade-São Caetano, Valéria, and Subúrbio, road axes with absent names reached 77%, 84%, and 84%, respectively.

Conclusion
Through multiple analyses performed in the study area, we examined the viability of using the OSM road axes features for mapping purposes by checking the veracity of geospatial data compared to authoritative mapping. This characteristic was observed in the completeness assessment, where the regions with the largest contributions in the past 12 years represented urban growth areas.
By evaluating the positional accuracy at a scale of 1:22,500, we were able to observe the limitations related to the precise identification of pathway locations and traces in the vertices. The layout of the OSM roads revealed a heterogeneous pattern, likely due to the period in which the editing was performed, the level of detail applied by the users, and the image quality.
We also evaluated the representativeness and found that 80% of the samples were representative of the dataset.
We expect that OSM will be adopted as the future VGI data standard, considering its potential as associative support in the studies related to urban growth in certain regions with outdated project areas. However, more studies are needed to analyze the data heterogeneity. Some European countries have been incorporating official cartographic data into their spatial data infrastructure. Despite OSM being a universal tool for spatial data infrastructure, in Brazil, this practice has not been incorporated yet.
As for future work recommendations, we suggest automating the method, establishing a conceptual framework for the OSM data quality assessment, and linking the analyses to different graphical primitives.