COMPARISON BETWEEN ABSOLUTE AND RELATIVE POSITIONAL ACCURACY ASSESSMENT-A CASE STUDY APPLIED TO DIGITAL ELEVATION MODELS

Comparação entre métodos de avaliação de acurácia posicional altimétrica absoluta e relativa – um estudo de caso aplicado a Modelos Digitais de Elevação Leandro Luiz Silva de França1,2 – ORCID: 0000-0003-0863-1926 Alex de Lima Teodoro da Penha1 – ORCID: 0000-0002-3275-1840 João Alberto Batista de Carvalho1,2 – ORCID: 0000-0002-2864-3302 1 Diretoria de Serviço Geográfico do Exército, 3o Centro de Geoinformação (3oCGEO), Olinda, Pernambuco, Brasil 2 Universidade Federal de Pernambuco, Departamento de Engenharia Cartográfica, Programa de Pós-Graduação em Ciências Geodésicas e Tecnologias da Geoinformação, Recife, Pernambuco, Brasil E-mail: geoleandro.franca@gmail.com; alexdapenha@gmail.com; jalbertobcarvalho@gmail.com


Introduction
The International Organization for Standardization (ISO) gives international norms about geographic information, being developed in its 19,100 series (ISO 19.157, 2013).The Brazilian standards were built based on these norms through the National Spatial Data Infrastructure (INDE), according to Decree No. 6,666, of November 27, 2008 (Brasil, 2008), with one of its objectives being control quality (DSG, 2016), with the aim of providing, in a standardized way, the methods for the evaluation of geographical data.
Among the technical specifications cited previously, the CQDG is the focus of this article, since it standardizes the quality evaluation criteria, establishing the required measures for each element of quality (Omission, Conceptual Consistency, Format Consistency, Absolute Positional Accuracy and Temporal Validity) considering different geospatial dataset (DSG, 2016).It should be noted that the first edition of the CQDG presents only measures for the evaluation of absolute positional accuracy and does not provide definition of measures for the evaluation of relative positional accuracy.This is the main reason of analyzing and discussing the results of this type measurement, judging their feasibility or not, for the quality control of Digital Elevation Models (DEM).
Another consideration on the relevance of this work is the fact that the relative method, in comparison with the absolute method, can be applied in a global way, since it makes use of the entire population of pixels, obtaining more realistic results, besides identifying regions with greater discrepancies and outliers.
The objective of this work was to compare quantitatively and qualitatively the statistical results of the assessment of the absolute positional accuracy of the Digital Elevation Digital Models (DEM) Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER) and Shuttle Radar Topography Mission (SRTM) with the results of the relative positional accuracy, calculated based upon a reference DEM with better accuracy.

Theoretical conceptualization
Aiming at the theoretical basis of this work, this chapter discusses the definitions used for digital models from altimetric data and describes the characteristics of the evaluated DEM (ASTER and SRTM).It is also reviewed the concepts of precision and accuracy, as well as the definitions of absolute and relative positional accuracy.

Definitions of Digital Model for Altimetry
The surface of the Earth can be modeled in three dimensions by means of Digital Elevation Model (DEM), Digital Terrain Model (DTM) or Digital Surface Model (DSM), however there are still dissonances about its definitions.Li et. al. (2005apud Santos et al., 2016) considers the DTM as a set of points that represent the spatial distribution of various types of information on the terrain, being the DEM a particularization when this information translates the altitude of the terrain that models the emerged or submerged land without vegetation canopy or buildings.
The PCDG (DSG, 2015), which is of interest in this work, defines an DEM as a cartographic product obtained from a mathematical model that represents a phenomenon, continuously, from adequately structured and sampled data of the real world.A DEM is called the DTM when it is associated with the representation of the bare-earth (exposed soil), disregarding the accidents found above the ground (buildings, bridges, vegetation, etc.).When the accidents on the ground are modeled, then the DEM is called as DSM.
To agree with the PCDG and, consequently, with the norms of the INDE, this work considers the generic name of all altimetric digital models as DEM, without the concern to differentiate the models that represent or not the bare-earth.

ASTER GDEM
The ASTER data are the result of a cooperative effort between the National Aeronautics and Space Administration (NASA) and the Ministry of International Trade and Industry of Japan (METI), Earth Resources Data Analysis Center (ERSDAC) and the United States Geological Survey (USGS) for the construction of an open access global DEM (USGS, 2014).
To obtain surface elevation data, ASTER sensors placed on TERRA satellite captured stereoscopic images between 2000 and 2010, operating in the bands of the near infrared spectrum.The GDEM is the most complete model of the Earth's surface, covering 99% of its surface, between the latitudes of 83 ° N and 83 ° S (Celestino and Philips, 2016;Santos et al., 2016).ASTER GDEM v.2 is provided by the USGS Earth Explorer in the GeoTIFF image format in 1°x1° geographic cutouts, with spatial resolution of 1 arc-second (about 30 meters near the equator).The files are in the WGS 84 geographic coordinate reference system, with the geological model EGM96 as altimetric reference (USGS, 2014).
According to Tachikawa et. al. (2011), ASTER GDEM v.2 has an absolute vertical accuracy of 17 meters with a 95% confidence level.The authors also verified that in forest areas the values of the model are about 8 meters above the correct reference and in areas of exposed soil the model tends to be 1 meter below the reference.

SRTM
The SRTM was a project of NASA and the National Geospatial-Intelligence Agency (NGA), which had the international collaboration of the German Space Agency (Deutsches Zentrum für Luft-und Raumfahrt -DLR) and the Italian Space Agency (Agenzia Spaziale Italiana -ASI) to acquire radar data with the objective of creating the first global altimetric model (USGS, 2015, Santos et al., 2016).
In order to obtain the elevation data, two Synthetic Aperture Radars (SAR) were used by interferometry (InSAR) in the C and X bands.The data were collected aboard the space shuttle Endeavor in a period of 11 days (11 to 22 February 2000) and cover more than 80% of the Earth's surface area, covering latitude 60 ° N to 56 ° S (Forkuor andMaathuis, 2012, Celestino andPhilips, 2014).
The SRTM MDE are provided by the USGS EarthExplorer in the GeoTIFF image format in 1°x1° geographic cutouts, with spatial resolution of 1 and 3 arc-seconds (about 30 and 90 meters near the equator line, respectively).The files are in the WGS 84 geographic coordinate reference system, with the geographic model EGM96 as altimetric reference (USGS, 2015).
It should be highlighted that, until recently, the vast majority of researches on SRTM was based on products with a resolution of 3 arc-seconds, since the models with resolution of 1 arc-second were made available to the public openly from November 2014 (LP DAAC, 2014).
The absolute vertical accuracy of the DEM SRTM of 3 arc-seconds is less than 16 meters, although this accuracy may vary, depending on the continent and region (Jozsa et al., 2014).Already for the model with 1 arc-second was obtained an accuracy better than 6 meters in the region of Saudi Arabia (Elkhrachy, 2017) and in the northeast of Brazil (França et. al, 2018).Sciences, 25(1): e2019003, 2019

Accuracy (or Exactness) versus Precision
The positional accuracy is one of the quality elements standardized in ISO 19.157 (ISO 19.157, 2013) which specifies how the geometric and positional characteristics of geographic data must be evaluated.
Nevertheless, for the evaluation of positional quality, several terms have been used, highlighting accuracy, exactness and precision, sometimes for reasons of idiomatic translation, leading to misinterpretations.Therefore, it is necessary to differentiate the concepts of precision and accuracy, being the latter synonymous of exactness (Monico et al., 2009).
According to Celestino and Philips (2014), the accuracy shows the degree to which spatial information is recorded in its true position, relative to a terrestrial reference or with reference to other adjacent information.They also define precision as the degree of agreement of a series of measurements made on similar conditions, translated by the standard deviation of the observations.Gemael (1994) states that the term accuracy is related to random and systematic effects and concerns the dispersion of values in relation to the exact value of the observation, whereas the precision is related to random effects and concerns the dispersion of values in relation to the average of the observations.According to Monico et. al. (2009) the concept of accuracy involves systematic and random errors, while the concept of precision involves only random errors.The accuracy, therefore, can be translated in terms of the following parameters: tendency and precision (Monico et al., 2009).Mathematically the tendency is calculated as the difference between the mean of the observations and the known (or expected) value and the precision is the standard deviation of the observations.The Accuracy measure proposed by Guass is known as the Mean Square Error (MSE) calculated by (1), where ² represents the dispersion of the measures (variance or uncertainty) in relation to the average of the observations and b represents the tendency, bias or vice of the estimator (Monico et al., 2009).In other words, the ² means random errors and b translates systematic errors.
For large samples, the MSE approximates to the quadratic mean of the errors (), where  correponds to the difference of an observed (or measured) value   and the reference (known) value   (Equation 2).

𝜀 = 𝑧 𝑜 − 𝑧 𝑟
(2) In this research, it was considered the mathematical formulation for tendency b and precision  given by ( 3) and ( 4), being the accuracy calculated by the Root Mean Square Error (RMSE), given by ( 5).

Absolute versus Relative Positional Accuracy
The concept of positional accuracy is well established in the cartographic literature and defined in the standards of ISO 19.113, ISO 19.114 and ISO 19.138 (Ariza-López et al., 2007).
The assessment of positional accuracy of geospatial data corresponds to a statistical analysis of the feature discrepancies identified on the map in relation to a given reference (DoD, 1990).Depending on the characteristics of the reference, positional accuracy has the following classification (Santos et al., 2017): ▪ Absolute (or external): statistics based upon the discrepancy of the observed coordinate in relation to a reference coordinate accepted as true.This commonly happens when the reference coordinate is measured with instruments that ensure their values faithfully translates the terrain reality.For example, with reference points collected by relative static positioning with dual frequency GNSS receivers.
▪ Relative (or internal): statistics based upon the discrepancy of observed coordinate in relation to coordinate of other dataset which respective position has better accuracy.For instance, the evaluation of features mapped to the scale 1: 25,000 having as a reference features mapped on the scale 1: 1,000.
For the relative positional accuracy assessment, Celestino & Philips (2014) suggest that the reference data should be three times better than the declared data for the tested data.In other words, Camargo et. al. (2009) confirm that the reference data must present positional error equal to or less than one-third of the standard error of the evaluated scale.
Although the assesment of absolute positional accuracy seems to be the most appropriate for mostly cases, the evaluation of DEM based on a small set of points may not be the most appropriate method (Józsa et al., 2014).

Brazilian Cartographic Accuracy Standard for Altimetry
All paper topographic charts produced officially in Brazil for systematic mapping must follow the positional accuracy accordingly to the Cartographic Accuracy Standard (Padrão de Exatidão Catográfica -PEC) which parameter values was established in Decree No. 89.817 of June 20, 1984 (Brasil, 1984), transcribed though the Table 1.With the recent evolution of digital cartography and the geothecnology's resources, new products require other quality parameters, including a refinement of the assessment for positional accuracy assessment (Ariza-López et. al, 2007) and computational programs to perform it (Nero et. al, 2017).
The Geographic Service Bureau (Diretoria de Serviço Geográfico -DSG, 2016) established the quality assessment parameters described in the Technical Specifications for Geospatial Set Products (ET-PCDG) and Geospatial Data Quality Control (ET-CQDG).

Study Region
The study region covers a part of the west of the Brazilian State of Bahia, corresponding to an extension of 3º of longitude by 2º of latitude.The analyzed models correspond to the DEM ASTER v.2 and SRTM v.3, both with resolution of 1 arc-second, distributed in the 1° x 1° joint by the United States Geological Survey (USGS, 2014; USGS, 2015), according to the frame shown in Figure 1.

Methodology
This chapter details the procedures for the evaluation of the absolute and relative positional accuracy of the DEM, showing how the data used as reference were obtained, as well as the method for comparing the results.
It is important to emphasize that all steps were performed with the General Public License (GNU), open-source software, being QGIS 2.18 and Python scripts based on the GDAL module.The developed scripts are available in the following Github repository: http://www.github.com/LEOXINGU/DEM_assessment.

Data Sources
The DEM ASTER and SRTM evaluated in this work and presented in figure 1 were obtained from the USGS EarthExplorer website: https://earthexplorer.usgs.gov.
The Ground Control Points (GCP) and DEM adopted respectively for absolute and relative reference are available in the Geographic Database of the Brazilian Army (Banco de Dados Geográfico do Exército -BDGEx) through the following website: http://www.geoportal.eb.mil.br/mediador/.
The DEM used as reference were obtained by digital photogrammetric flight performed by the company Engemap, through the airborne system of acquisition and post-processing of digital images (Sistema Aerotransportado de Aquisição e Pós-processamento de Imagens Digitais -SAAPI), during the years of 2009 and 2010 (Penha et al., 2012).

Absolute Positional Accuracy Assessment
For this evaluation were used GCP from GNSS orbital survey, through relative static method, executed by the Brazilian Army.It was collected more than 35,000 points in order to evaluate and approval the positional accuracy of the DEM for the Mapping Project of the State of Bahia (Penha et al., 2012).Of this total, 6,568 points were used in this research.Figure 2 shows the control points on the study region and Table 4 shows the number of GCP for each region.The orthometric altitude of the GCP has as altimetric reference the Datum of Imbituba, different of the vertical Datum of the evaluated DEM which is the EGM 96 WGS 84.However, as the difference between the altitudes in these references is in the order of 0.10 to 0.54 meters (Moura et al., 2014), the transformation between these vertical references was not performed, considering its smallness in relation to the values of EM and EP from PEC-PCD (Table 2).
Figure 3 illustrates, through a flowchart, the evaluation process of each DEM, being this process executed by the Python script "DEM Absolute Accuracy", where the output corresponds to an HTML file containing the PEC-PCD classification result and related statistics.The reference DEM also had their absolute positional accuracy assessed.For that, It was created mosaics with the reference DEM in order to verify if these models have better positional accuracy than the evaluated models.Those mosaics also was used for relative positional accuracy assessment.

Relative Positional Accuracy Assessment
In this evaluation, it was used the DEM generated by photogrammetry for the State of Bahia Mapping Project (Neto et al., 2014;Penha et al., 2012) as reference dataset.They have better resolution and positional accuracy than the ASTER and SRTM DEM.
The reference DEM has resolution of 5 meters in frames of 7.5' x 7.5', corresponding to the articulation of the scale 1: 25,000 for the Brazilian systematic mapping and referenced in the Vertical Datum of Imbituba and Horizontal Datum SIRGAS 2000.
Therefore, for each of the six ASTER and SRTM models in the study area a mosaic composed of 64 reference DEM was used, accordingly Figure 4.

Classification of PEC-PCD
The Standard of Cartographic Exactness (Padrão de Exatidão Cartográfica -PEC) for Digital Cartographic Products (Produtos Cartográficos Digitais -PCD) in the ET-CQDG (DSG, 2016), classifies geospatial data according to its accuracy in classes "A", "B", "C" or "D", based on the maximum permissible error (EM) and the standard error (EP ) to the altimetry.
Accordingly to Equation 2, the error of the altitude component in a reference position (x,y) is calculated by the difference of zo (the altitude calculated by interpolation of the pixels' values near the reference position) in relation to zr which is the orthometric altitude of the reference point.
For the determination of the altitude zo, the bilinear interpolation was adopted, similar to Reuter et. al. (2009), aiming to calculate the linear influence of the altitude values f (x,y) of the 4 pixels closest to the evaluated DEM in relation to the reference point, as shown in Figure 6 and Equation ( 6).  = (1 − )(1 − )(, ) + (1 − )( + 1, ) + (1 − )(,  + 1) + ( + 1,  + 1) (6) In the case of the relative accuracy assessment, the reference points correspond to the center of the pixels of the reference DEM, zt is also calculated by bilinear interpolation.
According to ET-CQDG (DSG, 2016), the classification of the PEC-PCD is given by observing the following criteria: ▪ Each error value  is compared with the maximum permissible error (EM) of the PEC-PCD table (according to class and scale), being classified where at least 90% of the absolute value of the errors is lower than EM for a specific class and scale.
▪ The result of the RMSE is compared with the EP from PEC-PCD table.If it is smaller, then the dataset is classified to that class, otherwise the RMSE is verified following each PEC-PCD class in a specific scale.If the result is not sorted in any of the classes, then it is said that the result is "not compliant", also called "R" class.
It is important to highlight that the Python scripts, developed by the authors, were written taking all care that the values of the null pixels, present both in the SRTM DEM and in the reference DEM, did not interfere with the calculations of the statistics.

Results and discussions
The following tables 5 and 6 present the results of the altimetric positional accuracy assessment for absolute and relative methods, respectively.It is shown statistical results of the calculated errors in meters (m) and the PEC-PCD category in which each ASTER and SRTM DEM have been classified for the following scales: 1:25,000 (25K), 1:50,000 (50k), 1:100,000 (100k) and 1:250,000 (250K).When the tables present the "R" for the PEC-PCD classification, it means that the data does not fit even the minimum standard (class "D") of the scale evaluated and therefore must be rejected.

Absolute positional accuracy Results
By calculating the average values of the results of Table 5, it is verified that the SRTM models have a tendency to be 3.69 meters displaced above the reference on the ground, however its precision of 2.01 meters is 3.22 times higher than the precision of 6.48 meters of ASTER DEM, that is, the altimetric errors are less dispersed (scattered) in the SRTM than in the ASTER, coming up very close to the precision of 1.1 meters of the photogrammetric DEM used as reference.Figure 7 presents the histograms of the altimetric discrepancies of the absolute accuracy assessment for the ASTER and SRTM DEM, as well as for the Reference DEM.It is observed, through the histograms and Table 5, the superiority in terms of accuracy (tendency and precision) of the Reference DEM in relation to the ASTER and SRTM models.Thus, the Reference DEM should be used for the relative positional accuracy assessment, in view that, on average, the accuracy of these models are 3.4 times better than SRTM and 6.1 times better than ASTER DEM.

Relative Positional Accuracy Results
Table 6 presents the statistical results (in meters) and the PEC-PCD for the SRTM and ASTER DEM, having as reference the photogrammetric DEM generated for Bahia Mapping Project.

Discussions
From Tables 5 and 6, it is verified that, in the relative method, the RMSE and the tendency are, on average, slightly smaller for both evaluated models.Differently, only, the precision values of the SRTM DEM proved to be larger (worse) in the relative method.
Table 6 presents the greatest discrepancies, in absolute terms, of the evaluated models for the study region, allowing to identify more clearly the "true" outliers, which is improbable to detect applying the absolute method.
About the classification of the PEC-PCD, the relative method led to better or equal results to the absolute assessment, with no case where the relative method was less than the absolute method.
Figures 8, 9 and 10 display images with altitude difference between the evaluated models and the reference model, as well as the difference between the two evaluated models, with the SRTM model as a reference to the ASTER.The location of the areas with tendency to be above the reference is represented in shades of green as well as the areas below the reference is in shades of purple.Figure 10 proves that the systematic error of the scan lines checked in Figure 8 is caused by the processing of the ASTER data, when it was used the SRTM models as reference.Also, in Figure 8, it is possible to observe that, in the two DEM in the east, the values of the altitudes in the crest lines and peak of the elevations tend to be smaller than the reference, unlike the funds of valleys and drainage lines, where the altitude values are higher than the reference.
Figure 9 confirms the tendency of the SRTM models to be above the reference, occurring the biggest differences in areas of natural and cultivated vegetable cover.
Therefore, from the data in Table 5 and 6 and the information contained in Figure 8, 9 and 10, some advantages of the relative assessment may be cited in relation to absolute: ▪ Use of a much higher population of reference observations, corresponding to the center of the pixels of the Reference DEM, making the role of the GCP.
▪ The reference DEM covers 100% of the study area and, therefore, all the pixels of the models, except for the null pixels, were evaluated, allowing to check the largest and smallest outliers.
▪ The image of the difference between the DEM makes it possible to clearly identify the areas of major discrepancies, enabling to identify objectively the potential causes.
However, it is important to emphasize that to the relative positional accuracy assessment reaches more reliable results and not distort the results, the positional accuracy of the reference model should be at least three times better than the evaluated model (DSG, 2016;Celestino & Philips, 2014;Camargo et al., 2009).

Conclusion
The absolute positional accuracy assessment was performed from GCP collected by GNSS survey.The evaluation result is translated by means of the RMSE (Monico et al., 2009).Thus, briefly, it can be stated that, on average, the absolute accuracy of the Reference DEM are better more than three times than both evaluated models; and the SRTM model is better than ASTER, as indicated below (7).
.= 1,2 <   = 4,2 <   = 7,6 The component that interferes most in the RMSE of the SRTM is the tendency (approximately 3.7 meters).If this systematic error is eliminated, the RMSE of the SRTM will be approximately equal to the accuracy (Monico et al., 2009), that is, 2.0 meters, which would imply PEC-PCD "A" and "B" in scales 1:50,000 and 1:25,000, respectively.
In the relative assessment, the results of positional accuracy were slightly better than the absolute evaluation, being the averages of the RMSE of the models given by (8).
= 3,7 <   = 7,1 The great accuracy of the DEM SRTM in relation to ASTER is due to the techniques used for the generation of the altimetric model, being the first carried out by radar and the second by photogrammetry, which was subject to interference of clouds and other variants (Goméz et al., 2012).
Although the ASTER models are inferior in positional accuracy than the SRTM, according to the results of the PEC-PCD (Tables 5 and 6), these models can be used for mapping in scales 1:100,000 and 1:250,000.
One advantage of ASTER DEM in relation to SRTM is the greater completeness, that is, do not present null pixels.Unlike the SRTM models of 1 arcseconds, which in some cases can contain many null pixels.
In this research, it was mainly aimed to define and present a practical example between the absolute and relative assessment in a case where the reference data has accuracy three times better than the evaluated models, being this is a fundamental verification in the evaluation of the positional accuracy.
Currently, the Brazilian norm for positional accuracy assessment of geospatial data (ET-CQDG) only adopts the absolute method and there are no measures which defines the relative method.
However, as the reference DEM accuracy is at least 3 times better than the other evaluated models, it was possible to prove that the results of the relative evaluation are similar or better statistically, being translated by PEC-PCD results.
The relative assessment gives the advantage of checking the entire pixel population of the models, allowing a global analysis and, thus, identifying a much larger amount of outliers, distortions and displacements, including being possible to delimit regions of outliers, which is less likely with a limited set of control points.

Figure 1 :
Figure 1: Study region located in the west of the State of Bahia, presenting the the SRTM and ASTER models

Figure 2 :
Figure 2: Spatialization of the GCP used in the research.

Figure 3 :
Figure 3: Flowchart of the Absolute Positional Accuracy Assessment.

Figure 5
Figure5shows the flowchart of the evaluation process of the ASTER and DEM, executed by the Python script "DEM Relative Accuracy", which generates, in a similar way to the evaluation of absolute accuracy, an HTML file with the results.

Figure 5 :
Figure 5: Flowchart of the Relative Positional Accuracy Assessment.

Figure 7 :
Figure 7: Histograms of discrepancies for ASTER, SRTM and Reference DEM.

Figure 8 :
Figure 8: Difference between the ASTER models and the Reference DEM.

Figure 9 :
Figure 9: Difference between the SRTM models and the reference DEM.

Figure 10 :
Figure 10: Difference between ASTER and SRTM models

Table 2 :
Mean Error (Erro Médio -EM) and Standard Error (Erro-Padrão -EP) values for the classification of PEC-PCD for DEM on small scales.

Table 3 :
Mean Error (Erro Médio -EM) and Standard Error (Erro-Padrão -EP) values for the classification of PEC-PCD for DEM on large scales.

Table 4 :
Number of control points for evaluation of absolute positional accuracy.

Table 5 :
Results of Absolute Positional Accuracy Assessment.

Table 6 :
Results of the relative positional accuracy assessment.