1. Introduction

Digital Elevation Models (DEM) are important geographic data arrangements for representing altimetry variations of the Earth surface. Although a DEM can be used to model almost any kind of attribute that has spatial distribution on the surface of the Earth, its most common use is in the topographic relief modeling (^{Burrough 1986}).

The applications using the widespread free distribution DEM have gained great importance, especially among academic activities, given the wide range of possible applications, such as mapping hilltops environmental protected areas (^{Oliveira 2015}), monitoring forest biomass (^{Solberg et al. 2010}), hydrological modeling (^{Ludwig and Schneider 2006}), soil losses estimation (^{Fornelos and Neves 2007}), aid in detecting neotectonic reactivations (^{Fonseca and Corrêa 2011}), geomorphologic mapping (^{Camargo et al. 2011}), relief shapes classification (^{Manfré, Nóbrega and Quintanilha 2015}), didactic resource for teaching and learning (^{Reis et al. 2005}).

Considering the use of open source DEM data, the choice between ASTER GDEM v2 and SRTM v3 has been causing some doubts among users since both models present the same one arc second spatial resolution (approximately 30 meters), but they are obtained from different remote sensing sources. While the ASTER GDEM v2 was obtained by optical spectrum satellite images stereoscopy, the SRTM v3 was produced by radar images interferometry method based in microwave C band data. Depending on the DEM obtaining method, any errors that may affect the altimetry accuracy of these models tend to behave in different ways. Both models are available, free of charge, from the United States Geological Survey (USGS). ASTER GDEM v2 since October 2011 and SRTM v3 since September 2014. Detailed information on the ASTER GDEM DEM principles and concepts can be obtained from ^{Lang and Welch (1999}) and the same on SRTM can be seen in ^{Farr et al. (2007}).

Several researches have been developed by scientific community to evaluate the altimetry accuracy of the free distribution DEM, as well as for identification of the possible factors that can influence this altimetry data quality. ^{Gorokhovich and Voustianiouk (2006}) concluded that the altimetry accuracy of the 90 m resolution SRTM C band data are influenced by the gradient and azimuth of the terrain slopes, where the most significant errors occur in areas with hillside slopes greater than 10 degrees. ^{Ludwig and Schneider (2006}) evaluated SRTM data collected in the microwave X band (30 m resolution) for a smooth relief area reporting the smallest errors (RMSE = 9.85 m) and, also, for an area with mountainous relief, where the biggest errors were identified (RMSE = 22.96) caused by radar shadow effects that occurred during the imaging process. ^{Chrysoulakis et al. (2011}) validated ASTER GDEM data for areas with different types of relief and soil cover, concluding that the typical urban areas presented the highest altimetry errors (RMSE 21.21 m). ^{Luana, Houa and Wang (2015}) identified a relation between the errors observed in the SRTM C band model (90 m resolution) and ASTER version 2 considering hillside slope and altitude of the terrain. They found RMSE of 13.74 m and 12.12 m for SRTM and ASTER, respectively.

The consistency of the altimetry errors observed in evaluated DEM for any application is directly related to the objective of the DEM use associated to the mapping scale applied. Barros (2006) and ^{Dias et al. (2011}) indicate a Class A Cartographic Accuracy Standard (PEC), associated to 1: 100,000 mapping scale, for both SRTM C band (90 m resolution) and ASTER GDEM. ^{Oliveira (2015}) found PEC, Class B, associated to mapping scale 1: 100,000 for ASTER and PEC, Class A, associated to mapping scale 1: 200,000 for the SRTM C band (90 m resolution). However, ^{Ferreira (2014}) reported PEC, Class B, associated to scale 1: 50,000, after the application of systematic errors corrections for both ASTER and SRTM products.

In such a context of multiple sources, evaluation uncertainties and little convergence of assessment results reported by literature, this paper proposes to evaluate the altimetry accuracy of the ASTER GDEM v2 and SRTM v3 DEM (30 m resolution), as well as investigating the relationship between observed errors and the terrain slope ranges. The reference data used for validation of evaluated models come from Light Detection and Ranging (LIDAR) technology with, 25 cm altimetry accuracy. The assessment was applied in urban coverage areas and considered, in addition to quantitative methods, the general principles of Earth resources remote sensing. Validating quantitative methods involved histograms analysis, topographic profiles, correlation index, regression analysis, Root Mean Square Error (RMSE) analysis of the distribution of errors and the National Cartographic Accuracy Standard of Digital Cartographic Products - PEC-PCD.

2. Methodological Approach

The methodology is focused on the altimetry evaluation of free DEM originated from remote sensing data (ASTER GDEM v2 and SRTM v3) where the behavior of altimetry errors in different topographic configurations is analyzed. In the present approach, the characterization of the study area is proceeded, the cartographic and computational processes are described and the most important details of the DEM comparison methods are established.

2.1 Study area

The study area corresponds to Belo Horizonte city boundaries and its neighborhoods. It was chosen because the availability of LIDAR data that present optimal altimetry quality to carry out the validation. In addition, the local hillside slopes variation ranges from 0 to 159%, which favors the analysis relating land slope and altimetry accuracy. The region is a densely populated urban area, that presents varying types of land cover, such as: vegetation, buildings, water bodies, streets and uncovered soil. The study region is located at the intersection of four ASTER GDEM v2 scenes and four SRTM v3 scenes, as can be observed in Figure 1.

2.2 Data Processing

The ASTER GDEM version 2 and SRTM version 3 DEM to be evaluated were obtained from the United States Geological Survey (USGS) website. The LIDAR data used as reference were provided by the Computing and Information Company of Belo Horizonte Municipality (PRODABEL). The LIDAR survey was performed by airborne laser scanning, that provides a 25 cm vertical accuracy and an average density of 1.5 points per square meter.

The main processing steps applied to assessment data can be summarized as follow:

Projection and datum conversion: ASTER GDEM v2 (4 scenes) and SRTM v3 (4 scenes) were converted to UTM Projection, 23 zone, South hemisphere and the Geodetic Reference System was set to WGS 84.

Data resampling: for effective comparison of raster data, the pixels of different scenes or sensors should be resampled to the same spatial resolution (30 meters adopted), since the pixel sizes of different scenes or sensors are generally only approximate (not exactly the same size and/or geographic position). All resampling operations used the Nearest Neighbor interpolation method.

Mosaicking and clipping: individual image scenes of ASTER GDEM v2 and SRTM v3 were joined to encompass the full study area. Following mosaics building the data were clipped to cover only the useful size of study area.

In similar way, the reference data was submitted to the processing steps that follow:

Format conversion and interpolation: LIDAR data, originally arranged in point cloud format (.las), was converted to raster format, using the Nearest Neighbor interpolation method to result a 30 m pixel size compatible to evaluation data .

Datum conversion: The Geodetic Reference System was converted from South American Datum-1969 to WGS 84. Original UTM Projection features were maintained.

Given these former basic steps the following processing were applied to both the reference and the assessment data.

Raster to vector conversion: This step resulted in coordinate points data sets (E=X, N=Y, H=Z) corresponding to geographic positions of each center pixel for all DEM. Data were summarized in a unique attribute table containing the three models corresponding altimetry values for each geographic position.

2.3 Relating assessment and reference data in areas with different land slope gradients

In order to analyze the influence of the terrain slope on the altimetry accuracy of the assessment models, a declivity map with 30 m pixel size was extracted from the reference DEM. The slope values were classified according to Brazilian Agricultural Research Corporation ^{(EMBRAPA, 1979}) standards, presented in Table 1.

2.4 Statistical analyzes

The Pearson correlation coefficient (r), given by Equation 1, is a measure of linear association between quantitative variables. The value ranges from 1 (indicating perfect positive correlation) to -1 (indicating perfect negative correlation) (^{Puth et al. 2014}).

Where

The regression analysis proposes to verify the existence of a functional relationship between a dependent variable and an independent variable. In the present analyzed situation, the altimetry values extracted from DEM ASTER GDEM v2 and SRTM v3 are the dependent variables and the altimetry values extracted from the LIDAR data are the independent variable. The functional relationship between these variables can be described by the regression line. According to ^{Reis (2002}) the explanatory power of the regression line can be measured by the coefficient of determination (R2), which shows how much the dependent variable Y is explained by the presence of variable X. The R2 value varies from 0 to 1, in such a way that the greater R2, the greater the adequacy of the function to describe the phenomenon.

In order to evaluate whether the distribution of the reference model data and of the assessment models are statistically the same, a frequency table of the residuals found from the differences between the LASER - ASTER GDEM v2 and LASER - SRTM v3 data was built. The Sturges formulation, given by Equation 2, was applied to find the number of classes to be used.

Tables and graphs generated from the frequency distribution of residuals were used to effective analyze the data set.

The Root Mean Square Error (RMSE), given by Equation 3, was calculated by estimating the error between the observed values (ASTER GDEM v2 or SRTM v3) and the reference values (LIDAR) in relation to the number of points checked (^{Chrysoulakis et al. 2011}). The calculation was performed for each relief declivity class.

Where:

The National Cartographic Accuracy Standard for Digital Mapping Products - PEC-PCD is a statistical indicator adopted among Brazilian mapping community created by the ET-ADGV standard (DSG 2015). To be assigned a given class associated to a mapping scale the cartographic product must meet the conditions that follow: standard error (EP) * 1.6449 must be equal to or less than the tolerance value called PEC provided in the standard for each mapping scale; and EP equal to or lower than that value provided in the standard for the associated scale. The official PEC and EP tolerance values for DEM and the reference mapping scales can be seen in DSG (2015). According to the PEC-PCD digital cartographic products can be classified in classes A, B, C and D in the mapping scales of 1: 1,000, 1: 2,000, 1: 5,000, 1: 10,000, 1: 25,000, 1: 50,000, 1: 100,000 and 1: 250,000 (^{Pereira and Nero 2012}; DSG 2015; ^{Santos et al. 2016}; ^{Santos 2010}).

3. Analysis of results

The assessment DEM were analyzed in order to identify pixels failures related to any lack of elevation data. In ASTER GDEM this task can be accomplished by looking the raster file called QA, which accompanies the DEM data. In the QA file positive values indicate the number of stereoscopic pairs used to calculate the DEM pixel elevation. So, the smaller the number of stereoscopic pairs used, the worse the accuracy of the recorded elevation. The negative values indicate locations where it was not possible to collect enough ASTER GDEM data for performing pixel representation, so the altimetry value is accomplished by interpolation from other elevation sources such as the 3-second C-band SRTM (^{NASA and METI 2011}). Low accuracy occurrences in ASTER GDEM elevations may be related to sites with clouds, shades, water bodies and low contrast soil coverage areas (^{Lang and Welch 1999}). In the present case of study area a concentration of pixels with number of stereoscopic pairs used less than five was observed only on water bodies (Pampulha Lagoon). In areas with stereoscopic pairs less than 5, RMSE values tend to increase (^{NASA and METI 2011}). Nevertheless, in the study area no pixels were identified, whose elevation origin is not from ASTER GDEM.

Pixels presenting extreme errors in SRTM are related to areas that the radar sensor could not capture response because of signal return absence or insufficiency in backscatter direction. These errors may be related to steep slope areas and low backscatter surfaces, such as water surface. In SRTM data processing gaps with less than 16 pixels were filled by interpolation methods. Those larger than 16 remained with gaps of altimetry representation being identified in the DEM by the pixel value -32768. No SRTM gaps were identified in the scenes of the study area.

The statistical tests were performed without considering the elevation data belonging to water bodies. It was observed that the presence of water bodies points influenced in a significant way the correlation and regression for the ASTER GDEM and in a significant way the values of RMSE obtained for the flat relief class were water surface is included. The RMSE of the ASTER GDEM flat relief class calculated taking into account the water bodies points resulted 7.05 m, while without considering water bodies points the RMSE value fell to 5.51 m. In the SRTM case water bodies points did not significantly influenced RMSE value. This problem is related to different water surface interaction with the different sensor systems that operate ASTER GDEM v2, SRTM v3 and LIDAR. Also, different processing methods are adopted in building process of each DEM type.

Figure 2 illustrates the above discussion, where the topographic profiles taken from de same path over the Pampulha Lagoon (starting coordinates: E = 605691.93 m, N = 7805001.28 m and ending coordinates: E = 606597.99 m; N = 7804578.14 m) show the different behavior among the three DEM. It can be seen that ASTER GDEM presented the largest discrepancies.

The LIDAR DEM used as reference in this research represents information about the behavior of the sensor and its interaction with water sheet, since the data collection used infrared laser beams with a wavelength of 1.064 nm (^{Temba 2008}). Analyzing Figure 2 it is possible to observe that the elevations in ASTER GDEM v2 profile show great variations. As pointed out previously, it is due to the low quality elevation measures caused by few stereoscopic pairs used to produce elevations in Pampulha Lagoon. The elevations of SRTM v3 presented a good similarity compared to reference model. However, these elevations are not exclusively based in SRTM original data, in fact this values can result of a masking process for water features applied on SRTM model generation that uses data from satellite images and complementary maps. This discussion supports the decision of excluding water bodies points in the statistical tests and quantitative calculations.

The models histogram analysis presented in Figure 3, shows the variability of elevations for the evaluated and reference data. It is observed that both elevation data do not match a normal distribution. In fact they are positively asymmetrical (positive skewness) and have similar flatness patterns (kurtosis). Presence of outliers can be observed in evaluated models comparing to the reference model. The different acquisition dates of the models data and the systemic errors associated to the acquisition process can contribute to this inconsistency. The basic statistics (maximum, minimum, mean and standard deviation) show close behavior between the three data types.

The Pearson correlation analysis found a correlation coefficient of 0.9986 between ASTER GDEM v2 and the reference model whereas the correlation coefficient found between SRTM v3 and the reference model was 0.9999. These values indicate the existence of a strong positive linear correlation between the evaluated models and the reference model.

This relationship was confirmed by the simple linear regression analysis, demonstrated by means of scatter plots, in which the analysis of the determination coefficients (R2) of the regression line shows that the reference model is adequate to describe the ASTER model in 99.73% and the SRTM model in 99.84% (Figure 4).

A check sample is used to evaluate the positional accuracy of spatial data. The positions of the spatial data are confronted with the positions of the check sample, thus obtaining a set of positional discrepancies. (^{Santos et al. 2016}). So the elevation values (altimetry position) of the ASTER GDEM v2 and SRTM v3 were subtracted from the reference model elevation values (ASTER GDEM v2 - LIDAR) (SRTM v3 - LIDAR). This simple arithmetic operation produced the residuals between the models, in metric units. Table 2 and Figure 5 show the frequencies of the residuals grouped in 19 classes, according to the Sturges formula (Equation 2).

Class | Class Intervals | ASTER GDEM v2 | SRTM v3 | |
---|---|---|---|---|

1 | -64.12 | -56.65 | 0 | 8 |

2 | -56.65 | -49.18 | 6 | 21 |

3 | -49.18 | -41.71 | 40 | 45 |

4 | -41.71 | -34.24 | 144 | 77 |

5 | -34.24 | -26.77 | 394 | 250 |

6 | -26.77 | -19.30 | 1529 | 909 |

7 | -19.30 | -11.83 | 9798 | 5704 |

8 | -11.83 | -4.36 | 64656 | 54917 |

9 | -4.36 | 3.11 | 174822 | 233145 |

10 | 3.11 | 10.58 | 97968 | 66520 |

11 | 10.58 | 18.05 | 1390 | 3684 |

12 | 18.05 | 25.52 | 1724 | 442 |

13 | 25.52 | 33.00 | 329 | 138 |

14 | 33.00 | 40.47 | 89 | 42 |

15 | 40.47 | 47.94 | 22 | 19 |

16 | 47.94 | 55.41 | 10 | 1 |

17 | 55.41 | 62.88 | 3 | 0 |

18 | 62.88 | 70.35 | 1 | 1 |

19 | 70.35 | 77.82 | 0 | 2 |

352925 | 365925 |

Table 2 and the Figure 5 show that class 9 grouped the smallest residuals and the highest frequencies. In the case of ASTER GDEM v2, 174,822 points out of a total 365,925, that is, 47.7% presented residuals between -4.36 m and 3.11 m. In the case of SRTM, 23,345 points out of a total 365,925, that is, 63.7% presented residuals between -4.36 m and 3.11 m. This evaluation highlights a gain of SRTM v3 over ASTER GDEM v2. Other significant percentages are located in class 8 that groups residuals between -11.83 m and -4.36 m and in class 10 that groups residuals between 3.11 m and 10.59 m. Adding the residuals of these two classes the ASTER GDEM grouped 44.44% of the residuals and the SRTM v3 33.18%.

The suitability of the errors observed in an evaluated DEM for an application is closely related to the objective of the DEM use and the mapping scale applied. The Cartographic Accuracy Standard calculations were supported by GeoPec software platform (^{Santos 2010}) which is based in the ET-ADGV standard, whose criteria were previously discussed in section 2.3. The most relevant results of classifications are presented in Table 3.

DEM | PEC-PCD | Scale | Contour interval | PEC | EP |
---|---|---|---|---|---|

ASTERv2 e SRTMv3 | Classe A | 1:100.000 | 50 m | 13,7 | 8,33 |

ASTERv2 | Classe C | 1:50.000 | 20 m | 10 | 6,66 |

SRTMv3 | Classe B | 1:50.000 | 20 m | 12 | 8 |

ASTER GDEM v2 and SRTM v3 models RMSE, taking LIDAR model as reference, calculated for full study area (entire city of Belo Horizonte), corresponding to 365,925 sample points indicated smaller errors for SRTM v3 (RMSE = 4.95 m) and larger errors for ASTER GDEM v2 (RMSE = 6.44 m). So, it can be inferred that for the total study area it is more advantageous to use the DEM SRTM v3 than ASTER GDEM v2.

The RMSE were calculated separately for each relief declivity class for both models ASTER GDEM v2 and SRTM v3. The results indicated an increase of the errors as the slope also increases (Figure 6).

A summary of ASTER GDEM v2 validations results presented by ^{Tachikawa et al. (2011}) reported for South America an RMSE of 8.68 m. In the present evaluation work, the Mountainous relief classes (RMSE = 12.13 m) and Steepest class (RMSE = 18.24) differ some degree from that perspective. The SRTM product description (^{Farr et al. 2007}) reports an absolute accuracy of ± 16 m, 90% confidence level. Steepest relief class evaluated in this work is also out of this perspective.

The SRTM v3 presented better performance for relief representation than ASTER GDEM v2, except for the Steepest relief class (declivity above 75%), where ASTER GDEM v2 presented smaller errors. The slightly larger sensitivity of the SRTM to the steeper slopes may be related to the shadow effect caused by the side looking imaging geometry of radar, as reported by ^{Ludwig and Schneider (2006}). According to ^{Rebelo (2007}) the shadow effect corresponds to areas that the microwave radar beam cannot illuminate the targets, occurring commonly behind (with respect to the sensor) mountains with steep slopes^{. Rebelo (2007}) also highlights the compression and inversion effects that cause erroneous representation of elevation values.

Topographic profiles were generated for the evaluated and reference DEM (Figure 7). The profile was drawn on relief that ranges from mountainous at the initial point (E = 605185.71 m; N = 7784553.89 m) to flat at the final point (E = 602685.71 m; N = 7784553.89 m). The visual analysis of these profiles reinforces the relationship between land slope and altimetry errors. In the flat relief, the profiles of evaluated data fit better to the profile generated from the reference data, whereas in the mountainous relief the disagreements are more evident.

4. Concluding remarks

Statistical methods and quantitative analyses were applied in this work to evaluate the embedded errors in the ASTER GDEM v2 and SRTM v3 DEM using LIDAR DEM as reference. Remote sensing principles and qualitative analyses concepts were also used to identify and understand the factors that influence these errors. So the concepts and results discussed in the present work can be worthy of value for driving studies addressing DEM ASTER GDEM v2 and SRTM v3 applications or its derivatives.

Quantitative tests indicated a high positive correlation between the evaluated and reference models. The regression analysis showed that the reference model is adequate to describe the ASTER GDEM v2 model in 99.73% and the SRTM v3 model in 99.84%.

The analysis of the frequency distribution of the residuals between the evaluated and reference sources indicates that these sources are statistically different from the reference source and therefore it is not possible to replace the reference source with one of the evaluated sources for medium and large surveys scales. The analysis of the errors distribution showed that in the case of ASTER GDEM 47.7% presented residuals between -4.36 m and 3.11 m. In the case of SRTM, 63.7% presented residuals between -4.36 and 3.11 m, indicating a gain of SRTM v3 in relation to ASTER GDEM v2.

The decision if errors are large or small is closely related to the purpose of the DEM use and the associated mapping scale. In the 1: 50,000 scale the evaluation of the PEC-PDC classified the DEM SRTM v3 as class B and the DEM ASTER GDEM v2 as class C. In the 1: 100,000 scale both models were assigned class A.

In both models, the altimetry errors increased as the terrain slope is accentuated. For the representation of flat, smooth corrugated, heavily corrugated and mountainous relief classes SRTM v3 presented smaller altimetry errors, whereas for representation of steepest relief class ASTER GDEM proved to be more advantageous.

The climatic conditions of the study area can be an additional factor that makes difference in obtaining better results. In rainy regions the SRTM may be more advantageous due to the nature of its active sensor system that operates in the microwave spectrum, being in no way affected by clouds and rainy weather. ASTER GDEM is based in passive sensor system that operates in the range of optical spectrum, and so is affected by clouds and rainy weather.

Both systems ASTER GDEM and SRTM presented some problems for mapping surfaces of low roughness such as water bodies.

Finally, it is also important to highlight some aspects that may affect the results of the evaluation proposed in this work. One is that the vertical datum used in ASTER GDEM v2 and SRTM v3 was EGM96, whereas in LIDAR it was used Imbituba-SC. The other is that difference of acquisition dates of evaluated and reference models may contribute to introduce unknown errors in the process. The dates of acquisition of the SRTM and LIDAR data were, respectively, February 2000 and November 2007. The ASTER GDEM was generated from multiple satellite scenes of different dates that are not published by system managers.