Elimination of some unknown parameters and its effect on outlier detection

Eliminação de alguns parâmetrtos desconhecidos e seu efeito na detecção de erros grosseiros

ARTIGOS

Elimination of some unknown parameters and its effect on outlier detection

Eliminação de alguns parâmetrtos desconhecidos e seu efeito na detecção de erros grosseiros

Serif Hekimoglu; Bahattin Erdogan; Nursu Tunalioglu

Yildiz Technical University Faculty of Civil Engineering Department of Geomatic Engineering Davutpasa Campus, 34220 - Esenler- Istanbul, Turkey [e-mail]; [e-mail]; [e-mail]

ABSTRACT

Outliers in observation set badly affect all the estimated unknown parameters and residuals, that is because outlier detection has a great importance for reliable estimation results. Tests for outliers (e.g. Baarda's and Pope's tests) are frequently used to detect outliers in geodetic applications. In order to reduce the computational time, sometimes elimination of some unknown parameters, which are not of interest, is performed. In this case, although the estimated unknown parameters and residuals do not change, the cofactor matrix of the residuals and the redundancies of the observations change. In this study, the effects of the elimination of the unknown parameters on tests for outliers have been investigated. We have proved that the redundancies in initial functional model (IFM) are smaller than the ones in reduced functional model (RFM) where elimination is performed. To show this situation, a horizontal control network was simulated and then many experiences were performed. According to simulation results, tests for outlier in IFM are more reliable than the ones in RFM.

Keywords: Elimination; Tests for Outliers; Reliability; Adjustment

RESUMO

Erros grosseiros nas observações afetam de forma negativa todos os parâmetros estimados e os resíduos, razão pela qual a detecção de erros grosseiros tem grande importância na estimativa confiável dos resultados. Diversos testes (por exemplo testes de Baarda e Pope) são utilizados frequentemente na detecção de erros grosseiros em aplicações geodésicas. Muitas vezes, visando reduzir o tempo computacional, é realizada a eliminação de alguns parâmetros que não são de interesse. Nesse caso, embora a estimativa dos parâmetros e resíduos não sofra modificações, a matriz cofatora dos resíduos e a redundância das observações mudam. Nesse estudo, são realizados testes para erros grosseiros e investigados os efeitos da eliminação dos parâmetros desconhecidos. Foi provado que quando a eliminação é realizada, as redundâncias no modelo funcional inicial (IFM - Initial Functional Model) são menores que no modelo funcional reduzido (RFM - Reduced Functional Model). Para ilustrar essa situação, uma rede de controle horizontal foi simulada e muitos experimentos foram realizados. De acordo com os resultados da simulação, testes para erros grosseiros com IFM são mais confiáveis que com RFM.

Palavras-chave: Eliminação; Testes para Erros Grosseiros; Confiabilidade; Ajustamento

1. INTRODUCTION

Outlier detection has a great importance in geodetic networks. The efficacy of the unknown parameters and their standard deviations depends on whether the observation set includes outliers or not. Sometimes, observations may contain one or more outliers. In this case, these outliers must be detected and removed or re-measured. Tests for outliers are mostly used for the outlier detection (BAARDA, 1968; POPE, 1976; KOCH, 1999). The efficacies of the tests for outliers change depending on the number of outliers and the magnitudes of outliers (HEKIMOGLU and KOCH, 2000). Tests for outliers can detect only one outlier reliably (HEKIMOGLU, 1997; BASELGA, 2007; HEKIMOGLU et al., 2011). If the observations include more than one outlier, the tests for outliers cannot detect them reliably due to masking effect or swamping effect, especially when the magnitudes of multiple outliers are small (HEKIMOGLU, 2000 and 2005).

Until recently, capacity of computers was bounded. Therefore, the number of unknown parameters in geodetic networks should be small so that some unknowns that are not related directly to the coordinates can be eliminated in adjustment model. The normal equations may then be inverted by using computer. Herewith, large geodetic networks are divided into sub-networks in Helmert-Block method and the coordinates of the points (except connection points) are eliminated (WOLF, 1968; HÖPCKE, 1980). Furthermore, elimination of the unknown parameters as bundle block adjustment is preferred in the photogrammetry (MIKHAIL and ACKERMANN, 1976; ALBERTZ and KREILING, 1989; KRAUS, 1997). In the process of GPS observations, we may want to eliminate the ambiguity unknowns (KING et al., 1987; STRANG and BORRE, 1997; GHILANI and WOLF, 2006); also in the adjustment model, constraints sometimes may be eliminated to reduce the number of parameters (MIKHAIL and ACKERMANN, 1976).

To apply outlier detection, the observations of the geodetic networks are adjusted as a free network. First, the unknown parameters, which are not related to the coordinates of the points, are eliminated to interpret the unconstrained adjustment model geometrically (NIEMEIER, 2002). A similar problem to eliminate the orientation parameters is commonly occurred in triangulation networks. Some unknown parameters that are not of interest are generally eliminated. In this context, two different functional models come up: (1) the initial functional model (IFM) and (2) the reduced functional model (RFM), where elimination is performed.

Although the estimated unknown parameters and residuals in RFM are the same as the ones in IFM, the cofactor matrix of residuals and redundancies r_i of the observations are different. In this study, firstly, we figure out this situation. Secondly, we investigate whether the differences among the redundancies may affect the outlier detection or not. Also, the following question is investigated: must the outlier detection be applied to RFM or IFM where a group of unknowns is not eliminated? To compare the reliabilities of tests for outliers in RFM and IFM, mean success rate (MSR) is used. The MSR was introduced to measure the efficacy of the tests for outliers (HEKIMOGLU and KOCH, 2000). The MSR is globally the number of successful detections over the number of experiments. The MSR also means the estimated power of the test (AYDIN, 2011).

2. ELIMINATION BY PARTITIONING BLOCKS

A general approach for eliminating some (or a group) unknown parameters is to split the functional model into blocks. The linearized IFM is given as follows:

where l is the observation vector, v is the residual vector, A is the coefficient (design) matrix of the unknown parameters, x is the unknown vector, is the variance of unit weight, P is the diagonal weight matrix, is the cofactor matrix of the observations and C₁₁ is the variance covariance matrix of the observations. Then the design matrix and the unknown vector given in Eq. (1) are divided into two sub-matrices and sub-vectors:

where the sizes of A₁, A₂, x₁ and x₂ are n x p, n x q, p x 1 and q x 1, respectively. X₁ contains the main unknown parameters, x₂ also contains the unknown parameters, which are eliminated. N is the number of observations, p is the number of main unknown parameters and q is the number of eliminated unknown parameters. In this case, Eq. (1) can be written as follows:

The stochastic model of Eq. (3) is the same as given in Eq. (1); it does not change (NIEMEIER, 2002). In principle, the normal equations and the right side of the equations can be expressed as in terms of divided blocks:

Here, if N₂₂ is invertible, in the Eq. (4) x₂ can be written as a function of x₁:

If Eq. (7) is put in the first equation of the Eq. (4),

Eq. (8) can be written shortly,

Eq. (10) is obtained for the inverse of the ,

In this way, the solution of the vector x₁ for RFM can be obtained from Eq. (9):

As calculating the block matrices in IFM, the sub-matrices of the cofactor matrix Q are determined. To obtain the sub-matrices, the following equations are usually used:

where E denotes the identity matrix. If the sub-matrix N₂₂ is regular, i.e. it is invertible; the following equations can be obtained as (Faddejew and Faddejewa, 1976):

Also, if Eq. (14) is considered in Eq. (12):

The above equation can be obtained, by taking advantage of the symmetry property, the following equation can be written as follows:

If Eq. (10) and Eq. (15) are compared to each other, it can be seen that the cofactor matrix of the x₁ is equal to the cofactor matrix Q₁₁ in IFM, i.e. Q₁₁= . Elimination does not change the cofactor matrix.

To determine the unknown vector x₂ in RFM, x₁ is obtained from Eq. (11) and put into Eq. (7). It is important to determine the residuals in RFM. If x₂ is obtained from Eq. (7) and put into Eq. (3), the residuals can be computed as follows:

If the terms of this equation are shortened as follows:

Eq. (18) can be written differently:

There will not be any change for the estimation value of the variance .

The equations in this section can be adapted to free network adjustment (HECK, 1975).

3. COMPARING THE REDUNDANCIES OF THE OBSERVATIONS IN RFM WITH THE ONES IN IFM

The H and R matrices in IFM are given as follows:

where H denotes the hat matrix in statistics and Q_xx matrix can be written from Eq. (4) as and the hat matrix can be written from Eq. (23) as below.

If Eq. (13) and Eq. (14) are considered, the following Eq. can be obtained:

Since the symmetry property, , Eq. (27) can be rewritten as below:

If Eq. (16) is considered in Eq. (28), Eq. (29) can be obtained:

Since;

H can be rewritten as follows:

For RFM, the following two equations from the Eq. (18) can be written:

Since , Eq. (31) can be written as below:

If Eq. (30) and (35) are taken into account, Eq. (36) can be obtained:

Since is a quadratic form, it can be expressed as 0. Therefore is always bigger than and also , since and .

The equations in this section can be adapted to free network adjustment (HECK, 1975).

4. TESTS FOR OUTLIERS

Outlier detection procedures were proposed by Baarda (1968) and Pope (1976) for geodesy. In these outlier detection processes, "good" observations originate from the same distribution, which is generally expressed as a normal distribution . Observations that contain outliers are called as "bad" observations.

Let an observation has an outlier with The hypothesis against is tested. If the observations are uncorrelated and the variance is known, the standardized residuals derived from IFM can be presented as:

where is the diagonal elements of Q_vv

If , which is the upper percentage point of the normal distribution, the observation is accepted as a bad observation where α is chosen as 0.001. This is called as Baarda's method. If there is more than one outlier among the observations, Baarda's method is used iteratively (BAARDA, 1968).

If the variance is not known before, the studentized residual is used for Pope's test:

where is given in Eq. (22). If the level of significance α is related to all observations, the level of each observation must be where .

If the following relations are taken into account,

the below relation can be written as:

If the Eqs. (37a) and (37b) are rewritten by considering Eq. (41), the following equations can be obtained

If rank A=q holds in the Gauss-Markov model, then , where α is generally chosen as 0.05 or 0.01 (Koch, 1999).

For RFM, we can generate the following equations similar to Eqs. (39) and (40).

If above two equations are considered, the following equation can be written as

and similar to Eqs. (42a) and (42b), the followings can be obtained:

Since and in IFM are bigger than and in RFM, respectively. It means that the effects of the outlier with small magnitude in IFM may be reflected stronger than the ones in RFM on the standardized or studentized residuals. Therefore, the MSRs of the tests for outliers in IFM become bigger than the MSRs of the ones in RFM.

5. CASE STUDIES

5.1 Elimination of orientation parameters in triangulation network

The method dating back to C. F. Gauss uses the elimination of the orientation parameters at triangulation networks. If we consider only one unknown parameter, which has the same coefficient in the residual equations, it can be regarded as a special case of the elimination. For elimination of the only one unknown x₂ (q=1 in Eq. (2)), the related design matrix can be written:

where the size of the A₂ is nx1 and k is the number of direction observations at one station. If P = I, the reduced approach according to Eq. (19) is given as follows:

where means column sum of the A₁ (it must be multiply by 1/k) (Niemeier, 2002). At the same time, it should be eliminated from the initial residual equations. is obtained similarly,

where column sum of is similarly divided by k and reduced from l_i. Mean residual equation that is eliminated from each residual equation is computed as follows:

In practice, Eq. (52) is computed for each station by which direction observations are made at the network. The linearized residual equations for a station, in which k direction measurements are made can be written with the orientation parameter o:

Reduced residual equations are obtained as follows:

where

5.2 Simulation

In this study, a Monte-Carlo simulation technique has been used to demonstrate as above described case. To measure the reliability of tests for outliers in IFM and RFM, a horizontal control network was simulated. Fig. 1 presents the positions of the points and observations for the horizontal control. The simulated horizontal control network given in Figure 1 consists of 7 points where n = 48, u = 21 and degrees of freedom f = 30. The MSRs for IFM and RFM are presented for the same network. All results have been obtained using MATLAB version R2006a.

The random errors e_i are generated from a normal distribution with expected value and variance by using the random number generator of MATLAB. The process of simulating the good observations l_i and bad observations are obtained as the same as Hekimoglu and Erenoglu (2007), Erenoglu and Hekimoglu (2010), Hekimoglu et al. (2011).

For the horizontal control network, the observations, such as direction measurements and distance measurements are computed from the coordinates of the points. They are free of random errors. The random errors are generated from a normal distribution as follows: for direction measurements with and for the distance measurements with where S_ij is the distance between i^th and j^th points. The random errors are added to the distance and direction measurements such as and . Hereby, the good observations are obtained. Then, the random error e_i is replaced by the outlier in the related observation as , thus the contaminated observations are obtained. The approximate values of the point coordinates are given in Table 1. The direction measurements and distance measurements are also given Table 2 and Table 3, respectively.

To test the reliabilities of the IFM and RFM for tests for outliers, 6^th observation (i.e. direction from 2 to 7) is contaminated by the outlier. The magnitude of the outlier is -1.255 mgon. The 6^th observation given in Table 2 includes this outlier. After adding the outlier to the observation, Baarda's method (i.e. assume that the a priori variance is known) and Pope's method (i.e. assume that the a priori variance is unknown before) are applied; and obtained redundancies, standardized and studentized residuals are represented for IFM and RFM in Table 4. In this study, is chosen 0.001 for Baarda's test and 0.05 for Pope's test, respectively. As it is seen from Table 4, the standardized and studentized residuals of the IFM are bigger than the ones of RFM. Although the outlier (6^th observation) can be detected in IFM, it cannot be detected in RFM. But, one sample is not enough to decide that the results of the IFM are more reliable than the RFM, that's why, a hundred different contaminated samples of are simulated for each of the data sets l. For a hundred different data sets l, totally 10000 different contaminated samples of , are obtained separately. Baarda's test and Pope's test are applied on these contaminated samples.

To measure the reliabilities of the tests for outliers, the MSR criterion has been handled. A test for outlier is regarded as successful when the test statistics can separate the null hypothesis H₀ from the alternative hypothesis H₁ at the significance level α. The mean success rate (MSR) is defined with dividing the number of success by the number of experiments. If a good sample is contaminated by replacing any number of the observations with arbitrary values, then a contaminated sample obtained. Many good samples can be obtained by generating the different subsets of random errors. Thus, for each good sample, many contaminated samples are generated by replacing any number of good observations with arbitrary values.

Since a simulation is used to generate the outliers, it is possible to know exactly whether an observation is contaminated or not, in advance of carrying out the analysis. After applying the outlier detection method, if the observation is identified as an outlier and it corresponds to truly contaminated observation, the method is regarded as successful. If the method fails, it is considered unsuccessful (HEKIMOGLU and ERENOGLU 2007, HEKIMOGLU and KOCH 2000, ERENOGLU and HEKIMOGLU 2010, HEKIMOGLU et al. 2011). Owing to using simulation techniques, a lot of samples can be generated easily. The MSR is globally the number of successful detections over the number of experiments.

The magnitudes of the small outliers (whose magnitudes lie between and ), and large outliers (whose magnitudes lie between and ), are generated separately. Also, both tests for outliers are iteratively applied. Only the observation with the largest normalized or studentized residual is tested and in case it is rejected, it is removed and the remaining observations are then adjusted again. But, in this case, a geometric defect of the network may occur. To prevent such a geometric defect, the detected observation is not removed; instead of this, the related weight of the observation is set smaller for the next iteration step, for example p_i=0.001 x p_i. In this case, the initial approximation of the orientation is estimated by using the weighted arithmetic mean.

The orientation parameters are eliminated in RFM; whereas they are estimated in IFM. Also, the observations at the network given in Fig. 1 were adjusted as free network. Tables 5 and 6 include the MSRs of both Baarda's and Pope's tests for IFM and RFM. The MSRs are increased by 11.5% (55.2% - 43.7%) and 19.0% (45.1% - 26.1%) for the Baarda's and Pope's tests, respectively, for one small outlier. Also, the reliability of IFM for two outliers is bigger than the ones of RFM. However, the MSRs of IFM are bigger than the RFM when there is no outlier in observation set. This is the type I error. The increase in type I error for IFM is 4% (5% - 1%) for Baarda's test and 2% (2% - 0%) for Pope's test. The advantage of IFM is 7.5% (11.5% - 4%) for Baarda's test and 17% (19% - 2%) for Pope's test. However, for one large outlier the MSRs for both tests are not increased significantly.

6. CONCLUSION

Elimination of the unknown parameters in adjustment model is sometimes preferred to shorten the calculation time. Although the estimated unknown parameters, residuals, cofactor matrix of the unknown parameters in IFM are the same as the ones in RFM, the cofactor matrix of the residuals and redundancies of the observations are different. The redundancies in IFM are smaller than the ones in RFM; this situation is proved in this study. Since the diagonal elements of the cofactor matrix of the residuals in RFM is bigger than the ones in IFM, the standardized residuals or studentized residuals in IFM are bigger than RFM. Therefore, the effects of the outliers do not appear strongly on the residuals for some cases of RFM where the magnitude of outlier is small, and the outliers cannot be detected.

In this study, two models are regarded for simulation. The orientation parameters are considered as unknowns in IFM and they are eliminated in RFM. To compare the reliabilities of tests for outliers in RFM and IFM, tests for outliers are applied on simulated horizontal control network. The reliability is measured by MSR. According to simulation results, the reliabilities of the tests for outliers in IFM are bigger than the ones in RFM for small and large outliers in condition that the observation set contains one and two outliers. For this reason, to apply tests for outliers on IFM should be preferred.

Recebido em março de 2012

Aceito em junho de 2012

• ALBERTZ, J.; KREILING, W. Photogrammetric Handbook, Herbert Wichmann, Karlsruhe,1989.
• AYDIN, C. On the Power of Global Test in Deformation Analysis. J. Surv. Eng. doi:10.1061/(ASCE)SU.1943-5428.0000064, 2011.
• BAARDA, W. A testing procedure for use in geodetic Networks. Publication on Geodesy, New Series 2, no.5, Netherlands Geodetic Commission, Delft, 1968.
• BASELGA, S. Critical limitation in use of τ test for gross error detection. J. Surv. Eng. 133(2): 52 - 55, 2007
• ERENOGLU, R. C.; HEKIMOGLU, S. Efficiency of robust methods and tests for outliers for geodetic adjustment models. Acta Geod. Geoph. Hung. 45(4):426-439, 2010.
• FADDEJEW, D. K.; FADDEJEWA, W. N. Numerische Methoden der linearen Algebra, 4. Aufl., München-Wien. R. Oldenbourg-Verlag, 1976.
• GHILANI, C.D.; WOLF, P. R. Adjustment Computations: Spatial Data Analysis, 4^th edition, John Wiley & Sons Inc., 2006.
• HECK, B. Die Genauigkeit eliminierter Unbekannter bei regularen und singularen Ausgleichungsproblemen. Allg. Vermess. Nachr, 82,345-348, 1975.
• HEKIMOGLU, S. The finite sample breakdown points of the conventional iterative outlier detection procedures. J. Surv. Eng.,123(1), 15-31, 1997.
• HEKIMOGLU, S.; KOCH, K. R. How can reliability of tests for outliers be measured? Allg. Vermess. Nachr 107/7: 247 - 254, 2000.
• HEKIMOGLU, S. Increasing Reliability of the Test for Outliers whose Magnitude is Small. Surv. Rev, 38(298), 274-285, 2005.
• HEKIMOGLU, S. Do robust methods identify outliers more reliably than conventional tests for outliers? Zeitschrift fuer Vermessungswesen, 130. Jahrgang, Heft 3, 2005.
• HEKIMOGLU, S.; ERENOGLU, R. C. Effect of heteroscedasticity and heterogeneousness on outlier detection for geodetic Networks. J. Geod, 81(2):137-148, 2007.
• HEKIMOGLU, S.; ERDOGAN, B.; ERENOGLU, R. C.; HOSBAS, R. G. Increasing the Efficacy of the Tests for Outliers for Geodetic Networks. Acta Geod. Geoph. Hung., 46(3), 291-308, 2011.
• HÖPCKE, W. Fehlerlehre und Ausgleichsprobleme. Walter de Gruyter, Berlin, 1980.
• KING, R. W.; MASTERS, E. G.; RIZOS, C.; STOLZ, A.; COLLINS, J. Surveying with Global Positioning System GPS. Ferd. Dümmler Verlag, Bonn, 1987.
• KOCH, K. R. Parameter estimation and hypothesis testing in linear models. 2nd Ed. Springer-Verlag, Berlin, 1999.
• KRAUS, K. Photogrammetry, Volume 2, 4th ed., Dümmler, Bonn, 1997.
• MIKHAIL, E. M.; ACKERMANN, F. E. Observations and Least-Squares. IEP SeriesCivil Engineering, New York, 1976.
• NIEMEIER, W. Ausgleichungsrechnung, Walter de Gruyter, Berlin, 2002.
• POPE, A J. The statistics of residuals and the outlier detection of outliers. NOAA Technical Report, NOS 65, NGS 1, Rockville, MD, 1976.
• STRANG, G.; BORRE, K. Linear algebra, geodesy, and GPS. Wellesley-Cambridge Press, 1997.
• WOLF, H. Ausgleichungsrechnung nach der Methode der kleinsten Quadrate. Dümmler, Bonn, 1968.

Publication Dates

• Publication in this collection
  09 Oct 2012
• Date of issue
  Sept 2012

History

• Received
  Mar 2012
• Accepted
  June 2012