PERFORMANCE EVALUATION OF THERMOGRAPHIC CAMERAS FOR PHOTOGRAMMETRIC DOCUMENTATION OF HISTORICAL BUILDINGS

Thermographic cameras record temperatures emitted by objects in the infrared region. These thermal images can be used for texture analysis and deformation caused by moisture and isolation problems. For accurate geometric survey of the deformations, the geometric calibration and performance evaluation of the thermographic camera should be conducted properly. In this study, an approach is proposed for the geometric calibration of the thermal cameras for the geometric survey of deformation caused by moisture. A 3D test object was designed and used for the geometric calibration and performance evaluation. The geometric calibration parameters, including focal length, position of principal point, and radial and tangential distortions, were determined for both the thermographic and the digital camera. The digital image rectification performance of the thermographic camera was tested for photogrammetric documentation of deformation caused by moisture. The obtained results from the thermographic camera were compared with the results from digital camera based on the experimental investigation performed on a study area.


INTRODUCTION
Digital cameras have become extremely popular due to their high resolution and automation possibilities in digital photogrammetric documentation.Digital cameras and thermal imaging systems are widely used in modern medical, automotive, industrial, architectural, and military applications (MCCAFFERTY, 2007).The uses of these thermal imaging systems and cameras have helped to increase the documentation of historical and cultural heritage, 3D modeling, deformation analysis, restoration and conservation applications (CLARK et al., 2003;GRINZATO et al., 1999;GRINZATO, 2012;MEOLA et al., 2005;LERMA et al., 2007).
Thermal imaging systems are sensor systems that record temperatures emitted by objects in the infrared region, particularly in the long infrared region (7-20 µm).Infrared (IR) thermography transforms thermal energy emitted by an object in the infrared band of the electromagnetic spectrum into a visible image.These sensor systems enable photogrammetric documentation and deformation analyses caused by moisture and isolation problems in wet areas (REZZI et al., 2007).IR thermography has been extensively used in cultural heritage investigations, in particular, for art work studies (PELAGOTTI et al., 2007;GHEDINI et al., 2003;AMBROSINI et al., 2010), monitoring (GRINZATO et al., 2002), diagnosis (PAOLETTI et al., 2013;IMPOSA, 2010;MOROPOULOU et al., 2013) and conversation (MOROPOULO et al., 2001;BOSILJKOV et al., 2010., AVDELIDIS andMOROPOULOU, 2004) of historical buildings.
Thermal imaging systems have been used mostly for naked eye analyses in conservation and restoration work of the historical buildings.The geometric information from the thermal images obtained with photogrammetric restitutions can be used for documentation of hidden structures, cracks pattern, structural failures, moisture and humidity problems in cultural heritage studies.For accurate photogrammetric documentation, the geometric calibration of the thermal cameras is vital importance since these cameras thermally calibrated and verified in order to guarantee an accurate temperature measurement.The geometric calibration of thermographic cameras is scarcely found in the literature (LUHMANN et al., 2010;LAGÜELA et al., 2011;BISON et al., 2012) in comparison with digital cameras which is very common (CHEN and SCHENK 1992;FRASER 1997;CRAMER, 2003aCRAMER, , 2003b;;LUCCHESE, 2005;LICHTI et al., 2010).Lagüela et al., (2011) and Bison et al., (2012) used the plane calibration fields for geometric calibration of thermographic camera whereas Luhmann et al., (2010) used the three dimensional (3D) calibration field.The plane calibration field is not ideal for precise determination of the thermographic cameras geometric calibration parameters (FRASER 1997).The selection of the target material in calibration field is also important in order to obtain clear targets in the IR spectrum.Rezzi et al., (2007) conducted tests on the emissivity of different materials to obtain clear targets in the IR spectrum.For performance evaluation of the thermographic camera, the comparison of the reference distances in standard artifact (LAGÜELA et al., 2011) and the comparison of the area values of surfaces from 3D models (BISON et al., 2012) were used.The performance evaluation of thermographic cameras for photogrammetric documentation of deformations based on the experimental investigation performed on a study area is not available in the literature.The primary objective of this paper is to evaluate the performance of thermographic cameras for photogrammetric documentation of deformation caused by moisture and isolation problems in historical buildings to better analyze existing situations.An overview of the theoretical background and methodologies related with the proposed geometric calibration and photogrammetric documentation approach is presented in the following section.The experimental section provides detailed information about the proposed approach for the geometric calibration and performance evaluation of thermographic cameras.The results of the experimental studies are discussed, and the performance of the thermographic camera in documenting deformation caused by moisture and isolation problems for architectural heritage restoration is assessed.

METHODOLOGY
Photogrammetry identifies an object's shape, size and location using photographs or records in the electromagnetic spectrum recorded with a detection system.For this purpose, measurements using photographs or images have to be performed, and orientation parameters must be determined with known camera geometry (BROWN, 1971;FRASER 1997;CRAMER, 2003aCRAMER, , 2003b;;HONKAVAARA et al., 2006;LICHTI et al., 2010;LUCCHESE, 2005).High-resolution digital Single Lens Reflex (SLR) cameras have the great potential for automation and precise photogrammetric measurements with a better image quality.Digital image rectification, monoscopic multi-image evaluation, and stereo digital photogrammetry are the methods used in digital photogrammetric software systems for photogrammetric documentation.The details of these methods for photogrammetric documentation of architectural heritage are given by YASTIKLI, 2007.In classic photogrammetry, camera geometry is expressed with the determined interior orientation parameters.The determination of the camera lens distortion is important for accuracy of the determined 3D coordinates.To obtain 3D information from images, at least two images must be recorded with known camera geometry (interior orientation parameters) from different positions in addition to signalized control points (REMONDINO and FRASER, 2006).These parameters are generally determined with geometric calibration in close-range photogrammetry with additional parameters to the bundle block adjustment.
Interior orientation parameters include the focal length of the photogrammetric cameras (c), the principal point position (x 0 , y 0 ), and the distortion model parameters.The principal point position in traditional photogrammetric cameras is defined in a photograph coordinate system with the help of fiducial marks.The lens systems used in cameras are composed of a combination of lenses rather than a single lens.The projection of a point P from the 3D object coordinate system to a 2D photograph system deviates from the assumed collinearity condition (collinearity of the camera's perspective center, the object point, and the corresponding image point).The difference is called lens distortion and has two components: radial and tangential distortions.
As mentioned earlier, geometric calibration is generally performed with the aforementioned additional parameters to the bundle adjustment using multiple photographs of the test object taken from different positions.The mathematics of the bundle adjustments is based on the collinearity equation: Where, x, y : photograph coordinates x 0 , y 0 : principal point photograph coordinates c : focal length X 0 , Y 0 , Z 0 : projection center coordinates in the reference coordinate system X, Y, Z : object coordinate in the reference coordinate system m ij = rotation matrix elements (M) between the photograph coordinate system and reference coordinate system Δx ve Δy = additional parameters.
Collinearity equations can be extended to include additional parameters for systematic errors.The most common physical model method was developed by Brown in 1971 to eliminate systematic errors.The additional parameters, Δx and Δy, are defined by the following expression (MCGLONE, 2004) :   2   2  2   1  3   6  2   4  1   2  0   2  1   2  2   3   6  2   4  1   2 Where, x , y is the photograph coordinates reduced to as the principal point r : radial distance K i : radial distortion coefficients P 1 , P 2 : tangential distortion coefficients.
The photograph coordinates referred to as the principal point and the radial distortion ∆r are defined by the following expressions: The additional parameters into the bundle block adjustment are estimated with the measured coordinate of the tie point and control points from all the photographs.The coordinates of the control points should be precisely known, where control points should be distributed at different depths and different locations.In general, a 3D test object is used for the geometric calibration process.The 3D test object is important for accurate estimation of the interior orientation parameters.
The distortion function (Equation 4) is dominated by the K 1 r 3 and the K 1 term usually suffices in medium accuracy application for the commonly encountered third-order barrel distortion seen in consumer-grade lenses (BROWN, 1971;FRASER 1997).The inclusion of the K 2 and K 3 terms is required for higher accuracy application and wide angle lenses.It is well-known that some geometric calibration parameters such as radial distortion coefficients (K 1 , K 2 and K 3 ) are correlated as well as the correlation between the interior and the exterior orientation parameters.
Digital image rectification is the preferred method in a photogrammetric documentation project if the architectural heritage façade can be considered as a plane.In general, the collinearity equation (Equation 1) and the projective transformation are used as mathematical model for digital image rectification (KRAUS, 1993;NOVAK, 1992).The collinearity equation is preferred when the interior orientation parameters of the camera are known for transformation from image plane to object plane.The minimum three control points are necessary to compute the exterior orientation parameters.The projective transformation describes the relationship between object plane and photograph plane defined by following expressions: 1 , are the projective parameters.
The image coordinates (pixel coordinates) can be used directly in projective transformation instead of photograph coordinates.In architectural photogrammetry applications, the projective transformation is commonly used method since the interior orientation parameters are not necessary.For accurate results, the lens distortions of the camera should be corrected before projective transformation.The minimum four control points are required for computation of eight projective parameters.
After geometric transformation with the collinearity equation or projective transformation, the tessellation of the transformed image does not coincide with the original image.The assignment of a proper gray value to the transformed image pixel is needed (SCHENK, 1999).In nearest neighbor resampling, the nearest gray value of the neighboring pixels is assigned as a gray value.This method is working very fast in computer but is insufficient because of the gaps and duplication.The alternative approach is based on the interpolation to assignment of the gray levels in transformed image.The bilinear resampling and bicubic convolution are the commonly used interpolation methods in digital image rectification (ATKINSON, 2001).
The geometric accuracy in photogrammetric documentation for deformation analyses and monitoring studies of historical buildings using thermographic cameras is primarily related to the geometric performance of the camera.In this study, the performance evaluation of the thermographic camera is aimed toward the possible use for photogrammetric documentation and deformation analyses caused by moisture and isolation problems in wet areas, which is a record of the historical and cultural heritage.

EXPERIMENT
The experimental section focuses on the geometric calibration of the thermographic and digital cameras and their performance evaluation for photogrammetric documentation.The details of the experiment are given in two subsections: geometric calibration and performance evaluation for photogrammetric documentation.

1. Geometric Calibration
The geometric calibration is vital in obtaining geometrical information through photogrammetric restitution.To determine the geometric calibration parameters at a high accuracy, a 3D test object is required that has control points at different depths.In this study, a 3D test object with 77 control points distributed at different depths was designed to perform precise geometric calibration of the thermographic and digital cameras, as shown in Figure 1.The locations of the control points were designed such that each control point does not overlap with others when the images are recorded from different positions and angles.In other words, the primary objective was to measure the maximum number of control points on the images recorded from different positions.The 3D design and modeling was performed using AutoCAD software.As observed in Figure 1, the distance between the three planes (red, green and blue) was approximately 70 cm, which allowed the control points to be distributed at different depths.The designed 3D test object was manufactured from iron, and targets made from plastic placed on the 77 control points.The target material and design are important since these targets should be clearly detectable in both the digital camera and thermographic camera.We preferred iron for manufacturing the 3D test object and plastic for manufacturing the targets to guarantee different spectral behavior in the IR region.In addition the different material, our target is including black and white regions to create different reflectivity and emissivity for better identification in the infrared images.The coordinates of the control points (center of the targets) were measured with a reflectorless total station with high accuracy.
For the performance evaluation, a Flir A320 thermographic camera with a spectral range of 7.5-13 μm was used.A Nikon D3X SLR digital camera was used as reference for comparison (Figure 2).The technical specifications of the thermographic and digital cameras used in this study are listed in Table 1.The digital images of the 3D test object were recorded with the Flir A320 thermographic camera and the Nikon D3X SLR digital camera from different positions and orientation angles.To define the photograph coordinate system, the location of the principal point was set as the center of the images for both the digital and thermographic camera as the initial value.The photograph coordinates of the control points were identified without any problem and measured manually on the digital and thermographic images (Figure 3).The PHIDIAS software was used for the image coordinate measurements and the bundle adjustment, which was developed by PHOCAD.The bundle block adjustment was performed first at a given focal length (see Table 1), and the principal point location (center of images) and then blunders were eliminated using the 77 control points.The additional parameters, which consist of the position of principal point and focal length (x 0 , y 0 , c), radial distortion (K 1 , K 2 , K 3 ) and tangential distortion (P 1 , P 2 ) parameters were included in the bundle block adjustment in the second stage.The statistical significance of introduced geometric calibration parameters were checked using the PHIDIAS software.As results of this check, the introduced geometric calibration parameters were significant statistically for the Flir A320 thermographic camera and the Nikon D3X SLR camera.The determined geometric calibration parameters with the bundle block adjustment for the Flir A320 thermographic camera and the Nikon D3X SLR digital camera are given in Table 2 and Table 3, respectively.
Based on the technical specifications, the focal lengths were 20 mm and 18 mm for the Nikon D3X SLR camera and the Flir A320 thermographic camera respectively (see Table 1).The principal point location was assumed 0 , 0 mm for x and y coordinates for both digital and thermographic cameras before geometric calibration.The computed corrections and standard deviations for the focal length, principal point location were larger for the Flir A320 thermographic camera (Table 2) compared with that of the Nikon D3X SLR digital camera (Table 3).For example, the computed correction for the focal length is -0.2418 mm with 0.0293 mm standard deviation for Nikon D3X SLR camera and -0.7738 mm with 0.1288 mm standard deviation for the Flir A320 thermographic camera.The computed radial and tangential distortion coefficients and their standard deviations are also larger for the Flir A320 thermographic camera.When comparing the geometric calibration results of the thermographic and digital camera, the pixel size should (a) (b) also be taken into account as a limiting factor for the thermographic camera because the pixel size was 25 µm for the thermographic camera and 6 µm for the digital camera.
For better evaluation of achieved results, the correlation matrix of the geometric calibration parameters can be used.In Table 4 and Table 5, the correlation matrices for the Nikon D3X SLR digital camera (R D ) and Flir A320 thermographic camera (R T ) are given.The correlations between the radial distortion coefficients are quite strong (≥ 0.8) for both correlation matrices (R D , R T ).The patterns of the correlation matrices produced from the geometric calibration results are similar for both digital and thermographic cameras.The high correlation between the radial distortion coefficients (K 1 , K 2 , K 3 ) and focal length can be seen in Table 4 and Table 5.The main difference between correlation matrices is the correlation between x 0 and y 0 with P 1 and P 2 respectively.The correlation between x 0 and y 0 with P 1 and P 2 are very strong (0.98) for the Nikon D3X SLR digital camera.The correlation matrix evaluation showed more robust calibration parameter estimation for the Nikon D3X SLR digital camera than the Flir A320 thermographic camera as expected.

2. Performance Evaluation for Photogrammetric Documentation
The historical Sokullu Mehmet Pasha Mosque was selected as the study area for testing the digital image rectification performance of the thermographic camera for photogrammetric documentation.The mosque is located along the shores of the Golden Horn in Azapkapi, Istanbul (Figure 4  The image coordinates of the control points and tie points between the images were measured, and then image orientation was performed with the bundle block adjustment using the Phidias software package.The 3D coordinate of the measured tie points were computed during the bundle block adjustment.The thermographic images of the northwest basement façade were recorded with the calibrated Flir A320 thermographic camera for digital image rectification.The image coordinate of the control points were measured in the thermographic image.In addition to the control points, the measured tie points in digital image were used as control points in digital image rectification process of digital and thermographic image since their 3D coordinate were computed in bundle block adjustment with high accuracy.Figure 5 shows the recorded digital and thermographic image of the same area with the measured control points for digital image rectification.Because the resolution of the Nikon D3X SLR digital camera was much better than the Flir A320 thermographic camera, the digital image covers a larger area.The effects of the lens distortions determined in the geometric calibrations were corrected separately for both the digital and thermographic images.The digital image rectification was performed using the Phidias software package.The projective transformation was used as a transformation equation between the photograph coordinate system and object coordinate system.The minimum condition for the digital image rectification is 4 control points; however, 6 to 8 control points show a good base for image rectification.In this study, more than 20 control points in the digital and thermographic images were used for rectification (see Figure 5).The obtained standard deviation for the control points used for the projective was 9 mm and 10 mm for the digital and thermographic images, respectively.The digital image rectifications were performed using the bilinear resampling, and georegistered rectified images were produced in a defined reference coordinate system with a ground sample distance (GSD) of 2 mm.The geo-registered rectified digital image and thermographic image are shown in Figures 6a and 6b.The swiped image of the geo-registered rectified digital image and thermographic image can be seen in Figure 6c.For the evaluation of the digital image rectification performance of the thermographic camera, geo-registered rectified digital image and thermographic image open together in other word blended with the transparency value of 50% (Figure 6d).As observed in Figures 6a  and 6b, the resolution of the rectified thermographic image is limited in comparison to the rectified digital image because of the larger pixel size (see Table 1).The areas near the basement in the rectified thermographic image (Figure 6b) show low temperatures, which were the areas that were seriously affected by moisture and isolation problems, and the texture analyses can be conducted easily.The areas seriously affected by moisture and isolation problems in the geo-registered rectified digital image (Figure 6a) are only slightly darker, and a detailed texture analyses is not easy.The blended image, prepared using the rectified thermographic and digital image (Figure 6d), offers high resolution with rich information context for detailed deformation and texture analyses.The rectified digital and thermographic images perfectly overlap in the swiped image, as observed in Figure 6c, which means that the digital rectification thermographic image was successful.The photogrammetric line drawing was also performed in a digital environment using the geo-registered rectified digital image.Figure 7 shows the superimposed photogrammetric line drawing on geo-registered rectified digital image thermographic image, swiped image and blended image at 50% transparency.As can be seen in Figure 7, photogrammetric line drawing produced using the geo-registered rectified digital image gives limited information about the deformation caused by moisture and isolation problems for better analyses of the existing situation.The results of the experiment verified that the geometric calibration of thermographic camera and digital image rectification can be performed successfully and these cameras can be used for photogrammetric documentation of deformation caused by moisture and isolation problems with the proposed approach.

CONCLUSIONS
A thermographic camera is an excellent tool for deformation and texture analysis of historical buildings and cultural heritage that have had moisture and isolation problems.To perform photogrammetric documentation of historical building deformation caused by moisture and isolation problems a geometric calibration of the thermographic camera is required.
In this paper, an approach is proposed for the geometric calibration and performance evaluation of thermographic cameras for photogrammetric documentation studies.A 3D test object was designed and manufactured to perform the geometric calibration.In the proposed approach, the geometric calibration parameters including the focal length, position of principal point and radial and tangential distortions can be estimated with the introduced additional parameters into the bundle block adjustment.The digital image rectification was preferred for photogrammetric documentation, which are the most preferred methods in a photogrammetric documentation project if the architectural heritage façade can be considered as a plane.
The achieved geometric calibration result of the thermographic camera was tested by comparing the results from the digital camera.The results of the experiment showed that the geometric calibration of the thermographic camera was successfully determined with the proposed approach.Within the experiment, results of the performance evaluation test verified that the thermographic cameras can be used successfully for the photogrammetric documentation of the historical building deformation caused by moisture and isolation problems after proper geometric calibration with the proposed approach.The combined use of the geo-registered rectified thermographic image and digital image may be helpful for a detailed texture analyses and photogrammetric documentation of hidden structures, cracks pattern, structural failures, moisture and humidity problems in cultural heritage studies to overcome the limitation of low-resolution thermographic cameras.

Figure 3 -
Figure 3-Measured photograph coordinates of the control points on the digital image (a) and thermographic image (b).
).The Sokullu Mehmet Pasha Mosque was designed by Ottoman imperial architect Mimar Sinan for Grand Vizier Sokollu Mehmet Pasha and was completed in 1578.The mosque was damaged by fires in 1807 and 1847.The mosque was restored after the fires.Large-scale restoration work was performed between 1938 and 1941.In 2012, Istanbul Special Provincial Administration asked the Division of Photogrammetry at Yildiz Technical University to conduct a geometric survey of the deformation caused by moisture and isolation problems during the restoration work on the Sokollu Mehmet Pasha Mosque.The northwest basement façade near the shores of Golden Horn was seriously affected by moisture and isolation problems.

Figure 5 -
Figure 5-Recorded digital (a) and thermographic image (b) of the northwest basement façades with the measured control points.

Table 1 -
Technical specifications of the thermographic camera and digital camera.

Table 2 -
Nikon D3X SLR digital camera geometric calibration parameters.

Table 4 -
The correlation matrix of the geometric calibration parameters for the Nikon D3X SLR digital camera.

Table 5 -
The correlation matrix of the geometric calibration parameters for the Flir A320 thermographic camera.