VALIDATION AND RELIABILITY BETWEEN EXTERNAL LOAD ANALYSIS DEVICES FOR SOCCER PLAYERS

VALIDACIÓN Y CONFIABILIDAD ENTRE DISPOSITIVOS DE ANÁLISIS DE CARGA EXTERNA PARA atletas DE FÚTBOL

ABSTRACT

Objective:

To test the reliability between two instruments with different analysis mechanisms, either by GPS (model GPSPORTS^®) or by video analysis (InStat For Players^®), relating the results of total distance covered and distance at high speed ≥ 20km/h (Very High-Intensity Running Distance, VHIR) during official soccer matches.

Study Design:

This is a methodological study. Data from 35 male professional soccer athletes from all tactical positions were included. Age 29.2 (± 4.8 years) and body fat 9.9 (± 1.7%), excluding goalkeepers (102 individual analyzes) were collected in official matches. In the data analysis, descriptive statistics procedures were used to characterize the sample and the intraclass correlation coefficient (ICC) was used to verify the agreement on the stability and internal consistency of the tests with 95% confidence intervals (CI).

Results:

The ICC in the case of the total distance traveled variable was significant 0,914 (0,876; 0,941) and indicated a very high agreement, with the linear correlation coefficient indicating a strong positive correlation (p <0.001). The ICC for the VHIR variable was not significant, although the linear correlation coefficient indicates a strong positive correlation (p <0.001).

Clinical Relevance Statment:

This study reveals that there is good agreement in the comparison of two systems designed to analyze the movement demands of each professional soccer athlete in relation to the total distance covered. Level of Evidence I; Methodological Study - Investigation of a diagnostic test.

Keywords:
Data Accuracy; Soccer; Athletic Performance; Materials testing

RESUMEN

Objetivo:

Probar la confiabilidad entre dos instrumentos con diferentes mecanismos de análisis, ya sea por GPS (modelo GPSPORTS^®) y por análisis de video (InStat For Players^®), relacionando los resultados de distancia total recorrida y distancia a alta velocidad ≥ 20km / h (Distancia de carrera de muy alta intensidad, VHIR) durante los partidos oficiales de fútbol.

Diseño del estudio:

se trata de un estudio metodológico. En los partidos oficiales se recopilaron datos de 35 atletas de fútbol profesionales masculinos, de todas las posiciones tácticas. Edad 29,2 (± 4,8 años) y grasa corporal 9,9 (± 1,7%), excluidos los porteros (102 análisis individuales). En el análisis de datos, se utilizaron procedimientos de estadística descriptiva para caracterizar la muestra y se utilizó el coeficiente de correlación intraclase (CCI) para verificar la concordancia sobre la estabilidad y consistencia interna de las pruebas con intervalos de confianza (IC) del 95%.

Resultados:

El CCI en el caso de la variable distancia total recorrida fue significativa 0,914 (0,876; 0,941) e indicó una concordancia muy alta, con el coeficiente de correlación lineal indicando una fuerte correlación positiva (p <0,001). El CCI para la variable VHIR no fue significativo, aunque el coeficiente de correlación lineal indica una fuerte correlación positiva (p <0,001).

Enunciado de relevancia clínica:

Este estudio revela que existe un buen acuerdo en la comparación de dos sistemas diseñados para analizar las demandas de movimiento de cada deportista de fútbol profesional en relación con la distancia total recorrida. Nivel de evidencia I; Estudio Metodológico - Investigación de una prueba diagnóstica.

Descriptores:
Exactitud de los Datos; Fútbol; Rendimiento Atlético; Ensayo de Materiales

RESUMO

Objetivo:

Testar a confiabilidade entre dois instrumentos com diferentes mecanismos de análise, seja por GPS (modelo GPSPORTS^®) ou por análise de vídeo (InStat For Players^®), relacionando os resultados de distância total percorrida e distância em alta velocidade ≥ 20km/h (Distância de Corrida de Intensidade Muito Alta, VHIR) durante partidas oficiais de futebol.

Desenho do Estudo:

Este é um estudo metodológico. Dados de 35 atletas de futebol profissional do sexo masculino, de todas as posições táticas foram incluídos. Idade 29,2 (± 4,8 anos) e gordura corporal 9,9 (± 1,7%), excluindo goleiros (102 análises individuais) foram coletados em jogos oficiais. Na análise dos dados, foram utilizados procedimentos de estatística descritiva para caracterizar a amostra e o coeficiente de correlação intraclasse (ICC) foi utilizado para verificar a concordância quanto à estabilidade e consistência interna dos testes com intervalos de confiança (IC) de 95%.

Resultados:

O ICC no caso da variável distância total percorrida foi significativo 0,914 (0,876; 0,941) e indicou concordância muito elevada, com o coeficiente de correlação linear indicando forte correlação positiva (p <0,001). O ICC para a variável VHIR não foi significativo, embora o coeficiente de correlação linear indique uma correlação positiva forte (p <0,001).

Declaração de Relevância Clínica:

Este estudo revela que há boa concordância na comparação de dois sistemas projetados para analisar as demandas de movimento de cada atleta profissional de futebol em relação à distância total percorrida. Nível de evidência I; Estudo metodológico - Investigação de um teste diagnóstico.

Descritores:
Confiabilidade dos dados; Futebol; Desempenho Atlético; Teste de materiais

INTRODUCTION

Quantifying the external load of athletes is performed to reduce the risk of injury and to improve performance.¹,² Tactical performance in a soccer match depends on the quality of the athletes’ actions in space and time during the game in order to be successful.³ Due to the multifactorial requirements for success in soccer, many attempts have been made to elucidate the physiological demands of a soccer match based on estimates of distance traveled and fluctuations in running intensity during a match.²,⁴ Technological innovations have led to new possibilities to capture accurate spatiotemporal information from all athletes and to unveil the dynamics and complexity of soccer games. The capture and analysis of this data is of paramount importance to elaborate the technical strategy of each athlete and the team as a whole, in addition to carrying out load control through generating and analyzing post-game and post-training data.¹,⁵,⁶

Such data in professional soccer can be captured using the Global Position System (GPS) and also by semi-automatic video analysis devices, both of which are responsible for generating positioning data, speeds and monitoring load demand.⁷,⁸ They enable evaluating the most important physical actions performed by the athletes, which are the total distance covered, the number of changes of direction and the distance covered at high speed which means ≥ 20km/h (Very High-Intensity Runnig Distance, VHIR).⁹ Although the use of electronic tracking devices is allowed, some studies have reported an intermittent loss of signal when downloading the raw data from the GPS device due to the fact that the engineering of some soccer stadiums ends up blocking the signal transmission, interfering in the data acquisition.²,¹⁰,¹¹, Thus, the use of video technology in official games was adhered to and being used together with GPS when possible, since the multi-camera technology does not suffer interference related to the environment structure, thus ensuring successful data acquisition.¹²,¹³

Some researchers report a lack of studies which compare the results obtained by the two methods, in addition to showing that several semi-automatic camera systems tend to report slightly to moderately greater distances covered in medium and high intensity than GPS technology.¹⁴

Thus, the objective of the present study was to compare the variables of total distance traveled and VHIR during professional soccer matches in different stadiums and arenas, collected simultaneously from the video system (InStat For Player) with the same data obtained from a GPS device (GPSports^®, model SPI HPU), and then to evaluate the agreement between them. All data in this study were collected during matches in official competitions and under real use conditions of the devices in the practice of professional soccer.

MATERIALS AND METHODS

This is a methodological study and it was described according to the Guidelines for Reporting Reliability and Agreement Studies – GRRAS¹⁵ and Consensus-based Standards for the Selection of Health Status Measurement Instruments – COSMIN¹⁶ initiative in order to improve the methodological quality of the study. The study was preceded by the approval of the Research Ethics Committee of the Faculdade de Ciências Médicas de Minas Gerais - 15737819.6.0000.5134. The performance data of 35 athletes from a Brazilian professional soccer team excluding goalkeepers from a total of 102 individual analyzes were collected in official championship games during the 2019 season.

Athletes from all tactical positions were included: central defenders, lateral defenders, central and lateral midfielders and forwards. The athletes belonged to the squad of a Brazilian first division soccer team and had professional experience of at least two years in training and competition at national and international level, recognized by the Brazilian Football Confederation (CBF) and South American Football Confederation (CONMEBOL).

The sample size was calculated to test the significance of the CCI.¹⁷ The significance of 5%, the minimum power of 80%, to test the significance of a CCI of 0.5 inoculation for at least 28 athletes. The athletes were informed about the objectives of the study and the confidentiality of their identity, and all signed the Free and Informed Consent Form.

A GPS unit, model GPSports^® brand GPS, SPI HPU model, was attached to each of the athletes individually before each one entered the field for an official soccer match.⁷ Semi-automatic video analysis devices such as InStat^® also achieve sports performance analysis in the same way as GPS, providing a professional tool for individual performance evaluation (Instat For Players). This technology is an external load analysis methodology based on multiple high definition cameras which accompany athletes placed around the soccer field. This system reproduces the trajectories of athletes around the field throughout the matche, and allows researchers with access to the trajectory data to study the athletes’ movements, speed, acceleration, total distance covered, VHIR and the interactions between them.⁸

The GPS data sets were adjusted according to the InStat data at the beginning of each time; however, data from some matches was eliminated because a signal loss was detected by downloading the raw data of all possible analyzes from the GPS device, making its use unfeasible. We used data from six teams of professional soccer games in three matches, in which the athletes played with the GPS device and were also analyzed by the InStat^®. These performance data were separately recorded from each game time in order to enable data comparison from the two systems and avoid evaluation errors, specifically using the variables total distance traveled and VHIR.

Data were obtained from 35 athletes in relation to the total distance covered and VHIR in matches situations (for some, first and second time) for a total of 102 measurements in order to assess the internal consistency of the measurements and the reproducibility of the measurements made by the GPS and InStat systems. The averages of each of the measures were obtained for each system, the average differences between the two systems, ICC, and Pearson’s linear correlation coefficient (r), with all of these measures being implemented with their respective 95% confidence intervals. The magnitude of the ICC was analyzed according to the criterion described by Pons:² trivial (≤0.1), small (0.11 to 0.30), moderate (0.31 to 0.5), large (0.51 to 0.7) and very large (0.71 to 0.9), and almost perfect (0.91 to 1).

The normality of the measurement differences was assessed using the Shapiro-Wilk test. The Student’s t-test for paired samples was used to compare the mean differences, and the significance of the ICC and r were also tested. The analysis was developed using the free R version 3.5.3 software program and p<0.05 was considered significant. Mean differences between the two systems, mean differences between the two systems, intraclass correlation coefficients (ICC), Pearson linear correlation coefficient (r), were made available as means for each of the systems, all of these measures with their interval intervals. 95% confidence. The magnitude of the ICC was analyzed according to the criterion described by Pons.²

The normality of the differences in the measurements was assessed using the Shapiro-Wilk test. To compare the mean differences, the Student’s t-test was used for the paired tests, and the significance of the ICC and the r were also tested. Simple linear regression models were built to describe the relationship between the measurements made by GPS and InStat. Scatter plots, regression equation and determination coefficient were produced. The quality of the adjustment was verified by analyzing the residues, where normality, homoscedasticity and presence of outliers were evaluated. The Bland-Altman graph was constructed to verify that the difference between the measures was reasonable to assume an agreement between them. The analysis was performed using the free software R version 4.0.0 and p <0.05 was considered significant.

RESULTS

The sample consisted of 35 male professional soccer players: age 29.2 (± 4.8 years), body mass 77.6 (± 7.2 kg), height 178.8 (± 8.2 cm), body fat 9.9 (± 1.7%), YoYo Intermittent Recovery Test - Level 1 (YIRT1: 1.820 ± 343 meters).

For the variables total distance covered and VHIR there was a significant difference in the measurements obtained via GPS and via InStat (p = 0.021 for distance and p <0.001 for VHIR). In the case of the distance variable, the ICC was significant and indicated almost perfect agreement and the linear correlation coefficient indicated a strong positive correlation (p <0.001). For the VHIR variable, the ICC was significant and indicated moderate agreement, and the linear correlation coefficient indicated a strong positive correlation (p <0.001) (Table 1).

Table 1
Analysis of the consistency of measurements performed via GPS and InStat.

Variables	Distance	VHIR
Mean (CI 95%)
GPS	4.937,1 (4.928,3; 4.945,9)	278,9 (276,7; 281,2)
InStat	4.978,9 (4.970,6; 4.987,3)	399,5 (396,9; 402,0)
CCI (IC 95%)	0,914 (0,876; 0,941)†	0,477 (0,312; 0,614)†
r (IC 95%)	0,919 (0,883; 0,945)†	0,827 (0,754; 0,880)†

• Notes: *Average of the differences (GPS - InStat). CI = Confidence interval; ICC = intraclass correlation coefficient; VHIR = Very High-Intensity Runnig Distance.
• ^†
  p-value ≤ 0.05; ^NSp-value> 0.05.

The simple linear regression models for the distance and distance VHIR variables showed good adjustments, with normal, homocedastic residues and without outliers. The values obtained by the GPS can be used to predict the InStat values, using the equations presented in Figures 1a and 1b.

Figure 1
Scatter diagram (a) distances measured by GPS and InStat, with linear regression equation (R2 84.34%), (b) VHIR measured by GPS and InStat, with linear regression equation (R2 68.15%).

Most of the differences in distance measurements made by GPS and InStat were between -402 (8% of the average between the two distances) and 318 (6% of the average between the two distances), being considered a reasonable variation (Figure 2a). In relation to the VHIR measures, most measures were between -270 (79.6% of the average between the two VHIR) and 28.99 (8.6% of the average between the two VHIR), in this case, a large variation that shows that the VHIR measured by GPS tend to be much smaller than those measured by InStat (Figure 2b).

Figure 2
Bland-Altman plots for (a) distances measured by GPS and InStat and (b) VHIR measured by GPS and InStat.

DISCUSSION

According to the results obtained in this study, we highlight that InStat moderately overestimated the values of the VHIR compared to the GPS. On the other hand, the variable total distance covered did not show any significant difference between the two technologies. The differences between the two systems were smaller than in previous studies, possibly due to technological advances,¹¹,¹⁸ such as the increase in the number of cameras (in the case of multi-camera technologies) to obtain greater coverage of the stadium from more angles and improvement in the resolution quality of the images, detecting and automatically tracking each player by their identification number.¹⁹ These results are consistent with the findings of the study by Edgecomb and Norton²⁰, concluding that both GPS technology and the computer/video tracking system involve systematic errors, overestimating the distance covered. It is noteworthy that the VHIR is one of the measures which may be related to injury prevention.¹,²

There was a significant difference in the measurements obtained via GPS and via InStat (p<0.001) in the total distance covered variable in our study, but the ICC was significant, indicating great agreement and a strong positive linear correlation coefficient. The ICC for the distance variable was 0.88, inferring a good agreement between the GPSports^® SPI HPU model and the InStat For Player, and therefore the combined use of the two instruments can be performed based on these results. Some studies have shown that several semi-automatic camera systems tend to report light to moderately greater distances⁴,¹⁴ covered at medium and high intensity than GPS technology. The results in these investigations highlighted that all systems similarly detected the external load produced during the start, and there were differences between the instruments in estimating the distances covered in each of the speed categories. Other authors²⁰,²¹,²² have also evaluated the measurement accuracy of the tracking technologies most used in professional team sports (i.e. semi-automatic video technology with multiple cameras, ultra-wide band - UWB, LPS and GPS), concluding that the differences between the technologies were not as pronounced in distances and speeds, but all technologies resulted in the magnitude of error increasing as the speed of the tracked object also increased. The results revealed technology-dependent variations in precision in the video tracking system. Some GPS units even measured two to six times more acceleration/deceleration occurrences than other units in these studies. According to the authors, this phenomenon could be explained due to the difference in the software version installed in each global positioning device.¹³,²², ²³

Other studies⁹,²⁴ evaluated the convergent validity and test-retest reliability between GPS devices of the same model in which it was observed that there was a difference in the results for the same variable and for the same test, concluding that some GPS models could vary in results between themselves, thus making it essential to carry out reliability studies to guarantee the reproducibility of the results from the equipment. On the other hand, most studies were carried out in non-ecological environments, creating circuits which simulate real conditions of competition¹³,²⁰,²⁵ and therefore it is interesting to note that our study compared Instat with real tracking data during official matches using a GPS device. This is due to the recurring signal loss from the engineering of certain soccer stadiums, especially those with metallic covers, as described in the literature,¹⁰,¹⁸ making it impossible to use these data from some matches in our study. The problem was overcome by using data from other games of the same season which did not experience this signal loss. We emphasize that the measurements were carried out in different environmental situations in stadiums in different geographical locations at different times, in comparison with other authors who carried out the study in a single stadium and at a specific time.²⁶

The imperfection in the correlation between the two systems remains a major challenge for analyzing training and game data. The difficulties in obtaining good quality GPS signal capture in games in indoor stadiums reinforced the increasing use of control systems through cameras. At the same time, the use of the camera system in the training camps also presents some difficulties such as the high installation cost and the demand for a longer time than the GPS for the data analysis and to prepare information reports. Thus, standardizing a single system to be used in training and official games still seems to be an important issue to be evaluated in other studies.

CONCLUSION

This study shows that there was a good agreement between the GPS systems and InStat in comparing the two instruments designed to analyze the movement of each player in a professional soccer match with regard to the total distance covered. In relation to the first, there is a tendency to moderately overvalue the variables analyzed in meters, and the statistical analysis indicated a significant ICC and a very large agreement between the two devices. Therefore, these data provide high relevance for analyzing real situations of correspondence related to physical demands. However, with the instruments respect to the VHIR variable were not reliable, with a significant difference in the measurements obtained between both technologies.

Practical implications

• GPS and InStat devices can only be interchangeably used to analyze the total distance covered.

• It is not recommended to simultaneously use both devices for the VHIR analysis.

• The same device must be maintained in all analyzes.

ACKNOWLEDGEMENTS

We thank Prof. Dr. Isabel Moura for her assistance in the statistical analysis of the data. The authors would also like to thank the Atlético Mineiro Club for providing the data and Minas Gerais Medical Sciences Faculty for their support in developing this project.

REFERENCES

• ¹
  Akenhead R, Nassis G. Training Load and Player Monitoring in High-Level Football: Current Practice and Perceptions. Int J Sports Physiol Perform. 2016;11(5):587–93.
• ²
  Pons E, García-Calvo T, Resta R, Blanco H, López del Campo R, Díaz García J, et al. A comparison of a GPS device and a multi-camera video technology during official soccer matches: Agreement between systems. PLOS ONE. 2019;14(8):e0220729.
• ³
  Van Beijsterveldt A, Anne M, Stubbe J, Schmikli S, van de Port I, Backx F. Differences in injury risk and characteristics between Dutch amateur and professional soccer players. J Sci Med Sport. 2015;18(2):145–9.
• ⁴
  Randers M, Mujika I, Hewitt A, Santisteban J, Bischoff R, Solano R, et al. Application of four different football match analysis systems: A comparative study. J Sports Sci. 2010;28(2):171–82.
• ⁵
  Buchheit M, Simpson B. Player-Tracking Technology: Half-Full or Half-Empty Glass? Int J Sports Physiol Perform. 2017;12(Suppl 2):S2–35–S2–41.
• ⁶
  Ryan M, Malone S, Collins K. An Acceleration Profile of Elite Gaelic Football Match-Play. J Strength Cond Res. 2018;32(3):812-20.
• ⁷
  GPSports, About us. 2019 [accessed on June 6, 2020]. Available at: https://www.catapultsports.com/products/evo
  » https://www.catapultsports.com/products/evo
• ⁸
  INStat. 2019 [accessed on June 6, 2020]. Available at: https://instatsport.com/football/instat_scout
  » https://instatsport.com/football/instat_scout
• ⁹
  Beato M, Bartolini D, Ghia G, Zamparo P. Accuracy of a 10 Hz GPS Unit in Measuring Shuttle Velocity Performed at Different Speeds and Distances (5 – 20 M). J Hum Kinet. 2016;54(1):15–22
• ¹⁰
  Aughey R. Applications of GPS Technologies to Field Sports. Int J Sports Physiol Perform. 2011;6(3): 295–310.
• ¹¹
  Castillo A, Gómez Carmona C, De la Cruz Sánchez E, Pino Ortega J. Accuracy, intra- and inter-unit reliability, and comparison between GPS and UWB-based position-tracking systems used for time–motion analyses in soccer. Eur J Sport Sci. 2018;18(4):450–7.
• ¹²
  Bradley P, O’Donoghue P, Wooster B, Tordoff P. The reliability of ProZone MatchViewer: a video-based technical performance analysis system. Int J Perform Anal Sport. 2007;7(3):117–29.
• ¹³
  Buchheit M, Allen A, Poon T, Modonutti M, Gregson W, Di Salvo V. Integrating different tracking systems in football: multiple camera semi-automatic system, local position measurement and GPS technologies. J Sports Sci. 2014;32(20):1844–57.
• ¹⁴
  Harley J, Lovell R, Barnes C, Portas M, Weston M. The Interchangeability of Global Positioning System and Semiautomated Video-Based Performance Data During Elite Soccer Match Play. J Strength Cond Res. 2011;25(8):2334–6.
• ¹⁵
  Kottner J, Audige L, Brorson S, Donner A, Gajewski B, Hróbjartsson A, et al. Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. J Clin Epidemiol. 2011;48(6):661–71.
• ¹⁶
  Mokkink L, de Vet H, Prinsen C, Patrick D, Alonso J, Bouter L, et al. COSMIN Risk of Bias checklist for systematic reviews of Patient-Reported Outcome Measures. Quality of Life Research. 2017;27(5):1171–9.
• ¹⁷
  Zou GY. Sample size formulas for estimating intraclass correlation coefficients with precision and assurance. Stat Med. 2012;31(29):3972-81.
• ¹⁸
  Castillo A, Gómez Carmona C, Pino Ortega J, de la Cruz Sánchez E. Validity of an inertial system to measure sprint time and sport task time: a proposal for the integration of photocells in an inertial system. Int J Perform Anal Sport. 2017;17(4):600–8.
• ¹⁹
  Imai T, Uchiyama A, Magome T, Higashino T. Play Recognition Using Soccer Tracking Data Based on Machine Learning. Lecture Notes on Data Engineering and Communications Technologies. 2018;875–884.
• ²⁰
  Edgecomb S, Norton K. Comparison of global positioning and computer-based tracking systems for measuring player movement distance during Australian Football. J Sci Med Sport. 2006;9(1-2):25–32.
• ²¹
  González M, Arcos A, Nakamura F, Gantois P, Ortega J. A comparison between UWB and GPS devices in the measurement of external load and collective tactical behaviour variables during a professional official match. Int J Perform Anal Sport. 2020;20(6):994-1002.
• ²²
  Linke D, Link D, Lames M. Validation of electronic performance and tracking systems EPTS under field conditions. PLOS ONE. 2018;13(7):e0199519.
• ²³
  Huggins R, Giersch G, Belval L, Benjamin C, Curtis R, Sekiguchi Y, et al. The Validity and Reliability of Global Positioning System Units for Measuring Distance and Velocity During Linear and Team Sport Simulated Movements. J Strength Cond Res. 2020;34(11):3070-77.
• ²⁴
  Barbero-Álvarez J, Coutts A, Granda J, Barbero-Álvarez V, Castagna C. The validity and reliability of a global positioning satellite system device to assess speed and repeated sprint ability (RSA) in athletes. J Sci Med Sport. 2010;13(2):232–5.
• ²⁵
  Cust E, Sweeting A, Ball K, Robertson S. Machine and deep learning for sport-specific movement recognition: a systematic review of model development and performance. J Sports Sci. 2019;37(5):1–33.
• ²⁶
  Polglaze T, Dawson B, Peeling P. Gold Standard or Fool’s Gold? The Efficacy of Displacement Variables as Indicators of Energy Expenditure in Team Sports. Sports Med. 2015;46(5):657–70.

Publication Dates

• Publication in this collection
  04 Apr 2022
• Date of issue
  Jul-Aug 2022

History

• Received
  06 Apr 2021
• Accepted
  29 Nov 2021