Acessibilidade / Reportar erro

AI BASED MOTION ANALYSIS SOFTWARE FOR SPORT AND PHYSICAL THERAPY ASSESSMENT

SOFTWARE DE ANÁLISIS DE MOVIMIENTO BASADO EN IA PARA LA EVALUACIÓN DEL DEPORTE Y LA FISIOTERAPIA

ABSTRACT

Introduction:

As the World Health Organization declared the novel coronavirus as a pandemic in March 2020, physical therapy is more difficult to execute, and social distancing is mandatory in the healthcare sector.

Objective:

In physical therapy, an online video analysis software that provides real-time graphic and numerical information about the patient’s movement executions without direct personal contact would mean a significant improvement in eHealth treatment.

Methods:

We have developed a software layer on top of OpenPose human body position estimation software that can extract the time series of angles of arbitrary body parts using the output coordinates from OpenPose processing the data recorded by two cameras simultaneously. To validate the procedure of determining the joint angles using the Openpose software we have used the Kinovea software.

Results:

The comparison of the determined maximal knee angle in our and the Kinovea software, which is widely used in biomechanical measurements, was not significantly different (2.03±1.06°, p<0.05)

Conclusion:

This indicates, that the developed software can calculate the appropriate joint angles with the accuracy that physiotherapy treatments require. As, to our knowledge no such software yet exists, with the help of this software development, therapists could control and correct the exercises in real-time, and also from a distance, and physical therapy effectiveness could be increased. Level of Evidence II; Experimental, comparative.

Keywords:
Artificial Intelligence; Computational Neural Networks; eHealth; Physical and Rehabilitation Medicine; Social Distancing

RESUMEN

Introducción:

Como la Organización Mundial de la Salud declaró el nuevo coronavirus como una pandemia en marzo de 2020, la fisioterapia es más difícil de ejecutar, el distanciamiento social es obligatorio en el sector de la salud.

Objetivo:

En la práctica de fisioterapia un software de análisis de vídeo online que proporcione información gráfica y numérica en tiempo real sobre las ejecuciones de movimiento del paciente sin contacto personal directo supondría una mejora significativa en el tratamiento de la eSalud.

Métodos:

Fue desarrollado una capa de software sobre el software de estimación de posición del cuerpo humano OpenPose que puede extraer la serie temporal de ángulos de partes arbitrarias del cuerpo utilizando las coordenadas de salida de OpenPose procesando los datos registrados por dos cámaras simultáneamente. Para validar el procedimiento de determinación de los ángulos articulares mediante el software Openpose fue utilizado el software Kinovea.

Resultados:

La comparación del ángulo máximo de rodilla determinado en nuestro software y Kinovea, que es ampliamente utilizado en mediciones biomecánicas, no fue significativamente diferente (2,03±1,06°, p<0,05).

Conclusión:

Esto indica que el software desarrollado puede calcular los ángulos articulares adecuados con la precisión que requieren los tratamientos de fisioterapia. Dado que aún no existe dicho software, con la ayuda de este desarrollo de software, los terapeutas podrían controlar y corregir los ejercicios en tiempo real, y también a distancia, y se podría aumentar la eficacia de la fisioterapia. Nivel de Evidencia II; Experimental, comparativo.

Descriptores:
Inteligencia Artificial; Redes Neuronales Computacionales; eSalud; Medicina Física y Rehabilitación; Distanciamiento social

RESUMO

Introdução:

Como a Organização Mundial da Saúde declarou o novo coronavírus como pandemia em março de 2020, a fisioterapia é mais difícil de executar, o distanciamento social é obrigatório no setor de saúde.

Objetivo:

Na prática da fisioterapia, um software de análise de vídeo online que fornece informações gráficas e numéricas em tempo real sobre as execuções de movimento do paciente sem contato pessoal direto significaria uma melhora significativa no tratamento eHealth.

Métodos:

Desenvolveu-se uma camada de software em cima do software de estimativa de posição do corpo humano OpenPose que pode extrair as séries temporais de ângulos de partes do corpo arbitrárias usando as coordenadas de saída do OpenPose processando os dados gravados por duas câmeras simultaneamente. Para validar o procedimento de determinação dos ângulos articulares utilizando o software Openpose utilizou-se o software Kinovea.

Resultados:

A comparação do ângulo máximo do joelho determinado em nosso e no software Kinovea, amplamente utilizado em medidas biomecânicas, não foi significativamente diferente (2,03±1,06°, p<0,05)

Conclusão:

Isso indica que o software desenvolvido pode calcular os ângulos articulares adequados com a precisão que os tratamentos de fisioterapia exigem. Como esse software ainda não existe, com a ajuda do desenvolvimento desse software, os terapeutas puderam controlar e corrigir os exercícios em tempo real, e também à distância, aumentando a eficácia da fisioterapia. Nível de Evidência II; Experimental, comparativo.

Descritores:
Inteligência Artificial; Redes Neurais Computacionais; eSaúde; Medicina Física e Reabilitação; Distanciamento Social

INTRODUCTION

The World Health Organization declared the novel coronavirus, which came from Wuhan, China to a pandemic on the 11th of March, 2020 and the disease is still spreading.11 World Health Organization. Director-General’s opening remarks at the media briefing on COVID-19 - 11 March 2020 [Internet]. 2020 [Access on 2021 Jan 20]. Disponibilizado em: https://www.who.int/director-general/speeches/detail/who-director-general-s-opening-remarks-at-the-media-briefing-on-covid-19---11-march-2020.
https://www.who.int/director-general/spe...
The COVID-19 pandemic situation provoked significant changes in every sector of society, including the health sector. For the healthcare sector, the situation is challenging, although, the relationship between patients and healthcare professionals was already changing before the pandemic. For the last three decades, direct personal contact was predominantly between an active decision-maker (doctor) and a passive follower (patient). This paternalistic approach was turned into a guidance-co-operation model first and through the years to a patient-centered model. The increase of internet usage was one of the factors that affected this change.22 Kaba R, Sooriakumaran P. The evolution of the doctor-patient relationship. Int J Surg. 2007;5(1):57-65. The role of self-care and active participation of patients in their rehabilitation process is increasing and mutual patient-doctor cooperation, simpler but direct communication and complex approach characterize healthcare in the digitalized world. Alternative solutions must be found for a new type, doctor-patient distance-based care since the number of personal contacts has reduced dramatically as a result of the COVID-19-induced social distancing norm. Moreover, prior to the novel coronavirus, other viruses had already been threatened by pandemic: SARS, H1N1, Influenza A, MERS, etc. It illustrates that there will be more and more diseases that physiotherapists and healthcare professionals need to adapt to.33 Landry MD, Geddes L, Moseman AP, Lefler JP, Raman SR, Wijchen J. Early reflection on the global impact of COVID19, and implications for physiotherapy. Physiother. 2020;107:A1-3. Consequently, the implementation of home rehabilitation and specifically telehealth methods have become essential for the continuity of rehabilitation programs, especially in case of patients with outpatient rehabilitation needs.44 Lew HL, Oh-Park M, Cifu DX. The War on COVID-19 Pandemic: Role of Rehabilitation Professionals and Hospitals. Am J Phys Med Rehabil. 2020;99(7):571-2. As using telehealth is not a new approach in healthcare,55 Jin K, Khonsari S, Gallagher R, Gallagher P, Clark AM, Freedman B, et al. Telehealth interventions for the secondary prevention of coronary heart disease: A systematic review and meta-analysis. Eur J Cardiovasc Nurs. 2019;18(4):260-71. doi:10.1177/1474515119826510.
https://doi.org/10.1177/1474515119826510...
either the idea of using it in physiotherapy.66 Holland AE. Telephysiotherapy: time to get online. J Physiother. 2017;63(4):193-5. doi:10.1016/j.jphys.
https://doi.org/10.1016/j.jphys...
Unlocking the potential of recent innovations, such as health apps for smartphones, laptops, iPads, etc., patient healthcare, healthy lifestyle, and physical activity is broadly supported.77 Ramey L, Osborne C, Kasitinon D, Juengst S. Apps and Mobile Health Technology in Rehabilitation: The Good, the Bad, and the Unknown. Phys Med Rehabil Clin N Am. 2019;30(2):485-97. doi:10.1016/j.pmr.2018.12.001.
https://doi.org/10.1016/j.pmr.2018.12.00...
,88 Moral-Munoz JA, Zhang W, Cobo MJ, Herrera-Viedma E, Kaber DB. Smartphone-based systems for physical rehabilitation applications: A systematic review. Assist Technol. 2021;33(4):223-36. doi:10.1080/10400435.2019.1611676.
https://doi.org/10.1080/10400435.2019.16...

Nowadays rehabilitation therapists can use several types of movement analyzing equipment and techniques for measuring the patients’ conditions, such as goniometer, dynamometer, EMG, gait analysis systems, stabilometry devices, force plate, etc. Various motion capture based systems are being used mostly in sports, as video analysis technology with reflective markers (Vicon, Qualisys, etc), which provide continuous data about the individuals’ vital kinematic and kinetic parameters: movement, speed, velocity, acceleration, force, power.99 Hurley OA. Part I./2. Types of Technology Commonly used in Sport. In: Sport Cyberpsychology. Abingdon: Routledge; 2018. Currently, the markerless motion capture analysis systems represent a separate category amongst the movement analysis devices. Its advantages include that the measurement preparation time is minimal, the systems can be used at home, do not impede the movement, and are almost as accurate as if the motion capture has been done with the usage of markers.1010 Colyer SL, Evans M, Cosker DP, Salo AIT. A Review of the Evolution of Vision-Based Motion Analysis and the Integration of Advanced Computer Vision Methods Towards Developing a Markerless System. Sports Med Open. 2018;5(4):24. doi:10.1186/s40798-018-0139-y.
https://doi.org/10.1186/s40798-018-0139-...
Probably the best-known is the Microsoft Kinect, which was created primarily as a video game. It includes a video camera, depth sensor, and multi-array microphone which sensors intended to classify different body motions.1111 Saputra MRU, Widyawan W, Putra GD, Santosa PI. Indoor human tracking application using multiple depth-cameras. 4th International Conference on Advanced Computer Science and Information Systems; 2012, Depok, Indonesia. p. 307-12. Severe authors claim that Kinect is useful with or without other technologies for remote musculoskeletal physiotherapy. For example, the Kinect sensor with an armband, inserted with IMU chip, which can calculate geo force can provide precise scanning of the patient’s activities during musculoskeletal physiotherapy sessions.1212 Durve I, Ghuge S, Patil S, Kalbande D. Machine Learning Approach for Physiotherapy Assessment. 2019 International Conference on Advances in Computing, Communication and Control (ICAC3); 2019. p. 1-5. doi:10.1109/icac347590.2019.90367.
https://doi.org/10.1109/icac347590.2019....
The Kinect as a rehabilitation tool was tested with positive outcomes among Parkinson’s disease patients.1313 Galna B, Jackson D, Schofield G, McNaney R, Webster M, Barry G, et al. Retraining function in people with Parkinson’s disease using the Microsoft kinect: game design and pilot testing. J Neuroeng Rehabil. 2014;11:60. doi:10.1186/1743-0003-11-60.
https://doi.org/10.1186/1743-0003-11-60...
metachromatic leukodystrophy (MLD) patients;1414 Ulaşlı AM, Türkmen U, Toktaş H, Solak O. The complementary role of the Kinect virtual reality game training in a patient with metachromatic leukodystrophy. PM R. 2014;6(6):564-7. doi:10.1016/j.pmrj.2013.11.010.
https://doi.org/10.1016/j.pmrj.2013.11.0...
cerebral palsy (CP) patients;1515 Chang YJ, Han WY, Tsai YC. A Kinect-based upper limb rehabilitation system to assist people with cerebral palsy. Res Dev Disabil. 2013;34(11):3654-9. doi:10.1016/j.ridd.2013.08.021.
https://doi.org/10.1016/j.ridd.2013.08.0...
stroke patients;1616 Ikbali Afsar S, Mirzayev I, Yemisci OU, Cosar Saracgil SN. Virtual Reality in Upper Extremity Rehabilitation of Stroke Patients: A Randomized Controlled Trial. J Stroke Cerebrovasc Dis. 2018;27(12):3473-8. doi:10.1016/j.jstrokecerebrovasdis.2018.08.007.
https://doi.org/10.1016/j.jstrokecerebro...
and in case of specific training such as balance training1717 Lange B, Chien-Yen C, Suma E, Newman B, Rizzo AS, Bolas M. Development and evaluation of low cost game-based balance rehabilitation tool using the microsoft kinect sensor. Annu Int Conf IEEE Eng Med Biol Soc. 2011;2011:1831-4. doi:10.1109/iembs.2011.6090521.
https://doi.org/10.1109/iembs.2011.60905...
and upper limb rehabilitation training what includes hand and finger movements.1818 Metcalf CD, Robinson R, Malpass AJ, Bogle TP, Dell TA, Harris C, et al. Markerless Motion Capture and Measurement of Hand Kinematics: Validation and Application to Home-Based Upper Limb Rehabilitation. IEEE Trans Biomed Eng. 2013;60(8):2184-92. doi:10.1109/tbme.2013.2250286.
https://doi.org/10.1109/tbme.2013.225028...
Furthermore, the Kinect may be able to prevent musculoskeletal injuries by performing ergonomic assessments in the workplace.1919 Dutta T. Evaluation of the Kinect™ sensor for 3-D kinematic measurement in the workplace. Appl Ergon. 2012;43(4):645-9. doi:10.1016/j.apergo.2011.09.011.
https://doi.org/10.1016/j.apergo.2011.09...
Another widely known motion analysis system is called Kinovea. It was created specifically to analyze sports motions,2020 Nor Adnan NM, Ab Patar MNA, Lee H, Yamamoto SI, Jong-Young L, Mahmud J. Biomechanical analysis using Kinovea for sports application. IOP Conf Ser Mater Sci Eng. 2018;342:012097. doi:10.1088/1757-899x/342/1/012097.
https://doi.org/10.1088/1757-899x/342/1/...
but is an undoubtfully useful tool for rehabilitation purposes too, for example, gait analysis,2121 Fernández-González P, Koutsou A, Cuesta-Gómez A, Carratalá-Tejada M, Miangolarra-Page JC, Molina-Rueda F. Reliability of Kinovea® Software and Agreement with a Three-Dimensional Motion System for Gait Analysis in Healthy Subjects”. Sensors (Basel). 2020;20(11):3154. doi:10.3390/s20113154.
https://doi.org/10.3390/s20113154...
measuring the kinematics of the wrist joint2222 El-Raheem RMA, Kamel RM, Ali MF. Reliability of using Kinovea program in measuring dominant wrist joint range of motion. Trends Appl Sci Res. 2015;10(4):224-30. or analyzing the range of motion of the cervical spine in the sagittal plane.2323 Elwardany SH, El-Sayed WH, Ali MF. Reliability of Kinovea computer program in measuring cervical range of motion in sagittal plane. Open Access Libr. J. 2015;2(9):e1916.

Recently using Artificial Intelligence (AI) is one of the most researched field within health-related motion analysis technologies. One form of AI is called Machine Learning (ML) where algorithms make predictions to interpret data and “learn” without static program instructions.2424 Tack C. Artificial intelligence and machine learning | applications in musculoskeletal physiotherapy. Musculoskelet Sci Pract. 2019;39:164-9. doi:10.1016/j.msksp.2018.11.012
https://doi.org/10.1016/j.msksp.2018.11....
It is evident, that AI has several advances and opportunities: time and cost-effective, and with machine learning (ML) diagnostics, decision-making, and measurement become simpler which can result in better patient care. The use of ML for medical imaging, pain phenotype prediction, wearable technology, risk prediction, and decision support in musculoskeletal diseases and conditions can be more efficacy.2525 Joo H, Liu H, Tan L, Gui L, Nabbe B, Matthews I, et al. The Panoptic Studio: A Massively Multiview System for Social Motion Capture (in ICCV 2015). [Internet]. 2015 [cited on 2020 Dec 2]. Available at: https://www.cs.cmu.edu/∼hanbyulj/panoptic-studio/.
https://www.cs.cmu.edu/∼hanbyulj/panopti...

OpenPose could be identified as the next generation of the existing markerless motion analysis tools for musculoskeletal rehabilitation as it is using AI. OpenPose is a state-of-the-art real-time free academic license pose estimation software based on Convolutional Neural Networks (CNNs) that can localize anatomical key points on humans. It was developed by researchers at Carnegie Mellon University, who collected massive amounts of data of people in various postures using their Panoptic Studio.2626 Kidziński L, Yang B, Lee Hicks J, Rajagopal A, Delp SL, Schwartz MH, et al. Deep neural networks enable quantitative movement analysis using single-camera videos. Nat Commun. 2020;11(1):4054. doi:10.1038/s41467-020-17807-z.
https://doi.org/10.1038/s41467-020-17807...
In the scientific literature, the accuracy of the OpenPose software was pre-validated and proved useful for gait analysis.2727 Cao Z, Hidalgo G, Simon T, Wei SE, Sheikh Y. OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields. IEEE Trans Pattern Anal Mach Intell. 2021;43(1):172-86. doi: 10.1109/TPAMI.2019.2929257.
https://doi.org/10.1109/TPAMI.2019.29292...

Our goal was to create and test a new solution for musculoskeletal physiotherapy based on the OpenPose software that can reform remote rehabilitation. In our opinion, an affordable online video analysis system that consists of commodity components provides real-time graphic and numerical information about the patient’s movements without direct therapist-patient contact during the rehabilitation process would be highly beneficial in general and in special situations, like COVID-19 pandemic.

MATERIALS AND METHODS

Code availability

We built our own executable server in phyton programming language from the source code of the OpenPose (github.com/openpose) that is running our local computer. Through the Application Programming Interface (API) of this server, we are able to obtain the detected coordinates of the joints. It works that our software is calling the OpenPose server in real-time with the camera images and then it calculates the selected angles from the returned values. The approval number of the ethics committee protocol: TE-KEB/07/2023.

The model is implemented in the OpenPose computer program (but can be wrapped on any other commercially available software, that works the same way as Openpose), which takes a color image or a video stream input and produces, as the output, the following key points of the human body: Nose, Neck; Right side: Shoulder, Elbow, Wrist, Hip, Knee, Ankle, Eye, Ear, Big Toe, Heel, Small Toe; Mid Hip; Left side: Shoulder, Elbow, Wrist, Hip, Knee, Ankle, Eye, Ear, Big Toe, Heel, Small Toe (for further information about code and keypoint tracking: github.com/openpose). Although OpenPose can determine the location of these key points, in order to determine whether the patient follows the prescribed exercises correctly, we have developed a software layer on top of OpenPose that can extract the time series of angles of arbitrary body parts using the output coordinates from OpenPose. We have used Python language (www.python.org) to generate a basic code to determine the joint angles based on the coordinates acquired from the Openpose layer that is being applied on our squat movement recording. To obtain the required joint angle, basic trigonometrical calculations were used. As Openpose determined the relative xi,yi coordinates for the needed A, B, and C joint point positions (Figure 1) we have calculated tgαAB=(YA-YB)/(XA-XB) and tgαCB=(YC-YB)/(XC-XB) for each individual frame respectively. Thereafter αABCB was calculated and visualized on the screen quantitatively as the definite joint angle of the knee for every frame.

Figure 1
Representation of knee joint angle using Openpose relative xi,yi coordinates for the A, B, C keypoints as hip, knee, ankle joint anatomical locations respectively.

One definite problem of cheap or free movement analysis systems is the usage of only one camera. Consequently, the information available about the individual’s movement can only process information acquired from one plane of the movement. There are 3D movement analysis systems like VICON, Qualysis etc. but these systems are very expensive, the attachment of markers is required on the individual, backup technicians are also required, and they are not intended for home use. To address the constraints of 2D camera images we have developed the system to simultaneously process data provided by two cameras, consequently although the system only uses 2D visual images, the observation of the patient’s movement can be achieved for the frontal and sagittal plane independently. (Figure 2) As the procedure is susceptive to adequate body positioning, as a joint positive-negative sign of the coordinate subtraction may lead to faulty results, the execution of the squat must be performed that the participant turns with his/her left side to the camera while executing the squat in the sagittal plane for one camera and in the frontal plane for the other.

Figure 2
Representative picture showing the running software. The software was developed to process data provided by two cameras simultaneously, consequently although the system only uses 2D visual images, the observation of the patient’s movement can be achieved for the frontal and the sagittal plane independently.

Validation

To validate the procedure of determining the knee joint angle using the Openpose software we have used the Kinovea software, as that is widely accepted in the field of rehabilitation and human movement analysis.2828 Dalal KK, Joshua AM, Nayak A, Mithra P, Misri Z, Unnikrishnan B. Effectiveness of prowling with proprioceptive training on knee hyperextension among stroke subjects using videographic observation- a randomised controlled trial. Gait Posture. 2018;61:232-7. Three of our co-authors have executed 10-10, altogether 30 squats, that were recorded by our camera. Thereafter we have determined the knee angles for the deepest body position and maximal knee flexion for each squat with the OpenPose and also with the Kinovea software (Figure 3). For the appropriate validation protocol of the Openpose procedure, we have subtracted the matching pairs of the two datasets and thereafter we have executed a single sample one-way t-test calculation (StatSoft Statistica 12) with the target value of zero (based on the Altman-Bland validation procedure) to determine whether the deviation from zero for the differences of the two procedures determining the knee angles is significant.

Figure 3
Representative picture indicating knee angle as being determined for the identical frame with Openpose (left) and Kinovea (right) for later comparison and validation of procedure based on Openpose to calculate joint angles.

RESULTS

Our results indicate that there was no significant statistical difference in the substracted Openpose-Kinovea knee joint angle values (Ave=0.33±2.29°) from the zero target value after performing a single sample one-way t-test (p<0.05), indicating that the procedure of determining the knee angles using OpenPose software is not significantly different from the determined knee joint angles using Kinovea and therefore these knee angles should be accepted. Although the comparison indicated, that there was no significant difference in the Openpose and Kinovea data, we have calculated the absolute differences: 2.03±1.06° and the percentages in differences between the data pairs: 2.28±1.18%.

Limitations:

We are aware that to more thoroughly validate the procedure of using OpenPose for the determination of human joint angles, more complex validating protocols should be executed using various widely accepted movement analysis systems.

DISCUSSION

As healthcare has been restructured because of COVID-19, patient direct contact meetings with health professionals have been radically reduced to delay and decrease the spread of the virus. This is the perfect time to exploit the opportunities offered by online technological developments that could increase the efficiency of healthcare, as the number of active internet users at present are more than 4.6 billion (59.5% of the global population) in the world that includes the number of active mobile internet users what is 4.32 billion.2929 Statista. Global digital population as of January 2021 [Internet]. 2021 [Access on 2021 Dec 10]. Disponibilizado em: https://www.statista.com/statistics/617136/digital-population-worldwide/.
https://www.statista.com/statistics/6171...
For retaining physical therapy sessions, solutions with online video connection could be a solution in the future as we know that continuous visual feedback concerning the execution of the movement is essential for patients in physiotherapy.3030 Kim HJ, Kramer JF. Effectiveness of Visual Feedback During Isokinetic Exercise. J Orthop Sports Phys Ther. 1997;26(6):318-23. doi:10.2519/jospt.1997.26.6.318.
https://doi.org/10.2519/jospt.1997.26.6....
Numerous previous research deals with the technological assistance of medicine.1414 Ulaşlı AM, Türkmen U, Toktaş H, Solak O. The complementary role of the Kinect virtual reality game training in a patient with metachromatic leukodystrophy. PM R. 2014;6(6):564-7. doi:10.1016/j.pmrj.2013.11.010.
https://doi.org/10.1016/j.pmrj.2013.11.0...
1818 Metcalf CD, Robinson R, Malpass AJ, Bogle TP, Dell TA, Harris C, et al. Markerless Motion Capture and Measurement of Hand Kinematics: Validation and Application to Home-Based Upper Limb Rehabilitation. IEEE Trans Biomed Eng. 2013;60(8):2184-92. doi:10.1109/tbme.2013.2250286.
https://doi.org/10.1109/tbme.2013.225028...
,2222 El-Raheem RMA, Kamel RM, Ali MF. Reliability of using Kinovea program in measuring dominant wrist joint range of motion. Trends Appl Sci Res. 2015;10(4):224-30.,2323 Elwardany SH, El-Sayed WH, Ali MF. Reliability of Kinovea computer program in measuring cervical range of motion in sagittal plane. Open Access Libr. J. 2015;2(9):e1916. One such area is the potential use of artificial intelligence and machine learning for posture detection and assessment of patients with musculoskeletal conditions.2424 Tack C. Artificial intelligence and machine learning | applications in musculoskeletal physiotherapy. Musculoskelet Sci Pract. 2019;39:164-9. doi:10.1016/j.msksp.2018.11.012
https://doi.org/10.1016/j.msksp.2018.11....
Our OpenPose-based software and camera system enables a remote treatment opportunity that gives not only real-time visual but also accurate numerical feedback on the actual body position of the patient using the benefits provided by AI technology and ML. According to our result, there was no significant statistical difference in the substracted Openpose-Kinovea knee joint angle values during the squat, which means that the software is capable of providing accurate numerical data of the movement. Moreover, the proposed system uses affordable, easily accessible components and its application fully complies with the current social distancing regulations. With the processing of two independent camera images simultaneously recording the movement of the participant in the frontal and the sagittal plane we could partially address the constraints of currently available free or cheap systems using one camera image. We believe that during the rehabilitation process, our OpenPose-based software can be used in at least two different ways. First, rehabilitation experts can get real-time video feedback on how their patients carry out the exercise tasks at home and correct, teach, and encourage them to achieve the most positive results. The second possible application is to easily and accurately measure the quantitative progress of the rehabilitation treatment by comparing the range of motion data on the sequentially recorded videos. Therefore, the longitudinal supervision of the patients’ rehabilitation program can be executed from a distance, based on the measured data. And although more thorough and precise validation procedures should be executed for the approval of the procedure, the deviation in our validation protocol between the Kinovea and OpenPose in the field of physiotherapy can be considered minimal.

CONCLUSIONS

To summarize our paper, in special cases, like the COVID-19 pandemic, it would be highly beneficial if we could combine the needs of society and the easy usability of accessible technology. Based on our research, we can conclude that our supplemented software has several advantages compared with currently used musculoskeletal rehabilitation supporting systems: it is fast and easy to use, can be used without doctor/physiotherapist-patient direct personal contact, and is much cheaper, than other high-quality movement analysis systems. We believe that as, to our knowledge no such software that uses real-time data and also processes two independent camera images simultaneously yet exists, by exploiting the advantages of the presented technology, physiotherapy effectiveness could be increased, and remote treatment services will be more widely available in the future, even after COVID-19 passes.

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

REFERENCES

  • 1
    World Health Organization. Director-General’s opening remarks at the media briefing on COVID-19 - 11 March 2020 [Internet]. 2020 [Access on 2021 Jan 20]. Disponibilizado em: https://www.who.int/director-general/speeches/detail/who-director-general-s-opening-remarks-at-the-media-briefing-on-covid-19---11-march-2020
    » https://www.who.int/director-general/speeches/detail/who-director-general-s-opening-remarks-at-the-media-briefing-on-covid-19---11-march-2020
  • 2
    Kaba R, Sooriakumaran P. The evolution of the doctor-patient relationship. Int J Surg. 2007;5(1):57-65.
  • 3
    Landry MD, Geddes L, Moseman AP, Lefler JP, Raman SR, Wijchen J. Early reflection on the global impact of COVID19, and implications for physiotherapy. Physiother. 2020;107:A1-3.
  • 4
    Lew HL, Oh-Park M, Cifu DX. The War on COVID-19 Pandemic: Role of Rehabilitation Professionals and Hospitals. Am J Phys Med Rehabil. 2020;99(7):571-2.
  • 5
    Jin K, Khonsari S, Gallagher R, Gallagher P, Clark AM, Freedman B, et al. Telehealth interventions for the secondary prevention of coronary heart disease: A systematic review and meta-analysis. Eur J Cardiovasc Nurs. 2019;18(4):260-71. doi:10.1177/1474515119826510.
    » https://doi.org/10.1177/1474515119826510
  • 6
    Holland AE. Telephysiotherapy: time to get online. J Physiother. 2017;63(4):193-5. doi:10.1016/j.jphys.
    » https://doi.org/10.1016/j.jphys
  • 7
    Ramey L, Osborne C, Kasitinon D, Juengst S. Apps and Mobile Health Technology in Rehabilitation: The Good, the Bad, and the Unknown. Phys Med Rehabil Clin N Am. 2019;30(2):485-97. doi:10.1016/j.pmr.2018.12.001.
    » https://doi.org/10.1016/j.pmr.2018.12.001
  • 8
    Moral-Munoz JA, Zhang W, Cobo MJ, Herrera-Viedma E, Kaber DB. Smartphone-based systems for physical rehabilitation applications: A systematic review. Assist Technol. 2021;33(4):223-36. doi:10.1080/10400435.2019.1611676.
    » https://doi.org/10.1080/10400435.2019.1611676
  • 9
    Hurley OA. Part I./2. Types of Technology Commonly used in Sport. In: Sport Cyberpsychology. Abingdon: Routledge; 2018.
  • 10
    Colyer SL, Evans M, Cosker DP, Salo AIT. A Review of the Evolution of Vision-Based Motion Analysis and the Integration of Advanced Computer Vision Methods Towards Developing a Markerless System. Sports Med Open. 2018;5(4):24. doi:10.1186/s40798-018-0139-y.
    » https://doi.org/10.1186/s40798-018-0139-y
  • 11
    Saputra MRU, Widyawan W, Putra GD, Santosa PI. Indoor human tracking application using multiple depth-cameras. 4th International Conference on Advanced Computer Science and Information Systems; 2012, Depok, Indonesia. p. 307-12.
  • 12
    Durve I, Ghuge S, Patil S, Kalbande D. Machine Learning Approach for Physiotherapy Assessment. 2019 International Conference on Advances in Computing, Communication and Control (ICAC3); 2019. p. 1-5. doi:10.1109/icac347590.2019.90367.
    » https://doi.org/10.1109/icac347590.2019.90367
  • 13
    Galna B, Jackson D, Schofield G, McNaney R, Webster M, Barry G, et al. Retraining function in people with Parkinson’s disease using the Microsoft kinect: game design and pilot testing. J Neuroeng Rehabil. 2014;11:60. doi:10.1186/1743-0003-11-60.
    » https://doi.org/10.1186/1743-0003-11-60
  • 14
    Ulaşlı AM, Türkmen U, Toktaş H, Solak O. The complementary role of the Kinect virtual reality game training in a patient with metachromatic leukodystrophy. PM R. 2014;6(6):564-7. doi:10.1016/j.pmrj.2013.11.010.
    » https://doi.org/10.1016/j.pmrj.2013.11.010
  • 15
    Chang YJ, Han WY, Tsai YC. A Kinect-based upper limb rehabilitation system to assist people with cerebral palsy. Res Dev Disabil. 2013;34(11):3654-9. doi:10.1016/j.ridd.2013.08.021.
    » https://doi.org/10.1016/j.ridd.2013.08.021
  • 16
    Ikbali Afsar S, Mirzayev I, Yemisci OU, Cosar Saracgil SN. Virtual Reality in Upper Extremity Rehabilitation of Stroke Patients: A Randomized Controlled Trial. J Stroke Cerebrovasc Dis. 2018;27(12):3473-8. doi:10.1016/j.jstrokecerebrovasdis.2018.08.007.
    » https://doi.org/10.1016/j.jstrokecerebrovasdis.2018.08.007
  • 17
    Lange B, Chien-Yen C, Suma E, Newman B, Rizzo AS, Bolas M. Development and evaluation of low cost game-based balance rehabilitation tool using the microsoft kinect sensor. Annu Int Conf IEEE Eng Med Biol Soc. 2011;2011:1831-4. doi:10.1109/iembs.2011.6090521.
    » https://doi.org/10.1109/iembs.2011.6090521
  • 18
    Metcalf CD, Robinson R, Malpass AJ, Bogle TP, Dell TA, Harris C, et al. Markerless Motion Capture and Measurement of Hand Kinematics: Validation and Application to Home-Based Upper Limb Rehabilitation. IEEE Trans Biomed Eng. 2013;60(8):2184-92. doi:10.1109/tbme.2013.2250286.
    » https://doi.org/10.1109/tbme.2013.2250286
  • 19
    Dutta T. Evaluation of the Kinect™ sensor for 3-D kinematic measurement in the workplace. Appl Ergon. 2012;43(4):645-9. doi:10.1016/j.apergo.2011.09.011.
    » https://doi.org/10.1016/j.apergo.2011.09.011
  • 20
    Nor Adnan NM, Ab Patar MNA, Lee H, Yamamoto SI, Jong-Young L, Mahmud J. Biomechanical analysis using Kinovea for sports application. IOP Conf Ser Mater Sci Eng. 2018;342:012097. doi:10.1088/1757-899x/342/1/012097.
    » https://doi.org/10.1088/1757-899x/342/1/012097
  • 21
    Fernández-González P, Koutsou A, Cuesta-Gómez A, Carratalá-Tejada M, Miangolarra-Page JC, Molina-Rueda F. Reliability of Kinovea® Software and Agreement with a Three-Dimensional Motion System for Gait Analysis in Healthy Subjects”. Sensors (Basel). 2020;20(11):3154. doi:10.3390/s20113154.
    » https://doi.org/10.3390/s20113154
  • 22
    El-Raheem RMA, Kamel RM, Ali MF. Reliability of using Kinovea program in measuring dominant wrist joint range of motion. Trends Appl Sci Res. 2015;10(4):224-30.
  • 23
    Elwardany SH, El-Sayed WH, Ali MF. Reliability of Kinovea computer program in measuring cervical range of motion in sagittal plane. Open Access Libr. J. 2015;2(9):e1916.
  • 24
    Tack C. Artificial intelligence and machine learning | applications in musculoskeletal physiotherapy. Musculoskelet Sci Pract. 2019;39:164-9. doi:10.1016/j.msksp.2018.11.012
    » https://doi.org/10.1016/j.msksp.2018.11.012
  • 25
    Joo H, Liu H, Tan L, Gui L, Nabbe B, Matthews I, et al. The Panoptic Studio: A Massively Multiview System for Social Motion Capture (in ICCV 2015). [Internet]. 2015 [cited on 2020 Dec 2]. Available at: https://www.cs.cmu.edu/∼hanbyulj/panoptic-studio/
    » https://www.cs.cmu.edu/∼hanbyulj/panoptic-studio/
  • 26
    Kidziński L, Yang B, Lee Hicks J, Rajagopal A, Delp SL, Schwartz MH, et al. Deep neural networks enable quantitative movement analysis using single-camera videos. Nat Commun. 2020;11(1):4054. doi:10.1038/s41467-020-17807-z.
    » https://doi.org/10.1038/s41467-020-17807-z
  • 27
    Cao Z, Hidalgo G, Simon T, Wei SE, Sheikh Y. OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields. IEEE Trans Pattern Anal Mach Intell. 2021;43(1):172-86. doi: 10.1109/TPAMI.2019.2929257.
    » https://doi.org/10.1109/TPAMI.2019.2929257
  • 28
    Dalal KK, Joshua AM, Nayak A, Mithra P, Misri Z, Unnikrishnan B. Effectiveness of prowling with proprioceptive training on knee hyperextension among stroke subjects using videographic observation- a randomised controlled trial. Gait Posture. 2018;61:232-7.
  • 29
    Statista. Global digital population as of January 2021 [Internet]. 2021 [Access on 2021 Dec 10]. Disponibilizado em: https://www.statista.com/statistics/617136/digital-population-worldwide/
    » https://www.statista.com/statistics/617136/digital-population-worldwide/
  • 30
    Kim HJ, Kramer JF. Effectiveness of Visual Feedback During Isokinetic Exercise. J Orthop Sports Phys Ther. 1997;26(6):318-23. doi:10.2519/jospt.1997.26.6.318.
    » https://doi.org/10.2519/jospt.1997.26.6.318

Edited by

Associate Editor responsible for the review process: Júlia Maria

Publication Dates

  • Publication in this collection
    07 July 2023
  • Date of issue
    2024

History

  • Received
    26 Jan 2022
  • Accepted
    23 Jan 2023
Sociedade Brasileira de Medicina do Exercício e do Esporte Av. Brigadeiro Luís Antônio, 278, 6º and., 01318-901 São Paulo SP, Tel.: +55 11 3106-7544, Fax: +55 11 3106-8611 - São Paulo - SP - Brazil
E-mail: atharbme@uol.com.br