Automated Forearm Prosthesis Controlling Using Fiber Bragg Grating Sensor

Abstract

This paper describes the automation of a forearm prosthesis using the signal collected by a Fiber Bragg Grating (FBG) sensor. The FBG sensor is applied to one subject's forearm to measure the deformation as a result of the index and middle fingers when moved individually. It is possible to control a one joint model prosthesis allied to a compliant hand mechanism through signal analyses. Each finger movement has its features, such as its amplitude, which opens the possibility of using those to control different parts of the prosthesis, joint rotation by the middle finger, and compliant hand movement by the index finger. This paper presents results regarding prosthesis assembling, Hypertext Transfer Protocol (HTTP) communication latency between prosthesis and computer and tests with pre-acquired and processed FBG signal data. The prosthesis wrist rotation movement is related to the middle finger signal, and its compliant mechanism actuation is due to index finger signal. The communication between prosthesis and the computer had a mean latency of 140 ms and a standard deviation of 18 ms. The results demonstrate the potential for using the sensor system and automated prosthesis in other applications involving real-time forearm sensing, multi-finger signal analysis, and prosthetic movement.

Index Terms
Automated Prosthesis; Two finger control; Fiber Bragg Grating (FBG); Forearm sensing

I. INTRODUCTION

A prosthesis is any device that replaces all or part of a limb, organ or tissue [¹]. According to the latest 2010 census data, more than 13 million Brazilians (about 6 % of the Brazilian population) suffer from a motor disability ranging from mild to complete [²]. This statistic suggests a huge field for automated prostheses development and application. Therefore, it is important to implement new methods and mechanical tools that help physiotherapists in the process of patient rehabilitation. In addition, the new prosthetic development can be used for human-controlled prosthetic robots deployed in hazardous environments to handle chemical components or objects at extremely high temperatures [³]–[⁷].

Several works present prosthesis development for human use. These works use different signals for prosthesis control. Among them, Electroencephalography signals collected through 16 sensors, distributed over a headset which makes contact with the scalp of the individual [⁸]. Similarly, Myoelectric signals combined with neural networks were utilized as method of training a prosthesis that recognize hand gestures [⁹]. Piezoelectric sensors were used to acquire signals from a human forearm with the purpose of controlling a prosthesis [¹⁰], and others [¹¹]–[¹⁶]. Electric sensors are the most commonly used for prosthesis control.

FBG-Based sensors generally exploit the sensibility of the sensor to strain or temperature to measure a physical quantity, they present numerous advantages such as low hysteresis error and higher durability [¹⁷]–[¹⁹]. Hence it can be applied in many fields such as biomedical and bio-mechanical [²⁰]–[²²], due to its non-invasive characteristic FBG sensors are ideal for applications such as measuring interface pressure within prosthetic sockets, as shown in [²³], and even being applied to measure inter-vertebral disc pressure [²⁴], showing great repeatability when compared to usual techniques. Another similar work [²⁵] that applied 16 sensors to a transtibial prosthesis to evaluate the user's gait by measuring the deformation along the prosthesis, showing that it is a much easier way of acquiring data for not requiring a bulk of electrical cables.

This paper presents the early development of a forearm prosthesis control system using a FBG-based sensor. The results are related to the previous work of Kalinowski, et al [²⁶] from which the input prosthesis data control is gathered. This paper describes an automated prosthesis control using an FBG sensor positioned on the subject forearm and is the extended version of the conference paper [²⁷].

This paper is organized as follows. Section II describes the analyses of previous works which gathered the data used in this paper [²⁶] and [²⁸], the data set characteristics regarding the FBG signal, the mechanical design of the prosthesis, hardware components, equations used to control the prosthesis servos, and the developed software architecture to transport the data and control the prosthesis. Section III describes the prosthesis assemble, the materials used to build it, its movements, and a data transmission latency analysis. Finally, Section IV presents the conclusions of this paper.

II. MATERIALS AND METHODS

Before using the FBG sensor signal to control the prosthesis, the acquisition and calibration of the sensor were performed as presented in Subsection II-A.

A. FBG Sensor data acquisition

The data set used in this work was acquired by an FBG sensor packaged within a silicone elastomer and attached to an elastic band fixed to the forearm of six different subjects, as shown in Fig. 1.

Fig. 1
Experimental setup used to monitor the fingers movements.

The FBG employed in this work were inscribed into a single-mode optical fiber using an excimer laser Xantos^TM laser operating at 193 nm wavelength at the Multi-User Photonics Facility Laboratory at the Universidade Tecnológica Federal do Paraná - UTFPR-CT. The gratings shown a peak reflectivity of about 50 %, a bandwidth of 0.3 nm, and Bragg wavelength resonance of 1532 nm. The strain and angle calibration of the packaged sensor was presented in [²⁸] and are 1.29 pm/μϵ and 64.23 pm/°, respectively.

The muscular contraction flexes the FBG sensor, thus modifying the Bragg wavelength of the optical signal, which the optical interrogator acquires and then sends to MATLAB^TM, where it is processed. The optical interrogator Micron Optics^TM SM130 was used to acquire the FBG signal at a 50 Hz sampling rate. The acquired signal was then analyzed using MATLAB^TM to extract features of each finger's movement. More detail about the data acquisition and analysis of the fingers movement can be found in [²⁶]. The FBG signal as an input to the automated prosthesis needs to be understood as a project requirement. Subsection II-B presents an analysis of the index and middle finger signal movement measured by the FBG.

B. Data set characteristics

Understanding the data set characteristics was fundamental to create the procedure to utilize it in controlling the prosthesis. The main focus of this analysis is directed at the signal's amplitude as well as the amplitude variation rate of the signal shown in Fig. 2, which represents the signals created by the movement of an individual's middle and index fingers. The graphic shown in Fig. 2 represents one subject fingers movement. Although presenting some differences from subject to subject, the data set maintains the same characteristics throughout all the subjects.

Fig. 2
Data characteristics of an Index and Middle fingers acquired at a 50 Hz frequency sampling rate.

By analyzing the signal shown in Fig. 2, it is possible to notice that the index finger movement provokes a red Bragg wavelength shift. In contrast, the middle finger causes a blue shift. Also, for both fingers, it is possible to notice that the amplitude variation rate, on rising or fall, is almost linear. It is essential to notice that those fingers were chosen to control the prosthesis because of their opposite Bragg wavelength response. This way, it is unlikely to misinterpret which finger was moving. It is relevant to mention that each finger was moved at a time.

Each graphic in Fig. 2 shows two time markers representing the interval between 10 % to 90 % of the total amplitude variation. The time difference between the markers is about 120 ms and represents a feature to be noted during the prosthesis assembling and automation process.

C. Prosthesis Mechanical Design

Six ellipses were measured along the forearm of an adult male, providing the necessary dimensions for modeling the forearm. The goal was to create a three-dimensional prosthesis model to house the electronic components of the prosthesis, including the servos responsible for rotating the hand coupler, and close the compliant hand mechanism as shown in Fig. 3.

Fig. 3
Exploded view of complete prosthesis.

The hand was modeled to provide a compatible mechanism for performing grasping motions that simulate a hand. The rotation of the servomotor creates tension in the strings, which converge with the fingers of the hand mechanism, similar to the work of muscles and tendons. Compliant mechanism use deformation as means to achieve movement [²⁹]. The functioning of the hand mechanism is shown in the upper section of Fig. 7.

Fig. 7
Compliant hand mechanism movement and wrist rotation movement. The upper section shows the contraction caused by the variation of the index finger signal. The lower section the rotation caused by the variation of the middle finger signal. The red arrows represent the movement of each part of the prosthesis, the blue arrows represent the direction of rotation o each servo motor and the yellow arrows represents the axis of rotation of each servo.

D. Hardware Components

The Raspberry^TM Pi 3B+ and two MG995 servo motors make up the list of integrated hardware components for this project. The Raspberry was chosen for its wide range of features, including connectivity over various communication protocols. In addition, this development board has easy access to its general purpose inputs/outputs (GPIOs), including pulse width modulation (PWM) outputs needed for controlling the servos. The MG995 servo motors were chosen for their easy control by PWM signal and their load capacity that can reach up to 15 kgf.

All the software developed for the prosthesis runs in different pieces of hardware that are divided into embedded and not embedded hardware. The not embedded is responsible for signal acquisition and pre-processing, while the embedded hardware is responsible for prosthesis control and movement. The hardware architecture can be better understood in Fig. 4.

Fig. 4
Hardware components.

E. Prosthesis Control

The prosthesis control is archived trough a linear relation between the corresponding finger and the PWM signal that controls a servo motor. Both the servo that controls the hand rotation and the one that controls the compliant mechanism grip, are controlled in the same way, only using different finger signals. Equation 1 represents the linear relation between a finger signal and the PWM signal output,

(1) $$mo=\left(\frac{up\omega m-lp\omega m}{usig-lsig}\right)\left(cs-lsig\right)+lp\omega m$$

where mo is the value of the PWM signal sent by MATLAB^TM to control the angle developed around the servo motor axis, upwm is the upper limit of the pwm value that can be sent to the motor, lpwm is the lower limit of the pwm value that can be sent to the motor, usig is the upper limit of the value sent by the optical interrogator, lsig is the lower limit of the value sent by the optical interrogator and cs is the current signal sent by optical interrogator.

The variables upwm, lpwm, usig and lsig can be altered by the prosthesis user with the purpose of personalizing the response of the prosthesis. This can be done through an user interface, which was built in Node-RED^TM. Those parameters dictate how the prosthesis responds to different users characteristic signal, the same amplitude variation can cause multiple responses by the prosthesis as shown in Fig. 5, therefore the prosthesis movement can be fully customized by the user.

Fig. 5
Different users parameterization examples.

F. Software architecture

The software was split across different hardware and integrated development environments (IDEs). Some of the code, created in MATLAB^TM, runs on a personal computer (PC) and the other, created in Node-RED^TM, runs on a Raspberry^TM Pi 3B+. The HTTP (Hypertext Transfer Protocol) GET request method worked in conjunction with the signal filter code in MATLAB^TM. As the Bragg wavelength varies, the optical interrogator sends a signal corresponding to the movement to the MATLAB^TM software. A degree corresponding to a specific PWM level is calculated for each servo by linear rationing and filtering, and this information is sent to Node-RED^TM via HTTP GET over Transmission Control Protocol/Internet Protocol (TCP/IP). This information is then treated by the prosthesis user parameters in Eq. (1), who can personalize its functionalities using a Dashboard. Finally, the code running on Node-RED^TM accesses the GPIO ports of the Raspberry^TM pi 3B+ and sets the PWM value that corresponds to the initial optical signal. The overall operation can be better understood by the software flowchart in Fig. 6.

Fig. 6
Software Architecture.

III. TESTS AND RESULTS

A. Prosthesis Assemble and movement

Two materials were used to 3D print the prosthesis, polylactic acid (PLA) [³⁰], and thermoplastic polyurethane (TPU) [³¹]. PLA was the plastic used in the rigid parts of the prosthesis, such as the forearm, the hand coupler, and the wrist servo clip, while TPU was used in the compliant hand mechanism. PLA was chosen due it's rigidity and ease of printing, while TPU was chosen due to its non-plastic deformation characteristics. The angle around the axis of the servo, which varies from 0 ° to 120 °, creates tension along the strings that are placed on the compliant fingers of the hand, causing the hand to contract as shown in Fig. 7. The angle around the axis of the wrist-mounted servo motor, which also varies between 0 ° and 120°, causes the hand to rotate, as shown in the lower section of Fig. 7.

B. Data transmission

One of the goals of the software developed for this work is to enable data exchange between MATLAB^TM and the prosthesis. Within MATLAB^TM, a payload containing a position for the wrist servo motor and a position for the hand servo motor was formed using the signal coming from the optical interrogator connected to the FBG sensor.

It is expected that the time between decision and action is as short as possible for prosthetic applications. In addition, it is necessary to ensure that all payloads are delivered so the prosthesis act as intended, those are some of the reason for using TCP instead User Datagram Protocol (UDP) [³²]. To measure the method of communication latency, 120 consecutive payloads containing a position for each servo were sent from MATLAB^TM to Node-RED^TM. The average latency of the payloads and the standard deviation of the latencies were measured and are shown in Fig. 8. Latency is the time between sending the payload with the positions of both servo motors and receiving confirmation that Node-RED^TM has successfully received the payload. While the standard deviation represents the spread between the 120 latencies. The mean latency of this study was 140 ms and the standard deviation was 18 ms, the variation and the transmission results can be seen in Fig. 8.

Fig. 8
Payloads transmission delays analysis.

C. Main contributions

1. FBGs bio-compatibility presents a non-invasive way of measuring biological signals, it also presents very little hysteresis when compared to electrical sensors and it can be multiplexed, using only one optical fiber for many sensors;

2. Prosthesis control through linear equations that can be easily modified as the user needs vary;

3. Well known hardware and communication protocols were used, making it easy to expand and modify the project as necessary;

4. 3D printing with different materials, making manufacture accessible.

IV. CONCLUSIONS

This paper presents the application of prostheses commanded by signals collected through FBG sensors. It is essential to mention that this paper, as well as the others that developed some sort of prosthesis that were previously mentioned, achieved their goals of controlling a prosthesis using a signal source in the human body, therefore this publication is not comparing itself to others and showing improvements made to older papers, it is but presenting a new way of controlling a prosthesis.

The FBG signals could have moved any mechanism that utilizes two servos or two actuators of any kind. It is worth noting that the software only covers a small part of what is possible to be done with the signals collected with the FBG sensor. Future works will focus on better FBG signal treatment and analyses through pattern recognition.

The equation mentioned in the paper also fulfilled its purpose. Based on the characteristics of the rate of rise and fall of the sensor signal, it was possible to translate it into a signal for the servo movement.

HTTP over TCP/IP has proven to be a secure and reliable form of communication. None of the 120 payloads sent were lost in transit. Most of the payloads took around 140 ± 18 ms to be delivered and those latencies are not compared with any other paper. Future works will measure if there are any faster and reliable ways of transmitting data. Therefore bringing this data is just a way of analyzing this method with future ones.

The compliant hand mechanism exhibited the expected malleability, suggesting that this mechanical design can be applied in future iterations of the prosthesis. Despite this, no mechanical stress tests were carried out.

Further improvements to this work may incorporate all the project elements to become viable, such as the computational part in MATLAB^TM and the optical interrogator for online fingers and prosthesis movement.

ACKNOWLEDGMENTS

The authors acknowledge CAPES (Coordination of Superior Level Staff Improvement), CNPQ (National Council for Scientific and Technological Development), FINEP (Funding Authority for Studies and Projects), Araucária Foundation, and SETI (State Secretary of Science, Technology, and Higher Education of Paraná), and Multi-User Photonics Facility - UTFPR-CT. This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001.

REFERENCES

• ^[1]
  “Confecção e manutenção de prótese de membros inferiores, Órteses suropodálicas e adequação postural em cadeira de rodas,” MINISTÉRIO DA SAÚDE, 2013. [Online]. Available: http://bvsms.saude.gov.br/bvs/publicacoes/confecao_manutencao_orteses_proteses.pdf
• ^[2]
  R. B. Gomide et al., “Fabricação de componentes injetados em insertos produzidos por estereolitografia,” 2000.
• ^[3]
  J. Trevelyan, W. R. Hamel, and S.-C. Kang, “Robotics in hazardous applications,” in Springer handbook of robotics Springer, 2016, pp. 1521–1548.
• ^[4]
  W. R. Hamel, “E-maintenance robotics in hazardous environments,” in Proceedings. 2000 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2000)(Cat. No. 00CH37113), vol. 2, pp. 838–842, 2000.
• ^[5]
  M. Javaid, A. Haleem, A. Vaish, R. Vaishya, and K. P. Iyengar, “Robotics applications in covid-19: A review,” Journal of Industrial Integration and Management, vol. 5, no. 04, pp. 441–451, 2020.
• ^[6]
  S. Narayanan and C. R. Reddy, “Bomb defusing robotic arm using gesture control,” International Journal of Engineering Research and Technology, vol. 4, no. 02, 2015.
• ^[7]
  S. J. Mihailov, “Fiber bragg grating sensors for harsh environments,” Sensors, vol. 12, no. 2, pp. 1898–1918, 2012.
• ^[8]
  A. H. Moreira, F. S. M. Barbosa, G. H. A. Ikeda, G. de Melo Carvalho, F. S. Madani, and L. G. Trabasso, “Development of a hybrid arm prosthesis controlled by eeg signals,” in 2017 2nd International Conference on Cybernetics, Robotics and Control (CRC), pp. 203–207, 2017.
• ^[9]
  B. Schabron, Z. Alashqar, N. Fuhrman, K. Jibbe, and J. Desai, “Artificial neural network to detect human hand gestures for a robotic arm control,” in 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 1662–1665, 2019.
• ^[10]
  J. Rodriguez-Labra, B. Narakathu, and M. Atashbar, “Development of a wireless robotic arm control system using piezoelectric sensors and neural networks,” in 2019 IEEE SENSORS, pp. 1–4, 2019.
• ^[11]
  D. Bandara, R. Gopura, K. Hemapala, and K. Kiguchi, “Development of a multi-dof transhumeral robotic arm prosthesis,” Medical Engineering & Physics, vol. 48, pp. 131–141, 2017.
• ^[12]
  P. Sihombing, R. B. Muhammad, H. Herriyance, and E. Elviwani, “Robotic arm controlling based on fingers and hand gesture,” in 2020 3rd International Conference on Mechanical, Electronics, Computer, and Industrial Technology (MECnIT), pp. 40–45, 2020.
• ^[13]
  K. Zhang, C. W. de Silva, and C. Fu, “Sensor fusion for predictive control of human-prosthesis-environment dynamics in assistive walking: A survey,” arXiv preprint arXiv:1903.07674, 2019.
• ^[14]
  M. Markovic, S. Dosen, D. Popovic, B. Graimann, and D. Farina, “Sensor fusion and computer vision for context-aware control of a multi degree-of-freedom prosthesis,” Journal of neural engineering, vol. 12, no. 6, p. 066022, 2015.
• ^[15]
  J. Silva, W. Heim, and T. Chau, “Mmg-based classification of muscle activity for prosthesis control,” in The 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, vol. 1, pp. 968–971, 2004.
• ^[16]
  K. Yuan, S. Sun, Z. Wang, Q. Wang, and L. Wang, “A fuzzy logic based terrain identification approach to prosthesis control using multi-sensor fusion,” in 2013 IEEE International Conference on Robotics and Automation, pp. 3376–3381, 2013.
• ^[17]
  K. O. Hill and G. Meltz, “Fiber bragg grating technology fundamentals and overview,” Journal of lightwave technology, vol. 15, no. 8, pp. 1263–1276, 1997.
• ^[18]
  K. O. Hill, Y. Fujii, D. C. Johnson, and B. S. Kawasaki, “Photosensitivity in optical fiber waveguides: Application to reflection filter fabrication,” Applied physics letters, vol. 32, no. 10, pp. 647–649, 1978.
• ^[19]
  K. O. Hill, B. Malo, F. Bilodeau, D. Johnson, and J. Albert, “Bragg gratings fabricated in monomode photosensitive optical fiber by uv exposure through a phase mask,” Applied Physics Letters, vol. 62, no. 10, pp. 1035–1037, 1993.
• ^[20]
  H. J. Kalinowski, “Fiber bragg grating applications in biomechanics,” in 19th International Conference on Optical Fibre Sensors, vol. 7004, pp. 441–444, 2008.
• ^[21]
  V. Mishra, N. Singh, U. Tiwari, and P. Kapur, “Fiber grating sensors in medicine: Current and emerging applications,” Sensors and Actuators A: Physical, vol. 167, no. 2, pp. 279–290, 2011.
• ^[22]
  E. Al-Fakih, N. A. A. Osman, and F. R. M. Adikan, “The use of fiber bragg grating sensors in biomechanics and rehabilitation applications: the state-of-the-art and ongoing research topics,” Sensors, vol. 12, no. 10, pp. 12 890–12 926, 2012.
• ^[23]
  E. A. Al-Fakih, N. A. A. Osman, F. R. M. Adikan, A. Eshraghi, and P. Jahanshahi, “Development and validation of fiber bragg grating sensing pad for interface pressure measurements within prosthetic sockets,” IEEE Sensors Journal, vol. 16, no. 4, pp. 965–974, 2015.
• ^[24]
  C. R. Dennison, P. M. Wild, D. R. Wilson, and P. A. Cripton, “A minimally invasive in-fiber bragg grating sensor for intervertebral disc pressure measurements,” Measurement Science and Technology, vol. 19, no. 8, p. 085201, 2008.
• ^[25]
  J. R. Galvao, C. R. Zamarreno, C. Martelli, J. C. C. da Silva, F. J. Arregui, and I. R. Matias, “Smart carbon fiber transtibial prosthesis based on embedded fiber bragg gratings,” IEEE Sensors Journal, vol. 18, no. 4, pp. 1520–1527, 2017.
• ^[26]
  A. Kalinowski, E. H. Dureck, U. J. Dreyer, C. R. Zamarreño, A. B. Leránoz, S. C. Sagasta, D. R. Pipa, C. Martelli, and J. C. da Silva, “Recognition of fingers movement using fiber bragg gratings in silicon elastomer packing,” in 2019 SBMO/IEEE MTT-S International Microwave and Optoelectronics Conference (IMOC), pp. 1–3, 2019.
• ^[27]
  P. Valera Rialto Júnior, E. Henrique Dureck, A. Kalinowski, D. Gomes Fernandes, J. Carlos Cardozo da Silva, and U. José Dreyer, “Automated prosthesis control by using fiber bragg grating forearm sensor,” in 2021 SBMO/IEEE MTTS International Microwave and Optoelectronics Conference (IMOC), pp. 1–3, 2021.
• ^[28]
  A. B. Socorro-Leranoz, S. Diaz, S. Castillo, U. J. Dreyer, C. Martelli, J. C. Cardozo da Silva, I. Uzqueda, M. Gomez, and C. R. Zamarreño, “Optical system based on multiplexed fbgs to monitor hand movements,” IEEE Sensors Journal, vol. 21, no. 13, pp. 14 081–14 089, 2021.
• ^[29]
  L. L. Howell, “Compliant mechanisms,” in 21st century kinematics Springer, 2013, pp. 189–216.
• ^[30]
  R. E. Drumright, P. R. Gruber, and D. E. Henton, “Polylactic acid technology,” Advanced materials, vol. 12, no. 23, pp. 1841–1846, 2000.
• ^[31]
  J. G. Drobny, Handbook of thermoplastic elastomers Elsevier, 2014.
• ^[32]
  G. Xylomenos and G. C. Polyzos, “Tcp and udp performance over a wireless lan,” in IEEE INFOCOM’99. Conference on Computer Communications. Proceedings. Eighteenth Annual Joint Conference of the IEEE Computer and Communications Societies. The Future is Now (Cat. No. 99CH36320), vol. 2, pp. 439–446, 1999.

Publication Dates

• Publication in this collection
  13 Mar 2023
• Date of issue
  Mar 2023

History

• Received
  04 Apr 2022
• Reviewed
  11 Apr 2022
• Accepted
  05 Dec 2022