An instrument for tongue performance assessment in activities associated with digital games: content and construct validity

ABSTRACT

Purpose:

to analyze the content and construct validity of an instrument for tongue performance assessment in activities associated with digital games.

Methods:

to analyze content validity, ten speech-language pathologists answered a questionnaire in which each item of the instrument and the set of items were judged as to its representativeness. The content validity index and the modified Kappa statistics were calculated. To analyze construct validity, 20 participants, with age between 8 and 13 years, (10 children with weak tongues and 10 children with normal tongue strength) performed a game activity with the T-Station, involving 12 targets with 2 N of strength and 5 s of time for sustained contraction for each target. The performance was compared, for each item of the instrument, between groups.

Results:

most of the items obtained satisfactory score according to the speech-language pathologist’s evaluation. Children with reduced tongue force had a poorer performance than those with normal tongue force, with a statistically significant difference in three items of the instrument.

Conclusion:

the instrument proved to be valid as a method for evaluating performance in activities associated with the T-Station.

Keywords:
Exercise Therapy; Muscle Strength; Myofunctional Therapy; Rehabilitation; Tongue; Video Games

RESUMO

Objetivo:

analisar a validade de conteúdo e de constructo do instrumento de avaliação do desempenho da língua em atividades associadas a jogos digitais.

Métodos:

para análise de conteúdo, dez fonoaudiólogos responderam um questionário em que cada item do intrumento e o conjunto de itens foram julgados quanto à sua representatividade. Foram calculados o índice de validade de conteúdo e a estatística kappa modificada. Para analisar a validade de constructo, 20 participantes, entre 8 e 13 anos (10 com força de língua adequada e 10 com alteração de força) realizaram uma atividade de jogo com o T-Station, com 12 alvos, 2 N de força e 5 s de sustentação da contração para cada alvo. O desempenho em cada item do instrumento foi comparado entre os grupos.

Resultados:

a maioria dos itens obteve pontuação satisfatória na avaliação dos fonoaudiólogos. Um item foi removido do instrumento. Os participantes com força da língua reduzida apresentaram pior desempenho do que os com força normal, com diferença estatistica para três itens do instrumento.

Conclusão:

o instrumento mostrou-se válido como método de avaliação do desempenho nas atividades associadas ao T-Station.

Descritores:
Terapia por Exercício; Força Muscular; Terapia Miofuncional; Reabilitação; Língua; Jogos de Vídeo

Introduction

Digital games have been increasingly used in speech-language pathology clinical practice therapy^1,2. They are effective therapeutic strategies to increase patient adherence to treatment, not only for entertaining and motivating, but also for providing positive reinforcement that improves information retention^3,4. Most of them are interfaced by computer, tablet or cell phone and do not require a specific device.

In Orofacial Motricity, some unconventional devices for interaction with digital games were developed for tongue motor rehabilitation. Miyaushi and collaborators developed a method to associate tongue mobility exercises with simple computer games, developed for children with Trisomy 21⁵. The method did not use an intraoral device, but the extraoral movements performed by the tongue were captured by Kinect (Microsoft®) and transformed into game commands⁵. Kothari and colleagues developed an input device for digital games, the Tongue Drive System, capable of recognizing different tongue positions inside the oral cavity and transforming them into commands in a digital game^6-11. To do so, they used a magnet attached to the lingual apex and magnetic sensors placed in an external apparatus attached to the individual's head.

The T-Station is an input instrument for digital games, developed by a team of researchers in Brazil, for the rehabilitation of tongue strength and mobility. The instrument was tested on adults¹² and children^13-15 and provides counter-resistance, isometric, and isotonic tasks to rehabilitate the musculature during its use. The innovation of this approach in relation to previous technologies is the possibility of adjusting the force and duration of contraction prior to the activity, and the data acquisition during the activity, allowing the therapist to assess the user's performance.

The instrument for performance assessment was developed to be used with the T-station, allowing the therapist to follow the patient's progress. In order to know if it fulfills this purpose, it is important that it be validated.

There are several ways to analyze the validity of an instrument¹⁶. Content validation is a process in which a group of experts establishes a consensus on the adequacy of the instrument to the field of interest¹⁷. Construct validation, on the other hand, indicates whether the instrument's results measure what it proposes, and can be performed in different ways, one of them being the comparison of groups that differ in terms of the construct of interest¹⁸.

This study aimed at analyzing the content and construct validity of the instrument for assessing tongue performance in activities associated with digital games.

Methods

This cross-sectional observational study was conducted after approval by the Research Ethics Committee of the Universidade Federal de Minas Gerais, Brazil, under the number 1,123,565, CAAE 18614313.3.0000.5149. All participants signed an Informed Consent Form or an Assent Term. The study had two parts: content analysis and construct analysis of the instrument to evaluate the performance in digital games activated by tongue.

Part 1 - Content Validation

Ten female speech-language pathologists, with clinical experience in Orofacial Myology and/or Dysphagia participated of this part of the study. Inclusion criteria were: undergraduation in Speech-Language Pathology; at least 5 years of experience in Orofacial Motricity and/or Dysphagia clinical practice; publications in journals and/or congresses. The inclusion criteria followed the recommendations of Grant and Davis¹⁹ for selecting experts for an instrument content validation. It was considered an exclusion criterion not to answer all the questions of the form.

Data collection was carried out in a presential and individual session with each participant. At first, the T-Station was presented and tested by the participant. The T-Station^12-15,20 is a device developed by a team of researchers in Brazil to rehabilitate tongue strength and mobility. It works like a video game joystick, but is actuated by the tongue. It consists of a body and a mouthpiece. The mouthpiece, made of thermoformable material, is for individual use, fits into the oral cavity and in its center there is a control rod moved by the tongue. Four tension springs provide resistance to the movement carried out by the tongue on the control rod and two Hall-effect sensors capture the displacements of the rod and transmit the information to a computer. The force applied by the user is calculated from the displacement information provided by the sensor and the elastic constant of the springs.

The participants used the equipment associated with a game, specifically developed for tongue strength training, in which targets, represented by fruit images, appeared sequentially in different regions of the computer screen, being scored as the user moved the control rod with the tongue in their direction and reached them. When moving the control rod in one direction, the game cursor also moves in the same direction, and the force exerted by the user to move the command rod is proportional to the cursor's movement on the screen. When the user reaches a target, a timer is triggered, being necessary to keep the movement for a predetermined period to score. Thus, the activity requires sustaining the muscle contraction, which is a condition of isometric exercise²¹. Four levels of difficulty were developed, differing only by the strength required to score in the game, which was 0.5 N at level one, 1 N at level two, 2 N at level three and 3 N at level four. The number of targets that appeared in each direction, the force to reach the target and the time the cursor had to be maintained within the target for the trial to be considered a success are all adjustable by the therapist.

Figure 1:
Levels of the game

To follow the patient's progress in therapy, after each game, the software generates a performance report. Seven variables were chosen to be registered by the instrument. They are: (a) number of attempts to score (number of times the user hits the target, even if he/she is not able to hold the force for the time necessary to score. The best possible performance happens when the number of attempts to score is equal to one, which means that the user only needed one attempt to score that target); (b) maximum force (higher strength, in Newtons, the user exerts during attempts to reach the target; (c) maximum time during which the target force was maintained (the longest time , in seconds, the user is able to maintain the target force level, considering all attempts made); (d) average time during which the target force was maintained (average time, in seconds, the user maintains the target level force, considering all attempts); (e) number of scored targets (number of times the patient reaches the target and sustains the contraction for the stipulated time); (e) area under the force-time curve (product of force by time, in N versus s, also called Impulse²²); (f) total time (time, in seconds, spent to score a target, after its appearance on the screen).

In addition to these variables, the performance report contains two types of graphs for each target in the game, one of them (Figure 2A) is two-dimensional, with information on time present on the X axis and the force on the Y axis. The hatched area (area under the force-time curve) is displayed at the top of the graph. The second graph (Figure 2B) is three-dimensional, contains time information on the Y axis, and, on the X and Z axis, the force values exerted in the vertical and horizontal directions, respectively. Through this graph, it is possible to visualize the path of the movement performed by the tongue, starting at time 0, with information on the force performed in each direction over time. The graph also indicates the difficulty level (target force level) and the direction of movement. The red circle at the end of the trajectory indicates that the target was scored and the printed value next to the circle refers to the total time.

Figure 2:
(a) Two-dimensional graph - it represents the force applied by the tongue on the Y axis and the muscle contraction time on the X axis. (b) Three-dimensional graph - it represents the information about the muscle contraction time on the Y axis, and the force applied by the tongue in the vertical and horizontal directions on the axes X and Z, respectively

After testing the four levels of the game, each speech-language therapist answered a questionnaire in which each item of the instrument, for each direction of movement, as well as the set of items of the instrument, was judged for its representativeness (relevance for the field of interest) by an ordinal scale from 1 to 4^19,23. A score of 1 indicated that the item evaluated was not representative, a score of 2 indicated that it was a little representative, 3 indicated that it was sufficiently representative, and 4 that the item was highly representative. The three and four responses options were considered adequate¹⁹. The Content Validity Index (CVI) for each item was calculated as the number of experts who gave the item a rating of 3 or 4, divided by the total number of experts. As a cutoff point for the item, the value of 0.78 was set, below which the item was considered not relevant to the domain of interest²³. The CVI for the set of items was calculated by averaging the CVI values for each item. The cutoff point to consider the instrument suitable for its domain was 0.8¹⁹.

The modified Kappa coefficient was also calculated. It is a complementary test to the CVI, which assesses the degree of agreement between experts for each item, taking into account the fact that agreement may occur by chance¹⁷. Kappa values above 0.74 are considered excellent, between 0.60 and 0.74 are good, and between 0.40 and 0.59 are moderate¹⁷.

All the items that obtained CVI scores above the cutoff point and acceptable agreement among professionals for modified Kappa were included in the final version of the instrument and the others were removed from the instrument¹⁷. The speech pathologists were also asked to evaluate the clarity of the items and to suggest changes, as well as propose the addition or deletion of items.

Part 2 - Construct Validation

Twenty children of both genders, aged between 8 and 13 years, speakers of Brazilian Portuguese, participated in this part of the study. They were divided into two groups according to tongue strength classification. Group 1 (G1) was composed of 10 children, eight males and two females, with low tongue strength determined by qualitative clinical evaluation, mean age of 9.8 years and standard deviation of 2 years. Group 2 (G2) consisted of 10 children with normal tongue strength (classified by qualitative clinical evaluation), five males and five females, with a mean age of 10.4 years and standard deviation of 1.8 years. Participants were recruited from the patients referred to the Speech-Language Pathology outpatient clinic of the UFMG Hospital.

The inclusion criteria were as follows: age between 8 and 13 years, having tongue strength classified as normal or weak by two speech-language pathologists, and have all the incisors teeth. Individuals with cognitive problems, pervasive developmental disorders, visual impairments, swallowing or speech disorders, oclusal alterations that interfered with the fitting of the T-Station in the oral cavity and an overly sensitive gag reflex (gag reflex triggered in the middle portion of the surface of the tongue) were excluded. Such information was obtained by interviewing the parents and reading the medical record.

To evaluate tongue strength, the participants were asked to press the tongue against a wooden spatula positioned in front of their lips for five seconds²⁴. The examiner classified the strength as normal or weak. Due to the subjectivity of this assessment, it was carried out independently by two examiners, both with specialization in Orofacial Motricity. Just the individuals that obtained the same classification were included in the study.

Each participant performed a game activity using the T-Station. They had to reach 12 targets, four in each direction (left, right, up and down). The force necessary to reach the target was previously adjusted to 2 N and the time to sustain the contraction to 5 s. During the activity, the participant remained seated in a chair with his back and feet supported, facing the computer screen, and holding the T-Station with his hands, with his elbows resting on a table.

Descriptive analysis of the variables was performed using measures of central tendency (mean and median) and variability (standard deviation). The Wilcoxon-Mann-Whitney non-parametric test was used to compare the variables between the groups, since the distribution did not show normality characteristics. A significance level of 5% was considered in the analyses.

Results

The speech-language therapists were all females, with clinical experience in Orofacial Motricity and/or Dysphagia between 6 and 25 years, with a mean of 13.9 and a standard deviation of 6.7 years. Table 1 shows the responses of each participant in the content validity assessment of each item in the instrument.

Table 1:
Evaluation of the representativeness of the instrument's items

Items	Direction	Participants										CVI	Kappa
		1	2	3	4	5	6	7	8	9	10
Number of attempts to score	Right	4	4	4	4	4	4	4	3	4	4	1	1
	Left	4	4	4	4	4	4	4	3	4	4	1	1
	Up	4	4	4	4	4	4	4	3	4	4	1	1
	Down	4	4	4	2	4	3	4	3	4	4	0.90	0.90
Maximum Force	Right	4	4	4	4	4	4	4	4	3	4	1	1
	Left	4	4	4	4	4	4	4	4	3	4	1	1
	Up	4	4	4	4	4	4	4	4	3	4	1	1
	Down	4	4	4	2	4	3	4	4	3	4	0.90	0.90
Maximum time during which the target force was maintained	Right	4	4	4	4	4	4	4	4	3	4	1	1
	Left	4	4	4	4	4	4	4	4	3	4	1	1
	Up	4	4	4	4	4	4	4	4	3	4	1	1
	Down	4	4	4	2	4	3	4	4	2	4	0.80	0.79
Average time during which the target force was maintained	Right	4	4	4	4	4	4	4	4	3	4	1	1
	Left	4	4	4	4	4	4	4	4	3	4	1	1
	Up	4	4	4	4	4	4	4	4	3	4	1	1
	Down	4	4	4	2	4	3	4	4	2	4	0.80	0.79
Scored Targets	Right	4	4	4	4	4	4	4	3	4	4	1	1
	Left	4	4	4	4	4	4	4	3	4	4	1	1
	Up	4	4	4	4	4	4	4	3	4	4	1	1
	Down	4	4	4	2	4	4	4	3	4	4	0.9	0.90
Area under the force-time curve	Right	4	4	4	1	4	4	4	2	2	3	0.70	0.66
	Left	4	4	4	1	4	4	4	2	2	3	0.70	0.66
	Up	4	4	4	1	4	4	4	2	2	3	0.70	0.66
	Down	4	4	4	1	4	4	4	2	2	3	0.70	0.66
Time to score	Right	4	4	4	4	4	4	4	4	4	4	1	1
	Left	4	4	4	4	4	4	4	4	4	4	1	1
	Up	4	4	4	4	4	4	4	4	4	4	1	1
	Down	4	4	4	2	4	4	4	4	4	4	0.90	0.90
2D Graphs		4	4	4	2	4	3	4	2	4	4	0.80	0.79
3D Graphs		3	4	4	2	4	4	4	4	2	4	0.80	0.79

• Captions: 1=not representative; 2=little representative; 3=sufficiently representative; 4=highly representative; CVI=content validity index

The CVI for the set of items was 0.92. Most of the items obtained a satisfactory score, except the item area under the force-time curve graph, which was, therefore, removed from the final version of the instrument. Items referring to the downward direction received lower scores for the CVI and modified kappa index than the same items in the other directions.

The speech-language pathologists suggested modifications to the utterance of two items: “time to score” instead of “total time” and “number of attempts to score” instead of “number of attempts”.

Table 2 presents the descriptive analysis of the values obtained for each variable of the instrument and the comparison of these values between groups of children with normal and reduced tongue strength. The group with reduced tongue strength performed worse than the group with normal tongue strength for the variables: number of attempts to score, average time during which the target force was maintained, and time to score.

Table 2:
Descriptive results for the variables and comparisons by group in each direction

Items	Right			Left			Up			Down
	Mean	SD	Median	Mean	SD	Median	Mean	SD	Median	Mean	SD	Median
Number of attempts to score
Group 1	6.4	7.5	4	4.0	4.8	2	1.9	1.7	1	1.9	1.5	1
Group 2	2.7	2.3	2	2.0	1.2	2	1.2	0.4	1	1.5	1.1	1
P-value	0.0333			0.0999			0.0453			0.2686
Maximum Force (N)
Group 1	2.0	0	2	2.0	0	2	2.0	0	2	2.0	0	2
Group 2	2.0	0	2	2.0	0	2	2.0	0	2	2.0	0	2
P-value	-			-			-			-
Maximum time during which the target force was maintained (s)
Group 1	4.9	0.5	5	4.9	0.6	5	4.9	0.5	5	5.0	0	5
Group 2	5.0	0	5	5.0	0	5	5.0	0	5	5.0	0	5
P-value	0.1538			0.3173			0.3173			-
Average time during which the target force was maintained (s)
Group 1	2.7	1.4	2.4	3.1	1.4	2.5	4.2	1.3	5	4.3	1.0	5
Group 2	3.8	1.1	3.7	3.6	1.3	2.8	4.8	0.6	5	4.5	0.9	5
P-value	0.0006			0.1541			0.0205			0.3368
Scored Targets
Group 1	2.8	0.4	3	2.9	0.3	3	2.9	0.3	3	3.0	0	3
Group 2	3.0	0	3	3.0	0	3	3.0	0	3	3.0	0	3
P-value	0.1462			0.3173			0.3173			-
Time to score (s)
Group 1	27.9	32.9	14.4	18.6	21.4	10.0	19.6	17.0	13.8	11.4	7.4	7.7
Group 2	11.5	8.6	8.7	8.0	3.6	6.3	8.3	5.3	6.5	8.4	4.2	6.7
P-value	0.0054			0.0271			0.0205			0.3368

• Wilcoxon-Mann-Whitney Test
• Captions: SD - standard deviation; Group 1 - children with reduced tongue force; Group 2 - children with normal tongue force

Discussion

Part I - Content Validation

According to the speech-language pathologist’s judgment, the instrument items, with the exception of the area under the force-time curve, are important for the domain of interest. The area under the curve is a measure of impulse²², it is present in some researches related to the evaluation of tongue and lips strength^25-28 and combines the applied force with the time of maintaining the muscle contraction. As it combines two variables, it does not allow inferring whether low values are related to difficulty in performing or maintaining the strength. An individual capable of maintaining a low strength for a long time can have a similar area as one who applies a high strength, but can maintain it just for a short time. Since this item did not reach the cutoff value recommended in the CVI, it was removed from the instrument evauation.

The instrument's average CVI (0.92) suggests adequacy of its content. According to Grant and Davis¹⁹, the minimum CVI for which the instrument is suited to its field of interest is 0.8. Other authors¹⁷, however, adopt the more conservative value of 0.9 as the acceptability limit. In both cases, the average CVI reached an acceptable value.

Kappa index was considered excellent for most of the items, indicating that there was agreement among experts. Only the item area under the time-force curve received a lower than expected index, but it is still considered good. Polit and Beck²³ point out the importance of the Kappa index to complement the CVI, as the first takes into account the fact that agreement between experts may occur by chance.

The items judgment was done by direction of movement, as in clinical practice strength training does not happen in the same way for all directions. Strength training in the upward direction is more common, as this is the direction in which the tongue exerts force in the oral phase of swallowing, for the propulsion of the bolus from the oral cavity to the oropharynx. Lateral movements are more frequent during chewing^29,30, which requires less force from the tongue. Therefore, lateral tongue movements are usually prescribed in tongue mobility training. For the downward movement, lower scores were observed in the CVI and modified kappa, because a relatively high number of speech-language pathologists judged the exercise in this direction as less relevant for clinical practice.

Part 2 - Construct Validation

There were differences in tongue motor performance between the groups of children and adolescents in three items of the instrument: number of attempts to score, average time during which the target force was maintained, and time to score, with the worst performance for the group with reduced tongue strength. Such differences only occurred in rightward and upward movements. High standard deviations were also verified for these three measures, indicating high inter-subject performance variability, even within the same group.

Tongue performance related to the downward movement did not differ between groups in any of the items evaluated. A hypothesis raised for this finding is the activation of the genioglossus muscle by the participants¹³ in downward movements while the other movements were performed mainly by the intrinsic muscles of the tongue. The sustained protrusion task against resistance, used to divide participants into groups, mainly assesses the intrinsic muscles of the tongue, with the genioglossus only serving as a stable platform for the extrinsic muscles to exert force against the spatula³¹. This finding is consistent with the assessment of the speech-language pathologists who assigned the lowest scores to the items in this direction of movement. Downward movement is not as often trained in orofacial myofunctional therapy as the others, possibly due its less relevant participation in the oral functions.

Significant difference between groups in leftward direction occurred only for the time to score. Another study¹⁴ found poor performance in this direction when using the instrument by children with poor tongue mobility and justified that the order of appearance of the targets on the screen, always starting from the left and ending down, combined with the need of performing two combined movements in order to shift the control rod from the bottom to the left, makes this the most difficult among the four movements. This may have generated similar difficulties in both groups.

All participants exerted the maximum stipulated strength in all directions. This happened because the necessary force to hit the targets, 2 N, was substantially lower than the force children and adolescent are capable of exerting using their tongues. The maximum tongue protrusion force reported by a study with children (aged 8 to 12 years) was 10.3 N for those with normal tongue strength and 8.1 N for those with mildly reduced tongue strength³². Regarding resistance, children without oral alterations, aged between 6 and 12 years, were able to sustain 50% of their maximum tongue force for 24 s. Even children with oral functions disorders were able to keep muscle contraction for more than 9 s³³. In this study, 5 seconds of contraction were enough to score, possibly for this reason no differences were found in the maximum contraction time between groups.

This study has limitations, including the small convenience sample in the construct validity analysis. For future research, expanding the sample when comparing the variables (instrument items) defined in this research, is suggested. Another limitation refers to the fact that tongue force was assessed using a qualitative method. It is known that subjective judgments vary among examiners. To address this limitation, qualitative assessments were performed by two speech therapists specialized in Orofacial Motricity.

Although the sample used for content validation was adequate in size¹⁹, there was a selection bias, as to enable presential meetings all speech-language pathologists were selected in the same city. The selection of specialists from different geographic locations would increase the chance of identifying the presence of inappropriate colloquial expressions¹⁹. In addition, the CVI has an inherent limitation, it concentrates the relevance on the items reviewed, but does not inform whether the instrument includes a complete set of items to adequately measure the construct of interest²³.

This was a preliminary study for the validation of the tongue performance assessment instrument in activities associated with digital games. The three instrument items "number of attempts to score", "average time during which the target force was maintained " and "time to score" can be considered the most relevant, because they presented adequate CVI and were good to differentiate the groups in the construct validity part of the study. This study does not close, but initiates reflections on the validity of this instrument. Literature points out that several studies are needed for the validity of an instrument to be established¹⁸. The instrument that this study deals with is used in association with the T-Station, but one believes that it can be useful in other contexts involving instrumental assessment of the tongue.

Conclusion

The instrument proved to be valid as a method for evaluating the performance in activities associated with the T-Station. It reached acceptable CVI and was able to show differences between children and adolescents with reduced and normal tongue strength in the items number of attempts to score, average time during which the target force was maintained, and time to score, with the worst performance in the group with reduced tongue strength.

Acknowledgments

We acknowledge support from the Brazilian research funding agencies: Fundação de Amparo a Pesquisa do Estado de Minas Gerais (FAPEMIG), Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) and Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq).

REFERÊNCIAS

• ¹
  Figueiredo KO, Cardenuto RRM, Amato CAH, Martins VF. Kera Puzzle: jogo digital educacional para apoio à intervenção fonoaudiológica. RISTI. 2021;41:503-15.
• ²
  Ceccon DL, Porto JB. Playing with the sounds: digital games in Down syndrome children's language development. Braz J Develop. 2021;7(1):1718-40.
• ³
  Schimid M. Reinforcing motor re-training and rehabilitation through games: a machine-learning perspective. Frontiers in Neuroengineering. 2009;2:1-2.
• ⁴
  Mirelman A, Bonato P, Deutsch JE. Effects of training with a robot-virtual reality system compared with a robot alone on the gait of individuals after stroke. Stroke. 2009;40(1):169-74.
• ⁵
  Miyauchi M, Kimura T, Nojima T. A tongue training system for children with down Syndrome. Proceedings of the 26th annual ACM symposium on user interface software and technology. St. Andrews: UIST´13, 2013. p.373-6.
• ⁶
  Kothari M, Svensson P, Basic A, Christiansen B, Vigsø M, Truc L et al. Influence of the ability to roll the tongue and tongue-training parameters on oral motor performance and learning. Arch Oral Biol. 2011;56(11):1419-23.
• ⁷
  Kothari M, Svensson P, Huo X, Ghovanloo M, Baad-Hansen L. Force and complexity of tongue task training influences behavioral measures of motor learning. Eur J Oral Sci. 2012;120(1):46-53.
• ⁸
  Kothari M, Svensson P, Huo X, Ghovanloo M, Baad-Hansen L. Motivational conditions influence tongue motor performance. Eur J Oral Sci. 2013;121(2):111-6.
• ⁹
  Kothari M, Svensson P, Jensen J, Kjærsgaard A, Jeonghee K, Nielsen JF et al. Training-induced cortical plasticity compared between three tongue-training paradigms. Neuroscience. 2013;246:1-12.
• ¹⁰
  Kothari M, Svensson P, Jensen J, Holm TD, Nielsen MS, Mosegaard T et al. Tongue-controlled computer game: a new approach for rehabilitation of tongue motor function. Arch Phys Med Rehabil. 2014;95(3):24-30.
• ¹¹
  Kothari M, Liu X, Baad-Hansen L, Kumar A, Bin G, Svensson P. Influence of visual observational conditions on tongue motor learning. Eur J Oral Sci. 2016;124(6):534-9.
• ¹²
  Furlan RMMM, Santana GA, Bischof WF, Motta AR, de Las Casas EB. A new method for tongue rehabilitation with computer games: pilot study. J Oral Rehabil. 2019;46(6):518-25.
• ¹³
  Furlan RMMM, Santana GA, Amaral MS, Motta AR, de Las Casas EB. The influence of tongue strength on children's performance in computer games reliant on lingual force generation. J Oral Rehabil. 2020;47(7):872-9.
• ¹⁴
  Amaral MS, Furlan RMMM, de Las Casas EB, Motta AR. The influence of tongue mobility on children's performance in computer games that depend on lingual movements. J Oral Rehabil. 2020;47(10):1233-41.
• ¹⁵
  Amaral MS, Furlan RMMM, Santana GA, Figueiredo CM, Las Casas EB, Motta RMMM. Feasibility of using a tongue - activated joystick by healthy pre-teen children. Int J Orofacial Myol. 2019;45:13-30.
• ¹⁶
  Souza AC, Alexandre NMC, Guirardello EB. Propriedades psicométricas na avaliação de instrumentos: avaliação da confiabilidade e da validade. Epidemiol Serv Saúde. 2017;26(3):649-59.
• ¹⁷
  Polit DF, Beck CT, Owen SV. Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing & Health. 2007;30(4):459-67.
• ¹⁸
  Martins GA. Sobre confiabilidade e validade. RBGN. 2006;8(20):1-12.
• ¹⁹
  Grant JS, Davis LL. Selection and use of content experts for instrumental development. Research in Nursing & Health. 1997;20(3):269-74.
• ²⁰
  Maia AV, Furlan RMMM, Moraes KO, Amaral MS, Medeiros AM, Motta AR. Tongue strength rehabilitation using biofeedback: a case report. CoDAS. 2019;31(5):e20180163.
• ²¹
  Rahal A. Exercícios utilizados na terapia de motricidade orofacial (quando e por que utilizá-los). In: Marchesan IQ, Silva HJ, Berretin-Felix G, organizadores. Terapia fonoaudiológica em motricidade orofacial. São José dos Campos: Pulso editorial, 2012. Cap.3, p.43-9.
• ²²
  McCormack J, Casey V, Conway R, Saunders J, Perry A. OroPress a new wireless tool for measuring oro-lingual pressures: a pilot study in healthy adults. J Neuroeng Rehabil. 2015;24(12):32-40.
• ²³
  Polit DF, Beck CT. The content validity index: are you sure you know what's being reported? Critique and recommendations. Research in Nursing & Health. 2006;29(5):489-97.
• ²⁴
  Berbert MCB, Brito VG, Furlan RMMM, Perilo TVC, Valentim AF, Barroso MFS et al. Maximum protrusive tongue force in healthy young adults. Int J Orofacial Myol. 2014;4056-63.
• ²⁵
  Nakatsuka K, Adachi T, Kato T, Oishi M, Murakami M, Okada Y et al. Reliability of novel multidirectional lip-closing force measurement system. J Oral Rehabil. 2011;38(1):18-26.
• ²⁶
  Murakami M, Adachi T, Nakatsuka K, Kato T, Oishi M, Masuda Y. Gender differences in maximum voluntary lip-closing force during lip pursing in healthy young adults. J Oral Rehabil. 2012;39(6):399-404.
• ²⁷
  Furlan RM, Motta AR, Valentim AF, Barroso MF, Costa CG, Las Casas EB. Protrusive tongue strength in people with severely weak tongues. Int J Speech Lang Pathol. 2013;15(5):503-10.
• ²⁸
  Kaede K, Kato T, Yamaguchi M, Nakamura N, Yamada K, Masuda Y. Effects of lip-closing training on maximum voluntary lip-closing force during lip pursing in healthy young adults. J Oral Rehabil. 2016;43(3):169-75.
• ²⁹
  Hori K, Ono T, Iwata H, Nokubi T, Kumakura I. Tongue pressure against hard palate during swallowing in post-stroke patients. Gerodontology. 2005;22(4):227-33.
• ³⁰
  Hori K, Ono T, Nokubi T. Coordination of tongue pressure and jaw movement in mastication. J Dent Res. 2006;85(2):187-91.
• ³¹
  Pittman LJ, Bailey EF. Genioglossus and intrinsic electromyographic activities in impeded and unimpeded protrusion tasks. J Neurophysiol. 2009;101(1):276-82.
• ³²
  Perilo TVC, Motta AR, Las Casas EB, Saffar JME, Costa CG. Objective evaluation of axial forces produced by the tongue of oral breathing children. Rev Soc Bras Fonoaudiol. 2007;12(3):184-90.
• ³³
  Robin DA, Somodi LB, Luschei ES. Measurement of tongue strength and endurance in normal and articulation disordered subjects. In: Moore CA, Yorkston KM, Beukelman DR, editores. Dysarthria and apraxia of speech: perspectives on management. Baltimore: Paul H Brookes Publishing; 1991. p.173-84.

• Study accomplished at Universidade Federal de Minas Gerais - UFMG, Belo Horizonte, Minas Gerais, Brazil.
• Reseach support source: Fundação de Amparo a Pesquisa do Estado de Minas Gerais (FAPEMIG), Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) and Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq).

Publication Dates

• Publication in this collection
  19 Nov 2021
• Date of issue
  2021

History

• Received
  14 Oct 2021
• Accepted
  08 Nov 2021