ABSTRACT
This study aimed to assess the effect of standardizing observation and rating criteria for the front crawl stroke technique on inter- and intra-rater reliability in order to propose an observation list. Two teachers and two competitive swimming coaches with different academic backgrounds were responsible for evaluating the front crawl stroke technique of 44 swimmers using video. Each swimmer completed a 25m course at a comfortable speed, having their displacement recorded in the lateral and frontal planes, both submerged and out of water, in the final 10 m of the pool. Throughout the recording, the swimmers did not breath laterally. Only the movement performed by the right arm was the raters’ focus of observation and was based on a checklist. After the shooting, the raters participated in three stages: in the first stage, there was an intervention to standardize the observation and rating criteria; inter-rater agreement was tested in the second stage, while intra-rater agreement was obtained in the third. In the second stage of the study, the agreement ranged from small to low (k=0.08 and k=0.38). In the third stage, some raters showed poor and low agreement (k=-0.13 and k=0.35), while others showed agreement between reasonable and virtually perfect (k=0.64 and k=0.87). From the results presented, it was concluded that the intervention proposal did not have the desired effect of standardizing the observation and rating criteria among the raters.
Keywords:
Swimming; Sport; Rating and Pedagogy
RESUMO
Esse estudo teve por objetivo avaliar o efeito da padronização dos critérios de observação e de avaliação da técnica da braçada do nado crawl na confiabilidade inter e intra avaliadores para uma proposta de lista de observação. Dois professores e dois treinadores de natação competitiva com formação acadêmica diferentes foram responsáveis por avaliar a técnica da braçada do nado crawl de 44 nadadores por meio de vídeo. Cada nadador realizou um percurso de 25m em velocidade confortável tendo seu deslocamento gravado nos planos lateral e frontal, ambos submersos e fora d’água, nos 10 m finais da piscina. No percurso de filmagem os nadadores não realizaram a respiração lateral. Somente o movimento realizado pelo braço direito foi foco de observação dos avaliadores e com base em uma lista de verificação. Após as filmagens os avaliadores participaram de três etapas: na primeira etapa houve uma intervenção para padronização dos critérios de observação e de avaliação. Na segunda etapa testou-se a concordância inter-avaliadores e, na terceira, a concordância intra-avaliador. Na segunda etapa do estudo a concordância variou de pequeno a baixa (k=0,08 e k=0,38). Na terceira etapa alguns avaliadores apresentaram concordância pobre e baixa (k=-0,13 e k=0,35), enquanto outros apresentaram concordância entre razoável e praticamente perfeita (k=0,64 e k=0,87). A partir dos resultados apresentados concluiu-se que a proposta de intervenção não surtiu o efeito desejado de padronização dos critérios de observação e avaliação entre os avaliadores.
Palavras-chave:
Natação; Esporte; Avaliação e Pedagogia
Introduction
Swimming techniques are influenced by the interaction of several components11. da Silva LM, Giuliano AF, de Souza Castro FA. Ensino, aprendizagem e avaliação da técnica do eggbeater em aulas de natação. R. Bras. Ci. e Mov. 2016;24(2):138-45.. DOI: https://doi.org/10.18511/rbcm.v24i2.6187.
https://doi.org/10.18511/rbcm.v24i2.6187...
), (22. de Oliveira TA, Apolinário MR, Freudenheim AM, Corrêa UC. Análise sistêmica do nado crawl. Bra. Journ. of Motor Behav. 2009;4(1):15-21. Acesso em: 05 de março de 2022. DOI: https://doi.org/10.20338/bjmb.v4i1.20
https://doi.org/10.20338/bjmb.v4i1.20...
),(33. Silveira RP, de Souza Castro FA, Figueiredo P, Vilas-Boas JP, Zamparo P. The effects of leg kick on swimming speed and arm-stroke efficiency in the front crawl. Int J Sports Physiol Perform. 2017Jul;12(6):728-35. DOI: https://doi.org/10.1123/ijspp.2016-0232
https://doi.org/10.1123/ijspp.2016-0232...
. In this sense, evaluation procedures are important in controlling learning44. Aleixo I, Vieira M. Análise do Feedback na instrução do treinador no ensino da Ginástica Artística. Motric. 2012[acesso em 06 mar 2021];8(2):849-59. Disponível em: https://www.proquest.com/openview/dc051d5d2e9f856ba7df3a756f88e560/1?pq-origsite=gscholar&cbl=616555
https://www.proquest.com/openview/dc051d...
),(55. Federolf P, Reid R, Gilgien M, Haugen P, Smith G. The application of principal component analysis to quantify technique in sports. Scand J Med Sci Sports. 2014Jun;24(3):491-9. DOI: https://doi.org/10.1111/j.1600-0838.2012.01455.x
https://doi.org/10.1111/j.1600-0838.2012...
, besides being a frequent task for teachers. This evaluation occurs by means of one observing the movement performed by the learner, who, subsequently, is given suggestions for correction44. Aleixo I, Vieira M. Análise do Feedback na instrução do treinador no ensino da Ginástica Artística. Motric. 2012[acesso em 06 mar 2021];8(2):849-59. Disponível em: https://www.proquest.com/openview/dc051d5d2e9f856ba7df3a756f88e560/1?pq-origsite=gscholar&cbl=616555
https://www.proquest.com/openview/dc051d...
),(66. Lees A. Technique analysis in sports: a critical review. J Sports Sci. 2002Oct;20(10):813-28. DOI: https://doi.org/10.1080/026404102320675657
https://doi.org/10.1080/0264041023206756...
),(77. Rosado A, Virtuoso L, Mesquita I. Relação entre as competências de diagnóstico de erros das habilidades técnicas e a prescrição pedagógica no voleibol. Rev. Br. de Ed. Fis. Esp. 2004;18(2):151. DOI:https://doi.org/10.1590/S1807-55092004000200003
https://doi.org/10.1590/S1807-5509200400...
. With this purpose, the literature suggests using checklists as a rating instrument88. Canossa S, Fernandes RJ, Carmo C, Andrade A, Soares SM. Ensino multidisciplinar em natação: reflexão metodológica e proposta de lista de verificação. MotriC.. 2007;3(4):82-99. DOI: https://doi.org/10.6063/motricidade.656
https://doi.org/10.6063/motricidade.656...
)-(1111. Wizer R, Franken M, Castro F. Concordância intra e inter-observador de protocolo de avaliação de habilidades aquáticas de crianças. Rev Br. Ci. e Mov. 2016;24(1):101-7. DOI: https://doi.org/10.18511/rbcm.v24i1.5835
https://doi.org/10.18511/rbcm.v24i1.5835...
) for a better analysis of the main components of the movement and adequate judgment55. Federolf P, Reid R, Gilgien M, Haugen P, Smith G. The application of principal component analysis to quantify technique in sports. Scand J Med Sci Sports. 2014Jun;24(3):491-9. DOI: https://doi.org/10.1111/j.1600-0838.2012.01455.x
https://doi.org/10.1111/j.1600-0838.2012...
. However, the checklist must be objective, valid, easy to use, and meet reliability requirements99. Tucher G, Quintão GF, Garrido ND, Castro FAdS. Inter-and intra-rater reliability of swimming teachers with different skill levels, in different conditions, evaluating front crawl arm movement in non-expert swimmers. Motriz: Rev. Educ. Fis.. 2020;26. DOI: https://doi.org/10.1590/s1980-6574202000010025
https://doi.org/10.1590/s1980-6574202000...
),(1111. Wizer R, Franken M, Castro F. Concordância intra e inter-observador de protocolo de avaliação de habilidades aquáticas de crianças. Rev Br. Ci. e Mov. 2016;24(1):101-7. DOI: https://doi.org/10.18511/rbcm.v24i1.5835
https://doi.org/10.18511/rbcm.v24i1.5835...
),(1212. Leandro C, Ávila-Carvalho L, Sierra-Palmeiro E, Bobo-Arce M. The evaluation rules in the view of the rhythmic gymnastics judges. Journ. of Sports. Sci. 2016;4:232-40. DOI: 10.17265/2332-7839/2016.04.007
https://doi.org/10.17265/2332-7839/2016....
.
Reliability refers to the condition for a rating or a measurement to present similar results on different occasions, as long as there is no interference with the objective of changing performance44. Aleixo I, Vieira M. Análise do Feedback na instrução do treinador no ensino da Ginástica Artística. Motric. 2012[acesso em 06 mar 2021];8(2):849-59. Disponível em: https://www.proquest.com/openview/dc051d5d2e9f856ba7df3a756f88e560/1?pq-origsite=gscholar&cbl=616555
https://www.proquest.com/openview/dc051d...
),(1111. Wizer R, Franken M, Castro F. Concordância intra e inter-observador de protocolo de avaliação de habilidades aquáticas de crianças. Rev Br. Ci. e Mov. 2016;24(1):101-7. DOI: https://doi.org/10.18511/rbcm.v24i1.5835
https://doi.org/10.18511/rbcm.v24i1.5835...
),(1313. Currell K, Jeukendrup AE. Validity, reliability and sensitivity of measures of sporting performance. Sports Med. 2008;38(4):297-316. DOI: https://doi.org/10.2165/00007256-200838040-00003
https://doi.org/10.2165/00007256-2008380...
. This is an important procedure that will ensure that the raters’ results are consistent1111. Wizer R, Franken M, Castro F. Concordância intra e inter-observador de protocolo de avaliação de habilidades aquáticas de crianças. Rev Br. Ci. e Mov. 2016;24(1):101-7. DOI: https://doi.org/10.18511/rbcm.v24i1.5835
https://doi.org/10.18511/rbcm.v24i1.5835...
),(1414. Peixoto JG, Dias AG, Miranda LM, Defilipo ÉC, Feitosa MB, Chagas PSdC. Análise de confiabilidade de medidas das pressões plantares estática e dinâmica de crianças e adolescentes com desenvolvimento normal. Fisioter e Pesqui. 2017;24:46-53. DOI: https://doi.org/10.1590/1809-2950/16222224012017
https://doi.org/10.1590/1809-2950/162222...
. Despite its importance, previous studies evaluating swimming techniques did not achieve the desired success in all its aspects99. Tucher G, Quintão GF, Garrido ND, Castro FAdS. Inter-and intra-rater reliability of swimming teachers with different skill levels, in different conditions, evaluating front crawl arm movement in non-expert swimmers. Motriz: Rev. Educ. Fis.. 2020;26. DOI: https://doi.org/10.1590/s1980-6574202000010025
https://doi.org/10.1590/s1980-6574202000...
),(1111. Wizer R, Franken M, Castro F. Concordância intra e inter-observador de protocolo de avaliação de habilidades aquáticas de crianças. Rev Br. Ci. e Mov. 2016;24(1):101-7. DOI: https://doi.org/10.18511/rbcm.v24i1.5835
https://doi.org/10.18511/rbcm.v24i1.5835...
. This can happen because teachers’ observation and rating criteria are different and, though particular, seem to lack internal consistency99. Tucher G, Quintão GF, Garrido ND, Castro FAdS. Inter-and intra-rater reliability of swimming teachers with different skill levels, in different conditions, evaluating front crawl arm movement in non-expert swimmers. Motriz: Rev. Educ. Fis.. 2020;26. DOI: https://doi.org/10.1590/s1980-6574202000010025
https://doi.org/10.1590/s1980-6574202000...
. In other words, a teacher does not use the same observation and rating criteria when analyzing a student twice in a row in a short period of time99. Tucher G, Quintão GF, Garrido ND, Castro FAdS. Inter-and intra-rater reliability of swimming teachers with different skill levels, in different conditions, evaluating front crawl arm movement in non-expert swimmers. Motriz: Rev. Educ. Fis.. 2020;26. DOI: https://doi.org/10.1590/s1980-6574202000010025
https://doi.org/10.1590/s1980-6574202000...
.
Considering the abovementioned aspects, the present study is justified by the need to devise a standardization proposal intended to evaluate the swimming learning process. Such standardization can bring about significant levels of intra- and inter-rater agreement. This agreement is important in practice because, with similar rating criteria, the swimming learning process would be made easier. Swimming studies indicate that there is a tendency to find greater intra-rater reliability among teachers with more academic training and experience99. Tucher G, Quintão GF, Garrido ND, Castro FAdS. Inter-and intra-rater reliability of swimming teachers with different skill levels, in different conditions, evaluating front crawl arm movement in non-expert swimmers. Motriz: Rev. Educ. Fis.. 2020;26. DOI: https://doi.org/10.1590/s1980-6574202000010025
https://doi.org/10.1590/s1980-6574202000...
),(1515. Zamparo P, Carrara S, Cesari P. Movement evaluation of front crawl swimming: Technical skill versus aesthetic quality. PLoS One. 2017;12(9):e0184171. DOI: https://doi.org/10.1371/journal.pone.0184171
https://doi.org/10.1371/journal.pone.018...
. However, these results are not completely satisfactory. In addition, throughout swimming learning, a student has classes with different teachers. Thus, it becomes imperative for procedures to be developed aiming at greater reliability regarding the observation and rating process among different teachers. After investigations on the subject, only one study99. Tucher G, Quintão GF, Garrido ND, Castro FAdS. Inter-and intra-rater reliability of swimming teachers with different skill levels, in different conditions, evaluating front crawl arm movement in non-expert swimmers. Motriz: Rev. Educ. Fis.. 2020;26. DOI: https://doi.org/10.1590/s1980-6574202000010025
https://doi.org/10.1590/s1980-6574202000...
was concerned with providing opportunities for interaction between evaluating teachers, discussing the adopted observation and rating criteria. However, this interaction was not systematized, and the hypothesis that it could improve inter- and intra-rater reliability was not fully confirmed.
In light of the foregoing, the objective of the present study was to assess the effect of standardizing observation and rating criteria for the front crawl stroke technique on inter- and intra-rater reliability. As a hypothesis, it was estimated that a pre-rating intervention, by enabling analysis and an organized and systematized discussion, would foster the standardization of technique observation and rating criteria by teachers.
Methods
Sample
Two swimming teachers and two competitive swimming coaches - referred to as raters, with different academic backgrounds and with a minimum of three years of experience teaching swimming -, were responsible for evaluating the front crawl stroke technique of 44 swimmers. Swimmers (9 to 11 years old) of both sexes were randomly selected; they were in the stage of learning/enhancing swimming techniques and on different levels of swimming skills. All individuals involved in the research (raters and swimmers) received oral and written instructions for voluntary participation in the present study, which was also approved by the Research Ethics Committee under number 4.578.750.
Procedures
The technical behavior of the swimmers’ stroke when performing the front crawl was observed by means of video recordings and following the qualitative movement analysis method1616. Knudson DV, Knudson CS. Análise qualitativa do movimento humano. Barueri-SP: Editora Manole Ltda; 2001.. The choice was to evaluate the stroke because it is considered the main propulsive agent of swimming33. Silveira RP, de Souza Castro FA, Figueiredo P, Vilas-Boas JP, Zamparo P. The effects of leg kick on swimming speed and arm-stroke efficiency in the front crawl. Int J Sports Physiol Perform. 2017Jul;12(6):728-35. DOI: https://doi.org/10.1123/ijspp.2016-0232
https://doi.org/10.1123/ijspp.2016-0232...
),(1515. Zamparo P, Carrara S, Cesari P. Movement evaluation of front crawl swimming: Technical skill versus aesthetic quality. PLoS One. 2017;12(9):e0184171. DOI: https://doi.org/10.1371/journal.pone.0184171
https://doi.org/10.1371/journal.pone.018...
. Each swimmer completed a 25-meter crawl swim at a comfortable speed; their displacements were recorded in the lateral plane (submerged and out of the water) and frontal plane (submerged and out of the water) in the final 10 m of the pool, in order to prevent the exit effect. The swimmers were instructed not to breath laterally during the recording for a standardization in swimming and prevention of possible technical changes resulting from this body action. For this purpose, markings on the bottom of the pool indicated the beginning and end of this distance.
The swimmers were filmed by sports cameras (GoPro Hero 8; 1080p at 240 frames per second), and the one used in the water was inside a diving box specifically designed for this purpose (GoPro) (0.5 m depth from the waterline). The four images were later digitized, synchronized and transformed into a single file for video editing using the Final Cut Pro software, version 10.4.9.
Only the movement performed by the swimmers’ right arm was used as an object of observation by the raters and was based on the same checklist, as already described in the literature99. Tucher G, Quintão GF, Garrido ND, Castro FAdS. Inter-and intra-rater reliability of swimming teachers with different skill levels, in different conditions, evaluating front crawl arm movement in non-expert swimmers. Motriz: Rev. Educ. Fis.. 2020;26. DOI: https://doi.org/10.1590/s1980-6574202000010025
https://doi.org/10.1590/s1980-6574202000...
. The checklist used, considered as the one that presents the most important movements to be observed for the stroke action, is composed of 5 items describing the stroke phases and the main variations of how the movement can be performed99. Tucher G, Quintão GF, Garrido ND, Castro FAdS. Inter-and intra-rater reliability of swimming teachers with different skill levels, in different conditions, evaluating front crawl arm movement in non-expert swimmers. Motriz: Rev. Educ. Fis.. 2020;26. DOI: https://doi.org/10.1590/s1980-6574202000010025
https://doi.org/10.1590/s1980-6574202000...
. The rater was free to watch the footage as many times as necessary. From the observation of the movement, it was indicated in which descriptor (A, B, C or D) the movement fit. Chart 1 presents the checklist used.
Checklist used by the raters to observe the front crawl stroke. The table presents the Items and their movement descriptors (MDs)
Intervention stages:
The study was divided into three stages, as described below.
1st Stage: intervention to standardize the criteria for observation and rating of the front crawl stroke by the teachers
One of the project’s researchers met remotely with the raters to discuss the criteria for observation and rating of the front crawl stroke. Three online meetings lasting about 60 minutes were held. In the first meeting, the crawl swimming technique was discussed, with an emphasis on the essential movements of the stroke. In this sense, there was: (1) a theoretical presentation with the purpose of providing a technical description of the swimming movement and discussing the procedures for the qualitative analysis of human movement, using textual elements and photographs, and (2) a presentation of the videos (in different observation planes) of swimmers on different levels of performance. At all times, the raters were free to intervene and ask questions. In the second meeting, (1) the technical checklist was presented (composed of five items), and the essential movements of the front crawl stroke were discussed through the presentation of the videos (in different observation planes) and (2) there was a discussion on how the computer resources would be used for a better observation of the movement, such as slow-speed viewing. The expectation for that meeting is that the raters would become familiar with the tool and develop similar observation criteria. In the third meeting, the raters had the opportunity to simultaneously watch three swimmers of different levels. Each observation and rating criterion was openly discussed among them. All meetings were held through video lessons.
2nd Stage: inter-rater reliability
Seven days after the end of the first stage, the raters received a link by email containing the videos of the swimmers identified by letters and had 30 days to fill out the observation form. The videos were transferred to each rater’s computer before observation and rating. Each rater used their own resources, criteria and methods to observe the movement. With the intention of preventing flaws in the observation process due to fatigue, a maximum of five swimmers were observed per day. The swimmers’ videos could be viewed as many times as the rater desired, but without contact with the other raters.
3rd Stage: intra-rater reliability
This stage consisted of assessing intra-rater reliability and was carried out 20 days after the end of the second stage. The raters received a link by email containing the videos of the swimmers identified by letters (different from those presented in the second stage) and had 30 days to re-evaluate them. The videos were transferred to their computers before observation and rating. Each rater used their own resources, criteria and methods to observe and rate the movement. With the intention of preventing flaws in the observation process due to fatigue, a maximum of five swimmers were observed per day. The swimmers’ videos could be viewed as many times as the rater desired, but without contact with the other raters.
Statistical analysis
The kinematic descriptors used in the present study can be classified as nominal. The level of inter-rater reliability (2nd stage) was tested using Fleiss’s Kappa Reliability Coefficient. On the other hand, the intra-rater reliability level (3rd stage) was tested using Cohen’s Kappa Reliability Coefficient. Results lower than zero represent poor reliability; between 0 and 0.20, small reliability; between 0.21 and 0.40, low reliability; between 0.41 and 0.60, moderate reliability; between 0.61 and 0.80, reasonable reliability, and above 0.81, a virtually perfect reliability((1717. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977Mar;33(1):159-74. DOI: https://doi.org/10.2307/2529310
https://doi.org/10.2307/2529310...
)). Using Fleiss’ Kappa, it was also possible to obtain the level of inter-rater reliability for each of the kinematic descriptors. In all cases, a significance level of 95% was considered, and reliabilities above 0.41 (moderate) were considered relevant. The data were processed with the aid of IBM SPSS Statistics, version 25.
Results
Table 1 presents the results for inter-rater reliability. Its values are between small and low, ranging from 0.13 to 0.38 (p<0.05).
Regarding the intra-rater reliability results, it can be highlighted that rater A showed moderate reliability in questions 1 and 5 (k=0.56 and p<0.001). In their turn, rater B showed reasonable reliability in questions 1, 3 and 5 (k=0.65 and p<0.001; k=0.68 and p<0.001; k=0.65 and p<0.001). Rater C presented the best reliability indexes among the 4 raters: virtually perfect reliability in questions 1, 3, 4 and 5 (k=0.82 and p<0.001; k=0.87 and p<0.001; k=1 and p< 0.001; k= 0.92 and p<0.001. Finally, rater D showed moderate reliability in question 5 (k=0.49 and p<0.001) and reasonable reliability (k=0.64 and p<0.001) in question 2. The intra-rater reliability classification for the front crawl stroke rating items is presented in Chart 2.
Discussion
The present study aimed to assess the effect of standardizing observation and rating criteria for the front crawl stroke technique on inter and intra-rater reliability. Values classified as small and low (0.13 to 0.38; p<0.05) were obtained for inter-rater reliability. This result fell short of the expectation and is an indication that the proposed intervention did not have the desired effect on this aspect of reliability. In its turn, as for intra-rater reliability, there was a large variation among the raters. Some of them obtained a “virtually perfect” classification for the proposed items, while others, low reliability. What was possible to notice, in this case, was that the swimming coaches presented, in general, better intra-rater reliability than the teachers did. However, even so, intra-rater reliability needs to be improved.
The reliability findings obtained with the present study did not confirm the hypothesis that the intervention offered with the aim of standardizing the observation and rating criteria would result in high inter- and intra-rater reliability values. Therefore, it seems that the way in which the intervention was carried out - remotely and with video classes -, did not have the desired effect on the standardization of these criteria. One of the limitations that may have occurred in the intervention may be precisely the use of video lessons. Adopting them as a pedagogical resource maintains a unidirectional relationship of knowledge, that is, the researcher who shot them was responsible for passing on the information. As much as studies show the benefits of video lessons aimed at training professionals1818. Lima VS, Azevedo NAdA, Guimarães JMX, Pereira MM, Neto JA, Souza LM, et al. Produção de vídeo-educacional: estratégia de formação docente para o ensino na saúde. Rev. Elet. de Com. 2019. acesso em 06 de março de 2022. DOI: 10.29397/reciis.v13i2.1594
https://doi.org/10.29397/reciis.v13i2.15...
),(1919. Mogetti RS, Brod FAT, Lopes JLB. VIDEOAULA INTERATIVA COMO RECURSO DE ENSINO PARA A EDUCAÇÃO PROFISSIONAL A DISTÂNCIA. Redin- Rev. Edu. Interd.. 2019;8(1). acesso em 07 de março de 2022. Disponível em:Disponível em:http://seer.faccat.br/index.php/redin/article/view/1531
http://seer.faccat.br/index.php/redin/ar...
, the present one did not achieve the expected success using this tool. This does not mean that the proposal is invalid, but that it needs to be improved.
The results found suggest the hypothesis that the opportunity for a discussion among the raters, with the latter reflecting on and presenting their own observation and rating criteria - which are oftentimes not clear to themselves even (as per the intra-rater reliability results), followed by guidance as to the standardization of these criteria, would be positive. Thus, interaction and communication among raters is important for the training and teaching process to be implemented2020. Lauriti NC. A comunicação na avaliação de desempenho docente no ensino superior. EccoS-Rev. Cient. 2002;4(1):111-30. DOI: https://doi.org/10.5585/eccos.v4i1.296
https://doi.org/10.5585/eccos.v4i1.296...
. Moreover, the use of different learning tools during the video lesson (such as animations, charts, and links to articles or texts based on the chosen theme) can contribute positively1919. Mogetti RS, Brod FAT, Lopes JLB. VIDEOAULA INTERATIVA COMO RECURSO DE ENSINO PARA A EDUCAÇÃO PROFISSIONAL A DISTÂNCIA. Redin- Rev. Edu. Interd.. 2019;8(1). acesso em 07 de março de 2022. Disponível em:Disponível em:http://seer.faccat.br/index.php/redin/article/view/1531
http://seer.faccat.br/index.php/redin/ar...
. In this way, for other studies addressing this topic, the use of these strategies is suggested. It is thus believed that the intervention process would be better structured and could result in significant inter and intra-rater reliability values. In any case, the offer of four possibilities for performing the movement makes it more difficult for random agreement to occur, and this factor must be considered when interpreting reliability results.
In a study conducted by Wizer et al.1111. Wizer R, Franken M, Castro F. Concordância intra e inter-observador de protocolo de avaliação de habilidades aquáticas de crianças. Rev Br. Ci. e Mov. 2016;24(1):101-7. DOI: https://doi.org/10.18511/rbcm.v24i1.5835
https://doi.org/10.18511/rbcm.v24i1.5835...
, which can be used for comparison with the second stage of the present study, inter-rater reliability ranged from poor to moderate. Said study evaluated the water skills of children during the phase of adaptation to the aquatic environment. Intra-rater reliability, in its turn, ranged from substantial to perfect. This reliability was lower for tasks that were rarely performed at this level of learning, mainly because they required greater confidence to be performed. In this way, the children may have presented different movement patterns, making the evaluation difficult due to inconsistency in the execution of motor tasks. Despite the reliability results presented, it is believed that, for this level of learning, the teacher has even greater difficulty establishing criteria and the internal standards of the movement.
On the other hand, Vidal et al.1010. Vidal JdM, Tucher G, Nogueira L, Novaes RC, Vale RGdS, Castro MORd, et al. Crawl technique observation sheet for beginning swimmers: an evaluation proposal for swimming teachers. Motriz: Rev. Educ. Fis. 2021;27. DOI: https://doi.org/10.1590/S1980-65742021016920
https://doi.org/10.1590/S1980-6574202101...
found reasonable reliability, in the inter-rater stage, when evaluating several technical components of crawl swimming. As for intra-rater reliability, it ranged from reasonable to substantial. Some limitations were presented by the authors, during both the intra and inter-rater stages.
Corazza et al.2121. Corazza ST, Pereira ÉF, Villis JMC, Katzer JI. Criação e validação de um teste para medir o desempenho motor do nado crawl. Rev Bras Cineantropom Desempenho Hum. 2006[acesso em 08 mar 2022];8(3):73-8. Disponível em: Disponível em: https://pesquisa.bvsalud.org/portal/resource/pt/lil-469812
https://pesquisa.bvsalud.org/portal/reso...
, in their turn, found virtually perfect agreement in a study that designed and validated a motor performance test for crawl swimming. These findings are still similar to those of other studies that reported reliability ranging from moderate to virtually perfect99. Tucher G, Quintão GF, Garrido ND, Castro FAdS. Inter-and intra-rater reliability of swimming teachers with different skill levels, in different conditions, evaluating front crawl arm movement in non-expert swimmers. Motriz: Rev. Educ. Fis.. 2020;26. DOI: https://doi.org/10.1590/s1980-6574202000010025
https://doi.org/10.1590/s1980-6574202000...
),(2222. Gollegã DG, de Oliveira TAC, Freudenheim AM, Madureira F, Rodrigues HF, Dubas JP. Validação de um instrumento para avaliação qualitativa do nado “Crawl”. Rev. Bras. de Ed. Fis. e Esp. 2008;22(4):273-84. DOI: https://doi.org/10.1590/S1807-55092008000400004
https://doi.org/10.1590/S1807-5509200800...
. As much as comparisons can be made between these studies, it is important to stress that swimmers’ levels, evaluation forms, observation mode (live or video recording), as well as the experience and level of training of the rating professionals are different. In the end, all these differences make it difficult to compare results between studies.
Another study evaluating only the front crawl stroke obtained reliability results ranging from mild to substantial, passing through moderate99. Tucher G, Quintão GF, Garrido ND, Castro FAdS. Inter-and intra-rater reliability of swimming teachers with different skill levels, in different conditions, evaluating front crawl arm movement in non-expert swimmers. Motriz: Rev. Educ. Fis.. 2020;26. DOI: https://doi.org/10.1590/s1980-6574202000010025
https://doi.org/10.1590/s1980-6574202000...
. It is worth mentioning that the intra-rater reliability testing stage was carried out after a meeting between the raters and one of the researchers responsible for the research. Even with the meeting and a debate to choose the most relevant points of the evaluation form, and the greater academic training of one of the teachers, the intra-rater agreement was not high for all items, especially for teachers with less academic training. As in our study, the authors point out that the intervention did not generate the expected effect of standardization among the raters. This was evident because they continued to use their previously constructed personal knowledge to rate the swimmers99. Tucher G, Quintão GF, Garrido ND, Castro FAdS. Inter-and intra-rater reliability of swimming teachers with different skill levels, in different conditions, evaluating front crawl arm movement in non-expert swimmers. Motriz: Rev. Educ. Fis.. 2020;26. DOI: https://doi.org/10.1590/s1980-6574202000010025
https://doi.org/10.1590/s1980-6574202000...
.
Two other studies evaluating crawl swimming obtained similar intra-rater reliability, which was virtually perfect2121. Corazza ST, Pereira ÉF, Villis JMC, Katzer JI. Criação e validação de um teste para medir o desempenho motor do nado crawl. Rev Bras Cineantropom Desempenho Hum. 2006[acesso em 08 mar 2022];8(3):73-8. Disponível em: Disponível em: https://pesquisa.bvsalud.org/portal/resource/pt/lil-469812
https://pesquisa.bvsalud.org/portal/reso...
),(2222. Gollegã DG, de Oliveira TAC, Freudenheim AM, Madureira F, Rodrigues HF, Dubas JP. Validação de um instrumento para avaliação qualitativa do nado “Crawl”. Rev. Bras. de Ed. Fis. e Esp. 2008;22(4):273-84. DOI: https://doi.org/10.1590/S1807-55092008000400004
https://doi.org/10.1590/S1807-5509200800...
. These findings are close to some of the results of the present study, in which considerable intra-rater reliability was obtained in questions 1, 3 and 5 for rater B, and virtually perfect for rater C in questions 1, 3 and 4. However, in the study by Corazza et al.2121. Corazza ST, Pereira ÉF, Villis JMC, Katzer JI. Criação e validação de um teste para medir o desempenho motor do nado crawl. Rev Bras Cineantropom Desempenho Hum. 2006[acesso em 08 mar 2022];8(3):73-8. Disponível em: Disponível em: https://pesquisa.bvsalud.org/portal/resource/pt/lil-469812
https://pesquisa.bvsalud.org/portal/reso...
, the observation of the swimmers was repeated six days after the first observation and by means of direct observation of the swimmers in the pool. This short time interval between observations may allow some evaluation criteria to be kept in one’s memory and influence the second evaluation. Moreover, the direct observation of the swimmer may allow an adjustment in the position of the observer from their observation focus needs. The same does not happen when one watches the video, which has a fixed observation plane throughout the entire movement. Even in the current study, which had four fixed observation points in two different planes (lateral and frontal), the observation capacity is predetermined. Furthermore, initially, the swimmer is behind the observation plane allowed by the camera, then aligned in parallel and, finally, positioned ahead.
The study by Madureira et al.2222. Gollegã DG, de Oliveira TAC, Freudenheim AM, Madureira F, Rodrigues HF, Dubas JP. Validação de um instrumento para avaliação qualitativa do nado “Crawl”. Rev. Bras. de Ed. Fis. e Esp. 2008;22(4):273-84. DOI: https://doi.org/10.1590/S1807-55092008000400004
https://doi.org/10.1590/S1807-5509200800...
took into account only the errors that the executor can make in each crawl swimming stage. Such a strategy may have facilitated agreement at this stage, as the rater’s only function was to evaluate possible execution errors by the swimmers, which is easier to do than evaluating the movement as a whole and fitting it into one of the options highlighted in the evaluation form, which was the case in our study. As much as this facilitates the evaluation by the observers, we have to emphasize that there is no rigid, immutable technique for crawl swimming, but variations of what may be the ideal technique. Therefore, one should bear in mind the characteristics and limitations of the practitioner and allow something within the spectrum of that which is desirable for the crawl swimming technique1010. Vidal JdM, Tucher G, Nogueira L, Novaes RC, Vale RGdS, Castro MORd, et al. Crawl technique observation sheet for beginning swimmers: an evaluation proposal for swimming teachers. Motriz: Rev. Educ. Fis. 2021;27. DOI: https://doi.org/10.1590/S1980-65742021016920
https://doi.org/10.1590/S1980-6574202101...
.
The training and experience of the professionals participating in the different studies presented should be discussed as well. This research hypothesizes that coaches are more concerned with observing the movement from the perspective of its contribution and propulsive efficiency. On the other hand, swimming teachers keep their focus on observing the movement from the aesthetics of the technique2323. Soares S, Fernandes R, Carmo C, Santos Silva J, Vilas-Boas J. Avaliação qualitativa da técnica em Natação. Apreciação da consistência de resultados produzidos por avaliadores com experiência e formação similares. Rev. Port. de Ci. do Desp. 2001[acesso em 08 mar 2022];1(3):22-32. Disponível em: Disponível em: https://www.researchgate.net/profile/J-Paulo-Vilas-Boas/publication/307781939_Avaliacao_qualitativa_da_tecnica_em_Natacao_Apreciacao_da_consistencia_de_resultados_produzidos_por_avaliadores_com_experiencia_e_formacao_similares/links/5836d0db08aec3fe331e0d56/Avaliacao-qualitativa-da-tecnica-em-Natacao-Apreciacao-da-consistencia-de-resultados-produzidos-por-avaliadores-com-experiencia-e-formacao-similares.pdf
https://www.researchgate.net/profile/J-P...
. For instance, the results found showed that the teachers had greater intra-rater reliability in questions 1 and 5. The coaches, in questions 1, 3, 4 and 5. Whereas questions 1 and 5 are descriptors of movements executed out of the water and which are not propulsive, questions 3 and 4 refer to movements executed in the water and with a real influence on propulsion. Therefore, coaches seem to be concerned with the movements performed in and out of the water, since the technique is built with a propulsive purpose. In this way, they develop their own and spontaneous criteria that allow this observation. On the other hand, teachers seem to be more concerned with the movement performed outside the water, developing observation criteria which are more related in this sense.
In any case, movements executed in the water are difficult to be perceived by the raters, even in underwater images1010. Vidal JdM, Tucher G, Nogueira L, Novaes RC, Vale RGdS, Castro MORd, et al. Crawl technique observation sheet for beginning swimmers: an evaluation proposal for swimming teachers. Motriz: Rev. Educ. Fis. 2021;27. DOI: https://doi.org/10.1590/S1980-65742021016920
https://doi.org/10.1590/S1980-6574202101...
, as this is not a common practice for most of them. All this influences the ability of these professionals to observe, perceive and evaluate the movement and, consequently, the inter and intra-evaluator agreement in each study. Also noteworthy is the difficulty presented by both groups in the current study in identifying the catch moment, described in question 2. Although this moment is not propulsive, the position of the upper limbs is important, because it is from them that the propulsive action of the stroke begins2424. Toussaint HM, Hollander AP, Van den Berg C, Vorontsov A. Biomechanics of swimming.In: Garrett WE, Kirkendall DT, editors. Exercise and Sport Science Philadelphia: Lippincott, Williams & Wilkins; 2000. [acesso em 08 mar 2022]:639-60. Disponível em:Disponível em:https://www.researchgate.net/profile/Peter-Hollander-3/publication/313617617_Biomechanics_of_swimming/links/60718dc14585150fe998b00d/Biomechanics-of-swimming.pdf
https://www.researchgate.net/profile/Pet...
. Given the above, a suggestion for studies on this theme is for the intervention to maintain the proposal of presenting the swimming technique, but to allow for a discussion among the raters, who will expose their own observation and rating criteria. This initiative may allow their criteria to be known and go through personal reflection and systematization. As a result, these criteria are expected to be standardized and contribute to raising inter- and intra-rater reliability in studies on swimming technique evaluation. An additional recommendation is the conduction of studies on other components of the crawl swimming technique, such as the movement of both arms (not just the right limb), the kicking movement, trunk position, and breathing phases.
Conclusion
The intervention carried out with the raters in an attempt to standardize the observation and rating criteria did not have the desired effect on inter- and intra-rater agreement. The results found are in line with those of other studies in the literature, but there was a belief in the possibility of improving reliability. Actually, despite the discussions raised in the intervention, what seems to have prevailed during the evaluations were the raters’ prior experiences. Thus, it is important to propose an intervention with quality and sufficient criteria to enable greater standardization as to the criteria for swimming teachers with different training and professional experiences to observe and rate swimmers.
Referências
-
1da Silva LM, Giuliano AF, de Souza Castro FA. Ensino, aprendizagem e avaliação da técnica do eggbeater em aulas de natação. R. Bras. Ci. e Mov. 2016;24(2):138-45.. DOI: https://doi.org/10.18511/rbcm.v24i2.6187.
» https://doi.org/10.18511/rbcm.v24i2.6187 -
2de Oliveira TA, Apolinário MR, Freudenheim AM, Corrêa UC. Análise sistêmica do nado crawl. Bra. Journ. of Motor Behav. 2009;4(1):15-21. Acesso em: 05 de março de 2022. DOI: https://doi.org/10.20338/bjmb.v4i1.20
» https://doi.org/10.20338/bjmb.v4i1.20 -
3Silveira RP, de Souza Castro FA, Figueiredo P, Vilas-Boas JP, Zamparo P. The effects of leg kick on swimming speed and arm-stroke efficiency in the front crawl. Int J Sports Physiol Perform. 2017Jul;12(6):728-35. DOI: https://doi.org/10.1123/ijspp.2016-0232
» https://doi.org/10.1123/ijspp.2016-0232 -
4Aleixo I, Vieira M. Análise do Feedback na instrução do treinador no ensino da Ginástica Artística. Motric. 2012[acesso em 06 mar 2021];8(2):849-59. Disponível em: https://www.proquest.com/openview/dc051d5d2e9f856ba7df3a756f88e560/1?pq-origsite=gscholar&cbl=616555
» https://www.proquest.com/openview/dc051d5d2e9f856ba7df3a756f88e560/1?pq-origsite=gscholar&cbl=616555 -
5Federolf P, Reid R, Gilgien M, Haugen P, Smith G. The application of principal component analysis to quantify technique in sports. Scand J Med Sci Sports. 2014Jun;24(3):491-9. DOI: https://doi.org/10.1111/j.1600-0838.2012.01455.x
» https://doi.org/10.1111/j.1600-0838.2012.01455.x -
6Lees A. Technique analysis in sports: a critical review. J Sports Sci. 2002Oct;20(10):813-28. DOI: https://doi.org/10.1080/026404102320675657
» https://doi.org/10.1080/026404102320675657 -
7Rosado A, Virtuoso L, Mesquita I. Relação entre as competências de diagnóstico de erros das habilidades técnicas e a prescrição pedagógica no voleibol. Rev. Br. de Ed. Fis. Esp. 2004;18(2):151. DOI:https://doi.org/10.1590/S1807-55092004000200003
» https://doi.org/10.1590/S1807-55092004000200003 -
8Canossa S, Fernandes RJ, Carmo C, Andrade A, Soares SM. Ensino multidisciplinar em natação: reflexão metodológica e proposta de lista de verificação. MotriC.. 2007;3(4):82-99. DOI: https://doi.org/10.6063/motricidade.656
» https://doi.org/10.6063/motricidade.656 -
9Tucher G, Quintão GF, Garrido ND, Castro FAdS. Inter-and intra-rater reliability of swimming teachers with different skill levels, in different conditions, evaluating front crawl arm movement in non-expert swimmers. Motriz: Rev. Educ. Fis.. 2020;26. DOI: https://doi.org/10.1590/s1980-6574202000010025
» https://doi.org/10.1590/s1980-6574202000010025 -
10Vidal JdM, Tucher G, Nogueira L, Novaes RC, Vale RGdS, Castro MORd, et al. Crawl technique observation sheet for beginning swimmers: an evaluation proposal for swimming teachers. Motriz: Rev. Educ. Fis. 2021;27. DOI: https://doi.org/10.1590/S1980-65742021016920
» https://doi.org/10.1590/S1980-65742021016920 -
11Wizer R, Franken M, Castro F. Concordância intra e inter-observador de protocolo de avaliação de habilidades aquáticas de crianças. Rev Br. Ci. e Mov. 2016;24(1):101-7. DOI: https://doi.org/10.18511/rbcm.v24i1.5835
» https://doi.org/10.18511/rbcm.v24i1.5835 -
12Leandro C, Ávila-Carvalho L, Sierra-Palmeiro E, Bobo-Arce M. The evaluation rules in the view of the rhythmic gymnastics judges. Journ. of Sports. Sci. 2016;4:232-40. DOI: 10.17265/2332-7839/2016.04.007
» https://doi.org/10.17265/2332-7839/2016.04.007 -
13Currell K, Jeukendrup AE. Validity, reliability and sensitivity of measures of sporting performance. Sports Med. 2008;38(4):297-316. DOI: https://doi.org/10.2165/00007256-200838040-00003
» https://doi.org/10.2165/00007256-200838040-00003 -
14Peixoto JG, Dias AG, Miranda LM, Defilipo ÉC, Feitosa MB, Chagas PSdC. Análise de confiabilidade de medidas das pressões plantares estática e dinâmica de crianças e adolescentes com desenvolvimento normal. Fisioter e Pesqui. 2017;24:46-53. DOI: https://doi.org/10.1590/1809-2950/16222224012017
» https://doi.org/10.1590/1809-2950/16222224012017 -
15Zamparo P, Carrara S, Cesari P. Movement evaluation of front crawl swimming: Technical skill versus aesthetic quality. PLoS One. 2017;12(9):e0184171. DOI: https://doi.org/10.1371/journal.pone.0184171
» https://doi.org/10.1371/journal.pone.0184171 -
16Knudson DV, Knudson CS. Análise qualitativa do movimento humano. Barueri-SP: Editora Manole Ltda; 2001.
-
17Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977Mar;33(1):159-74. DOI: https://doi.org/10.2307/2529310
» https://doi.org/10.2307/2529310 -
18Lima VS, Azevedo NAdA, Guimarães JMX, Pereira MM, Neto JA, Souza LM, et al. Produção de vídeo-educacional: estratégia de formação docente para o ensino na saúde. Rev. Elet. de Com. 2019. acesso em 06 de março de 2022. DOI: 10.29397/reciis.v13i2.1594
» https://doi.org/10.29397/reciis.v13i2.1594 -
19Mogetti RS, Brod FAT, Lopes JLB. VIDEOAULA INTERATIVA COMO RECURSO DE ENSINO PARA A EDUCAÇÃO PROFISSIONAL A DISTÂNCIA. Redin- Rev. Edu. Interd.. 2019;8(1). acesso em 07 de março de 2022. Disponível em:Disponível em:http://seer.faccat.br/index.php/redin/article/view/1531
» http://seer.faccat.br/index.php/redin/article/view/1531 -
20Lauriti NC. A comunicação na avaliação de desempenho docente no ensino superior. EccoS-Rev. Cient. 2002;4(1):111-30. DOI: https://doi.org/10.5585/eccos.v4i1.296
» https://doi.org/10.5585/eccos.v4i1.296 -
21Corazza ST, Pereira ÉF, Villis JMC, Katzer JI. Criação e validação de um teste para medir o desempenho motor do nado crawl. Rev Bras Cineantropom Desempenho Hum. 2006[acesso em 08 mar 2022];8(3):73-8. Disponível em: Disponível em: https://pesquisa.bvsalud.org/portal/resource/pt/lil-469812
» https://pesquisa.bvsalud.org/portal/resource/pt/lil-469812 -
22Gollegã DG, de Oliveira TAC, Freudenheim AM, Madureira F, Rodrigues HF, Dubas JP. Validação de um instrumento para avaliação qualitativa do nado “Crawl”. Rev. Bras. de Ed. Fis. e Esp. 2008;22(4):273-84. DOI: https://doi.org/10.1590/S1807-55092008000400004
» https://doi.org/10.1590/S1807-55092008000400004 -
23Soares S, Fernandes R, Carmo C, Santos Silva J, Vilas-Boas J. Avaliação qualitativa da técnica em Natação. Apreciação da consistência de resultados produzidos por avaliadores com experiência e formação similares. Rev. Port. de Ci. do Desp. 2001[acesso em 08 mar 2022];1(3):22-32. Disponível em: Disponível em: https://www.researchgate.net/profile/J-Paulo-Vilas-Boas/publication/307781939_Avaliacao_qualitativa_da_tecnica_em_Natacao_Apreciacao_da_consistencia_de_resultados_produzidos_por_avaliadores_com_experiencia_e_formacao_similares/links/5836d0db08aec3fe331e0d56/Avaliacao-qualitativa-da-tecnica-em-Natacao-Apreciacao-da-consistencia-de-resultados-produzidos-por-avaliadores-com-experiencia-e-formacao-similares.pdf
» https://www.researchgate.net/profile/J-Paulo-Vilas-Boas/publication/307781939_Avaliacao_qualitativa_da_tecnica_em_Natacao_Apreciacao_da_consistencia_de_resultados_produzidos_por_avaliadores_com_experiencia_e_formacao_similares/links/5836d0db08aec3fe331e0d56/Avaliacao-qualitativa-da-tecnica-em-Natacao-Apreciacao-da-consistencia-de-resultados-produzidos-por-avaliadores-com-experiencia-e-formacao-similares.pdf -
24Toussaint HM, Hollander AP, Van den Berg C, Vorontsov A. Biomechanics of swimming.In: Garrett WE, Kirkendall DT, editors. Exercise and Sport Science Philadelphia: Lippincott, Williams & Wilkins; 2000. [acesso em 08 mar 2022]:639-60. Disponível em:Disponível em:https://www.researchgate.net/profile/Peter-Hollander-3/publication/313617617_Biomechanics_of_swimming/links/60718dc14585150fe998b00d/Biomechanics-of-swimming.pdf
» https://www.researchgate.net/profile/Peter-Hollander-3/publication/313617617_Biomechanics_of_swimming/links/60718dc14585150fe998b00d/Biomechanics-of-swimming.pdf
Edited by
Publication Dates
-
Publication in this collection
29 May 2023 -
Date of issue
2023
History
-
Received
30 Nov 2022 -
Reviewed
28 Mar 2023 -
Accepted
28 Mar 2023