Validation of the Objective Structured Assessment of Technical Skill in Brasil

Campos, Marcelo Esteves Chaves; Oliveira, Marcelo Magaldi Ribeiro de; Assis, Lilian Bambirra de; Reis, Augusto Barbosa; Gonçalves, Flávio Bambirra

doi:10.1590/1806-9282.66.3.328

SUMMARY

BACKGROUND

The aim of this study was to perform a cross-cultural adaptation of the Objective Structured Assessment of Technical Skill (OSATS) tool into Brazilian Portuguese and to determine its reproducibility and validity in Brasil.

METHODS

A Brazilian Portuguese version of OSATS was created through a process of translation, back-translation, expert panel evaluation, pilot testing, and then its validation. For the construct and the concurrent validities, twelve participants were divided into a group of six experts and six novices, who had to perform tasks on a simulation model using human placentas. Each participant was filmed, and two blinded raters would then evaluate their performance using the traditional subjective method and then the Brazilian Portuguese version of OSATS.

RESULTS

The Brazilian Portuguese version of OSATS had the face, content, construct, and concurrent validities achieved. The average experts’ score and standard deviations were 34 and 0.894, respectively, for Judge 1 and 34.33 and 0.816 for Judge 2. In the case of novices, it was 13.33 and 2.388 for Judge 1 and 13.33 and 3.204 for Judge 2. The concordance between the judges was evident, with the Correlation Coefficient (Pearson) of 0.9944 with CI 95% between 0.9797 and 0.9985, with p < 10-10, evidencing the excellent reproducibility of the instrument.

CONCLUSION

This preliminary study suggests that the Brazilian Portuguese version of OSATS can reliably and validly assess surgical skills in Brasil.

Educational measurement; Simulation; Training; Surgical procedures, operative/education; Surveys and questionnaires

[1] CORRESPONDING AUTHOR: Marcelo Esteves Chaves Campos Rua Congonhas 792 - 301 Santo Antônio – Belo Horizonte, Minas Gerais, Brasil, 30330-100 Tel.: 55 31 991967784 E-mail: camposmec@yahoo.com.br

Escala de Classificação Global de Instrumento de Avaliação Objetiva e Estruturada de Habilidades Técnicas Operatórias (Global Rating Scale of Objective Structured Assessment of Technical Skills)
Cuidados com o Tecido (Respect for tissue)	1. Utilizou frequentemente de força desnecessária sobre o tecido ou causou danos ao mesmo pelo uso inapropriado dos instrumentos. (Frequently used unnecessary force on tissue or caused damage by inappropriate use of instruments.)	2	3. Manipulou cuidadosamente o tecido, mas ocasionalmente, causou danos inadvertidos. (Careful handling of tissue but occasionally caused inadvertent damage.)	4	5. Consistentemente manipulou o tecido de forma apropriada, causando danos mínimos. (Consistently handled tissues appropriately with minimal damage.)
Economia de Tempo e Movimentos (Time and motion)	1. Muitos movimentos desnecessários. (Many unnecessary moves.)	2	3. Movimentos eficientes, mas alguns desnecessários. (Efficient time/motion but some unnecessary moves)	4	5. Evidente economia de movimentos e máxima eficiência. (Economy of movement and maximum efficiency.)
Manuseio dos Instrumentos (Instrument handling)	1. Constantemente faz movimentos hesitantes ou desajeitados com os instrumentos. (Repeatedly makes tentative or awkward moves with instruments.)	2	3. Uso competente dos instrumentos, embora, ocasionalmente, apresenta-se travado ou desajeitado. (Competent use of instruments although occasionally appeared stiff or awkward.)	4	5. Movimentos ajustados e fluidos com os instrumentos. (Fluid moves with instruments and no awkwardness.)
Conhecimento dos Instrumentos (Knowledge of instruments)	1. Frequentemente usou ou solicitou instrumentos inapropriados. (Frequently asked for the wrong instrument or used an inappropriate instrument.)	2	3. Conhecia o nome da maioria dos instrumentos e os utilizou adequadamente para a tarefa. (knew the names of most instruments and use appropriate instrument for the task.)	4	5. Evidentemente familiarizado com os instrumentos requisitados e com os seus respectivos nomes. (Obviously familiar with the instruments required and their names.)
Fluxo operatório e antecipação no planejamento cirúrgico (Flow of operation and forward planning)	1. Frequentemente interrompeu o procedimento operatório ou necessitou discutir sobre o próximo passo. (Frequently stopped operating or needed to discuss next move.)	2	3. Demonstrou capacidade de antecipação no planejamento operatório com progressão contínua do procedimento. (Demonstrate ability for forward planning with steady progression of operative procedure.)	4	5. Evidentemente planejou o curso da operação, sem esforços para avançar no passo-a-passo da cirurgia. (Obviously planned course of operation with effortless flow from one move to the next.)
Uso de Auxiliares (Use of assistants)	1. Consistentemente alocou mal os auxiliares ou falhou ao utilizá-los. (Consistently placed assistants poorly or failed to use assistants.)	2	3. Bom uso dos auxiliares na maior parte do tempo. (Good use of assistants most of the time.)	4	5. Utilizou os auxiliares estrategicamente, com o máximo proveito durante todo o tempo. (Strategically used assistant to the best advantage at all times.)
Conhecimento do Procedimento Operatório Específico (Knowledge of specific procedure)	1. Conhecimento deficiente. Necessitou de instrução específica na maioria dos passos operatórios. (Deficient knowledge. Needed specific instruction at most operative steps.)	2	3. Conhecia todos os aspectos importantes da operação. (Knew all important aspects of the operation.)	4	5. Demonstrou familiaridade em todos os aspectos da operação . (Demonstrate familiarity with all aspects of the operation.)