Construct validation of educational technology for patients through the application of the Delphi technique

Abstracts

OBJETIVO: Descrever o processo de validação de uma tecnologia educativa para pacientes submetidos à cirurgia ortognática, mediante a aplicação da técnica Delphi. MÉTODOS: Participaram do estudo, dez juízes (quatro cirurgiões bucomaxilofaciais, duas enfermeiras, duas nutricionistas e duas fonoaudiólogas) que indicaram os assuntos a serem abordados no material destinado à primeira fase da técnica Delphi. Após a construção do material e a obtenção dos resultados da primeira fase, iniciou-se as segunda e terceira fases dessa técnica com validação da pertinência das informações contidas nessa tecnologia educativa. RESULTADOS: Na segunda rodada, obteve-se, pelo menos, 90,0% de concordância em todas as categorias. Na terceira, houve um aumento de 12% para o item conteúdo, melhora de 6,7% em linguagem, de 12,5% em ilustrações, de 3,3% no layout, de 10% em cultura, mantendo-se constante o item motivação. CONCLUSÃO: A técnica Delphi, pode ser uma importante ferramenta na construção e validação de tecnologias educativas.

Educação em saúde; Técnica Delfos; Prospecto para educação de pacientes; Estudos de validação

EXPERIENCE REPORT

Construct validation of educational technology for patients through the application of the Delphi technique^*

Validación de un constructo de tecnología educativa para pacientes mediante la aplicación de la técnica Delphi

Cristina Silva Sousa^I; Ruth Natalia Teresa Turrini^II

^IMaster of Science at Nursing School of USP. PhD student in the Post Graduation program PROESA of the Medical-Surgical Nursing Department of the Nursing School of USP

^IIPhD in Public Health. Professor in the Department of Medical-Surgical Nursing School of USP

Corresponding Author

ABSTRACT

OBJECTIVE: To describe the validation process of an educational technology for patients undergoing orthognathic surgery, through the application of the Delphi technique.

METHODS: Participating in the study were ten experts (four maxillofacial surgeons, two nurses, two nutritionists and two speech therapists) who indicated the subjects to be covered in the material for the first phase of the Delphi technique. After constructing the material and obtaining the results of the first phase, the second and third phases of this technique were initiated with validation of the pertinence of the information contained in this educational technology.

RESULTS: In the second round we obtained at least 90.0% agreement in all categories. In the third, the following improvements were noted: 12% for the content item, 6.7% for language, 12.5% for illustrations, 3.3% for layout, 10% for culture; the motivation item was unchaged.

CONCLUSION: The Delphi technique can be an important tool in the construction and validation of educational technologies.

Keywords: Health education; Delphi technique; Patient education handout; Validation studies

RESUMEN

OBJETIVO: Describir el proceso de validación de una tecnología educativa para pacientes sometidos a cirugía ortognática, mediante la aplicación de la técnica Delphi.

MÉTODOS: Participaron en el estudio, diez jueces (cuatro cirujanos bucomaxilofaciales, dos enfermeras, dos nutricionistas y dos fonoaudiólogas) que indicaron los asuntos a ser abordados en el material destinado a la primera fase de la técnica Delphi. Después de la construcción del material y la obtención de los resultados de la primera fase, se inició la segunda y tercera fases de esa técnica con validación de la pertinencia de las informaciones contenidas en esa tecnología educativa.

RESULTADOS: En la segunda vuelta, se obtuvo, por lo menos, el 90,0% de concordancia en todas las categorías. En la tercera, hubo un aumento del 12% para el item contenido, mejora del 6,7% en lenguaje, del 12,5% en ilustraciones, del 3,3% en el esbozo, del 10% en cultura, manteniéndose constante el item motivación.

CONCLUSIÓN: La técnica Delphi, puede ser una herramienta importante en la construcción y validación de tecnologías educativas.

Descriptores: Educación en salud; Técnica Delfos; Folleto informativo para pacientes; Estudios de validación

INTRODUCTION

Printed educational materials contribute favorably in the communication process, besides increase adherence to the treatment and decision-making power, as they provide consistent information, enable portability, flexibility, patient feedback and reinforce the instruction verbalized⁽¹⁾. In literature, there is a consensus on the verbal guidance with writing as more effective, which increases patient understanding and promotes better recovery⁽²⁾.

Normally, the creation of an educational material requires two types of research: one, thematic kind and other of diagnosis, in which you need to read what authors say on the subject, know the opinion of experts, or incorporate some of these experts in the production team. After that, you must choose the central ideas that will be addressed by the material, as well as the theme through which to seek to generate a learning experience. However, the opinion of experts or reading texts will not be sufficient, because it is also necessary to know the pedagogical contexts and especially the persons to whom the material is intended^(3).

A methodology that uses the Delphi technique may be useful in the development of educational materials with the participation of experts.

A technique called "Delphi" allows to consult a group of judges regarding future events, through a questionnaire, which is continuous times passed on until a convergence of responses is obtained, a consensus, which is the consolidation of intuitive judgment group^(4).

Based on opinions, points of agreement and disagreement among the experts, it is intended to reach some consensus elements. Unlike other research strategies, their goal is not to derive a simple answer or only reach a consensus, but getting answers and opinions of quality for a given question presented to the expert panel⁽⁵⁾.

It begun in the 50s, by researchers at the Rand Corporation, these scholars sought a method of predicting the future; when they found a method of feedback, they called as Delphi. It was based on the assumption that "two heads are better than one"; and was understood to have more opinions about a subject better guided decision-making^(6).

The technique remained forgotten for years, but it was retaken in new areas such as administration, social planning, research and education. In the 1990s, some work in nursing applied this technique⁽⁷⁾; the technique has essential characteristics that make it useful: anonymity (judges are not exposed and so feel free to provide feedback), feedback (responses and opinions of judges, guiding the researcher) and flexibility (freedom of schedules, time and place to judges respond to the questionnaire)^(6,8). In practice, this controlled feedback consists in referring to all participants in successive rounds, the information generated at each step⁽⁹⁾.

The Delphi process technique begins with the formulation of the problem and the choice of judges, followed by an initial questionnaire which acts as a strategy to generate ideas to find out what it says about the subject of the study (brainstorming). The feedback of the answers is offered in a second round and a new questionnaire is designed to gather opinions on what was raised. An analysis of the responses of judges and other rounds with the answers of these are performed in order to make them reflect and opine until the acceptable consensus on the given topic^(6,8,10).

The disadvantage is that multiple rounds can take months to complete, and requires engagement of the professionals involved in the response for not giving up among cycles of rounds.

For this work, the application of the Delphi technique aimed helping the construction of an educational material for patients undergoing orthognathic surgery and evaluate with the multidisciplinary team the relevance of information contained in this technology.

METHODS

In this article, the data presented are part of a thematic project that aimed to develop an educational technology for patients undergoing orthognathic surgery, approved by the Ethics Committee of the University of São Paulo Nursing School (Process #972/2010). For the participants who consented with the research a Statement of Informed Consent were obtained in two copies.

The methodological approach followed two steps: building an educational material for patients undergoing orthognathic surgery and validation of information contained with the help of judges.

For the Delphi technique were proposed three rounds of evaluation: the first consists of an open question for brainstorming, "what is important for the guidance of patient undergoing orthognathic surgery?"; the second, evaluation of educational materials built as to the coherence/pertinence and Illustration of the information; the third, final evaluation of the material after corrections performed based on the suggestions of the second round. The rounds of the Delphi technique used can be viewed in figure 1.

For each judge was mailed a tool for characterization of the group with the first issue of the Delphi technique and a letter of introduction of the educational material.

After the first round of the Delphi Technique, the educational material called "Orthognathic Surgery for Patients" was constructed from data obtained in relevant subprojects from the thematic project: literature review ⁽¹¹⁾, the results obtained from the focus group, searches in blogs and from the first round of the Delphi technique jointly to the judges. The program Microsoft Publisher^® was used for building material, written with plain language directed to the lay public (patients), following the recommendations for making educational material cited in the literature ^(12-14). The illustrations were done by a cartoonist for better understanding and adherence to reading.

The instrument developed for evaluations of the judges after the first round consisted of 22 evaluation items divided into six categories (content, language, illustration, layout, motivation and culture) with Likerttype responses, with five levels (strongly disagree, disagree, neither agree nor disagree, agree, strongly agree) and three essay questions that refer to errors or ideas harmful, lack of some information and comments.

The Likert scale items were assessed by proportions. The internal consistency evaluation of the instrument was made by Cronbach's alpha and agreement among the judges by intraclass correlation coefficient with a significance level of 5%, using the software Stathistical Package for Social Science.

RESULTS

The results for validation of the content of the educational material will be presented following the phases of development of the Delphi technique.

Characterization of the judges: The panel of judges was composed of ten healthcare professionals who treat patients undergoing orthognathic surgery (four maxillofacial surgeons, two nurses, two nutritionists and two speech-language pathologist). The age of the professionals ranged from 23 to 45 years; three professionals had 0-5 years of graduation, three of them ranged from 6 to 10 years, three from 11 to 15 years and only one above 20 years of graduation.

First round: This round was intended to know what the judges deemed important to inform the patient during procedures pertaining to the orthognathic surgery. The information provided related to guidelines in the pre-operative, intraoperative and postoperative, as listed below: documentation and surgical guide; risks inherent in the procedure; surgical time expected; need for additional procedures; absolute fasting; description of the type of anesthesia (general); anesthetic recovery; bleedings; paresthesia; facial edema; bed head raised to reduce edema and temporary hearing change; use of ice compresses; chewing and breathing difficulties; voice and postoperative muscle movements alterations; liquid diet; eating light meals for two months (without solid foods); standard chewing and swallowing; oral hygiene; use of elastic inter-occlusion; hematoma; pain; use of lip lubricant; use of nasal decongestants; risk of infection; outpatient follow-up of 30 days; break of expectation about aesthetics; head, body, lips and tongue posture; loss weight; nausea/vomiting/constipation; sun exposure after four months; continuity in the use of braces; not blowing the nose; communication strategies and speech enhancement; difficulty in acceptance from the family members; suture dehiscence; absence from work for a specified time.

Second round: The second round of the Delphi technique consisted of sending educational material built for judges to validate the material though of Likert-type instrument (results in Chart 1), and three open questions.

The analysis of open responses contributed to changes in the text. The datas of Chart 1 shows the result of the question about writing mistakes and changes.

The second question, concerning the lack of any information, brought up items related to drugs, multidisciplinary team and some suggestions for adding information or content review. The third question was about the reviews, leaving the judge free to contribute their experience.

The changes that were considered relevant by the researcher were made and a new educational material was printed, sent back to the judges, along with the assessment tool, giving rise to the third round of the Delphi technique.

Third round: The results obtained with the instrument were analyzed quantitatively.

Table 1

Table 2

In this round, the essay questions regarding the missing ideas or unclear were presented by only two judges and covered aspects related to provision of subtitles, excerpt key, grammar tenses and spelling (^{Chat 2}).

The proposed amendments did not affect the content of the material and had no impact on the layout. Thus, ended the validation in the third round of the Delphi technique.

DISCUSSION

In the first round, results achieved our objective and guided the researcher to construct the first proof of educational material. Oral hygiene was most recommended by judges (70%). From 44 items listed, 56.8% were mentioned only once.

In the second round, considering the items in each assessment category it was obtained 50 responses to content, 30 to language, 40 to illustrations is, layout to 60, 30 to motivation and 10 to culture, making a total of 220 responses.

For the highest level of the Likert scale, it was obtained "strongly agree" in 32 (64%) of the category of content responses, 15 (50%) in language, 25 (62.5%) in the illustrations, 37 (61.7%) in layout, 21 (70%) to motivation and 5 (50%) to culture. By aggregating the data level "agree" responses would reach at least 90.0% for all categories.

The internal consistency of the instrument in the second round of the Delphi technique was 0.953 and the intraclass correlation coefficient was 0.476 (p <0.000).

For the lowest percentage of "strongly agree" obtained in the language category, could be asked: would the judges think that the patient does not understand the writing? or do the judges have little knowledge about writing methods? With this, this item would be reassessed in the third round of the Delphi technique or later in the evaluation with the patient, along with the culture category for its relationship with language.

When evaluating ^{Chart 2}, it was perceived that the suggested changes resulted details that improve patient understanding and were accepted as being relevant to the content and bring greater clarity to the item.

In the second qualitative question, regarding the lack of information shall mean that the use of continuous medication must be reported to the professional during the preoperative period. Therefore, the observation made by the judge was inserted in excerpt "Preparation for surgery" as a warning to the patient as follows: "remember to tell the surgeon the medicines you are taking and illness you have."

The quotation of the multidisciplinary team was held at the beginning of the educational material through the phrase "post-operative, the professionals, nurse, nutritionists, speech-language pathologist may be present." The description of the surgical language has been improved, but technical terms are difficult to change to a lay language. It was decided to wait for the evaluation of the patient.

The third question about the comments, most of them showed satisfaction with the material presented. However, three comments presented critiques to the educational material, description related to diet, esthetic and text format.

The proposed diet is a generalized protocol, but some patients may not be able to perform it, and they followed their own planning to return to the normal diet. In case the patient presents an eating disorder or if their weight loss is significant, an appointment with a nutritionist is recommended.

The esthetic was related to the illustrations for not having the same style among the images. For the judge, the images should have the same colors and design features, which for the researcher is not interesting because it would leave the work monotonous and would not arouse the interest of the patient.

Another professional judged the material to be lengthy, but a large quantity of information is sometimes necessary, and line spacing facilitates reading material. Due to the reduced size of the paper and images need, the material would remain with 24 pages.

In the third round, an analysis by category shows that with regard to the content 38 (76%) "strongly agreed", an increase of 12% when compared to the second round of the Delphi technique. In language also improved by 6.7%, 12.5% in illustrations, 3.3% in layout, 10% of culture and motivation remained constant.

Internal consistency in the third round Delphi was 0.972 and the intraclass correlation coefficient was 0.601 (p<0.000). Despite maintaining the significance, it was observed in this round an improvement in the intra coefficient value.

This analysis by category showed that changes made to the first draft of the material provided significant changes to improve the material, according to the perspective of judges and the researcher.

Some suggestions were proposed by the judges, such as: placing text together to illustrations, highlight the search for a speech-language pathologist. These were seen as improvements and heeded by the researcher. The comments presented by the five judges were acknowledgments and congratulations for the work.

In this research, professional agreement with the educational material presented results growing between the second and third phase of the Delphi technique. The levels of the Likert scale, "agree" and "strongly agree" were respectively 30.9% and 68.6% in the third round, which together would represent 99.5% concordance with the built educational materials.

A study about construction of educational materials for women with mastectomies⁽¹⁵⁾ also considered the levels "agree" and "strongly agree" as similar, which results can be combined. Other evaluation studies of educational materials^(16-18) used different instruments, precluding its use for discussion of the present results.

Even without standardization of instruments for assessment of the educational materials and validation method, the construction of educational materials geared to the target audience, with the participation of experts, may raise acceptance and patient compliance in using this method for health education.

CONCLUSIONS

We noted that there is not a standardized instrument for assessing educational materials. Some studies apply

REFERENCES

• 1. Hoffmann T, Worrall L. Designing effective written health education materials: considerations for health professionals. Disabil Rehabil. 2004;26(19):1166-73.
• 2. Alexander RE. Patient understanding of postsurgical instruction forms. Oral Surg Oral Med Oral Pathol Oral Radiol Endod. 1999;87(2):153-8.
• 3. Kaplún G. [Educational material: a learning experience]. Comun Educ. 2008;9(27):46-60. Portuguese
• 4. Wright JTC, Giovinazzo RA. [Delphi - a support tool for the prospective planning ]. Cad Pesqui Adm. 2000;1(12):54-65. Portuguese.
• 5. Gupta UG, Clarke RE. Theory and applications of the Delphi technique: A bibliography (1975-1994). Technol Forecast Soc Change. 1996;53(2):185-211.
• 6. Goodman CM. The Delphi technique: a critique. J Adv Nurs.1987;12(6):729-34.
• 7. Mancussi e Faro AC. [The Delphi technique to validate the nursing interventions]. Rev Esc Enferm USP. 1997;31(2):259-73. Portuguese.
• 8. Sousa PA, Frade MH, Mendonça DM. [A model of organization and nursing exchange information: a delphi study]. Acta Paul Enferm. 2005;18(4):368-81. Portuguese
• 9. Moreno-Casbas T, Martin-Arribas C, Orts-Cortes I, Comet-Cortes P. Identification of priorities for nursing research in Spain: a Delphi study. J Adv Nurs. 2001;35(6):857-63.
• 10. Keeney S, Hasson F, McKenna HP. A critical review of the Delphi technique as a research methodology for nursing. Int J Nurs Stud. 2001;38(2):195-200.
• 11. Sousa CS, Turrini RN. Complications in orthognathic surgery: a comprehensive review. J Oral Maxillofac Surg Med Pathol.2012;24:67-74.
• 12. Houts PS, Doak CC, Doak LG, Loscalzo MJ. The role of pictures in improving health communication: a review of research on attention, comprehension, recall, and adherence. Patient Educ Couns. 2006;61(2):173-90.
• 13. Pierce LL. How to choose and develop written educational materials. Rehabil Nurs. 2010;35(3):99-105.
• 14. Aldrigde MD. Writing and designing readable patient education materials. Nephrol Nurs J. 2004;31(4):373-7.
• 15. Oliveira MS, Fernandes AF, Sawada NO. [Educational handbook for self care in women with mastectomies: a validation study]. Texto & Contexto Enferm. 2008;17(1):115-123. Portuguese
• 16. Reberte LM, Hoga LA, Gomes AL. Process of construction of an educational booklet for health promotion of pregnant women]. Rev Latinoam Enferm. 2012;20(1):101-8.
• 17. Gonçalves MB, Barbieri M, Gabrielloni MC. Teste de Papanicolau: construction and validation of an educacional booklet for users of healthcare services [Pap Test: http://redalyc.uaemex.mx/src/inicio/ArtPdfRed.jsp?iCve=84202003] Saúde Coletiva. 2008;5(20):39-44. Portuguese.
• 18. Fonseca LM, Scochi CG, Rocha SM, Leite AM. [Educational guideline for the maternal orientation concerning the care with preterm infants]. Rev Latinoam. Enferm. 2004;12(1):65-75. Portuguese.

*

Extraído da Dissertação de Mestrado "Educação pós-operatória: construção e validação de uma tecnologia educativa para pacientes submetidos à cirurgia ortognática", defendida em 2011 na Escola de Enfermagem da USP.

Publication Dates

• Publication in this collection
  04 Jan 2013
• Date of issue
  2012

History

• Received
  26 Feb 2012
• Accepted
  31 May 2012