Construction and validation of an Educational Content Validation Instrument in Health

Construcción y validación de un Instrumento de Validación de Contenido Educativo en Salud

ABSTRACT

Objective:

to construct and validate the Educational Content Validation Instrument in Health.

Method:

methodological study that includes the establishment of the conceptual structure; definition of objectives and population; construction of items and response scale; selection and arrangement of items; instrument structuring; opinion of experts; pre-test and content validation.

Results:

an instrument with 15 items was constructed and, after expert evaluation, eighteen items were obtained, divided into three domains: objectives (four), structure/presentation (nine), and relevance (two). Six items were modified since they presented a percentage of agreement below 0.8. Items of the total instrument presented good internal consistency (0.877) regarding domains.

Conclusion:

an Educational Content Validation Instrument in Health was elaborated and validated, presenting good reliability, and may contribute to the practice of researchers and health professionals in the development of educational content.

Descriptors:
Methodological Research in Nursing; Health Education; Data Collection; Validation Studies; Nursing

RESUMEN

Objetivo:

construir y validar el Instrumento de Validación de Contenido Educativo en Salud.

Método:

estudio metodológico contemplando el establecimiento de la estructura conceptual; definición de los objetivos y población; construcción de los ítems y escala de respuesta; selección y organización de los ítems; estructura del instrumento; opinión de expertos; pre-test y validación de contenido.

Resultados:

se construyó un instrumento con quince ítems y después de la evaluación de expertos, se obtuvieron dieciocho ítems, divididos en tres áreas: objetivos (cuatro), estructura/presentación (nueve) y relevancia (dos). Seis ítems fueron modificados, pues presentaron un porcentaje de concordancia inferior a un 0,8. Los ítems del instrumento total presentaron una buena consistencia interna (0,877) en cuanto a sus dominios.

Conclusión:

fue elaborado y validado el Instrumento de Validación de Contenido Educativo en Salud, que presentó buena confiabilidad, contribuyendo a la práctica de investigadores y profesionales de las áreas de salud en la elaboración de contenidos educativos.

Descriptores:
Investigación Metodológica en Enfermería; Educación en Salud; Recolección de Datos; Estudios de Validación; Enfermería

RESUMO

Objetivo:

construir e validar o Instrumento de Validação de Conteúdo Educativo em Saúde.

Método:

estudo metodológico contemplando estabelecimento da estrutura conceitual; definição dos objetivos e população; construção dos itens e escala de resposta; seleção e organização dos itens; estruturação do instrumento; opinião de especialistas; pré-teste e validação de conteúdo.

Resultados:

construiu-se instrumento com quinze itens e após avaliação dos especialistas obtiveram-se dezoito itens, divididos em três domínios: objetivos (quatro), estrutura/apresentação (nove) e relevância (dois). Seis itens foram modificados, pois apresentaram percentual de concordância inferior a 0,8. Itens do instrumento total apresentaram boa consistência interna (0,877) quanto aos seus domínios.

Conclusão:

foi elaborado e validado Instrumento de Validação de Conteúdo Educativo em Saúde, que apresentou boa confiabilidade, podendo contribuir para a prática de pesquisadores e profissionais das áreas de saúde na elaboração de conteúdos educativos.

Descritores:
Pesquisa Metodológica em Enfermagem; Educação em Saúde; Coleta de Dados; Estudos de Validação; Enfermagem

INTRODUCTION

Materials with educational content are facilitators of the teaching-learning process, enabling the transfer of knowledge, through the individual’s participation and involvement, and the exchange of experiences conducive to the improvement of skills⁽¹⁾. Such materials have been widely used for health education, which is a vehicle of knowledge socialization for contributing to the improvement of living conditions and health of the population⁽²⁾.

Educational materials should be properly prepared and evaluated before being used by the target population. One of the essential steps for developing an effective educational material is the content validation, a process that evaluates its representativeness in properly addressing the universe with which it intends to deal and aims to measure or attend to the absence of unnecessary elements⁽³⁾.

Currently, measuring instruments have been used to aid the content validation, constituting tools that measure indicators and attribute numeric values to abstract concepts, which can be observable and measurable. Thus, they contribute to enhancing the health praxis⁽⁴⁾.

To develop tools that validate educational contents, an essential step is defining the construct to be investigated⁽⁵⁾. Among the aspects evaluated by such instruments, issues involving promotion, prevention, and recovery of health are frequent, such as quality of life, vitality and limitations, adherence to treatment, and emotional and psychosocial factors⁽⁶⁾.

However, there are recurring problems concerning the validation of instruments related to educational contents in health. It should be noted that inappropriate validation forms, without rigorous methodological criteria, the absence of instruments validated by capable professionals, as well as the fragmentation of educational instruments for health in topics or areas of interest, contribute to this process not always be effective, hindering the improvement and dissemination of knowledge about the phenomenon under study⁽⁷⁾. This may result in the provision of educational materials with mistaken technical and didactic-pedagogical aspects. Thus, it is imperative that tools to validate educational material contents in the health areas can evaluate any themes, featuring proper reliability and validity. This will avoid unnecessary adaptation of instruments that validate educational contents for new research contexts, in addition to optimizing the time of researchers in this stage of data collection.

In the scientific literature, one can observe the absence of a universal validation instrument for contents with educational purposes. Given this context, the construction and validation of a reliable instrument through a scientific basis becomes necessary to validate several contents in health.

OBJECTIVE

To construct and validate the Educational Content Validation Instrument in Health (ECVIH), to be used by researchers and health professionals in the orientation and development of educational content.

METHOD

Ethical aspects

The study development complied with national and international ethical standards of research with human beings. This research was approved by the Research Ethics Committee of the Federal University of Ceará (UFC). Experts who participated in this study were clarified as to the purposes of the research and the nature of data collection. Those who agreed in participate signed the Informed Consent Form (ICF).

Study design, location, and period

This is a methodological study that includes the establishment of the conceptual structure; definition of objectives and population; construction of items and response scale; selection and arrangement of items; instrument structuring; opinion of experts; pre-test, and content validation⁽⁴⁾.

For the construction of the instrument, a narrative review⁽⁸⁾ in the databases PubMed, Scopus, and Cinahl, and a theoretical framework based on the principles of the instructional design of Filatro⁽⁹⁾ was used.

The instrument construction and validation steps were carried out from March to December 2016. The sixth and seventh step, referring to the experts’ opinion and to pre-testing, were performed in person in the Laboratory of Communication and Health (Labcom_Saúde) of UFC. The last step, content validation, occurred virtually.

Population or sample, inclusion and exclusion criteria

A total of 27 experts participated in the study. Were selected, by trial, the experts who met the following inclusion criteria: Ph.D. and/or Master’s Degree in Health areas, professional experience (clinical, teaching, or research), and an article published in indexed journals in the study area of interest. Experts who have not responded to the invitation to participated in the study were excluded.

Study protocol

The process of instrument construction, on the first step, deals with the theoretical foundation of what will be studied for defining construct operations and dimensionality. Such step was based on the narrative review⁽⁸⁾ and in the principles of instructional design, of Filatro⁽⁹⁾.

The narrative review occurred in the databases PubMed, Scopus, and Cinahl, using the controlled keywords “Validation Studies” and “Teaching Materials”, and the non-controlled “Measuring Instruments”. The following crossings were performed: “Validation Studies” and “Teaching Materials”; “Validation Studies” and “Measuring Instruments”; “Teaching Materials” and “Measuring Instruments”. This step was based on the guiding question: which evaluation criteria were used to validate educational contents?

As inclusion criteria, we defined: research available in full and with free access in the selected databases, in English, Portuguese, or Spanish languages, which used instruments to evaluate the content in educational materials and were published from 2006 to 2016. Editorials, letters to the editor, reflective studies, experience reports, annals of scientific events (abstracts), and duplicate publications were excluded. Thus, 1,600 articles were found, and fifty were included in the research. However, after reading the full texts, 36 articles were excluded, totaling fourteen analyzed articles.

The examined literature showed that items of instruments used to evaluate educational material in health focus three groups of content-related factors: objectives, structure and presentation, and relevance. Items belonging to the group “objects” were related to purpose, goals, or target of the use of educational materials. The group “structure and presentation” contained information concerning the general organization, structure, strategy, consistency, and sufficiency of the presentations. In “relevance”, the items assessed the significance degree of the educational content and its ability to cause impact, motivation and/or interest.

For the theoretical support of the evaluated instruments, it was also used a theoretical reference based on the principles of instructional design, which is defined as the systematic actions of planning and construction of teaching methods and/or materials that can be used for classroom teaching, blended education, or distance learning. According to these principles, educational materials should provide interactivity, be attractive and written in an appropriate language to the target audience, promote relevant and contextualized activities, allow the exchange of experiences, and present quality information⁽⁹⁾. In this study, the items of the ECVIH sought to cover such instructions, which are regarded as essential for the development of materials for educational purposes.

Then, goals and population were defined. The ECVIH aims to provide a scientific basis for the content validation of educational materials in health. It is destined to higher-level health professionals who wish to construct and validate educational contents to any target audience.

Instrument construction refers to the preparation or selection of its content according to the construct operational definitions. The conceptual structure step allowed the construction of nineteen items, with response options by a Likert-type scale, being 0 = disagree, 1 = partially agree, and 2 = totally agree. Following, items were selected and organized to evaluate the relevance of the drafting, adopting the behavioral criterion, in addition to objectivity, simplicity, clarity, accuracy, validity, relevance, and interpretability⁽¹⁰⁾. After that, eighteen items remained.

Subsequently, the instrument was structured. This step aimed at consolidating the previous ones, namely, organizing the content to obtain a logical presentation order and thus establish the general format. Therefore, the ECVIH was structure by subdividing it into objectives, structure/presentation, and relevance. The last three steps were held with experts in the area of interest in measuring instrument, educational material, and health education.

Five experts in health, education, and statistics participated in the step of opinions and validation of the constructed items, to expand the diversity in the studied area of knowledge. All of them were Ph.Ds., had a scientific production and more than 10 years of experience in the area of interest of this study. The nominal group technique was adopted, being interactive, face-to-face, simultaneous, and held in group consensus⁽¹¹⁾.

Eleven experts participated in the instrument pre-testing, all of them having from three to five years of experience in the development of methodological studies in the area of health. Data collection happened only once, in and individual encounter, and came because of the discussion on instrument items and associated appendices.

The validation process counted with eleven experts – nine nurses and two pedagogues. The mean age was 38.9 ± 10,7 years. As for professional qualification, nine were Ph.Ds. and two had Master’s degree. We emphasize that tens experts were professors in Brazilian public universities (UFC, UECE, UNILAB, UNICAMP, and UFPI) and all had more than five years of scientific production in the study area. This process was made possible by electronic mail, via invitation letter and request for assessing the ECVIH regarding clarity, pertinence, relevance, and dimension in the writing of each item.

Analysis of results and statistics

Data were analyzed by the intraclass correlation coefficient (ICC), being regarded as great reliability (ICC > 0.9), good reliability (0.7 ≤ ICC ≤ 0.8), and poor reliability for values below 0.6⁽¹²⁾. The significance level adopted was 95%. An agreement percentage of 80% was considered as a criterion for deciding on the pertinence of the instrument items, or its modification.

RESULTS

The initially constructed instrument comprised eighteen items and, after the five-expert consensus, fifteen remained (Chart 1). It should be noted that, after pre-testing, there was no modification in the items devised. Eleven experts have then performed the content validation process for the instrument items, following the criteria: clarity, practical pertinence, theoretical relevance, and theoretical dimension (Table 1). Items that did not reach an 80% agreement percentage were reformulated.

Table 1
Distribution of items of the Educational Content Validation Instrument in Health (ECVIH) according to the percentage of expert agreement, Fortaleza, Ceará, Brazil, 2017

Items	CL*	PP*	TR*	TD*
	(%)	(%)	(%)	(%)
Objectives
Contemplates the proposed theme	100.0	100.0	100.00	100.00
Suits the teaching-learning process	90.9	90.9	90.9	90.9
Clarifies doubts on the addressed subject	90.9	81.8	90.9	81.8
Stimulates the learning about the addressed subject	81.8	100.0	100.0	81.8
Structure/presentation
Proper language	54.5	90.9	90.9	100.0
Interactive language	72.7	100.0	100.0	100.0
Correct information	72.7	90.9	100.0	100.0
Objective information	81.8	100.0	100.0	100.0
Understandable information	54.5	81.8	90.9	90.9
Logical sequence of ideas	81.8	90.9	100.0	100.0
Current thematic content	72.7	100.0	100.0	90.9
Appropriate text size	90.9	100.0	100.0	100.0
Content free of discrimination or prejudice	90.9	90.9	90.9	90.9
Relevance
Provides reflection on the subject	81.8	100.0	90.9	81.8
Encourages behavior change	81.8	90.9	81.8	72.7

• Note:
• ^*
  CL: clarity of language; PP: practical pertinence; TR: theoretical relevance; TD: theoretical dimension.

Chart 1
Original items, excluded, modified and/or maintained in the Educational Content Validation Instrument in Health (ECVIH) according to the experts' evaluations, Fortaleza, Ceará, Brazil, 2017

ECVIH* Itens	Evaluation
1. Contemplates the universe of the proposed theme	Modified
2. Has a direct relationship with the target audience	Excluded
3. Suits the teaching-learning process	Maintained
4. Clarifies doubts on the addressed subject	Maintained
5. Stimulates learning about the addressed subject	Maintained
6. Proper language	Maintained
7. Interactive language	Maintained
8. Correct information	Maintained
9. Clear information	Excluded
10. Objective information	Maintained
11. Understandable information	Maintained
12. Logical sequence of ideas	Maintained
13. Current content	Maintained
14. Appropriate text size	Maintained
15. Free of discrimination or prejudice	Maintained
16. Provides reflection on the subject	Maintained
17. Encourages behavior change	Maintained
18. Stresses the importance of the content	Excluded

• Note:
• ^*
  Educational Content Validation Instrument in Health.

In the reliability assessment, the ECVIH presented a satisfactory total internal consistency (ICC > 0.8) and a significant p-value (p < 0.05), similarly to the domains “structure/presentation” and “relevance”. However, in the domain “objectives”, a weak internal consistency was identified (ICC = 0.432) and p-value > 0.05, i.e. indicating the need for adjustments to increase domain reliability (Table 2).

Table 2
Analysis of the internal consistency of the domains analyzed in the Educational Content Validation Instrument in Health (ECVIH), Fortaleza, Ceará, Brazil, 2017

Domains	ICC*	IC95%	p value
Objectives	0.432	[-0.212; 0.819]	0.072
Structure and presentation	0.857	[0.702; 0.954]	<0.0001
Relevance	0.853	[0.672; 0.954]	<0.001
ECVIH** Total	0.877	[0.746; 0.960]	<0.0001

• Note:
• ^*
  ICC: Intraclass Correlation Coefficient;
• ^**
  ECVIH: Educational Content Validation Instrument in Health

The final configuration of the ECVIH has 18 items, divided into three areas: objectives, structure/presentation, and relevance. The total score of the instrument is calculated through the sum of all domains (Chart 2).

Chart 2
Educational Content Validation Instrument in Health, Fortaleza, Ceará, Brazil, 2017

OBJECTIVES: purposes, goals, or targets	0	1	2
1. Contemplates the proposed theme
2. Suits the teaching-learning process
3. Clarifies doubts on the addressed theme
4. Provides reflection on the theme
5. Encourages behavior change
STRUCTURE/PRESENTATION: organization, structure, strategy, consistency, and sufficiency	0	1	2
6. Language appropriate to the target audience
7. Language appropriate to the educational material
8. Interactive language, enabling active involvement in the educational process
9. Correct information
10. Objective information
11. Enlightening information
12. Necessary information
13. Logical sequence of ideas
14. Current theme
15. Appropriate text size
RELEVANCE: significance, impact, motivation, and interest	0	1	2
16. Encourages learning
17. Contributes to knowledge in the area
18. Arouses interest in the theme

• Note: Items valuation: 0 disagree; 1 partially agree; 2 totally agree.

DISCUSSION

Developing and validating the ECVIH contributes to the clinical and scientific praxis because this instrument is an innovative tool to be used to validate educational content available in materials such as videos, albums, booklets, games, websites, and software. Moreover, considering that it does not specify information about the theme, target audience, and application circumstances, the instrument can support educational activities in health.

The use of the nominal group technique was advantageous for the ECVIH construction since the consensus on the maintenance, modification, or exclusion of items much contributed for the construction of a more reliable instrument due to the evaluators’ expertise⁽¹¹⁾.

The prior use of the instrument in a small sample, characterizing a pre-test, allowed us to reach a preliminary version with more harmonic methodological choices.

For the ECVIH validation process, the experts evaluated three domains, namely: objectives, structure/presentation, and relevance. In the first domain, the modifications consisted in the inclusion of the items “provides reflection on the subject” and “encourages behavior changes”, as well as the offset of the item “stimulates the learning” to another domain. Although the five items belonging to this domain presented an agreement percentage of more than 80% among the experts in all criteria assessed, the researchers complied with the modifications suggested by the experts to improve the ECVIH items.

Educational materials evaluated by content instruments in the health area should aim at facilitating the work of health professionals in orienting and educating patients and family⁽¹³⁾. Furthermore, they must aim at promoting healthy, ensuring the effectiveness of care for the target population. Therefore, the domain “objective” of an instrument is required for the understanding on the content to be studied.

In the domain “structure/presentation,” the items complied with the criteria of practical pertinence and theoretical relevance and dimension, with an agreement of more than 80% among experts, requiring adjustments in five items to make them more understandable since they featured low agreement on the criterion language clarity. The main modifications were in relation to the items writing, such as replacing “proper language” for “language appropriated to the target audience” and “interactive language” for “Interactive language, enabling active involvement in the educational process”. We also decided to standardize nomenclatures, such as “thematic content” and “theme”.

Paying attention to the evaluation of educational materials in health is essential, through specific content instruments, especially regarding a consistent, cohesive, organized, sufficient structure, and an appropriate language. Regarding cohesion and coherence, the content must remain focused on the proposed theme, and in its topics, paragraphs, or portions there should be a logical sequence of ideas, starting from general and impersonal issues until more delicate or specific subjects. Sufficiency is one of the aspects closely related to material validity since sufficient educational contents explain relevant topics of the proposed theme in a comprehensive way, in addition to preventing doubts of the reader^(9,14).

Among the necessary aspects for the language of a good measuring instrument, objectivity and clarity are highlighted. Long and detailed sentences may lead to dispersion and make the reading tiresome, and an extensive instrument can compromise the willingness in replying it. Ambiguity caused by the use of vague terms is another potential problem related to the language, which may result in inadequate and inaccurate responses^(15-16). To avoid such problems, one of the aspects this study has prioritized was the use of a clear and direct language. The ECVIH items are objective, comprising solely the words required for understanding what is being measured. The size of the instrument also contributes to its fast and simple filling since it consists of only eighteen items. We also emphasize that each item measures only one attribute. Therefore, the conciseness is considered the strong point of this instrument.

Regarding “relevance”, the experts suggested significant changes in the items of this domain, such as the exchange of some items with the dimension “objective”, in addition to the inclusion of the items “contributes to knowledge in the area” and “arouses interest in the topic”.

For the content evaluated by health areas instruments to be current and relevant, it is essential that the researcher is inserted in the social context of the target population, knowing its needs and circumstances. Theme and topics covered in educational contents should meet the needs of the participant, promoting empowerment on the addressed subject. A previous study on education through a simple educational brochure on low-carbohydrate and low-sugar diet shows the increased knowledge on the theme after using the resource, in addition to emphasizing the importance of educational materials with this purpose⁽¹⁷⁾.

In the reliability assessment, the total ECVIH presented a good level of internal consistency, as well as the domains “structure/presentation” and “relevance”, and a weak level only for the domain “objective”. It is important to mention that, although the instrument present this weakness, the recommended modifications enabled the improvement of reliability in the domains “objective”, considering the experience of the experts in devising and validating the measuring instrument and in educational contents in the area of health.

Thus, the ECVIH points to several relevant aspects to educational content, which are essential to fulfill the purposes of this kind of material and provide education as efficiently as possible. It is therefore considered that the instrument was able to meet the purposes for which it was developed, being able to orientate the construction of educational contents in health, in different formats, since it was overall considered appropriate by the experts.

Study limitations

As a limitation, we emphasize that the instrument here constructed and validated aims solely to evaluate the written content of education materials, being excluded pictures, images, and drawings. It is understood that these items will be evaluated in the subsequent validation process of construct appearance.

Contributions to the field of nursing, health, or public policies

This study will contribute to the practice of researchers and health professionals in developing contents for educational materials that go beyond health and traditional education.

It is believed that the elaboration of a research instrument based on general standards may be the first step to enhance educational intervention in health, evaluating multi-professional phenomena in different health contexts.

CONCLUSION

This study constructed and validated the ECVIH, which presented a good reliability (ICC > 0.8) and value p < 0.05. The participation of specialists in the development and validation of this instrument was essential to avoid inaccurate results or biased measures that could lead to wrong conclusions. It should be noted that the evaluation of instruments in the field of health must be multi-professional, aiming to increase phenomenon objectivity.

Thus, the final configuration of the instrument has eighteen items, divided into three domains, with response options ranging from 0 to 2. We considered the ECVIH a valid and reliable alternative to evaluate educational contents in health.

It is expected that, in subsequent studies, the clinical validation will be performed, to evaluate the efficiency of the instrument here constructed and validated as a facilitator in obtaining data on educational materials in the health area.

• FUNDING
  The Ceará Foundation of Support to the Scientific and Technological Development allowed financial support to the research.

Publication Dates

• Publication in this collection
  2018

History

• Received
  21 Sept 2017
• Accepted
  27 Jan 2018