Development of an appearance validity instrument for educational technology in health

Desarrollo de un instrumento de validez de apariencia para tecnología de salud educativa

ABSTRACT

Objectives:

to develop and evaluate the convergence of the instrument for the appearance validity of educational technologies in health.

Methods:

methodological study conducted in two steps. In step 1, the instrument items were developed, with subsequent content validity by nine specialists in the development of educational technologies in health. In step 2, the convergent validity between another instrument and the appearance instrument was performed. Correlation results above r> 0.3 and p <0.05 were considered as plausible convergent validity.

Results:

the ten items of the initial version of the appearance instrument were submitted to content validity that resulted in a final version with 12 items (Content Validity Index = 0.93). The correlation indexes were strong with the objective and appearance domains; moderate with motivation, organization and total; and weak with writing style.

Conclusions:

the appearance instrument demonstrated content validity and convergent validity, in addition to a strong correlation with the other instrument.

Descriptors:
Validation Studies; Educational Technology; Technological Development; Methodological Research in Nursing; Health Education

RESUMEN

Objetivos:

desarrollar y evaluar la convergencia del instrumento de validez de apariencia para tecnologías educativas en salud.

Métodos:

estudio metodológico realizado en dos fases. En la fase 1, se desarrollaron los ítems del instrumento, con posterior validez de contenido por nueve especialistas en el desarrollo de tecnologías educativas en salud. En la fase 2, se realizó la validez convergente entre otro instrumento y el instrumento de apariencia. Los resultados de correlación superiores a r> 0.3 y p <0.05 se consideraron como validez convergente plausible.

Resultados:

los diez ítems de la versión inicial del instrumento de apariencia fueron sometidos a validez de contenido, que resultó en una versión final con 12 ítems (Índice de Validez de Contenido = 0.93). Los índices de correlación fueron fuertes con los dominios objetivo y de apariencia; moderado con motivación, organización y total; y débil con estilo de escritura.

Conclusiones:

el instrumento de apariencia demostró validez de contenido y validez convergente, además de una fuerte correlación con el otro instrumento.

Descriptores:
Estudios de Validez; Tecnología Educacional; Desarrollo Tecnológico; Investigación Metodológica en Enfermería; Educación para la Salud

RESUMO

Objetivos:

desenvolver e avaliar a convergência do instrumento para validação de aparência de tecnologias educacionais em saúde.

Métodos:

estudo metodológico realizado em duas fases. Na fase 1, foram elaborados os itens do instrumento, com posterior validação de conteúdo por nove especialistas em elaboração de tecnologias educacionais em saúde. Na fase 2, foi realizada a validade convergente entre outro instrumento e o instrumento de aparência. Os resultados de correlações acima de r>0,3 e p<0,05 foram considerados como validação de convergência plausível.

Resultados:

os dez itens da versão inicial do instrumento de aparência tiveram seu conteúdo validado, resultando em uma versão final com 12 itens (Índice de Validade de Conteúdo=0,93). Os índices de correlação foram fortes com os domínios objetivos e aparência; moderados com motivação, organização e total; e fracos com estilo de escrita.

Conclusões:

o instrumento de aparência demonstrou validade de conteúdo e convergência, além de forte correlação com o outro instrumento.

Descritores:
Estudos de Validação; Tecnologia Educacional; Desenvolvimento Tecnológico; Pesquisa Metodológica em Enfermagem; Educação em Saúde

INTRODUCTION

Nurses are attentive to the current population health panorama and propose the development of technologies such as booklets for the prevention of childhood diarrhea⁽¹⁾, prevention of metabolic syndrome for adolescents⁽²⁾, educational practices on complementary feeding⁽³⁾, phototherapy manual for guidance of family members of jaundiced newborns⁽⁴⁾, among others. In this scenario, educational technology is an important tool that helps health care by promoting innovation and interaction⁽⁵⁾, favoring the exchange of experiences and the search for healthy behaviors by daily users of health services.

The creation and use of educational technologies demands thinking about their construction, the purpose of their use, their users and their prior validation⁽⁵⁾. Validation is the process performed using assessment instruments to check the presence of the validity attribute. It is designated as a hypothetical attribute that the technology will be able to perform the task that it proposes to perform⁽⁶⁾. There are different types of validity; of construct, content, appearance, criterion, predictive and concurrent, among others^(6-7). The main types applied for the validation of educational technologies are content and appearance validity. Content validity is the representation of a sample of the content universe⁽⁷⁾ that the educational technology must contain.

Appearance validity is the aesthetic representation comprising lines, shapes, colors and movement of the images that must harmonize with the information content. This has been used in the validation of educational technologies based on non-specific instruments. The creation of a specific instrument for the appearance validity of educational health technologies is a gap in the knowledge of nursing and the health area.

The appearance validity of technologies created by nurses and other health professionals has often been done by items contained within content validity instruments^(1-4,8-10). The absence of specific instruments for appearance validity has led to the use of instruments that do not involve the constitutive features of appearance.

The relevance of the appearance validity of educational technologies lies in how pictures can facilitate the understanding of messages. Theorists in the field point out that illustrations can persuade readers to read the educational material, increase their attention by 43%, stimulate emotions, avoid distractions by anticipating previous experiences on the topic and direct readers to the main information of the message⁽¹¹⁾. The information above shows the relevance of the appearance validity of educational materials and the need to use a specific instrument for this function. The use of a specific appearance validity instrument may optimize the effectiveness of the educational technology with the target audience.

OBJECTIVES

To develop and evaluate the convergence of the instrument for the appearance validity of educational technologies in health (Portuguese acronym: IVATES).

METHODS

Ethical aspects

The study was approved by the Research Ethics Committee of the Universidade Estadual do Ceará under number 723.860.

Study design, period and location

Development and validation study of a measuring instrument in which the GRRAS tool (Guidelines for Reporting Reliability and Agreement Studies) was used to report the results. The phase of instrument development was between July and October 2014 in Fortaleza-Ceará, Brazil. The validation phases were between December 2017 and July 2018 using online forms.

Population, inclusion and exclusion criteria

Twenty-four experts participated in the study. Initially, nine specialists in the development and validation of educational technologies in health were invited to validate the content of IVATES. The criteria for choosing experts at this step was to have a doctorate with a thesis on the development and validation of educational technologies, experience in the development of some type of educational technology and conducting research in this area.

Other 15 specialists applied the IVATES to an educational technology with the purpose of generating data for the criterion validation of the proposed instrument. In this step, criteria for inclusion of specialists were having a doctorate or master’s, one year of clinical practice, research and articles published in the area of educational technology used as a model for the application of IVATES. Researchers who modified the research area in the five prior years were excluded.

Study protocol

The methodological study was conducted in two steps. Step 1 was the development of the instrument and its content validity. For the development of the instrument, a reference framework that deals with the development of educational materials for teaching people with low literacy was used⁽¹¹⁾. The theoretical dimension of the latent trait “appearance of educational technologies in health” was defined, with the construction of constitutive and item definitions. The theoretical dimensionality was based on the search for understanding the concept of appearance as a latent trait to be measured. To this end, the definition of appearance adopted had constitutive features that could be addressed in a measuring instrument. After defining the theoretical dimensionality, the constitutive definitions of the latent trait were constructed. In this step, the constituents of the definition adopted were addressed by drawing a parallel with the adopted framework. Such framework describes the characteristics of illustrations with emphasis on aspects related to colors, shapes and relevance for understanding the information, association with the daily life of the target audience, quantity and size of figures and harmonization with the text, which favors the exposure of the theme.

Still in step 1, the constitutive definitions were drafted and addressed in operational terms. The items of the instrument were prepared, composing an initial version, and exposed to content validity by a panel of nine specialists in the development and validation of educational technologies in health. Criteria for choosing experts at this step were having a doctorate with a thesis on the development and validation of educational technologies, experience in the development of some type of educational technology and conducting research in this area. To perform the validation, an invitation letter explaining the activities conducted was sent to the specialists, together with a characterization questionnaire, the items for content validity of the appearance validity instrument, and the informed consent form.

The experts analyzed the items and indicated if each was appropriate to assess the appearance of educational technologies. At the end, they expressed their agreement through an adjective scale of five response options: 1 = strongly disagree, 2 = disagree, 3 = partially disagree, 4 = agree and 5 = strongly agree. These results were analyzed using the Content Validity Index⁽¹²⁾. After content validity, a version with the items and the five-point adjective scale was composed.

Step 2 comprised the convergent validity of the proposed instrument, in which an instrument presents a convergent correspondence with some instrument that measures a latent trait theoretically in the same direction as the proposed test⁽⁷⁾. In lack of a specific instrument for appearance validity and by knowing there are items for evaluating appearance in some content assessment instruments, it has been postulated that appearance validity is in the same direction as content validity. The convergent validity was tested based on the correspondence between the scores obtained between the appearance validity instrument and a content validity instrument.

To operationalize this step, 15 experts who did not participate in step 1 applied both instruments to an educational technology⁽¹³⁾. The criteria for choosing these specialists considered the experience and qualification of the committee members⁽⁶⁾. Specialists were given an invitation letter, a document with instructions, the educational technology, a characterization questionnaire, the content validity instrument⁽¹⁴⁾ and the appearance validity instrument for educational materials.

The content validity instrument used was the “Suitability Assessment of Materials” version adapted to Brazilian Portuguese, which assesses educational technologies. This instrument consists of 30 items with a 0-2 score scale distributed in six domains: objectives, organization, writing style, appearance, motivation and cultural appropriateness⁽¹⁴⁾. For this study, the cultural appropriateness domain was not considered because the educational technology evaluated was developed for the Brazilian context.

Analysis of results and statistics

To show content validity, the Content Validity Index (CVI) was calculated and items marked 4 or 5 by the specialists were considered positive. Items with CVI ≥ 0.78 and total instrument CVI ≥ 0.90 were considered valid according to the following standard: CVI ≥ 0.78 excellent; CVI between 0.60 and 0.77 good; and CVI <0.59 poor⁽¹²⁾.

To show the convergent validity of the appearance validity instrument for educational materials, results were summarized by a score based on the sum of responses of the five-point adjective scale. The convergence technique used was based on the correlation between the scores of two scales with contents directed to the same theoretical meaning⁽⁶⁾. Pearson’s correlation calculation was performed between the sum of scores of the content validity instrument and the scores of the appearance validity instrument. Correlations were made between the total scores and the domains of the content validity instrument, and the score of the appearance validity instrument. Results of correlations above r> 0.3 and p <0.05 were considered as plausible convergent validity⁽¹⁵⁾.

RESULTS

The ten items had their contents validated by a panel of specialists in the development of educational technologies in health. Nine specialists participated in this validation, almost all (88%) were female, physicians, nurses, 88% were aged between 31 and 40 years, with an average training time of 13 years. All conducted research in the area of educational technology development. Table 1 shows the results of the CVI.

Table 1
Distribution of items according to the Content Validity Index of the Instrument for appearance validity of educational technology in health, Fortaleza, Ceará, Brazil, 2018

Items	CVI*
1. Illustrations are suitable for the target audience.	1.0
2. Illustrations are clear and easy to understand.	1.0
3. Illustrations are relevant for the content understanding by the target audience.	1.0
4. The colors and shapes of illustrations are suitable for the type of material.	0.88
5. Illustrations depict the daily life of the target audience of the intervention.	0.88
6. The layout of figures is in harmony with the text.	0.88
7. The pictures used are related to the text of education technology in health and elucidate the content.	0.77
8. Illustrations help to expose the theme and are in a logical sequence.	1.0
9. Illustrations are in appropriate quantity and size to the proposed technology.	1.0
10. Illustrations help to change the behavior and attitudes of the target audience.	0.88
Total CVI*	0.93

• Note:
• ^*
  CVI - Content Validity Index.

In content validity, experts requested changes in items 4, 7 and 9 of the Instrument. The suggestion in item 4 was to separate the color and shape indicators; in item 7, was suggested that the text for “the figures used elucidate the content of the educational material” was rewritten; in item 9, was suggested the separation of quantity and size indicators. Thus, the content validity resulted in a panel of 12 items.

The convergent validity included 15 specialists, almost all (86.6%) were female and nurses (93.3%), more than half (60%) were aged between 30 and 40 years, almost all (93.3%) conducted research on the theme, 53.3% of specialists were doctors and 46.7% masters.

Table 2 shows that IVATES showed convergence by total scores and by domain scores with the content validity instrument. The IVATES correlation indexes were strong with the objective and appearance domains; correlation was moderate with motivation, organization and total domains; and considered statistically significant (p<0.05). It presented a weak correlation with the writing style domain, although all of them reached the criterion adopted of r above 0.3.

Table 2
Convergent validity of the Instrument for appearance validity of educational technology in health, Fortaleza, Ceará, Brazil, 2014

CI domains*	IVATES†	95% CI‡	p value§
Objectives	0.825§	[0.240; 0.939]	<0.001
Organization	0.448§	[-0.157; 0.834]	0.095
Writing style	0.311§	[-0.329; 0.882]	0.260
Appearance	0.807§	[0.663; 0.965]	<0.001
Motivation	0.505§	[-0.398; 0.792]	0.005
Total	0.655§	[0.064; 0.889]	0.008

• Note:
• ^*
  CI - Content Instrument⁽¹⁴⁾;
• ^†
  IVATES - Instrument for Appearance Validation of Educational Technology in Health;
• ^‡
  CI - Confidence Interval;
• ^§
  Pearson's correlation.

Chart 2 presents the final version of IVATES, validated and ready to be used in future studies in the field of nursing and health in general.

Chart 1
Items of the Instrument for appearance validity of educational technology in health, Fortaleza, Ceará, Brazil, 2018

Theoretical dimension	Constitutive definition	Operational definitions	Items
The appearance shows the reality of the objects as they are shown to people who will make the judgment and is related to what is revealed as the essence of objects(16).	The illustration is a figurative image, abstract or not, followed by an explanatory character with the aim to add information, synthesize, decorate or visually represent a text(17).	The style of the picture allows that readers recognize and identify with the subject and it facilitates people's learning.	1. Illustrations are suitable for the target audience. 2. Illustrations are clear and easy to understand. 3. Illustrations are relevant for the content understanding by the target audience. 4. The colors and shapes of illustrations are suitable for the type of material.
		The family scenario helps in understanding the message.	5. Illustrations depict the daily life of the target audience of the intervention.
		Procedural messages with pictures of behavior in small procedures make the performance of the action easier.	6. The layout of the figures is in harmony with the text. 7. The pictures used are related to the text of education technology in health and elucidate the content. 8. Illustrations help to expose the theme and are in a logical sequence. 9. Illustrations are in appropriate quantity and size to the proposed technology. 10. Illustrations help to change the behavior and attitudes of the target audience.

Chart 2
Final validated version of the Instrument for appearance validity of educational technology in health, Fortaleza, Ceará, Brazil, 2014

Items	1	2	3	4	5
	Strongly disagree	Disagree	Partially disagree	Agree	Strongly agree
1. Illustrations are suitable for the target audience.
2. Illustrations are clear and easy to understand.
3. Illustrations are relevant for the content understanding by the target audience.
4. The colors of illustrations are suitable for the type of material.
5. The shapes of illustrations are suitable for the type of material.
6. Illustrations depict the daily life of the target audience of the intervention.
7. The layout of figures is in harmony with the text.
8. The pictures used elucidate the content of the educational material.
9. Illustrations help to expose the theme and are in a logical sequence.
10. Illustrations are in appropriate quantity in the educational material.
11. Illustrations are in appropriate size in the educational material.
12. Illustrations help to change the behavior and attitudes of the target audience.

Considering the correspondence between the convergent validity between instruments, a appearance validity index (FVI) is proposed based on the method of estimating the content validity index (CVI)⁽¹²⁾. On a 5-point adjective scale (1=strongly disagree; 2=disagree; 3=partially disagree; 4=agree; 5=strongly agree), the FVI for each item (FVI-I) is computed by the number of specialists who answered 4 or 5 divided by the total number of specialists. For the total FVI (FVI-T), the sum of FVI-I is divided by the total of items.

The item with FVI> 0.78 is considered excellent; between 0.60 and 0.77 indicates the need for adaptation to improve the appearance of the educational technology in health; items with FVI<0.60 are classified as poor and the material must be redeveloped from the key point of the item⁽¹²⁾. For example, FVI-I=0.50 in item 1 indicates that the material must be redeveloped to adapt the illustrations to the target audience. This procedure will impact on improvement of the FVI-T that should be> 0.90⁽¹²⁾.

DISCUSSION

The proposed IVATES brings the appearance elements represented by its items. The development of this appearance instrument was based on an assumption of harmonization of the elements constituting the educational technology in health, such as shapes, colors, images, text, quantity and sizes of figures that can provide greater efficiency of the technology⁽¹¹⁾.

A broad-looking concept with several constitutive features was found, which made it difficult to develop a clearly one-dimensional instrument accommodating all items. Then, the analysis of the concept turned to demarcation of the constitutive definition in terms of illustration by understanding that it represents the essence of the object studied.

The representation of the latent trait “appearance of educational technologies in health” through the concept of illustration enabled the construction of twelve items that hypothetically include the principle of unidimensionality⁽⁶⁾. In this sense, the content validity results of the IVATES showed coherence between the developed items and the theory representing the latent trait.

The content validity of the IVATES showed all items with excellent CVI, suggesting that the instrument is representative of appearance. The items on clear, relevant illustrations that help in exposing the theme, arranged in a logical sequence, adequate quantity and sizes, which helps the understanding by the target audience, obtained CVI equal to one. These results are in agreement with the foundations of the development of educational materials for people with low literacy, which postulate on elements necessary for the understanding of the key points of the message, since the primary function of the visual resource is to playfully inform simple messages that are often difficult to be addressed in health education actions⁽¹¹⁾.

The items addressing the colors and shapes of illustrations that portray the daily life, the layout of figures and the effect of illustrations in changing the behaviors and attitudes of the target audience had CVI equal to 0.88. In this perspective, the use of illustrations that bring the characters of educational technology closer to the real life context where it is supposed to intervene facilitates the understanding of the message⁽¹¹⁾. A study that developed educational technology in the format of a manual for patients with head and neck cancer undergoing radiation therapy showed the important role played by figures in communication, through the use of photos obtained in patients’ own treatment environment, representing the real scenario of this population⁽⁸⁾.

Only one item received a good CVI, less than 0.78. It addresses the relationship of the picture with the text of the educational material, which helps to elucidate the content. The difficulty of representing this relationship between text and image may explain the lower CVI of this item. Health messages of hard understanding are more easily transmitted with the use of pictures, because these involve a greater amount of sense organs than text presentation using only written or even oral language without the visual resource⁽¹¹⁾. In a study that built and validated an educational technology in venous ulcer care, was used the strategy of mixing fully colored pictures that needed to be highlighted, and black and white pictures with only some elements that needed greater emphasis in color⁽⁹⁾.

IVATES also showed convergence with the total scores and the objective, organization, writing style, appearance and motivation domains of the content validity instrument. The IVATES scores showed a moderate total correlation index with the content validity instrument, thereby showing the convergent validity of the proposed instrument. The absence of an appearance validity instrument that could be used as a gold standard in comparison with IVATES led to the choice of the content validity instrument for such a comparison. This choice was due to the relationship between content and appearance concepts and the representation of both in the development of educational technologies, as already demonstrated in other studies^(1-2,8-9).

Convergent validity is often considered a challenge for researchers, as a gold standard measure is required for comparisons with a chosen instrument, which is not found in all areas of knowledge. There is also the possibility of investigating if the evaluated instrument has a strong correlation with other existing and validated measures⁽¹⁸⁾. Regarding the domains of content scores to which IVATES was tested for convergence, the objective domain showed a strong correlation. This domain of content validity is composed of items evaluating the transmission of the educational technology message globally. IVATES assesses the transmission of the health message through illustrations, thereby explaining the strong correlation found.

The appearance domain of the content validity instrument showed a strong correlation with IVATES. This was already expected, as this domain consists of items that assess characteristics contained in IVATES. On the other hand, the motivation and organization domains showed a moderate correlation. The first assesses the extent to which educational technology content stimulates interaction and the second encompasses the organization of educational technology, the logical sequencing of topics and the coherence of textual and image information. Such characteristics are intrinsically related to the appearance of educational technologies and require accurate attention in the development of these materials.

Regarding the writing style domain, the correlation with IVATES was weak. This result may have occurred because the textual pattern of educational technology involving the level and clarity of the message are not objects of analysis of the appearance instrument. A study developed an orientation manual for oropharyngeal dysphagia home care with application of a convergent validity analytical technique similar to that of this work. A strong correlation between the appearance scores and the objective, structure, writing style, appearance and motivation domains was found⁽¹⁹⁾. These results clarify the relationship between the degree of adequacy of the theoretical content of constructs with the manifested appearance of the educational technology.

Encompassing the components of the latent trait “appearance of educational technologies in health” is paramount in the development of these technologies, with a view to greater effectiveness of the health education implemented. In this sense, the application of IVATES for the appearance validity elucidates how appropriate the development of educational technology was in terms of appearance and directs improvements in the material that strengthen the persuasion characteristics and improve the interface between the technology and the target audience.

Study limitations

The limitation of the study was the lack of an appearance validity instrument that could be used as gold standard. However, the analytical technique adopted demonstrated that IVATES contributed as an option of appearance validity instrument.

Contributions to the Area

In the construction process of educational technologies conducted by nurses, the developed instrument qualifies the educational products and materials used in health education practices at the collective level or in individual care. They need to undergo a validation phase, and it is important to use accurate instruments to measure the presence of essential characteristics that will impact on their effects. IVATES is the first specific instrument addressing the process of appearance validity of educational technologies in which the visual resource is used as a facilitating element in the health learning process.

CONCLUSIONS

The IVATES was built based on a framework that deals with the development of materials for teaching people with low literacy. It showed theoretical correspondence with the adopted postulates of appearance, was based on the concepts of appearance and illustration, and resulted in an instrument with 12 items in an adjective scale of five points. It demonstrated internal validity through content validity with excellent CVI and external validity through convergent validity with strong and positive correlations with the content validity instrument and its domains.

The use of IVATES will allow new interpretative horizons in the development of educational technologies in health. Its results can be used as aids in their validity and by redirecting their development in order to promote greater effectiveness in the communication of educational health technologies with the target audience. The use of IVATES is expected to contribute to the validation of technologies that facilitate health education processes. In addition, other studies may be conducted in search of the demarcation of the IVATES construct validity, especially by applying multivariate techniques that demarcate dimensionality such as Exploratory and Confirmatory Factor Analysis.

REFERENCES

• ¹
  Sabino LMM, Ferreira AMV, Joventino ES, Lima FET, Penha JC, Lima KF, et al. Elaboration and validation of a reader on childhood diarrhea prevention. Acta Paul Enferm. 2018;31(3):233-9. doi: 10.1590/1982-0194201800034
  » https://doi.org/10.1590/1982-0194201800034
• ²
  Moura IH, Silva AFR, Rocha AESH, Lima LHO, Moreira TMM, Silva ARV. Construction and validation of educational materials for the prevention of metabolic syndrome in adolescents. Rev. Latino-Am. Enfermagem. 2017;25:e2934. doi: 10.1590/1518-8345.2024.2934
  » https://doi.org/10.1590/1518-8345.2024.2934
• ³
  Viana LR, Barreto MM, Girard CCP, Teixeira E. Tecnologia educacional para mediar práticas educativas sobre alimentação complementar na Amazônia: estudo de validação. RISTI. 2018;28:29-40. doi:10.17013/risti.28.29-40
  » https://doi.org/10.17013/risti.28.29-40
• ⁴
  Jesus EB, Esteves AVF, Teixeira E, Medeiros HP, Nascimento MHM, Saboia VM. Validation of educational technology on phototherapy to guide family members of icteric neonates. Rev Enferm UERJ. 2018;l(26):e21789. doi: 10.12957/reuerj.2018.21789
  » https://doi.org/10.12957/reuerj.2018.21789
• ⁵
  Cardoso RSS, Sá SPC, Domingos AM, Sabóia VM, Maia TN, Padilha JMFO, et al. Tecnologia educacional: um instrumento dinamizador do cuidado com idosos. Rev Bras Enferm. 2018;71(Suppl 2):786-92. doi: 10.1590/0034-7167-2017-0129
  » https://doi.org/10.1590/0034-7167-2017-0129
• ⁶
  Boateng GO, Neilands TB, Frongillo EA, Melgar-Quiñonez HR and Young SL. Best practices for developing and validating scales for health, social, and behavioral research: a primer front. Public Health. 2018;6:149. doi: 10.3389/fpubh.2018.00149
  » https://doi.org/10.3389/fpubh.2018.00149
• ⁷
  Medeiros RKS, Ferreira JMA, Pinto DPSR, Vitor AF, Santos VEP, Barichello E. Modelo de validação de conteúdo de Pasquali nas pesquisas em Enfermagem. Rev Enferm Ref. 2015;(4):127-35. doi: 10.12707/RIV14009
  » https://doi.org/10.12707/RIV14009
• ⁸
  Cruz FOAM, Ferreira EB, Vasques CI, Mata LRF, Reis PED. Validation of an educative manual for patients with head and neck cancer submitted to radiation therapy. Rev Latino-Am Enfermagem. 2016;24:e2706. doi:10.1590/1518-8345.0949.2706
  » https://doi.org/10.1590/1518-8345.0949.2706
• ⁹
  Benevides JL, Coutinho JFV, Pascoal LC, Joventino ES, Martins MC, Gubert FA, et al. Development and validation of educational technology for venous ulcer care. Rev Esc Enferm USP. 2016;50(2):309-16. doi: 10.1590/S0080-623420160000200018
  » https://doi.org/10.1590/S0080-623420160000200018
• ¹⁰
  Wild CF, Nietsche EA, Salbego C, Teixeira E, Favero NB. Validation of educational booklet: an educational technology in dengue prevention. Rev Bras Enferm. 2019;72(5):1318-25. doi: 10.1590/0034-7167-2018-0771
  » https://doi.org/10.1590/0034-7167-2018-0771
• ¹¹
  Doak C, Doak L, Root J. Teaching patients with low literacy skills. Philadelphia: JB Lippincott; 1996 [2018 Nov 02]. Available from: https://www.hsph.harvard.edu/healthliteracy/resources/teaching-patients-with-low-literacy-skills/
  » https://www.hsph.harvard.edu/healthliteracy/resources/teaching-patients-with-low-literacy-skills/
• ¹²
  Polit DF, Beck CT, Owen SV. Is the CVI an acceptable indicator of content validity? appraisal and recommendations. Res Nurs Health. 2007;30(4):459-67. doi: 10.1002/nur.20199
  » https://doi.org/10.1002/nur.20199
• ¹³
  Souza ACC, Moreira TMM, Oliveira ES, Menezes AVB, Loureiro AMO, Silva CBA, et al. Effectiveness of educational technology in promoting quality of life and treatment adherence in hypertensive people. PLoS One. 2016;11(11):e0165311. doi: 10.1371/journal.pone.0165311
  » https://doi.org/10.1371/journal.pone.0165311
• ¹⁴
  Sousa CS, Turrini RNT, Poveda VB. Tradução e adaptação do instrumento “Suitability, assessment of materials”(SAM) para o português. Rev Enferm UFPE. 2015;9(5):7854-61. doi: 10.5205/r euol.6121-57155-1-ED.0905201515
  » https://doi.org/10.5205/r euol.6121-57155-1-ED.0905201515
• ¹⁵
  Neri AL, Borim FSA, Batistoni SST, Cachioni M, Rabelo DF, Fontes AP, et al. Nova validação semântico-cultural e estudo psicométrico da CASP-19 em adultos e idosos brasileiros. Cad Saúde Pública. 2018;34(10):e00181417. doi: 10.1590/0102-311x00181417
  » https://doi.org/10.1590/0102-311x00181417
• ¹⁶
  Abbagnano N. Dicionário de Filosofia. 6 ed. São Paulo: Martins Fontes ; 2012 [2020 Mar 01]. Available from: https://www.estantevirtual.com.br/livros/nicola-abbagnano/dicionario-de-filosofia/2750838412
  » https://www.estantevirtual.com.br/livros/nicola-abbagnano/dicionario-de-filosofia/2750838412
• ¹⁷
  Maurício CHS. Ilustrar-me: ilustração como meio de descoberta da expressão emocional [Dissertação] [Internet]. Aveiro (PT): Universidade de Aveiro; 2014 [2018 Nov 02]. Available from: https://ria.ua.pt/bitstream/10773/14437/1/Tese.pdf
  » https://ria.ua.pt/bitstream/10773/14437/1/Tese.pdf
• ¹⁸
  Souza AC, Alexandre NMC, Guirardello EB. Psychometric properties in instruments evaluation of reliability and validity. Epidemiol Serv Saúde. 2017;26(3):649-59. doi: 10.5123/s1679-49742017000300022
  » https://doi.org/10.5123/s1679-49742017000300022
• ¹⁹
  Pereira KFPO, Pereira AS, Zeigelboim BS, Santos RS. Attention to oropharingeal dysfunction in home care: speech therapy management. appearance and content validation study of a guidance manual. Rev CEFAC. 2018;20(5):640-47. doi:10.1590/1982-021620182052918
  » https://doi.org/10.1590/1982-021620182052918

Edited by

EDITOR IN CHIEF: Dulce Barbosa

ASSOCIATE EDITOR: Maria Elisabete Salvador Graziosi

Publication Dates

• Publication in this collection
  21 Dec 2020
• Date of issue
  2020

History

• Received
  03 Oct 2019
• Accepted
  23 Apr 2020