Development, judgment of the validity and reliability of an instrument of assessment of Oral Health Literacy among diabetics

Desenvolvimento, julgamento da validade e confiabilidade de um instrumento de avaliação da Alfabetização em Saúde Bucal entre diabéticos

ABSTRACT

Objective

The study purposed to perform content validation and verification of the reliability of an instrument in the form of a questionnaire, called “ Oral Health Literacy for Diabetics “. It was designed to investigate whether there was access, understanding, appraise and practice of information related to oral health among diabetics.

Methods

This is a methodological research that was carried out through the application of the Oral Health Literacy for Diabetics between 109 diabetic patients, since at least 60 participants should be considered in studies using test/retest. The research complied with the ethical principles of research. The content validation was performed by dentists, acknowledging the relevance and the ability to measure the levels of literacy of each item of the Oral Health Literacy for Diabetics. The reliability/reproducibility was estimated by the test/retest in an interval of seven to fifteen days by Kappa, using the SPSS^®.

Results

The results showed that the content of the Oral Health Literacy for Diabetics presented relevance and ability to measure the levels of literacy in oral health among diabetics. The results of the Kappa ranged from -0.09 to 1. Only 16 of the 150 questions did not have satisfactory levels of agreement, i.e., Kappa smaller or equal to 0.60. We chose to synthesize the Oral Health Literacy for Diabetics initially with 150 questions, for a version with 30 questions.

Conclusion

It is concluded that the Oral Health Literacy for Diabetics was considered valid with respect to the content and that a general form its reliability was satisfactory. It is recommended to use the Oral Health Literacy for Diabetics in academia and in health services aiming to improve the quality of life of diabetic patients.

Indexing terms
Data accuracy; Diabetes mellitus; Oral health; Reproducibility of results

RESUMO

Objetivo

Conduzir a validação de conteúdo e a verificação da confiabilidade de um instrumento em forma de questionário, denominado “Alfabetização em Saúde Bucal entre diabéticos” que foi idealizado para investigar se houve acesso, compreensão, avaliação e aplicação de informações relacionados à saúde bucal entre diabéticos.

Métodos

Trata-se de uma pesquisa metodológica que foi realizada através da aplicação do Instrumento entre 109 participantes, uma vez que no mínimo 60 participantes devem ser considerados em estudos teste/reteste. A investigação respeitou os princípios éticos da pesquisa. A avaliação de conteúdo foi feita por cirurgiões-dentistas, que verificaram a relevância e a capacidade de mensuração dos níveis de alfabetização do Instrumento. A confiabilidade/reprodutibilidade foi estimada pelo teste/reteste em um intervalo de três a sete dias pelo Kappa, utilizando o SPSS^®.

Resultados

A validade de conteúdo do Instrumento evidenciou capacidade do instrumento em mensurar os níveis de alfabetização em saúde bucal entre diabéticos. O Kappa variou de -0,09 a 1. Das 150 perguntas somente 16 não apresentaram níveis de concordância satisfatórios, ou seja, Kappa ? 0,60. Optou-se por condensar o Alfabetização em Saúde Bucal inicialmente com 150 perguntas, para uma versão com 30 perguntas.

Conclusão

do Alfabetização em Saúde Bucal entre diabéticos foi considerado válido quanto ao conteúdo e de forma geral a sua confiabilidade foi satisfatória. Recomenda-se o uso do Instrumento na academia e nos serviços de saúde visando melhorar a qualidade de vida dos diabéticos.

Termos de indexação
Confiabilidade dos dados; Diabetes Mellitus; Saúde bucal; Reprodutibilidade dos testes

INTRODUCTION

Access to health can be assessed from four different perspectives: availability, acceptability, payment capacity, and information; indicating the complexity of the concept of access. This concept has changed over time: in the 1970s, access referred to geographical (availability) and financial (ability to pay) aspects. However, the literature began to address cultural, educational and socioeconomic aspects. Thus, it became evident that information forms the basis of access to health, promoting the empowerment of people when taking health decisions through Health Literacy (HL). Improving access to health and guaranteeing greater equity depends on intersectorial actions and social and economic policies that allow the dissipation of differences in income and education [¹]. Considering this context, in 2007, Solar & Irwin developed an “action-oriented” health care model to identify the social determinants of health inequities and suggest action proposals. The determinants considered as most important for health are the so-called structural, because structural and social rank differences cause unjust iniquities in health [²]. There is a need to address health demands in an expanded way, promoting interventions that have an effect on the determinants of human health conditions in order to minimize inequities.

Non-communicable chronic diseases (NCD), a global public health problem, have a costly impact, especially for low- and middle-income countries. It is estimated that, in 2007, 72% of deaths in Brazil were secondary to NCD. Industrialization growth, rapid demographic transition, and changes in nutritional habits have contributed to an increase in people’s weight, increasing the risk for NCD [³,⁴]. For Diabetes Mellitus (DM), one of the main NCD related to the morbimortality of the population, there is a clear trend towards its increased prevalence as well as financial and social impact [⁴-⁷]. A report by the Pan American Health Organization (PAHO) presented a Regional Oral Health Plan for the Americas. The document recognizes that oral health throughout the Americas is critical. It evidenced the association of oral health with risk factors for NCD and the capacity of health care providers to implement effective interventions for preventing disease. Goals were defined for the integration of oral health in Primary Health Care (PHC), as well as strategies to amplify access to health care. The importance of multidisciplinary approaches including the private sector, academic institutions and civil society in order to improve oral health and general health in the Americas was emphasized. Certain recommendations should be considered: oral health should be recognized as important for general health as well as a factor that directly affects life quality of the population; countries should continue to strengthen capacity of workers regarding primary health care [⁸].

Oral health is multifaceted, interfering with the ability to speak, smile, chew, swallow and may be related to the sense smell and taste, as well as transmission of emotions. It may reflect the physiological, social, and psychological attributes essential to life quality of people [⁹]. Dental practice was modified due to the complexity of oral health, with emphasis on health promotion and disease prevention, as well as curative care. Healthcare promotion and diseases prevention have brought results that indicate improvement of oral health conditions, as they promote joint actions and face the demands for healthcare in a more integral manner [¹⁰]. The evaluation of Integrated Diabetes Management Program of Mexico recognizes the importance of HL in empowering people [¹¹]. Regarding the oral health conditions of Brazilians, the national epidemiological surveys carried out by the Ministry of Health in 1986, 1996, 2003 and 2010 show improvements [¹²]. These conditions can be further improved, and health promotion and preventive actions are required, which propose the development of personal skills related to self-care, through the dissemination of information and the increase of HL levels, aiming to improve the ability of people to exercise greater control over their own health, making them able to make choices that lead to a better quality of life [¹³]. There is evidence of a link between satisfactory socioeconomic conditions and oral health [¹²]. Actions to promote oral health and to prevent oral diseases should consider, whenever necessary, seek an increase HL levels on people.

HL involves people’s knowledge, motivation and skills to access, to understand, to appraise and to apply health information, making daily judgment and decision-making about health and disease prevention possible, aiming to maintain or improve the quality of life [¹⁴]. The term functional literateness in health designates the ability to use reading and writing for practical purposes on daily life in health: to know the causes of precarious conditions of oral health; adopt self-care measures; communicate with dental providers; put your name on waiting lists for dental treatment; find the way to the dental clinic; fill in the forms; return in follow-up visits; and follow prescription medication. In this perspective, HL differs from health literateness and is considered broader since it covers the domains of appraisal and understanding of information [¹⁵], in addition to functional literateness in health. The instruments that measure health literateness have focused only on word recognition and reading ability [¹⁶-²¹]. The translation, cross-cultural adaptation to the Portuguese language of Brazil [¹⁵] of an instrument originally in English [¹⁶] was conducted, it was verified that the instrument presented satisfactory reliability [¹⁵].

Despite the availability of instruments for the measurement of oral health literateness, a proper instrument to appraise the levels of literacy in oral health considering access to understanding, appraisal and application of information [¹⁴] related to oral health was not found. Thus, it is necessary to develop and judge the content validity and reliability of an instrument to appraise oral health literacy. The validation of a data collection instrument is essential, since health planning decisions are made based on research results obtained through data collection instruments. Otherwise, there is a great risk of inaccurate or biased results, which may lead to inadequate conclusions [²²]. The question arises: what characteristics of an evaluation instrument on conditions or events related to health should be considered in the judgment of its quality?

In a Delphi study, the researchers consulted fifty-seven experts, blindly, to clarify which features should be considered in judging the quality of data collection instruments in the health area; the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) was obtained as product. According to the COSMIN checklist, for a data collection instrument to be considered appropriate, its psychometric properties must be analyzed, that is, one must measure the confidence, validity and responsiveness of this instrument. The degree of confidence is analyzed based on three categories: (1) internal consistency; (2) reproducibility; and (3) measure of error. Internal consistency is estimated by Cronbach’s alpha, reproducibility and error measurement are measured based on testing and retesting. The validity of an instrument is investigated when three criteria are analyzed: (1) content validity; (2) criterion validity, subdivided into concurrent validity and predictive validity; and (3) construct validity, subdivided into structural validity, hypothesis testing and cross-cultural validity [²²] (figure 1). The purpose of this work was the creation and presentation of an instrument to appraise oral health literacy, in addition to verifying its content validity and its reliability / reproducibility.

Figure 1
Domains assessed to determine the quality and validity of Health Related-Patient Reported Outcomes.

METHODS

The study is the product of a project that evaluates oral health literacy among diabetic users of two Primary Health Care Units (PHCU) of the Family Health Strategy (FHS) of Montes Claros / MG. Inclusion criteria were: to be over 18 years old; have being diagnosed with Diabetes Mellitus by a doctor; if elderly (60 years of age or older), do not present cognitive impairment in the Mini Mental State Examination (MMSE); those of exclusion were: to have acute decompensation informed by the FHS team; diagnosis of four concurrent comorbidities; not be found at home after three attempts. The Mini-mental score, according to the validated version for Brazil, took into account the interviewees’ level of education in the identification of those who presented cognitive problems. Those who scored below those set for the following study years were excluded because they were considered as having cognitive deficits: among illiterates, scores lower than 21; among those with low education level (≤ 5 years), scores lower than 22; among those with average education level (6-11 years), score lower than 23 and among those with high education level (≥ 12 years), score lower than 24 [²³].

The judgment of the instrument was conducted as proposed by the – Check List COSMIN (CL-COSMIN) [²²]. The validity verification of content was done by dental professionals who evaluated the relevance of the items contained in the instrument, making sure that they were able to measure literacy in oral health. To evaluate reliability, as reproducibility [²²], 109 diabetics were invited to participate in the process, since this type of analysis does not necessarily require the estimation of probabilistic samples [²⁴]. The retest was applied seven to fifteen days after the first application, that is, the test. The interviews were conducted by a previously trained interviewer to follow the application instructions evenly to the others interviews. Statistical analysis was carried out with the aid of SPSS (version 24.0, SPSS INC., Chicago, Illinois, USA). The test-retest reliability for each variable was calculated using the simple Kappa coefficient, considering values between 0.6 and 0.8 as acceptable and above 0.8 as close to perfect [²⁵].

This study was approved by the Research Ethics Committee of the State University of Montes Claros, CONEP number 764.743.

RESULTS

Participant characterization

The proposed sample consisted of 109 people; however, after losses, it was reduced to 101 people (Answer Rate = 92.67%). Among the 101 people diabetics, 85 (84.15%) were female. The mean age was 54.9 years (SD = 9.97), the minimum age was 29 and the maximum 77 years. Schooling ranged from 0 to 12 years or more of study (mean = 5.63, SD = 3.99). The average family income was R$ 2,216.73 (SD = 1,854.13, IC95% = 1,995.62-2,465.55), minimum R$ 0.00 and maximum R$ 12,000.00. Schooling ranged from 0 to 12 years or more of study (mean = 7.55, SD = 4.34, 95% CI = 7.03-8.05).

Description of the proposed Oral Health Literacy for Diabetics (OHL-D) Assessment Tool

The instrument is divided into five parts for different purposes. The first one, composed of ten questions, inquires whether there was access, understanding, appraisal and application of information related to oral health. Generally, the frequency of access, and how comprehension, appraisal and application of this information occurred (table 1) are also verified. The second part proposes to assess which professionals or individuals (doctor, nursing team, dentist/dental team, health agent and others) are involved in the transmission of information. The third part aims to identify which oral health subjects people could access (cavities, canal treatment, gingival/gingival bleeding problems, dental mobility/bone loss, oral hygiene, trauma, oral cancer, dental prosthesis, implant and orthodontic appliance). The fourth part aims to determine if the interviewee had read information about oral health (dental prescriptions, package leaflets for dental products, pre and postoperative guidelines, newspaper, magazine, poster, billboard, flyer/folder/booklet, internet and others). The fifth part aims to identify if the interviewee listened to and/or watched information about oral health and in what media (educational videos, television, radio, telephone, internet, movies/cinemas and others). In addition, the instrument proposes to investigate how often people access, read, listen to and/or watch information about oral health, if any, and how understanding, appraisal, and application of such information occurred (table 2).

Table 1
Oral Health Literacy for Diabetics (OHL-D) Assessment Tool with Kappa values corresponding to each of its variables.

1. Have you received any information about oral health?						1
2. When was the last time you received oral health information?						0,755
3. How often do you receive information about oral health?						0,823
4. Did you understand the information you have already received about oral health?						0,717
5. Can you appraise whether the information you have already received about oral health is true or false?						0,827
6. Can you rank in more or less important the information you have already received about oral health?						0,805
7. Can you identify the information quality you have already received about quality oral health?						0,726
8. Can you appraise the advantages and disadvantages of different dental treatments considering the information you have already received about oral health?						0,730
9. Do you put into practice in your daily life the information you have received about Oral Health?						0,798
10. Do you maintain a proper behavior, considering the information you have received on Oral Health?						0,714
From which health professional(s) have you received information about oral health?
	Yes/No	Last time	Understanding	Appraisal	Practice
Doctor	0,901	0,780	0,470	1	0,669
Nurse/Crew	0,955	0,821	-0,176	0,615	0,528
Dentist/Dental Team	0,957	0,828	0,805	0,772	0,773
Community Health Agent	0,901	0,899	-0,090	0,555	0,647
Other? Who?	0,970	0,823	0,727	0,823	0,666
What issues were considered when you received information about oral health?
	Yes/No	Last Time	Understanding	Appraisal	Practice
Caries	0,980	0,689	0,823	0,753	0,787
Dental Canal treatment	0,960	0,626	0,786	0,834	0,771
Gingival problems / Gingival bleeding	0,970	0,771	0,774	0,797	0,772
Bone loss around teeth / Dental mobility	0,980	0,746	0,416	0,538	0,508
Dental trauma	0,950	0,746	0,863	1	0,400
Bruxism / clenching	0,960	0,576	0,750	0,777	0,708
Oral cancer	0,950	0,722	0,777	0,882	0,760
Dental prosthesis	0,960	0,782	0,742	0,739	0,623
Dental implants	0,970	0,768	0,739	0,733	0,884
Orthodontic treatment	0,930	0,832	0,502	0,729	0,785
Other?	0,980	1	1	1	0,499
Have you read information about oral health in printed or electronic materials?
	Yes/No	Last Time	Understanding	Appraisal	Practice
Dental recipes	0,930	0,627	0,850	0,907	0,945
Package leaflets for medicinal products / Oral Rinse and / or Toothpaste Labels	0,930	0,723	0,521	0,488	0,672
Pre-and / or post-dental care guidelines	0,910	0,762	0,793	0,878	0,757
Poster	0,960	0,798	0,681	0,777	0,683
Reviews	0,920	0,829	0,865	0,625	0,766
Newspapers	0,970	0,869	0,624	0,625	0,857
Flyer / Folder / Booklet	0,924	0,853	0,485	0,504	0,808
Internet	0,960	0,750	0,646	0,778	0,729
Other?	0,980
Have you ever watched and/or heard about oral health information in print or electronic materials?
	Yes/No	Last Time	Understanding	Appraisal	Practice
Educational video	0,910	0,660	0,710	0,710	0,799
Television	0,901	0,755	0,820	0,813	0,903
Radio	0,950	0,781	1	1	0,877
Internet	0,960	1	1	1	0,400
Lecture	0,950	0,818	0,817	0,798	0,810
Movie / cinema	0,940	0,684	0,744	0,912	0,823
Other, which?	0,980	1	1	1	1

Table 2
Oral Health Literacy for Diabetics (OHL-D) Assessment Tool.

1. Have you received any information about oral health?		1	0 No 1 Yes
2. When was the last time you received oral health information?		2	0 Never received information 1More than 2 years ago 2In the last two years 3In the last year	4 In the last 6 months 5In the last month 99No answer/ did not know
3. How often do you receive information about oral health?		3	0 Never received information 1Interval bigger than 2 years 2from 2 to 2 years 3Once each year	4 Once in the last 6 months 5Once in the last month 99No answer/ did not know
4. Did you understand the information you have already received about oral health?		4	0 Never received information 1Did not understand 2Understood poorly 3Understood partially	4 Understood almost everything 5Understood everything 99No answer/ did not know
5. Can you appraise whether the information you have already received about oral health is true or false?		5	0 Never received information 1Unable 2Able with major difficulty 3Able with some difficulty	4 Able with minor difficulty 5Easily able 99No answer/ did not know
6. Can you rank in more or less important the information you have already received about oral health?		6	0 Never received information 1Unable 2Able with major difficulty 3Able with some difficulty	4 Able with minor difficulty 5Easily able 99No answer/ did not know
7. Can you identify the information quality you have already received about quality oral health?		7	0 Never received information 1Unable 2Able with major difficulty 3Able with some difficulty	4 Able with minor difficulty 5Easily able 99No answer/ did not know
8. Can you appraise the advantages and disadvantages of different dental treatments considering the information you have already received about oral health?		8	0 Never received information 1Unable 2Able with major difficulty 3Able with some difficulty	4 Able with minor difficulty 5Easily able 99No answer/Did not know
9. Do you put into practice in your daily life the information you have received about Oral Health?		9	0 Never received information 1Never 2Seldom 3Sometimes	4 Frequently 5Always 99No answer/ did not know
10. Do you maintain a proper behavior, considering the information you have received on Oral Health?		10	0 Never received information 1Never 2Seldom 3Sometimes	4 Frequently 5Always 99No answer/ did not know
11. From which health professional(s) have you received information about oral health?	12. When was the last time you received information about oral health from this\these professional(s)?	13. Did you understand the information you received about oral health from the(se) professional(s)?	14. Can you appraise the quality of the information you received about oral health from the(se) professional(s)?	15. Do you put into practice the information you received about oral health from the(se) professional(s)?
0 Doctor 0 Nurse/Crew 0 Dental Surgeon(s)/Team 0 Community Health Agent 0 Other? Who? _______________________	0 Never received information 1Interval bigger than 2 years 2In the last 2 years 3In the last year 4In the last 6 months 5In the last month 98 Not applicable 99No answer/ did not know	0 Never received information 1Did not understand 2Understood poorly 3Understood partially 4Understood almost everything 5Understood everything 98 Not applicable 99No answer/Did not know	0 Never received information 1Unable 2Able with major difficulty 3Able with some difficulty 4Able with minor difficulty 5Easily able 98 Not applicable 99No answer/Did not know	0 Never received information 1Never 2Seldom 3Sometimes 4Frequently 5Always 98 Not applicable 99No answer/Did not know
16. What issues were considered when you received information about oral health?	17. When was the last time you received information about oral health from this\these issues?	18. Did you understand the information you received about oral health from the(se) issues?	19. Can you appraise the quality of the information you received about oral health from the(se) issues?	20. Do you put into practice the information you received about oral health from the(se) issues?
0 Caries 0 Dental Canal treatment 0 Gingival problems / Gingival bleeding 0 Bone loss around teeth / Dental mobility 0 Oral Hygiene 0 Dental trauma 0 Bruxism / clenching 0 Oral cancer 0 Dental prosthesis 0 Dental implants 0 Orthodontic treatment 0 Other, which? _______________________	0 Never received information 1Interval bigger than 2 years 2In the last 2 years 3In the last year 4In the last 6 months 5In the last month 98 Not applicable 99No answer/Did not know	0 Never received information 1Did not understand 2Understood poorly 3Understood partially 4Understood almost everything 5Understood everything 98 Not applicable 99No answer/Did not know	0 Never received information 1Unable 2Able with major difficulty 3Able with some difficulty 4Able with minor difficulty 5Easily able 98 Not applicable 99No answer/Did not know	0 Never received information 1Never 2Seldom 3Sometimes 4Frequently 5Always 98 Not applicable 99No answer/Did not know
21. Have you read information about oral health in printed or electronic materials?	22. When was the last time you read information about oral health in print or electronic materials?	23. Did you understand the information about oral health you read from those printed or electronic materials?	24. Can you appraise the quality of oral health information from those printed or electronic materials?	25. Do you put oral health information from printed or electronic materials into practice?
0 Dental recipes 0 Package leaflets for medicinal products / Oral Rinse and / or Toothpaste Labels 0 Pre-and / or post-dental care guidelines 0 Poster 0 Reviews 0 Newspapers 0 Billboard 0 Flyer / Folder / Booklet 0 Internet 0 Other, which? ______________________	0 Never received information 1Interval bigger than 2 years 2In the last 2 years 3In the last year 4In the last 6 months 5In the last month 98 Not applicable 99No answer/Did not know	0 Never received information 1Did not understand 2Understood poorly 3Understood partially 4Understood almost everything 5Understood everything 98 Not applicable 99No answer/Did not know	0 Never received information 1Unable 2Able with major difficulty 3Able with some difficulty 4Able with minor difficulty 5Easily able 98 Not applicable 99No answer/Did not know	0 Never received information 1Never 2Seldom 3Sometimes 4Frequently 5Always 98 Not applicable 99No answer/Did not know
26. Have you ever watched and/or heard about oral health information in print or electronic materials?	27. What was the last time you watched and/or heard oral health information in print or electronic materials?	28. Did you understand the information you watched and/or heard about oral health from those printed or electronic materials?	29. Can you appraise the quality of the information about oral health you watched and/or heard from those printed or electronic materials?	30. Do you put oral health information you watched and/or heard from printed or electronic materials into practice?
0 Educational video 0 Television 0 Radio 0 Internet 0 Lecture 0 Movie / cinema 0 Other, which? _______________________	0 Never received information 1Interval bigger than 2 years 2In the last 2 years 3In the last year 4In the last 6 months 5In the last month 98 Not applicable 99No answer/Did not know	0 Never received information 1Did not understand 2Understood poorly 3Understood partially 4Understood almost everything 5Understood everything 98 Not applicable 99No answer/Did not know	0 Never received information 1Unable 2Able with major difficulty 3Able with some difficulty 4Able with minor difficulty 5Easily able 98 Not applicable 99No answer/Did not know	0 Never received information 1Never 2Seldom 3Sometimes 4Frequently 5Always 98 Not applicable 99No answer/Did not know

After judging the validity of content, conducted by the committee of experts, it was evident that the 150 questions with their respective options of answers are adequate to evaluate the construct “Oral Health Literacy for Diabetics”, thus it was verified that OHL-D was considered valid as to its content and to the construct it proposes to appraise.

After the test and retest, reliability/reproducibility was estimated by the simple Kappa coefficient, it was found that the results ranged from -0.09 to 1. Out of the 150 questions, only 16 did not present satisfactory levels of agreement, that is, Kappa smaller or equal to 0.60 (table 1).

During data collection, it was found that the application of the instrument took a considerable time, ranging from 15 to 25 minutes. In addition, it was observed that many of the questions received the code “not applicable”, because access to information was not verified to be passed on by some professionals, through reading or listening. Thus the characterization of the information regarding the understanding, appraisal and application was not made. We chose to synthesize the OHL-D instrument. The result after this step was shown in table 2.

I will ask some questions about how you obtain, understand, appraise and apply information on oral health. After reading each question, I will read through all of the answer choices so that you can choose an option. Did you understand? Can we start? (Make sure the interviewee understood how the interview will be conducted, then start the interview. Do not read: “Never received information” and “No answer/did not know”).

DISCUSSION

The HL assessment should consider people’s ability to access, understand, appraise and interpret health information, since high levels of HL help people make appropriate health-related decisions. Low HL can influence professional-patient communication, access and adherence to health promotion and disease prevention. The development of instruments to measure HL levels has great relevance, since knowing the level of HL among The people can help to improve the communication between professional and patient, maximizing the understanding on the part of the patients and consequently increasing the probability of them applying the past orientations by professionals more effectively. A health literate population can better manage their health and, consequently, decrease the demand for services in the health system [¹⁵-²⁶].

There are tools available in the literature that assess the level of literacy in oral health. One of them evaluates the reading and possible recognition by the interviewee of words related to oral health, which would be associated to the knowledge of the concept of these words [¹⁵]. This tool, therefore, only evaluates access and comprehension of people on some matters related to oral health, but does not explore the evaluation and application of information related to oral health. Thus, OHL-D presents itself as an instrument that intends to measure oral health literacy in a broader way, that is, to verify access, understanding, appraisal and application of oral health information. In addition, according to the values obtained in this study by the Kappa coefficient, excellent reproducibility was found.

When conducting a research, instruments should be chosen that show reliability, validity of their results, responsiveness and interpretability; otherwise, mistaken results could be obtained [²²]. The validity (of content, criterion and construct) verifies whether the instrument measures exactly what it proposed to measure, that is, it evaluates the competence of the instrument to accurately measure the phenomenon to be studied [²²,²⁷,²⁸]. The OHL-D presented satisfactory results regarding content validity. The judgments of criterion validity (gold standard requirement) and construct validity (minimum number of respondents close to 400 people) as well as the judgment of responsiveness (longitudinal study) were not considered in this investigation, although experts participating in a Delphi study on the subject recommend these judgments [²²]. The judgments not considered extrapolated the purpose of this investigation, in addition to demanding methodological care incompatible with logistical project questions, such as the cross-sectional study design, the amount of human resources and the time available to conduct the research.

Reliability is the ability to reproduce a result consistently in time and space or with different observers [²⁷,²⁹]. The following aspects are used for the evaluation of reliability: stability (measured by means of the test and retest), homogeneity, and inter-interviewed equivalence [²⁸,³⁰]. On the other hand, the results of a Delphi study propose that the reliability of an instrument is measured when its reproducibility, error measurement and internal consistency are measured [²²]. Reproducibility refers to the ability of the instrument’s findings to be reproducible, regardless of who applies the questionnaire or when it is applied [³¹].

In this study, reproducibility was calculated by the simple Kappa coefficient. Kappa assesses the agreement between multiple assessments of the same phenomenon and can be defined as the ratio of the proportion of times respondents agree (corrected by coincidence due to chance) to the maximum proportion of times respondents might agree (also corrected by agreement due to chance). This coefficient becomes appropriate when the data are categorical and are in nominal scale. Kappa values range from -1 to 1, which represent total absence of agreement and total agreement, respectively. Values of Kappa between -1 and 0 are considered low, between 0 and 0.2 are considered discrete, between 0.2 and 0.4 are considered regular, between 0.4 and 0.6 are considered moderate, between 0.6 and 0.8 are considered as substantial and above 0.8 are considered close to perfect [²⁵]. Some studies identified in the literature used a similar methodology to evaluate the reproducibility/reliability of research instruments [³²-³³].

In relation to the first part of the instrument, which objective was to ascertain if there was access, understanding, evaluation and application of oral health knowledge, the Kappa coefficient result was satisfactory, with a minimum value of 0.714 among the ten questions. Within the ten variables, six obtained an acceptable Kappa coefficient (between 0.6 and 0.8) and, therefore, satisfactory. The other variables had values close to perfect.

The second part of the instrument aimed to evaluate from which professionals the interviewees received information about oral health. In two variables (those who checked if there was access to knowledge about oral health and how often it was accessed) there was no Kappa value below acceptable, with a minimum value of 0.780 observed. However, the variable that examined the understanding of the information received by the interviewees presented three values of Kappa below the acceptable level, being less than 0.500. There were also values below 0.600 in the variable “evaluated” and in the variable “applied”. Only for the professional dental surgeon, all the values obtained were satisfactory. This may suggest greater safety on the part of the interviewees as to the information passed on by dental surgeons.

In the third part of the instrument, referring to oral health subjects to which people were exposed, four variables were found with coefficients of maximum value, that is, 1, showing that there was total agreement among the interviewees. Also in the third part, six Kappa values were identified below 0.600, and three of these values were found in the “practiced” variable. For dental cavities, canal treatment, bruxism/clenching, oral cancer, dental prosthesis and dental implant, all values obtained were satisfactory. The subjects dental trauma and orthodontic treatment presented only one unsatisfactory value, which should be investigated, since there may have been difficulty on understanding the meaning of these expressions by the interviewees.

In the fourth part, there were four Kappa values below the acceptable one. The minimum observed value was 0.485 in the question that asked about the understanding of the information read in flyers/folders/booklets, while the maximum was 0.970, in the question that examined whether the interviewee had read information in newspapers. For the dental prescriptions, pre and post treatment guidelines, poster, magazine, newspaper and internet, all Kappa values obtained were satisfactory. Package leaflet for medicinal products and flyers/folders/booklets each had two unsatisfactory values.

Finally, in the fifth and last part of the instrument, it was identified whether the interviewee listened to and/or watched oral health information and in what ways he/she listened and/or watched it. Nine values of total agreement were found. Only the variable “practiced”, referring to the information obtained by telephone, showed a coefficient below the acceptable level. The media of educational videos, television, radio, internet and lecture presented acceptable Kappa values. Unsatisfactory values in the “understanding” and “practice” dimensions may suggest that respondents were reluctant to admit any deficiency regarding their ability to understand and practice the information accessed, causing discrepancy between test and retest responses.

In general, it was observed that the instrument presented good reproducibility of the results, indicating a good level of confidence. Among the 150 questions, only 16 did not present satisfactory levels of agreement, that is, Kappa less than or equal to 0.60.

It should be pointed out that, due to the lack of similar instruments to evaluate oral health literacy among diabetics, it was difficult to compare the results obtained using the OHL-D with results from the other instruments.

However, the reproducibility achieved through the application of this instrument among elderly diabetics was greater than that observed through the application of Scale Oral Health Outcomes for Five-Year-Old Children and Oral Health Impact Profile for Children and Adolescents, respectively [³²]. In evaluating the psychometric properties of the Self-Care Activities Questionnaire with Diabetes, were also found satisfactory values regarding the reliability and validity of the instrument [³³], in agreement with this study.

A limitation of the study was to use a sample that, although recommended for reliability assessment, did not allow for the estimation of responsiveness, criterion and construct validity, which are important evaluations in the assessment of research instruments. Only the content and reliability/reproducibility validity were checked, so it is necessary to evaluate the other inherent properties of the OHL-D. The results presented, through the calculation of simple Kappa, indicated that there was good reproducibility of the results between the test and the retest. In addition, the instrument also had validity of desirable content. This means that the OHL-D has achieved a good level of reliability and validation.

CONCLUSION

The test of psychometric properties is an essential step in the process of constructing instruments for the evaluation of health related conditions. The OHL-D was considered valid for content and, in general, its reliability was satisfactory. Among the 150 questions, only 16 did not present satisfactory levels of agreement, that is, Kappa less than or equal to 0.60. The shortening of the instrument into a version with 30 questions was decided. The use of the OHL-D in the academy and in the health services is recommended for the evaluation and subsequent understanding of the levels of health literacy among diabetic people. Therefore, it becomes possible to adapt interventions and actions with found results, aiming to improve the service delivery and life quality of diabetics. Further studies are necessary in order to later evaluate the other psychometric properties inherent to OHL-D.

How to cite this article

• Martins AMEBL, Amorim MMTA, Carvalho BO, Pinto RA, Fróes DTC, Santos ASF. Development, judgment of the validity and reliability of an instrument of assessment of Oral Health Literacy among diabetics. RGO, Rev Gaúch Odontol. 2020;68: e20200039. http://dx.doi.org/10.1590/1981-86372020000393588

REFERENCES

• ¹
  Sanches RM, Ciconelli RM. Conceitos de acesso à saúde. Rev Panam Salud Publica. 2012;31(3):260-8. http://dx.doi:10.1590/S1020-49892012000300012
  » https://doi.org/10.1590/S1020-49892012000300012
• ²
  Solar O, Irwin A. A conceptual framework for action on the social determinants of health. Social Determinants of Health Discussion, Paper 2 (Policy and Practice). Geneva: World Health Organization; 2010.
• ³
  Schmidt MI, Duncan BB, Silva GA, Menezes AM, Monteiro CA, Barreto SM et al. Doenças crônicas não transmissíveis no Brasil: carga e desafios atuais. In: Victora CG. Saúde no Brasil: a série The Lancet. Rio de Janeiro: Fiocruz; 2011. p. 61-74.
• ⁴
  Brasil. Ministério da Saúde. Secretaria de Vigilância em Saúde. Departamento de Análise de Situação de Saúde. Plano de ações estratégicas para o enfrentamento das doenças crônicas não transmissíveis (DCNT) no Brasil 2011-2022. Brasília: Ministério da Saúde; 2011.
• ⁵
  Toscano CM. As Campanhas nacionais para detecção das doenças crônicas não transmissíveis: diabetes e hipertensão arterial. Ciência Saúde Colet. 2004;9(4):885-95. http://dx.doi.org/10.1590/S1413-81232004000400010
  » https://doi.org/10.1590/S1413-81232004000400010
• ⁶
  Haddad MCL, Bortoletto MSS, Silva RS. Amputação de membros inferiores de portadores de diabetes mellitus: análise dos custos de internação em hospital público. Cienc Cuid Saude. 2010;9(1):107-113. https://doi.org/10.4025/cienccuidsaude.v9i1.10536.
  » https://doi.org/10.4025/cienccuidsaude.v9i1.10536
• ⁷
  Milman MHSA, Leme CBM, Borelli DT, Kater FR, Baccili ECDC, Rocha RCM, et al. Pé diabético: avaliação da evolução e custo hospitalar de pacientes internados no Conjunto Hospitalar de Sorocaba. Arq Bras Endocrinol Metab. 2001;45(5):447-51. http://dx.doi.org/10.1590/S0004-27302001000500007.
  » https://doi.org/10.1590/S0004-27302001000500007
• ⁸
  Organização Pan-americana de Saúde. PAHO. Proposed 10-year regional plan on oral health for the Americas: final report. 160th session of the executive committee Washington, D.C., USA, 26-30 June 2017 [cited 2017 Jan 29]. Available from: <http://www.paho.org/hq/index.php?option=com_content&view=article&id=13174&Itemid=1160⟨=pt>.
  » http://www.paho.org/hq/index.php?option=com_content&view=article&id=13174&Itemid=1160⟨=pt
• ⁹
  FDI World Dental Federation. Vision 2020 Think Tank: A new definition for oral health [cited 2017 Jan 29]. Available from: <http://www.fdiworlddental.org/sites/default/files/media/images/oral_health_definition-exec_summary-en.pdf>.
  » http://www.fdiworlddental.org/sites/default/files/media/images/oral_health_definition-exec_summary-en.pdf
• ¹⁰
  Organização Pan-Americana da Saúde. A política nacional de saúde bucal no Brasil: registro de uma conquista histórica. Brasília: Organização Pan-Americana de Saúde, 2006 health [cited 2017 Sept 3]. Available from: <http://189.28.128.100/dab/docs/publicacoes/geral/serie_tecnica_11_port.pdf>
  » http://189.28.128.100/dab/docs/publicacoes/geral/serie_tecnica_11_port.pdf
• ¹¹
  Blanco-Cornejo M, Riva-Palacio-Chiang-Sam IL, Sánchez-Díaz I, Cerritos A, Tena-Tamayo C, LópezHernández D. New model for diabetes primary health care based on patient empowerment and the right to preventive health: the MIDE program. Rev Panam Salud Publica. 2017;41:1-10.
• ¹²
  Rodrigues CC, Leite ICG, Paula MVQ. Conhecimento e atitudes em saúde bucal de usuários do serviço público. Revista APS. 2004;7(1):08-16.
• ¹³
  World Health Organization. Ottawa Letter for Health Promotion [cited 2017 Sept 3]. Available from: <http://www.euro.who.int/__data/assets/pdf_file/0004/129532/Ottawa_Charter.pdf>.
  » http://www.euro.who.int/__data/assets/pdf_file/0004/129532/Ottawa_Charter.pdf
• ¹⁴
  Sørensen K, Broucke SVD, Fullam J, Doyle G, Pelikan J, Slonska Z, et al. Health literacy and public health: a systematic review and integration of definitions and models. BMC Public Health. 2012;12(1):1-13. http://doi.org/10.1186/1471-2458-12-80.
  » https://doi.org/10.1186/1471-2458-12-80
• ¹⁵
  Junkes MC, Fraiz FC, Sardenberg F, Lee JY, Paiva SM, Ferreira FM. Validity and Reliability of the Brazilian Version of the Rapid Estimate of Adult Literacy in Dentistry – BREALD-30. Plos One. 2015;10(7):1-11. http://doi.org/10.1371/journal.pone.0131600.
  » https://doi.org/10.1371/journal.pone.0131600
• ¹⁶
  Lee SY, Bender DE, Ruiz RE, Cho YI. Development of an easy-to-use Spanish Health Literacy test. Health Serv Res. 2006;41(4Pt1):1392-412. http://doi.org/10.1111/j.1475-6773.2006.00532.x
  » https://doi.org/10.1111/j.1475-6773.2006.00532.x
• ¹⁷
  Richman JA, Lee JY, Rozier RG, Gong DA, Pahel BT, Vann WF Jr. Evaluation of a word recognition instrument to test health literacy in dentistry: the REALD-99. J Public Health Dent. 2007 Spring;67(2):99-104. http://doi.org/10.1111/j.1752-7325.2007.00022.x
  » https://doi.org/10.1111/j.1752-7325.2007.00022.x
• ¹⁸
  Gong DA, Lee JY, Rozier RG, Pahel BT, Richman JA, Vann WF Jr. Development and testing of the Test of Functional Health Literacy in Dentistry (TOFHLiD). J Public Health Dent. 2007;67(2):105-12. http://doi.org/10.1111/j.1752-7325.2007.00023.x
  » https://doi.org/10.1111/j.1752-7325.2007.00023.x
• ¹⁹
  Sabbahi DA, Lawrence HP, Limeback H, Rootman I. Development and evaluation of an oral health literacy instrument for adults. Community Dent Oral Epidemiol. 2009;37(5):451-62. http://doi.org/10.1111/j.1600-0528.2009.00490.x
  » https://doi.org/10.1111/j.1600-0528.2009.00490.x
• ²⁰
  Macek MD, Haynes D, Wells W, Bauer-Leffler S, Cotten PA, Parker RM. Measuring conceptual health knowledge in the context of oral health literacy: preliminary results. J Public Health Dent. 2010;70(3):197-204. http://doi.org/10.1111/j.1752-7325.2010.00165.x
  » https://doi.org/10.1111/j.1752-7325.2010.00165.x
• ²¹
  Stucky BD, MA, Lee JY, Lee S-YD, Rozier RG. Development of the Two Stage Rapid Estimate of Adult Literacy in Dentistry (TS-REALD). Community Dent Oral Epidemiol. 2011;39(5):474-480. http://doi.org/10.1111/j.1600-0528.2011.00619.x
  » https://doi.org/10.1111/j.1600-0528.2011.00619.x
• ²²
  Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. J Clin Epidemiol. 2010;63(7):737-45. http://doi.org/10.1016/j.jclinepi.2010.02.006
  » https://doi.org/10.1016/j.jclinepi.2010.02.006
• ²³
  Bertolucci PHF, Brucki SMD, Campacci SR, Juliano Y. O mini-exame do estado mental em uma população geral: impacto da escolaridade. Arq. Neuro-Psiquiatr. 1994;52(1):1-7. https://doi.org/10.1590/S0004-282X1994000100001
  » https://doi.org/10.1590/S0004-282X1994000100001
• ²⁴
  Hair JF, Black WC, Babin BJ, Anderson RE, Tatham RL. Análise multivariada de dados. São Paulo: Bookman Editora; 2009.
• ²⁵
  Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159-174. http://doi.org/10.2307/2529310
  » https://doi.org/10.2307/2529310
• ²⁶
  Martins AMEBL, Almeida ER, Oliveira CC, Oliveira RCN, Pelino JEP, Santos ASF, et al. Alfabetização em saúde bucal: uma revisão da literatura. Rev Assoc Paul Cir Dent. 2015;69(4):328-34.
• ²⁷
  Alexandre NMC, Coluci MZO. Validade de conteúdo nos processos de construção e adaptação de instrumentos de medida. Ciênc Saúde Coletiva. 2011;16(7):3061-3068. http://dx.doi.org/10.1590/S1413-81232011000800006
  » https://doi.org/10.1590/S1413-81232011000800006
• ²⁸
  Roberts P, Priest H, Traynor M. Reliability and validity in research. Nurs Stand. 2006;20(44):41-45. http://doi.org/10.7748/ns2006.07.20.44.41.c6560
  » https://doi.org/10.7748/ns2006.07.20.44.41.c6560
• ²⁹
  Contandriopoulos AP, Champagne F, Potvin L, Denis JL, Boyle P. Saber preparar uma pesquisa. 3. ed. São Paulo: Hucitec; 1999.
• ³⁰
  Burns N, Grove SK. The practice of nursing research: conduct, critique & utilization. 3 ed. Philadelphia: Saunders Company; 1997.
• ³¹
  Cristiani BS, Resende JM, Chamlian TR. Validação e repro-dutibilidade da versão em Português da Lower Limb Amputee Measurement Scale. Acta Fisiatr. 2013; Acta Fisiatr. 2013;20(4):187-193. https://doi.org/10.5935/0104-7795.20130031
  » https://doi.org/10.5935/0104-7795.20130031
• ³²
  Oliveira MAC. Reprodutibilidade de questionários de quali-dade de vida relacionada à saúde bucal (SOHO-5 e OHIP-14) [dissertação]. João Pessoa: Universidade Federal da Paraíba; 2015.
• ³³
  Michels MJ et al. Questionário de Atividades de Autocuidado com o Diabetes: tradução, adaptação e avaliação das pro-priedades psicométricas. Arq Bras Endocrinol Metab. 2010;54/7. https://doi.org/10.1590/S0004-27302010000700009
  » https://doi.org/10.1590/S0004-27302010000700009

Publication Dates

• Publication in this collection
  05 Oct 2020
• Date of issue
  2020

History

• Received
  12 Sept 2018
• Reviewed
  31 July 2020
• Accepted
  02 Aug 2020