Development, judgment of the validity and reliability of an instrument of assessment of Oral Health Literacy among diabetics Desenvolvimento, julgamento da validade e confiabilidade de um instrumento de avaliação da Alfabetização em Saúde Bucal entre diabéticos

Objective: The study purposed to perform content validation and verification of the reliability of an instrument in the form of a questionnaire, called “ Oral Health Literacy for Diabetics “. It was designed to investigate whether there was access, understanding, appraise and practice of information related to oral health among diabetics. Methods: This is a methodological research that was carried out through the application of the Oral Health Literacy for Diabetics between 109 diabetic patients, since at least 60 participants should be considered in studies using test/retest. The research complied with the ethical principles of research. The content validation was performed by dentists, acknowledging the relevance and the ability to measure the levels of literacy of each item of the Oral Health Literacy for Diabetics. The reliability/reproducibility was estimated by the test/retest in an interval of seven to fifteen days by Kappa, using the SPSS®. Results: The results showed that the content of the Oral Health Literacy for Diabetics presented relevance and ability to measure the levels of literacy in oral health among diabetics. The results of the Kappa ranged from -0.09 to 1. Only 16 of the 150 questions did not have satisfactory levels of agreement, i.e., Kappa smaller or equal to 0.60. We chose to synthesize the Oral Health Literacy for Diabetics initially with 150 questions, for a version with 30 questions. Conclusion: It is concluded that the Oral Health Literacy for Diabetics was considered valid with respect to the content and that a general form its reliability was satisfactory. It is recommended to use the Oral Health Literacy for Diabetics in academia and in health services aiming to improve the quality of life of diabetic patients. Indexing terms: Data accuracy. Diabetes mellitus. Oral health. Reproducibility of results. AMEBL MARTINS et al. 2 RGO, Rev Gaúch Odontol. 2020;68:e20200039 RESUMO Objetivo: Conduzir a validação de conteúdo e a verificação da confiabilidade de um instrumento em forma de questionário, denominado “Alfabetização em Saúde Bucal entre diabéticos” que foi idealizado para investigar se houve acesso, compreensão, avaliação e aplicação de informações relacionados à saúde bucal entre diabéticos. Métodos: Trata-se de uma pesquisa metodológica que foi realizada através da aplicação do Instrumento entre 109 participantes, uma vez que no mínimo 60 participantes devem ser considerados em estudos teste/reteste. A investigação respeitou os princípios éticos da pesquisa. A avaliação de conteúdo foi feita por cirurgiões-dentistas, que verificaram a relevância e a capacidade de mensuração dos níveis de alfabetização do Instrumento. A confiabilidade/reprodutibilidade foi estimada pelo teste/reteste em um intervalo de três a sete dias pelo Kappa, utilizando o SPSS®. Resultados: A validade de conteúdo do Instrumento evidenciou capacidade do instrumento em mensurar os níveis de alfabetização em saúde bucal entre diabéticos. O Kappa variou de -0,09 a 1. Das 150 perguntas somente 16 não apresentaram níveis de concordância satisfatórios, ou seja, Kappa ≤ 0,60. Optou-se por condensar o Alfabetização em Saúde Bucal inicialmente com 150 perguntas, para uma versão com 30 perguntas. Conclusão: do Alfabetização em Saúde Bucal entre diabéticos foi considerado válido quanto ao conteúdo e de forma geral a sua confiabilidade foi satisfatória. Recomenda-se o uso do Instrumento na academia e nos serviços de saúde visando melhorar a qualidade de vida dos diabéticos. Termos de indexação: Confiabilidade dos dados. Diabetes Mellitus. Saúde bucal. Reprodutibilidade dos testes.


INTRODUCTION
Access to health can be assessed from four different perspectives: availability, acceptability, payment capacity, and information; indicating the complexity of the concept of access. This concept has changed over time: in the 1970s, access referred to geographical (availability) and financial (ability to pay) aspects. However, the literature began to address cultural, educational and socioeconomic aspects. Thus, it became evident that information forms the basis of access to health, promoting the empowerment of people when taking health decisions through Health Literacy (HL). Improving access to health and guaranteeing greater equity depends on intersectorial actions and social and economic policies that allow the dissipation of differences in income and education [1]. Considering this context, in 2007, Solar & Irwin developed an "action-oriented" health care model to identify the social determinants of health inequities and suggest action proposals. The determinants considered as most important for health are the so-called structural, because structural and social rank differences cause unjust iniquities in health [2]. There is a need to address health demands in an expanded way, promoting interventions that have an effect on the determinants of human health conditions in order to minimize inequities.
Non-communicable chronic diseases (NCD), a global public health problem, have a costly impact, especially for low-and middle-income countries. It is estimated that, in 2007, 72% of deaths in Brazil were secondary to NCD. Industrialization growth, rapid demographic transition, and changes in nutritional habits have contributed to an increase in people's weight, increasing the risk for NCD [3,4].
For Diabetes Mellitus (DM), one of the main NCD related to the morbimortality of the population, there is a clear trend towards its increased prevalence as well as financial and social impact [4][5][6][7]. A report by the Pan American Health Organization (PAHO) presented a Regional Oral Health Plan for the Americas. The document recognizes that oral health throughout the Americas is critical. It evidenced the association of oral health with risk factors for NCD and the capacity of health care providers to implement effective interventions for preventing disease. Goals were defined for the integration of oral health in Primary Health Care (PHC), as well as strategies to amplify access to health care. The importance of multidisciplinary approaches including the private sector, academic institutions and civil society in order to improve oral health and general health in the Americas was emphasized. Certain recommendations should be considered: oral health should be recognized as important for general health as well as a factor that directly affects life quality of the population; countries should continue to strengthen capacity of workers regarding primary health care [8].
Oral health is multifaceted, interfering with the ability to speak, smile, chew, swallow and may be related to the sense smell and taste, as well as transmission of emotions. It may reflect the physiological, social, and psychological attributes essential to life quality of people [9]. Dental practice was modified due to the complexity of oral health, with emphasis on health promotion and disease prevention, as well as curative care. Healthcare promotion and diseases prevention have brought results that indicate improvement of oral health conditions, as they promote joint actions and face the demands for healthcare in a more integral manner [10]. The evaluation of Integrated Diabetes Management Program of Mexico recognizes the importance of HL in empowering people [11]. Regarding the oral health conditions of Brazilians, the national epidemiological surveys carried out by the Ministry of Health in 1986, 1996, 2003 and 2010 show improvements [12]. These conditions can be further improved, and health promotion and preventive actions are required, which propose the development of personal skills related to self-care, through the dissemination of information and the increase of HL levels, aiming to improve the ability of people to exercise greater control over their own health, making them able to make choices that lead to a better quality of life [13]. There is evidence of a link between satisfactory socioeconomic conditions and oral health [12]. Actions to promote oral health and to prevent oral diseases should consider, whenever necessary, seek an increase HL levels on people.
HL involves people's knowledge, motivation and skills to access, to understand, to appraise and to apply health information, making daily judgment and decisionmaking about health and disease prevention possible, aiming to maintain or improve the quality of life [14]. The term functional literateness in health designates the ability to use reading and writing for practical purposes on daily life in health: to know the causes of precarious conditions of oral health; adopt self-care measures; communicate with dental providers; put your name on waiting lists for dental treatment; find the way to the dental clinic; fill in the forms; return in follow-up visits; and follow prescription medication. In this perspective, HL differs from health literateness and is considered broader since it covers the domains of appraisal and understanding of information [15], in addition to functional literateness in health. The instruments that measure health literateness have focused only on word recognition and reading ability [16][17][18][19][20][21]. The translation, cross-cultural adaptation to the Portuguese language of Brazil [15] of an instrument originally in English [16] was conducted, it was verified that the instrument presented satisfactory reliability [15].
Despite the availability of instruments for the measurement of oral health literateness, a proper instrument to appraise the levels of literacy in oral health considering access to understanding, appraisal and application of information [14] related to oral health was not found. Thus, it is necessary to develop and judge the content validity and reliability of an instrument to appraise oral health literacy. The validation of a data collection instrument is essential, since health planning decisions are made based on research results obtained through data collection instruments. Otherwise, there is a great risk of inaccurate or biased results, which may lead to inadequate conclusions [22]. The question arises: what characteristics of an evaluation instrument on conditions or events related to health should be considered in the judgment of its quality?
In a Delphi study, the researchers consulted fiftyseven experts, blindly, to clarify which features should be considered in judging the quality of data collection instruments in the health area; the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) was obtained as product. According to the COSMIN checklist, for a data collection instrument to be considered appropriate, its psychometric properties must be analyzed, that is, one must measure the confidence, validity and responsiveness of this instrument. The degree of confidence is analyzed based on three categories: (1) internal consistency; (2) reproducibility; and (3) measure of error. Internal consistency is estimated by Cronbach's alpha, reproducibility and error measurement are measured based on testing and retesting. The validity of an instrument is investigated when three criteria are analyzed: (1) content validity; (2) criterion validity, subdivided into concurrent validity and predictive validity; and (3) construct validity, subdivided into structural validity, hypothesis testing and cross-cultural validity [22] (figure 1). The purpose of this work was the creation and presentation of an instrument to appraise oral health literacy, in addition to verifying its content validity and its reliability / reproducibility.

METHODS
The study is the product of a project that evaluates oral health literacy among diabetic users of two Primary Health Care Units (PHCU) of the Family Health Strategy (FHS) of Montes Claros / MG. Inclusion criteria were: to be over 18 years old; have being diagnosed with Diabetes Mellitus by a doctor; if elderly (60 years of age or older), do not present cognitive impairment in the Mini Mental State Examination (MMSE); those of exclusion were: to have acute decompensation informed by the FHS team; diagnosis of four concurrent comorbidities; not be found at home after three attempts. The Mini-mental score, according to the validated version for Brazil, took into account the interviewees' level of education in the identification of those who presented cognitive problems. Those who scored below those set for the following study years were excluded because they were considered as having cognitive deficits: among illiterates, scores lower than 21; among those with low education level (≤ 5 years), scores lower than 22; among those with average education level (6-11 years), score lower than 23 and among those with high education level (≥ 12 years), score lower than 24 [23].
The judgment of the instrument was conducted as proposed by the -Check List COSMIN (CL-COSMIN) [22]. The validity verification of content was done by dental professionals who evaluated the relevance of the items contained in the instrument, making sure that they were able to measure literacy in oral health. To evaluate reliability, as reproducibility [22], 109 diabetics were invited to participate in the process, since this type of analysis does not necessarily require the estimation of probabilistic samples [24]. The retest was applied seven to fifteen days after the first application, that is, the test. The interviews were conducted by a previously trained interviewer to follow the application instructions evenly to the others interviews. Statistical analysis was carried out with the aid of SPSS (version 24.0, SPSS INC., Chicago, Illinois, USA). The test-retest reliability for each variable was calculated using the simple Kappa coefficient, considering values between 0.6 and 0.8 as acceptable and above 0.8 as close to perfect [25].
This study was approved by the Research Ethics Committee of the State University of Montes Claros, CONEP number 764.743.

Description of the proposed Oral Health Literacy for Diabetics (OHL-D) Assessment Tool
The instrument is divided into five parts for different purposes. The first one, composed of ten questions, inquires whether there was access, understanding, appraisal and application of information related to oral health. Generally, the frequency of access, and how comprehension, appraisal and application of this information occurred (table 1) are also verified. The second part proposes to assess which professionals or individuals (doctor, nursing team, dentist/dental team, health agent and others) are involved in the transmission of information. The third part aims to identify which oral health subjects people could access (cavities, canal treatment, gingival/gingival bleeding problems, dental mobility/bone loss, oral hygiene, trauma, oral cancer, dental prosthesis, implant and orthodontic appliance). The fourth part aims to determine if the interviewee had read information about oral health (dental prescriptions, package leaflets for dental products, pre and postoperative guidelines, newspaper, magazine, poster, billboard, flyer/folder/booklet, internet and others). The fifth part aims to identify if the interviewee listened to and/ or watched information about oral health and in what media (educational videos, television, radio, telephone, internet, movies/cinemas and others). In addition, the instrument proposes to investigate how often people access, read, listen to and/or watch information about oral health, if any, and how understanding, appraisal, and application of such information occurred (table 2).    After judging the validity of content, conducted by the committee of experts, it was evident that the 150 questions with their respective options of answers are adequate to evaluate the construct "Oral Health Literacy for Diabetics", thus it was verified that OHL-D was considered valid as to its content and to the construct it proposes to appraise.
After the test and retest, reliability/reproducibility was estimated by the simple Kappa coefficient, it was found that the results ranged from -0.09 to 1. Out of the 150 questions, only 16 did not present satisfactory levels of agreement, that is, Kappa smaller or equal to 0.60 (table 1).
During data collection, it was found that the application of the instrument took a considerable time, ranging from 15 to 25 minutes. In addition, it was observed that many of the questions received the code "not applicable", because access to information was not verified to be passed on by some professionals, through reading or listening. Thus the characterization of the information regarding the understanding, appraisal and application was not made. We chose to synthesize the OHL-D instrument. The result after this step was shown in table 2.
I will ask some questions about how you obtain, understand, appraise and apply information on oral health. After reading each question, I will read through all of the answer choices so that you can choose an option. Did you understand? Can we start? (Make sure the interviewee understood how the interview will be conducted, then start the interview. Do not read: "Never received information" and "No answer/did not know").

DISCUSSION
The HL assessment should consider people's ability to access, understand, appraise and interpret health information, since high levels of HL help people make appropriate health-related decisions. Low HL can influence professional-patient communication, access and adherence to health promotion and disease prevention. The development of instruments to measure HL levels has great relevance, since knowing the level of HL among The people can help to improve the communication between professional and patient, maximizing the understanding on the part of the patients and consequently increasing the probability of them applying the past orientations by professionals more effectively. A health literate population can better manage their health and, consequently, decrease the demand for services in the health system [15][16][17][18][19][20][21][22][23][24][25][26].
There are tools available in the literature that assess the level of literacy in oral health. One of them evaluates the reading and possible recognition by the interviewee of words related to oral health, which would be associated to the knowledge of the concept of these words [15]. This tool, therefore, only evaluates access and comprehension of people on some matters related to oral health, but does not explore the evaluation and application of information related to oral health. Thus, OHL-D presents itself as an instrument that intends to measure oral health literacy in a broader way, that is, to verify access, understanding, appraisal and application of oral health information. In addition, according to the values obtained in this study by the Kappa coefficient, excellent reproducibility was found.
When conducting a research, instruments should be chosen that show reliability, validity of their results, responsiveness and interpretability; otherwise, mistaken results could be obtained [22]. The validity (of content, criterion and construct) verifies whether the instrument measures exactly what it proposed to measure, that is, it evaluates the competence of the instrument to accurately measure the phenomenon to be studied [22,27,28]. The OHL-D presented satisfactory results regarding content validity. The judgments of criterion validity (gold standard requirement) and construct validity (minimum number of respondents close to 400 people) as well as the judgment of responsiveness (longitudinal study) were not considered in this investigation, although experts participating in a Delphi study on the subject recommend these judgments [22]. The judgments not considered extrapolated the purpose of this investigation, in addition to demanding methodological care incompatible with logistical project questions, such as the cross-sectional study design, the amount of human resources and the time available to conduct the research.
Reliability is the ability to reproduce a result consistently in time and space or with different observers [27,29]. The following aspects are used for the evaluation of reliability: stability (measured by means of the test and retest), homogeneity, and inter-interviewed equivalence [28,30]. On the other hand, the results of a Delphi study propose that the reliability of an instrument is measured when its reproducibility, error measurement and internal consistency are measured [22]. Reproducibility refers to the ability of the instrument's findings to be reproducible, regardless of who applies the questionnaire or when it is applied [31].
In this study, reproducibility was calculated by the simple Kappa coefficient. Kappa assesses the agreement between multiple assessments of the same phenomenon and can be defined as the ratio of the proportion of times respondents agree (corrected by coincidence due to chance) to the maximum proportion of times respondents might agree (also corrected by agreement due to chance). This coefficient becomes appropriate when the data are categorical and are in nominal scale. Kappa values range from -1 to 1, which represent total absence of agreement and total agreement, respectively. Values of Kappa between -1 and 0 are considered low, between 0 and 0.2 are considered discrete, between 0.2 and 0.4 are considered regular, between 0.4 and 0.6 are considered moderate, between 0.6 and 0.8 are considered as substantial and above 0.8 are considered close to perfect [25]. Some studies identified in the literature used a similar methodology to evaluate the reproducibility/reliability of research instruments [32][33].
In relation to the first part of the instrument, which objective was to ascertain if there was access, understanding, evaluation and application of oral health knowledge, the Kappa coefficient result was satisfactory, with a minimum value of 0.714 among the ten questions. Within the ten variables, six obtained an acceptable Kappa coefficient (between 0.6 and 0.8) and, therefore, satisfactory. The other variables had values close to perfect.
The second part of the instrument aimed to evaluate from which professionals the interviewees received information about oral health. In two variables (those who checked if there was access to knowledge about oral health and how often it was accessed) there was no Kappa value below acceptable, with a minimum value of 0.780 observed. However, the variable that examined the understanding of the information received by the interviewees presented three values of Kappa below the acceptable level, being less than 0.500. There were also values below 0.600 in the variable "evaluated" and in the variable "applied". Only for the professional dental surgeon, all the values obtained were satisfactory. This may suggest greater safety on the part of the interviewees as to the information passed on by dental surgeons.
In the third part of the instrument, referring to oral health subjects to which people were exposed, four variables were found with coefficients of maximum value, that is, 1, showing that there was total agreement among the interviewees. Also in the third part, six Kappa values were identified below 0.600, and three of these values were found in the "practiced" variable. For dental cavities, canal treatment, bruxism/clenching, oral cancer, dental prosthesis and dental implant, all values obtained were satisfactory. The subjects dental trauma and orthodontic treatment presented only one unsatisfactory value, which should be investigated, since there may have been difficulty on understanding the meaning of these expressions by the interviewees.
In the fourth part, there were four Kappa values below the acceptable one. The minimum observed value was 0.485 in the question that asked about the understanding of the information read in flyers/folders/ booklets, while the maximum was 0.970, in the question that examined whether the interviewee had read information in newspapers. For the dental prescriptions, pre and post treatment guidelines, poster, magazine, newspaper and internet, all Kappa values obtained were satisfactory. Package leaflet for medicinal products and flyers/folders/booklets each had two unsatisfactory values.
Finally, in the fifth and last part of the instrument, it was identified whether the interviewee listened to and/or watched oral health information and in what ways he/she listened and/or watched it. Nine values of total agreement were found. Only the variable "practiced", referring to the information obtained by telephone, showed a coefficient below the acceptable level. The media of educational videos, television, radio, internet and lecture presented acceptable Kappa values. Unsatisfactory values in the "understanding" and "practice" dimensions may suggest that respondents were reluctant to admit any deficiency regarding their ability to understand and practice the information accessed, causing discrepancy between test and retest responses.
In general, it was observed that the instrument presented good reproducibility of the results, indicating a good level of confidence. Among the 150 questions, only 16 did not present satisfactory levels of agreement, that is, Kappa less than or equal to 0.60.
It should be pointed out that, due to the lack of similar instruments to evaluate oral health literacy among diabetics, it was difficult to compare the results obtained using the OHL-D with results from the other instruments.
However, the reproducibility achieved through the application of this instrument among elderly diabetics was greater than that observed through the application of Scale Oral Health Outcomes for Five-Year-Old Children and Oral Health Impact Profile for Children and Adolescents, respectively [32]. In evaluating the psychometric properties of the Self-Care Activities Questionnaire with Diabetes, were also found satisfactory values regarding the reliability and validity of the instrument [33], in agreement with this study.
A limitation of the study was to use a sample that, although recommended for reliability assessment, did not allow for the estimation of responsiveness, criterion and construct validity, which are important evaluations in the assessment of research instruments. Only the content and reliability/reproducibility validity were checked, so it is necessary to evaluate the other inherent properties of the OHL-D. The results presented, through the calculation of simple Kappa, indicated that there was good reproducibility of the results between the test and the retest. In addition, the instrument also had validity of desirable content. This means that the OHL-D has achieved a good level of reliability and validation.

CONCLUSION
The test of psychometric properties is an essential step in the process of constructing instruments for the evaluation of health related conditions. The OHL-D was considered valid for content and, in general, its reliability was satisfactory. Among the 150 questions, only 16 did not present satisfactory levels of agreement, that is, Kappa less than or equal to 0.60. The shortening of the instrument into a version with 30 questions was decided. The use of the OHL-D in the academy and in the health services is recommended for the evaluation and subsequent understanding of the levels of health literacy among diabetic people. Therefore, it becomes possible to adapt interventions and actions with found results, aiming to improve the service delivery and life quality of diabetics. Further studies are necessary in order to later evaluate the other psychometric properties inherent to OHL-D.

Collaborators
AMEBL MARTINS idealization and participation in all stages of the manuscript, revision of the manuscript. MMT AMORIM participated in the idealization, statistical analysis and writing of the manuscript. BO CARVALHO, RA PINTO and DTC FRÓES data collection, statistical analysis and manuscript writing.