Elaboration and validity and reliability analysis of a questionnaire to assess the knowledge of primary care physicians and nurses about trachoma

Abstract

Objective:

To develop and analyze the validity and reliability of a questionnaire to assess the knowledge of primary care physicians and nurses about trachoma.

Methods:

This is a methodological study of elaboration and analysis of validity and reliability of an instrument (questionnaire). The following steps were developed: 1) identification of updated source literature on the subject; 2) elaboration of the questionnaire items by the researchers; 3) content validation by experts; 4) construct validation with instrument application and hypothesis test analysis; 5) internal consistency analysis and; 6) temporal stability analysis (test-retest).

Results:

The study included 205 primary health care physicians and nurses and 10 specialists in ophthalmology and infectology. The instrument, initially with 52 items, was left with 41 items after all stages. The instrument was able to adequately discriminate professionals with greater and lesser knowledge, according to the hypothesis test (p <0.001). Cronbach's alpha was 0.86 and the test-retest recorded an agreement greater than 60% for most items.

Conclusions:

The final instrument presented satisfactory validity and reliability. It may be a useful tool to assess knowledge of primary health care professionals about trachoma and assisting in the development of educational strategies for these professionals.

Keywords:
Trachoma; Validation studies; Primary Health Care; Surveys and questionnaires

Resumo

Objetivo:

Elaborar e analisar a validade e confiabilidade de um questionário para avaliar o conhecimento de médicos e enfermeiros da atenção primária sobre o tracoma.

Métodos:

Trata-se de um estudo metodológico de elaboração e análise de validade e confiabilidade de instrumento (questionário). Foram desenvolvidas as seguintes etapas: 1) identificação da literatura fonte atualizada sobre o tema; 2) elaboração dos itens do questionário pelos pesquisadores; 3) validação de conteúdo por experts; 4) validação de construto com aplicação do instrumento e análise de teste de hipóteses; 5) análise de consistência interna e; 6) análise de estabilidade temporal (teste-reteste).

Resultados:

Participaram da pesquisa 205 médicos e enfermeiros da atenção primária e 10 especialistas em oftalmologia e infectologia. O instrumento, inicialmente com 52 itens, ficou com 34 itens após todas as etapas. O instrumento mostrou-se capaz de discriminar adequadamente profissionais com maior e menor conhecimento, segundo o teste de hipóteses (p<0,001). O alfa de Cronbach foi de 0,86 e o teste-reteste registrou uma concordância superior a 60% para a maioria dos itens.

Conclusões:

O instrumento final apresentou validade e confiabilidade satisfatórios. Poderá ser um instrumento útil para aferir conhecimentos de profissionais da atenção primária sobre o tracoma e auxiliar na elaboração de estratégias educacionais para estes profissionais.

Descritores:
Tracoma; Estudos de validação; Atenção Primária à Saúde; Inquéritos e questionários

Introduction

Trachoma results of infection by bacterium Chlamydia trachomatis show that it remains the leading global cause of preventable infectious blindness. This disease is endemic in some developing countries; therefore, it is categorized as Neglected Tropical Disease. Trachoma is associated with poor socioeconomic and sanitary conditions^(1-4); in view of its severity, WHO launched an initiative to eradicate trachoma as blindness-causing disease.^(5-8)

There was significant decrease in trachoma detection during the “economic miracle” (1970s) in Brazil. The positive results took trachoma out of the spot as public health issue; however, it led to the false belief that trachoma had been eradicated.^(9,10) Epidemiological research has evidenced that trachoma remains a public health issue in Brazil and, as such, it should be included in potential chronic conjunctivitis diagnoses.^(9,10) Misconception of its eradication persisted within the Brazilian scientific community, which neglected education on trachoma etiology, diagnosis and treatment.^(9,10)

Trachoma diagnosis is essentially clinical and can be obtained through external eye examination, with the aid of a 2.0-3.0x binocular loupe. Trachoma should be diagnosed upon the presence of at least two of the following clinical signs: follicles in the upper tarsal conjunctiva, limbal follicles, typical conjunctival scarring and superior limbal pannus (in-growth of new blood vessels). ^(1-4)

Health professionals — mainly physicians and nurses working in the Family Health Strategy program — must be prepared to diagnose trachoma cases, since primary care is the gateway to the health system. The aim of the present study was to analyze the validity and reliability of a questionnaire applied to assess the knowledge of primary care physicians and nurses about trachoma.

Methods

Methodological study aimed at analyzing the validity and reliability of a questionnaire applied to assess the knowledge of primary care physicians and nurses about trachoma.

The present study was carried out with physicians and nurses who work at the expanded health services in Jequitinhonha (Northeastern Minas Gerais) and Montes Claros (Northern Minas Gerais) cities. These cities were chosen because they were more likely to present trachoma cases, since they account for the worst socioeconomic status in Minas Gerais State.

The questionnaire was developed from: updated literature on trachoma, preparation of items related to the theme, content validity analysis by experts, construct validity (hypothesis testing), internal consistency analysis and temporal stability analysis or test–retest reliability, as detailed below.

Questionnaire preparation

Literature identification

MEDLINE, SCIELO and LILACS were the databases of choice for trachoma literature identification and questionnaire preparation. The following meshes were entered into the databases in Portuguese, English and Spanish: trachoma and primary care. References were selected based on their relevance and topicality.^(1-10)

Preparation of Questionnaire Items

Research material worked as reference to prepare 52 questionnaire items focusing the main trachoma features (epidemiology, etiopathogenesis, clinical signs, diagnosis and treatment). All items were considered true statements. The original text was kept and sentences were developed to be as short as possible.

Validity Analysis

Content validity

Content validity was assessed by 4 ophthalmologists and 1 epidemiologist, all of whom held PhD and/or master's degree, worked as professors and/or researchers, as well as had been working in the healthcare field for more than 20 years. Their assessors answered the first 52 questions by highlighting the relevance of each one of them and the adequacy of their statements to the expected knowledge of physicians and nurses. Relevance questions comprised the following answers: very relevant, relevant, slightly relevant and not relevant. Statement adequacy questions only comprised yes or no answers. Statements considered adequate and questions considered relevant or very relevant by at least 75% of participants in the survey were included in the questionnaire. In total, 11 questions were deleted from the questionnaire due to their irrelevance or little relevance and/or to their inadequate statements (two referring to pathophysiological features, five to clinical features and four to diagnosis and treatment), according to more than 75% of experts. Next, 30% of the remaining questions were drawn and turned into false statements.

All answers followed a 5-point Likert scale describing the agreement degree among respondents, wherein the first point corresponded to full agreement (“strongly agree”) and the last point, to full disagreement (“strongly disagree”). Middle rows made it possible choosing middle statements between the extreme ones.

Construct validity

Construct validity analysis was performed through hypothesis testing. Success scores of general practitioners and nurses were assessed and compared to the scores of ophthalmologists and infectious disease experts. The questionnaire was assigned to primary care physicians and nurses (general practitioners and community-family physicians) and to ophthalmologists and infectious disease experts.

Answers showing partial or full agreement to true statements and partial or full disagreement to false statements were considered correct. All answers were turned into scores by adding Likert scale agreement degrees to questionnaire items. One point was assigned to each correct answer. Hypothesis testing was based on a construct validity strategy aimed at verifying whether the questionnaire could separate ophthalmologists (likely to achieve the highest scores) from other professionals (likely to achieve the lowest scores).

Respondents' mean scores were calculated and compared to other variables (professional education, sex and age) in order to find correlation between the assessed groups. Scores were compared through Mann-Whitney and Kruskal-Wallis U-tests, which followed a non-normal distribution. Significance level was set to 5% (p < 0.05).

Reliability analysis

Internal consistency analysis

Internal consistency analysis was performed through Cronbach's alpha, since it is the most common instrument to measure reliability.^(11,12) This coefficient represents the correlation, or covariance degree, among a set of questionnaire items, e.g. the extent to which the items are linked to each other when the same construct is measured. Values higher than 0.7 are indicative of acceptable internal consistency.⁽¹³⁾

Furthermore, mean inter-item correlation and its influence on Cronbach's alpha value were assessed after questionnaire-item deletion. This procedure, somehow, validates the internal consistency analysis: mean inter-item correlation decreases as Cronbach's alpha coefficient decreases, and vice versa. In other words, high correlations indicate that items measure the same construct, and it fulfills reliability test requirements.^(14,15) Overall, mean correlations among superior items are appropriate when their score is higher than 0.3.⁽¹⁶⁾ Cronbach's alpha coefficient was calculated in IBM-SPSS software.

Temporal stability analysis (test-retest reliability)

Cohen's Kappa coefficient was calculated after reassigning the questionnaire (test-retest reliability) to approximately 15% of its respondents for 12 days, on average. Answers were divided into “right” and “wrong”, based on the same procedure used to determine the aforementioned scores. Kappa values were statistically classified according to the Landis-Koch scale: Poor or slight agreement (< 0.40); moderate agreement (0.41 to 0.60); substantial agreement (0.61 to 0.80); almost perfect agreement (> 0.80).⁽¹⁷⁾

Data collection was carried out after two steps, namely: 1) Research approval by the local health departments in the selected regions; 2) identification of phone numbers, names and workplaces of local primary care physicians and nurses. All professionals were personally contacted at their workplaces by a member of the research team and asked to answer the questionnaire. The Informed Consent Form (ICF) was signed by all respondents.

Inclusion criteria:

1. Being one of the following professionals: family physician, general practitioner, nurse, ophthalmologist or infectious disease specialist;

2. Agreeing to participate in the research and signing the informed consent term.

Exclusion criteria:

1. Not working full-time (being away for some reason);

2. Not practicing primary health care or medical specialties (ophthalmologists and infectious disease specialists).

All ethical considerations were addressed to conduct the present study. Participants filled out the informed consent form and they were assured of the confidentiality of their personal data. The research project was evaluated by the Research Ethics Committee of Montes Claros State University. It was registered and approved under opinion No. 2624699.

Results

After the most relevant trachoma topics were selected, 52 items (statements) were designed and divided into four categories: epidemiology; etiopathogenesis; clinical aspects; diagnosis and treatment. These items were assessed for content validity by medical experts who have reduced the total number of them to 41. The remaining items were assessed for statement relevance and adequacy; they were approved by more than 75% of experts. Suggestions to change the writing of some items without changing their main content were also welcomed.

The 41-item questionnaire was answered by 205 professionals. Seven items (two referring to clinical aspects and five referring to diagnosis and treatment) reached correct response rates lower than 10% (including blank answers); they were deleted because they were considered too complex for PHC professionals.

The main features of professionals in the groups are shown in Table 1. In total, 31.7% of respondents were male and 46.8% were in the age group 30 to 39 years old. More than 80% of participants were physicians and 72.7% of them did not hold or had not completed any specialization degree.

Table 1
Profile of physicians and nurses who participated in the psychometric property assessment of trachoma knowledge questionnaire

Variables	(n)	(%)
Sex
Male	65	31.7
Female	140	68.3
Marital status
Single	110	53.7
Married/In a (common-law) partnership	95	46.3
Age Group (years)
< 30	72	35.1
30 - 39	96	46.8
> 40	37	18.1
Profession
Physicians	172	83.9
Nurses	33	16.1
Medical Specialty
No/In progress	149	72.7
Yes	56	27.3
Length of work experience (years)
<=1	57	27.8
1-5	62	30.2
> 5	86	41.9
Workplace
Urban area	180	87.8
Rural area	25	12.2

Table 2 shows the mean scores of some group categories / variables. Statistical significant difference was only observed for the professional-category variable, which compared ophthalmologists and infectious disease experts to primary caregiving professionals (p = 0.001).

Table 2
Comparison among the mean scores of physicians and nurses who participated in the psychometric property assessment of the trachoma knowledge questionnaire.

Variable	Mean	SD	P value
Professional group 1			0.001*
Generalists/Family Physicians	22.88	6.15
Ophthalmologists/ infectious
disease experts	29.60	1.90
Professional group 2			0.351*
Physicians	23.06	6.25
Nurses	21.93	5.57
Sex			0.262*
Male	23.44	6.56
Female	22.62	5.95
Age Group			0.294*
< 30 years old	22.47	6.68
> = 30 years old	23.56	5.90
Marital Status	0.707*
Single	22.74	6.01
Married	23.04	6.32
Length of work experience			0.866**
< = 1 year	23.59	5.57
> 1 year	22.61	6.35
> 5 years	23.01	6.18
Workplace		0.880*
Urban area	22.93	6.22
Rural area	22.52	5.65

• ^*
  Mann-Whitney U-test;
• ^**
  Kruskal-Wallis U-test.

Table 3
Kappa statistic of trachoma knowledge questionnaire's reproducibility test.

Kappa statistic	Agreement Degrees	Items (Questions)*
≤ 0.40	Poor or slight	Q3, Q5, Q20 and Q28
0.41 - 0.60	Moderate	Q1, Q8, Q9, Q12, Q17, Q18, Q23, Q24 and Q29,
0.61 - 0.80	Substantial	Q6, Q7, Q11, Q13, Q16, Q21, Q22, Q27, Q35, Q36 and Q37.
> 0.80	Almost perfect	Q2, Q4, Q10, Q19, Q25, Q26, Q34, Q38, Q40 and Q41.

• ^*
  Questions 14, 15, 30, 31, 32, 33 and 39 were deleted prior to reliability analysis due to their low correct response rate.

Discussion

Primary care physicians and nurses' knowledge about trachoma was assessed through an independently prepared questionnaire, since no validated trachoma questionnaire was found prior to the present study. Psychometric tests showed acceptable questionnaire validity based on hypothesis testing. Cronbach's alpha reliability test revealed adequate internal consistency, whereas test-retest reliability showed good temporal stability.

Hypothesis testing evidenced that ophthalmologists and infectious disease experts were more informed on trachoma than primary care professionals. Data such as sex, age, marital status, length of professional experience and workplace did not influence the results.

Although the questionnaire originally comprised 52 items, they were reduced to 34, since items in all three questionnaire sections were deleted throughout the preparation process. Item deletion was encouraged both by experts' arguments during the content validity process and by their complexity (alleged by respondents). Researchers may have regarded too many trachoma data as relevant after reviewing the literature. Thus, they did not consider that some data may be too complex for primary care professionals. Accordingly, specialists play crucial role in ensuring coherence between questionnaires and their respective respondents.⁽¹⁸⁾ It is also worth noticing that questions with very low correct response rates are not suitable to assess construct validity.

Questionnaire assignment to different health care groups — a satisfactory condition for construct validity analysis, according to Pasquali⁽¹⁹⁾ — enabled hypothesis-testing performance. It also enabled the detection of trachoma knowledge gaps of primary care professionals. This result implies lack of education on trachoma since college, presumably due to misconception on disease eradication.⁽¹⁰⁾

Actions must be taken to eradicate trachoma in Brazil, since it remains a blindness-causing disease in many Brazilian regions. Accordingly, health professionals — mainly Family Health team physicians and nurses — must be trained to properly identify and tackle trachoma, given that primary health care is the gateway to the health system.

The final questionnaire has the potential to identify knowledge weaknesses of primary care physicians and nurses. In addition, it can help health managers to prepare training plans for their teams, as well as help assessing the impact of trachoma education strategies.

Figure 1
Questionnaire Items

STATEMENTS	Answer/Agreement
	1	2	3	4	5
Q1*. Trachoma is the 3rd major cause of preventable infectious blindness worldwide.	☐	☐	☐	☐	☐
Q2. Trachoma is considered a neglected tropical disease.	☐	☐	☐	☐	☐
Q3. Measures such as surgery (if necessary) antibiotics using, facial cleansing and environmental care are recommended to combat trachoma.	☐	☐	☐	☐	☐
Q4. Trachoma is usually associated with poor basic sanitation, hygiene and education.	☐	☐	☐	☐	☐
Q5. Estimates show 21 million people with active trachoma, of whom 2 million are blind or have severe visual impairment due to it.	☐	☐	☐	☐	☐
Q6. Active trachoma prevalence decreases with age.	☐	☐	☐	☐	☐
Q7. High prevalence of trachomatous trichiasis, scarring and corneal opacity in elderly is associated with early exposure to trachoma.	☐	☐	☐	☐	☐
Q8. The 50s-70s economic development period in Brazil encouraged significant decrease in trachoma, which created the "trachoma eradication misconception".	☐	☐	☐	☐	☐
Q9*. Studies have shown that trachoma occurs in few Brazilian regions.	☐	☐	☐	☐	☐
Q10. Trachoma is caused by the bacterium Chlamydia trachomatis.	☐	☐	☐	☐	☐
Q11*. Trichiasis is featured by the presence of inverted cilia irritating the cornea.	☐	☐	☐	☐	☐
Q12. Chlamydia trachomatis, mostly associated with genital infection, can also cause ophthalmia neonatorum in infants and inclusion conjunctivitis in adults.	☐	☐	☐	☐	☐
Q13. Trachomatous scarring has been associated with the presence of pathogens other than Chlamydia trachomatis.	☐	☐	☐	☐	☐
Q14. Trachomatous Inflammation - Follicular (TF) is featured by five or more follicles of at least 0.5 mm in diameter in the upper palpebral conjunctiva	DELETED
Q15. Trachomatous Inflammation - Intense (TI) occurs when conjunctival thickening in the upper eyelid obscures more than 50% of the normal deep tarsal vessels.	DELETED
Q16. Trachomatous Scarring (TS) is featured by apparent scars in the upper palpebral conjunctiva.	☐	☐	☐	☐	☐
Q17. Trachomatous trichiasis (TT) is featured by one or more eyelashes rubbing against the eyeball, or by scarring after upper eyelid surgery.	☐	☐	☐	☐	☐
Q18. Trachomatous Corneal Opacification (CO) in case of corneal opacity covering the pupillary margin.	☐	☐	☐	☐	☐
Q19*. Entropion is a condition in which the eyelid margin turns inward and causes eyelashes to move away from the cornea.	☐	☐	☐	☐	☐
Q20*. The World Health Organization created the trachoma grading system so that experts can quickly assess trachoma prevalence and severity within populations.	☐	☐	☐	☐	☐
Q21*. Chlamydia trachomatis may cause chronic conjunctivitis followed by intense symptoms, such as severe pruritus, hyperemia and eye discharge.	☐	☐	☐	☐	☐
Q22. Trachoma is often mistaken for allergic conjunctivitis, since both can coexist in the same patient.	☐	☐	☐	☐	☐
Q23*. Trachoma diagnosis is primarily clinical.	☐	☐	☐	☐	☐
Q24. WHO advises that trachoma should be diagnosed if at least two of the following symptoms are detected: follicles on upper tarsal conjunctiva, limbal follicles, typical conjunctival scarring and superior limbal pannus.	☐	☐	☐	☐	☐
Q25. The active trachoma types are: Trachomatous Inflammation - Follicular (TF) and Trachomatous Inflammation - Intense (TI).	☐	☐	☐	☐	☐
Q26. The scarring or sequelae types of trachoma are Trachomatous Trichiasis (TT) and Corneal Opacification (CO).	☐	☐	☐	☐	☐
Q27. Trachomatous scarring requires surgical intervention.	☐	☐	☐	☐	☐
Q28. Although trachoma prevalence in Brazil has been evidenced, this disease is not regularly discussed at medical schools and specialization courses.	☐	☐	☐	☐	☐
Q29. Trichiasis and entropion tend to recur after surgical treatment.	☐	☐	☐	☐	☐
Q30*. The treatment of choice for active trachoma is single-dose, orally administered ciprofloxacin.	DELETED
Q31. Erythromycin and doxycycline feature among trachoma systemic treatment alternatives.	DELETED
Q32*. Ciprofloxacin eye drops, tetracycline ointment and sulfa eye drops are modern treatment alternatives.	DELETED
Q33*. Mass treatment is recommended when trachoma prevalence rate is higher than 50%.	DELETED
Q34*. Trachoma has not yet been eradicated in most developed countries even after improved access to water, sanitation and housing.	☐	☐	☐	☐	☐
Q35. Conjunctival folliculosis, toxic follicular conjunctivitis, inclusion conjunctivitis and acute follicular conjunctivitis should receive differential trachoma diagnosis.	☐	☐	☐	☐	☐
Q36*. Children younger than 3 years old are the main reservoir of trachoma's infectious agent in endemic trachoma areas.	☐	☐	☐	☐	☐
Q37. Trachoma can only be transmitted when there are active lesions, which are more severe at the beginning of the disease and when they are caused by bacterial infections.	☐	☐	☐	☐	☐
Q38*. A single case of infectious conjunctivitis caused by etiologic agents is enough to determine clinical trachoma.	☐	☐	☐	☐	☐
Q39*. Active trachoma patients should be discharged one year after treatment had started, when trachoma symptoms are no longer observed.	DELETED
Q40. Suspected trachoma cases are considered serious when patients have history of recurrent conjunctivitis or persistent eye symptoms, such as: burning, itching, foreign body sensation, photophobia, excessive tearing and eye discharge.	☐	☐	☐	☐	☐
Q41*. Health education is unnecessary for trachoma prevention and control.	☐	☐	☐	☐	☐
(*) False statements.

Conclusions

The herein developed questionnaire presented good internal consistency and reliability results. Therefore, it was considered effective in developing educational strategies for primary care physicians and nurses.

• Research Institution: Montes Claros State University

Referências

• ¹
  Taylor HR, Burton MJ, Haddad D, West S, Wright H. Trachoma. Lancet. 2014;384(9960):2142-52.
• ²
  Hu VH, Harding-Esch EM, Burton MJ, Bailey RL, Kadimpeul J, Mabey DC. Epidemiology and control of trachoma: systematic review. Trop Med Int Health. 2010;15(6):673-91.
• ³
  Araújo Silva MB, Andrade MS, de Oliveira YV, Santiago Marques MG, Mourão Pinho C, Araújo de Miranda Lopes K. Trachoma as cause of blindness: literature review. Int Arch Med. 2017;10(56):1-9.
• ⁴
  Lansingh VC. Trachoma. BMJ Clin Evid. 9;2016:0706.
• ⁵
  World Health Organization (WHO). Alliance for the Global elimination of trachoma by 2020: progress report on elimination of trachoma, 2014-2016. Geneva: WHO; 2017.
• ⁶
  World Health Organization (WHO). Planning for the global elimination of Trachoma (GET): report of a WHO Consultation Geneva 1997. Geneva: WHO; 1997.
• ⁷
  World Health Organization (WHO). 51 st World Healty Assembly, WHA51. 11Global elimination of blinding trachoma. Geneva: WHO; 1998.
• ⁸
  Gómez DV, Lopes MF, Medina NH, Luna EJ. Tracoma: aspectos epidemiológicos e perspectivas de eliminação como problema de saúde pública no Brasil. eOftalmo. 2018;4(4):147-53.
• ⁹
  Brasil. Ministério da Saúde. Fundação Nacional de Saúde. Manual de controle do tracoma. Brasília (DF): Fundação Nacional de Saúde; 2014. p. 52.
• ¹⁰
  Schellini SA, Sousa RL. Tracoma: ainda uma importante causa de cegueira. Rev Bras Oftalmol. 2012;71(3):199-204.
• ¹¹
  Beeckman D, Vanderwee K, Demarré L, Paquay L, Van Hecke A, Defloor T. Pressure ulcer prevention: development and psychometric validation of a knowledge assessment instrument. Int J Nurs Stud. 2010 ;47(4):399-410.
• ¹²
  Bonett DG, Wright TA. Cronbach's alpha reliability: interval estimation, hypothesis testing, and sample size planning. J Organ Behav. 2015;36(1):3-15.
• ¹³
  Bland JM, Altman DG. Cronbach's alpha. BMJ. 1997;314(7080):572.
• ¹⁴
  Streiner DL. Starting at the beginning: an introduction to coefficient alpha and internal consistency. J Pers Assess. 2003;80(1):99-103.
• ¹⁵
  Cortina JM. What is coefficient alpha? An examination of theory and applications. J Appl Psychol. 1993;78(1):98-104.
• ¹⁶
  Streiner DL, Norman GR, Cairney J. Health measurement scales. A practical guide to their development and use. Oxford University Press; 2015.
• ¹⁷
  Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159-74.
• ¹⁸
  Coluci MZ, Alexandre NM, Milani D. Construção de instrumentos de medida na área da saúde. Cien Saude Colet. 2015;20(3):925-36.
• ¹⁹
  Pasquali L. Psicometria. Rev Esc Enferm USP. 2009;43(Esp):992-9.

Publication Dates

• Publication in this collection
  05 Feb 2021
• Date of issue
  Nov-Dec 2020

History

• Received
  26 June 2020
• Accepted
  23 Sept 2020