Validation of an instrument to identify actions for screening and detection of breast cancer

Abstracts

Objective:

To develop and validate a questionnaire to identify the actions performed in screening and detection of breast cancer in Brazil, and to determine its applicability.

Methods:

A methodological study, with the participation of three experts and a pilot test with 85 users of four primary health care services, with a descriptive data analysis.

Results:

Of the 132 questions formulated and organized in the structure and process dimensions undergoing validation, there was a 96.7% and 78.8% agreement of the evaluators in the first and second rounds, respectively. Most of the questions were understood by those involved in the investigation. The absence of the medical record resulted in the exclusion of 40 questions, resulting in 83 questions in the final version.

Conclusion:

The content of the instrument was adequate to evaluate actions to control breast cancer in primary care. The pilot test confirmed its applicability, and the need for improvements in documenting information.

Breast neoplasms/diagnosis; Primary health care; Health evaluation; Disease prevention; Validation studies

Objetivo:

Construir e validar um questionário para identificar as ações realizadas no rastreio e diagnóstico do câncer de mama no Brasil e determinar sua aplicabilidade.

Método:

Estudo metodológico, com participação de três especialistas e teste piloto junto a 85 usuárias de quatro serviços de saúde, com análise descritiva dos dados.

Resultados:

Das 132 questões formuladas e organizadas nas dimensões de estrutura e processo submetidas à validação, houve 96,7% e 78,8% de concordância dos avaliadores na 1ª e 2ª rodadas respectivamente. A maioria das questões foi compreendida pelos envolvidos na investigação. A ausência de registro no prontuário resultou no descarte de 40 questões, ficando 83 na versão final.

Conclusão:

O conteúdo do instrumento mostrou-se adequado para avaliar as ações para controle do câncer de mama na atenção básica. O teste piloto confirmou sua aplicabilidade e a necessidade de melhorias no registro das informações.

Neoplasia da mama/diagnóstico; Atenção primária à saúde; Avaliação em saúde; Prevenção de doenças; Estudos de validação

Introduction

Since 1984, Public Health Programs and Policy have directed measures for breast cancer control in Brazil. However, this condition is still a public health issue, as it remains the second most common cancer among women.^(1,2)

Within primary care, the focus of the government program over the past ten years has been: an annual clinical breast exam in women over 40; a mammogram every two years and an annual clinical breast exam between 50 to 69 years of age; an annual clinical breast exam and a mammogram in high risk groups starting at 35 years old; and a monthly breast self-examination as a complementary strategy for self-awareness of the body.

The World Health Organization emphasizes the priority of cancer control, and the implementation of actions, monitoring of those actions, and their continuous evaluation in order to guide decision making across the available resources.⁽³⁾ In Brazil, although the screening program for breast cancer was instituted between 2001 and 2006, a study conducted in 28 health public services with 2155 women affected by this condition showed that 39% of them were in an advanced stage (III and IV). In addition, 17% of these evaluations were inconclusive.⁽⁴⁾ These data suggest potential failures in breast cancer screening.

Validity corresponds to precision and the degree to which an instrument measures what it should measure.^(5,6) Although there is no perfect measurement, this is one of the essential requirements that a data collection instrument must have, and the disregard of content validity of an instrument may compromise its accuracy and, consequently, produce unreliable results.^(5,6) In this context, this study describes the development, validation and applicability of a questionnaire directed to the public health care users, to identify the actions performed for screening and diagnosing breast cancer in Brazil.

Methods

This methodological study describes the development, content validation and testing of a data collection instrument in three phases.^(5–7)

First phase - development of the instrument

1. Review of articles and documents about actions for breast cancer control in Brazil.

2. Framework selection: the National Breast Cancer Control Program was adopted as a benchmark for the questionnaire content, and Donabedian's model was adopted for evaluation of the health care service.⁽⁸⁾ This author proposes evaluation through the systematization of measurable attributes that represent the quality of services and/or stages of production (structure, process and outcomes).⁽⁸⁾ Structure refers to the resources used by the health care service, and the selected attributes were: availability of the physical structure and equipment, staffing and team qualification, existence and operation of logistical resources. The process corresponds to the set of activities developed between professionals and users. For evaluation, the presence and execution of flows and protocols and the availability of professional training were considered.⁽⁸⁾

3. Definition of the informant and the form of data collection: users of the primary health care service were chosen as informants about the use of the structure and services in this level of care. In order to minimize losses due to registration failures or recall bias, data collection through interviews (86 questions) and consulting of the medical records (46 issues) were proposed.

Second stage -content validation of the instrument^(5–7)

1. Selection of the validation technique - we chose the Delphi technique, which has advantages by eliminating the influence of direct interaction, distance communication, the production of large amounts of high-quality ideas and specificity, in addition to low cost of execution.

2. Selection of evaluators - content validation requires a subjective judgment about whether a measure makes sense intuitively. It refers to the degree to which an instrument represents a domain or the relevance of the items. However, the literature does not mention an ideal number of judges. Thereby, through convenience sampling, five experts were asked to participate. They were trained in the mastology area and/or evaluation of policies or health program focused on primary care. Only three responded to the questionnaires.

3. Degree of agreement analysis - the experts were asked to evaluate, by means of an instrument, the set of variables considered important by choosing one of the options: strongly agree, partially agree, and disagree. The criteria adopted for the consensus level of the evaluators were: 1 - Maintain the question whenever there was complete agreement among all evaluators; 2 - Redesign whenever the agreement was partial, or whenever only one evaluator disagreed whereas two of them completely agreed; 3 - Delete the question when there was partial agreement, or disagreement between more than one evaluator. Suggestions were also considered, which resulted in the creation of new questions and changes made by the authors, later justified and submitted to the judgment of experts in the subsequent round of evaluation. For each validation round, the mean of the proportions of the questions (items) considered relevant was calculated, i.e., those that obtained a complete agreement and/or a partial agreement by only one of the evaluators.⁽⁶⁾ As some authors suggest, we considered the minimum agreement of 70% for instrument validation.⁽⁷⁾

Third step - test of the questionnaire applicability, conducted in two weeks in February 2011

1. Study area – of the five municipal regions of São Paulo, the southeast region was selected because it is an area of education and university research. Within this territory of 211,89Km², 90 primary health care service exist that attended 585,120.00 women <20 years old per month.

2. Sample inclusion criteria - basic health units constructed after January 2006, and users aged≥35 years, being followed-up for more than three years in the service, who signed the Term of Free and Informed Consent Statement.

3. Sampling -. the budgetary and time constraints, as well as the population heterogeneity and extent of the study area hampered enlisting the women in the study area, leading to a complex sampling plan in two stages.⁽⁹⁾ This type of sample consists of selecting individuals belonging to subunits that concentrate on groupings forming conglomerates.⁽⁹⁾ A confidence level of 95% was considered, design effect equal to two with a sampling error of 5%, resulting in a sample of 760 users of 38 services. However, the instrument was administered by five trained interviewers to 85 users in four primary health care services, corresponding to approximately 10% of the sample.

4. Evaluation of the participants’ understanding and difficulties in the field – a content analysis of the reports of the trained interviewers was performed, about the questions that they and the users found difficult to understand, as well as difficulties encountered during data collection.

5. Time taken for instrument administration - the start and the end time of data collection was documented on the instrument itself, which enabled calculation of the mean time of the interviews.

6. Calculation of missing data - performed according to the distribution of the missing responses in relation to the number of interviews and records investigated, considering a 95% confidence interval.

   The development of the study met national and international standards of ethics in research involving human subjects.

Results

In the literature review, the actions aimed at primary health care of the breast cancer control program in Brazil were identified as priorities and selected as the standard for the construction of the instrument, as presented in chart 1.

Chart 1
Actions for the control of breast cancer in primary care

Actions	Periodicity	Indication	Necessary structure
Clinical breast exam	Annual	Prioritized women aged ≥ 40 years.	Trained nurse or physician, and offices.
Mammography	Depends on age	Women ≥ 50 years - every 2 years. Women ≥ 35 years at high risk for breast cancer - annual.	One mammography device per 240,000 inhabitants, referral availability, trained professionals.
Breast self- exam	Monthly	Self-awareness of the body of every woman. Any date for climacteric women and a week after menstruation for other women.	Nursing staff, physician, consultation office and educational material.
Educational meetings	Not mentioned	Every woman in the target group aimed at adherence to the actions.	Health professional and educational material.
Recall	Not mentioned	Prioritize women who do not have the tests done, those missing, and those with abnormal tests.	List of target women and health professionals.
Woman appointment book	Not mentioned	Every woman in the target group, aimed at adherence to the actions.	Women's and health professionals' schedule.
Mammography Information System (SISMAMA)	Not mentioned	Each worker in primary health care service and the imaging service completes the digital form of each mammography performed.	Computer with operating system and skilled administrative professional.

The actions listed in chart 1 were organized according to the attributes of the structure and process dimensions. The variables related to structure were: reason for rebooking or not having the mammogram and breast ultrasound, and schedule availability of the women. The reference variables for process were: identification of risk factors; implementation and teaching of clinical breast examination; request, execution and guidance regarding mammography; guidance and teaching of breast self-exam; performance and guidance regarding pap smear test of nursing and medical consultation. chart 2 provides a breakdown of the variables considered in the study and their grouping into nine blocks.

Chart 2
Organization of the user questionnaire questions to assess the actions for breast cancer control in primary care

	Number of questions
Blocks	Interview	Medical records	Variables
General information	8	0	Interview date, interviewer code, interviewer's name, type and name of the primary health care service, specification if Family Health Strategy staff, start and end time of data collection.
User identification	5	0	Medical record number, initials, address, telephone number and date of registration.
User information	8	0	Age, marital status, race, education, income source, household income, health insurance, criteria for use of the health insurance.
Risk factor	7	7	Family history of breast or ovarian cancer, breast cancer before 50 years of age, bilateral breast cancer at any age, male breast cancer, ovarian cancer at any age, breast biopsy, type of tumor biopsied.
General actions	9	2	Medical/nursing consultation in the last four years, frequency of visits per year, woman appointment book, professional who investigated risk factor, call to the primary health care service to make mammogram or clinical breast exam appointment, participation in appointment.
Breast self-exam	7	8	Age started, if menstruating, if breast self-exam is performed, frequency, how they learned, when they perform, reason why they do not perform.
Clinical breast exam	15	9	Age indication, performed by the primary health care service or complimentary health, year performed, difficulties, who made the request, the elapsed time between request and the result, site where performed, changes identified, conduct, reason for not having the exam at the basic health unit.
Mammography	15	10
Breast ultrasound	12	10
Total	86	46	=132

The experts participating in the content validation of the user instrument had ten to 20 years of experience; two were active in teaching and research in public health, and in the care and research in mastology.

In the first version of the instrument, eight of the 132 questions related to the identification of the primary health care service and the interviewer were not sent.

In the first round of the 124 questions evaluated in June of 2010, 66.9% (83) had full agreement between the three evaluators, 29.8% (37) had partial agreement, with 24.2% (30) of one evaluator and 5.6% (7) of two evaluators, and; 3.3% (4) had a disagreement of one evaluator, resulting in 42 questions being maintained, 61 reformulated, 21 excluded, and 11 created. From this analysis, the 114 questions added to the eight others relating to the identification totaled 122 questions in the second version.

In the second round of validation, performed in September 2010, 80 questions of the total 122 were evaluated, of which 63.8% (51) had full agreement of three evaluators, partial agreement was 12.5% (10) of one evaluator and 2.5% (2) of two evaluators; 21.2% (17) had disagreement of one evaluator; resulting in 47 questions maintained, 30 reformulated, three excluded, and five created, generating the third version with 124 questions.

In the first round, there was a 91.2% complete and/or partial agreement of only one of the evaluators. In the second round, 76.2% was obtained and in both rounds, the mean agreement was 83.7%.

After the instrument was field-tested and analyzed, 11 questions were identified that were considered difficult to understand by the interviewers, and included the following justifications and suggestions: item was not formulated as a question, reason for not performing a clinical breast exam independent of the patient, lacked alternative response, and the space for description of some data was considered insufficient. For the seven questions considered difficult to understand by the interviewees, the justifications and suggestions were: change the way of asking about income and education, replacing them with number of minimum wages and series, respectively; translate unfamiliar technical terms into popular language (biopsy, breast exam, breast ultrasound, breast self-exam and basic health unit); the term referral guide of the examination had better understanding than just referral, and the answer choice “housewife” caused discomfort.

The difficulties mentioned by the field team were: long form, with little space to provide the address and the medical record number; lack of explanatory text that elucidated the terms that were difficult to understand; need to change response options, and to avoid repetition of questions with the development of sub-questions.

The time spent in the interview of users varied, on average, from three to 18 minutes. In collecting data from the files of users, the poor quality of records and of their archiving resulted in low use of this source of information, since of the 40 questions collected, lost responses were ≥40%, and then were excluded. The suggestions were accepted and one question was created, resulting in the final version with 83 questions (Chart 3).

Chart 3
Organization of the user instrument questions in the versions resulting from the content validation process

First version			Second version		Third version		Final version
Block and number of questions			Block and number of questions		Block and number of questions		Block and number of questions
Source - interview
	General information	08	General information	07	General information	07	General information	07
	User identification	05	User identification	05	User identification	05	User identification	05
	User information	08	User information	08	User information	09	User information	09
	Risk factor	07	Risk factor	08	Risk factor	08	Risk factor	09
	Clinical breast exam	15	Clinical breast exam	11	Clinical breast exam	11	Clinical breast exam	09
	Mammography	15	Mammography	16	Mammography	17	Mammography	17
	Breast ultrasound	12	Breast ultrasound	14	Breast ultrasound	14	Breast ultrasound	14
	Breast self-exam	09	Breast self-exam	08	Breast self-exam	08	Breast self-exam	08
	General actions	07	General actions	05	General actions	05	General actions	05
	Sub-total	86	Sub-total	82	Sub-total	84	Sub-total	83
Source – medical records
	Risk factor	07	Risk factor	07	Risk factor	07		0
	Clinical breast exam	09	Clinical breast exam	06	Clinical breast exam	06		0
	Mammography	10	Mammography	10	Mammography	10		0
	Breast ultrasound	10	Breast ultrasound	10	Breast ultrasound	10		0
	Breast self-exam	08	Breast self-exam	04	Breast self-exam	04		0
	General actions	02	General actions	03	General actions	03		0
	Sub-total	46	Sub-total	39	Sub-total	40	Sub-total	0
Total geral		132		122		124		83

Discussion

The assessment of a measurement accuracy through construct, criterion and/or content validation of a data collection instrument is an item considered desirable in scientific research; additionally, the literature also recommends that the reproducibility is measured by other tests, namely measure of reliability or psychometric tests.^(5–7) Thus, other validation tests such as the criterion and construct and reliability measures may be applied to the questionnaire presented here, which was only submitted to content validation. It should be noted that obtaining the research authorizations of individuals and all the institutions involved (University, Evaluators, City Health Department, Southeast Regional Health Coordination, manager of primary health care service, and the user) required significant time. In addition, the magnitude of the phenomenon studied generated an extensive questionnaire, requiring even more time and articulation to complete its validation (eight months) and pilot test (two weeks), thus made impossible the measurement of its reliability.

Even so, this study allowed for the construction, content validation and measuring the applicability of the questionnaire for the evaluation of screening actions of the Brazilian program for breast cancer control.

Content validation requires availability of time to do the analysis from the evaluator, in addition to competence in the subject. This last factor probably contributed to only three of the five specialists who were invited to participate in the study, and resulted in a long period of time for them to send their responses. Supported by the literature, which mentions that no ideal number of judges exists,^(5–7) as well as the fact that the variables of the instruments have been extracted from a national public health program, previously obtaining consensus by specialists, it was considered that the assessment made by the three judges achieved its goal, since they considered that the instrument had incorporated most of the essential elements of the investigation.

The degree of agreement obtained for the user instrument, whether on the first round of validation, or the second, demonstrated the relevance of the questions. It should be noted that the observations of experts helped improve the content of the questions and the grouping of actions.

Part of the sample was used for the validation of the questionnaire, as well as logistical evaluation and feasibility studies. Thus, supporting the literature,^(10,11) the application of validated instruments and field team notes enabled measuring the mean time for data collection, the level of understanding of the content, helped identify the main difficulties, the possible conditioning factors, and the means to circumvent them. The suggestions and comments of the interviewers about the content of the instrument helped make the language of some questions accessible to the target population.

With regard to interviews with users, despite having been referred by the interviewers that some technical terms appeared to be unknown to them, the low absence of response rates to the questions suggests that their formulation favored understanding by the target audience saving time, probably associated with the training offered, suggesting that this model could be applied in a larger sample.

In an instrument, many missing data may indicate poor formulation of a particular item or difficulty in data collection.^(12,13) It must be considered that a variable can also be investigated in a cluster of related questions (sub-questions) which, depending on the alternative chosen, could lead to no response to the others. This situation was identified in this study, a fact that led to the maintenance of many questions in the instrument, although there were significant losses. Regarding the medical records of the users in the primary health care service, the decision to dismiss them was mainly due to the absence of records of the professionals, and the low quality of archiving of the information.

Missing data made it difficult to analyze the results of the research, because the majority of these procedures were not designed for them. Although not the main focus of research, missing data is usually a nuisance and handling it has been a computational challenge.^(12,13) Missing data may generate two major problems. The first is the reduction of statistical power, namely, reduced power to find an association between a data set; and, the second is the possibility of directing a biased estimate. Among the various possibilities of existing treatments, the literature supports the disposal of the variable that does not have an important effect along with the outcome.^(12,13)

The expansion of health care through decentralization and focusing on preventive actions has been gradually occurring since 1988.⁽¹⁴⁾ In the city of São Paulo, this network reorganization started in the year 2000, and during this period, research that used data from medical records showed the poor quality of records and storage of information. After 11 years, the same situation is perpetuated, indicating the existence of gaps in clinical consultations of physicians and nurses, failures in auditing services, and the fragility of this source. The medical record is a collection of documents in which health professionals describe patient data in a standardized, organized and concise manner. These recordings guarantee the continuity of care, security of professional and patient. It is also useful for teaching and conducting research and audits. The absence or poor quality of records makes it difficult to monitor and evaluate health practices, as well as to meet needs, failing, in this case, to contribute to improvements in public service care delivery in a manner that can resolve the population's needs.

The absence of medical record showed that this source of information is inadequate to monitor the practices and needs improvement.

Conclusion

The validation process resulted in adequacy of the content in the questionnaire developed to measure screening actions recommended by the National Program for Breast Cancer Control. Also, the reduced missing data in the interviews, which were the reference for understanding of most questions, as well as the few difficulties in the field and the time spent on data collection, indicate that the validated instrument is applicable.

Acknowledgements

We thank the National Council of Science and Technology (CNPq) for granting the level 1B productivity in research grant, and the Higher Education Personnel Training Coordination (CAPES) for granting of the master–s scholarship.

Referências

• ¹
  Goss PE, Lee BL, Badovinac-Cmjevic T, Strasser-Weippl K, Chavarri-Guerra Y, St Louis J, et al. Planning cancer control in Latin America and the Caribbean. Lancet Oncol. 2013; 14(5):391–436.
• ²
  Ferlay J, Shin HR, Bray F, Forman D, Mathers C, Parkin DM. Cancer incidence and mortality worldwide: GLOBOCAN 2008. [cited 2014 Nov 21]. Available from: http://globocan.iarc.fr
  » http://globocan.iarc.fr
• ³
  World Health Organization. Prevention. Fight Against Cancer: Strategies that prevent, cure and care. Geneva: WHO; 2007. 28 p.
• ⁴
  Liedke PE, Finkelstein DM, Szmonifka J, Barrios CH, Chavarri-Guerra Y, Bines J, Vasconcelos C, Simon SD, Goss PE. Outcomes of breast cancer in Brazil related to health care coverage: a retrospective cohort study. Cancer Epidemiol Biomarkers Prev. 2014;23(1):126–33.
• ⁵
  Keeney S, Hasson F, McKenna H. Consulting the oracle: ten lessons from the Delphi Technique in nursing research. J Adv Nurs 2006;2(53):205–12.
• ⁶
  Brod M, Tesler LE, Christensen TL. Qualitative research and content validity: developing best practices based on science and experience. Qual Life Res. 2009;18(9):1263–78.
• ⁷
  Akins RB, Tolson H, Cole BR. Stability of response characteristics of a Delphi panel: application of bootstrap data expansion. BMC Med Res Method 2005;5:37.
• ⁸
  Donabedian A. The quality of care. How can it be assessed? JAMA. 1988; 260(12):1743–8.
• ⁹
  Szwarcwald CL, Damacena GN. Amostras complexas em inquéritos populacionais: planejamento e implicações na análise estatísticas dos dados. Rev Bras Epidemiol. 2008; 11(Supl 1):38–45.
• ¹⁰
  Rees C, Shepherd M. Students’ and assessors’ attitudes towards students’ self-assessment of their personal and professional behaviors. Med Educ. 2005; 39(1):30–9. Erratum in Med Educ. 2006;40(1):88.
• ¹¹
  Davis RE, Couper MP, Janz NK, Caldwell CH, Resnicow K. Interviewer effects in public health surveys. Health Educ Res. 2010;25(1):14–26.
• ¹²
  Williams R. Fitting heterogeneous choice models with olgm. Stata Journal. 2010; 10(4):540–67.
• ¹³
  Schafer JL, Graham JW. Missing data: our view of the state of the art. Psychol Methods. 2002;7(2):147–77.
• ¹⁴
  Paim J, Travassos C, Bahia CA, Macinko J. The Brazilian health system: history, advances and challenges. Lancet: Health in Brazil 1. 2011 may. DOI:10.1016/S0140-6736(11)60054-8.
  » https://doi.org/10.1016/S0140-6736(11)60054-8

• Erratum

  In the article published in Acta Paul Enferm. 2015; 28(2):183-89, Marques CA, Figueiredo EN, Gutiérrez MG; “Validation of an instrument to identify actions for screening and detection of breast cancer”, the authors requested to publish the following errata: Include the instrument, final product, of the validation process.
  

  
  Form to identify the breast cancer early detection action - to Female patient
  

Publication Dates

• Publication in this collection
  Mar-Apr 2015

History

• Received
  28 Sept 2014
• Accepted
  26 Nov 2014