Strategies and development of quality assurance and control in the ELSA-Brasil

Abstracts

The ELSA-Brasil (Estudo Longitudinal de Saúde do Adulto - Brazilian Longitudinal Study for Adult Health) is a cohort study composed of 15,105 adults followed up in order to assess the development of chronic diseases, especially diabetes and cardiovascular disease. Its size, multicenter nature and the diversity of measurements required effective and efficient mechanisms of quality assurance and control. The main quality assurance activities (those developed before data collection) were: careful selection of research instruments, centralized training and certification, pretesting and pilot studies, and preparation of operation manuals for the procedures. Quality control activities (developed during data collection and processing) were performed more intensively at the beginning, when routines had not been established yet. The main quality control activities were: periodic observation of technicians, test-retest studies, data monitoring, network of supervisors, and cross visits. Data that estimate the reliability of the obtained information attest that the quality goals have been achieved.

Quality Control; Quality Indicators, Health Care; Organization and Administration; Multicenter Studies as Topic, methods; Cohort Studies

O Estudo Longitudinal de Saúde do Adulto (ELSA-Brasil) é um estudo de coorte composto de 15.105 adultos acompanhados para avaliar o desenvolvimento de doenças crônicas, especialmente diabetes e doença cardiovascular. Seu porte, natureza multicêntrica e diversidade de medidas exigiram mecanismos ágeis e efetivos de garantia e controle de qualidade. Entre as atividades de garantia de qualidade (aquelas desenvolvidas antes de iniciar a coleta de dados), destacam-se: seleção criteriosa dos instrumentos de pesquisa, treinamento e certificação centralizados, pré-testes e estudos pilotos, e elaboração de manuais de operações para os procedimentos. As atividades de controle de qualidade (realizadas durante a coleta e processamento dos dados) foram efetuadas mais intensivamente no início, quando as rotinas ainda não estavam estabelecidas. Entre elas, ressaltam-se: observação periódica dos aferidores, estudos de teste reteste, monitoramento dos dados, rede de supervisores e visitas cruzadas. Dados que estimam a confiabilidade das informações obtidas atestam que as metas de qualidade foram alcançadas.

Controle de Qualidade; Indicadores de Qualidade em Assistência à Saúde; Organização e Administração; Estudos Multicêntricos como Assunto, métodos; Estudos de Coortes

INTRODUCTION

Health research aims primarily to find answers to diverse health issues, and the integrity of results is strongly determined by the quality of the information produced.^{1 , 5 , 6} To guarantee the quality of the data in the Estudo Longitudinal de Saúde do Adulto (ELSA-Brasil - Brazilian Longitudinal Study for Adult Health), Quality Assurance and Control (QAC) tools were developed, like the ones described by Szklo & Nieto,⁷ for the planning, execution and analysis of epidemiological studies.

ELSA-Brasil is a cohort study that aims to follow up 15,000 participants, in six research centers, based on interviews and tests of varied complexities. These characteristics - large size, longitudinal nature, multicenter organization, measurements of distinct complexities - structured the strategic definition of the QAC system to be adopted.

Experience in planning and conducting large longitudinal studies, especially from the 1950s onwards, has enabled the development of QAC tools to be used in cohort studies⁸ and in randomized clinical trials.⁴ The QAC system of Elsa-Brasil was based on this international experience, and necessary adaptations were performed by the Steering Committee and its Advisory Committees, founded on principles outlined by the QAC Committee.

The present study aimed to describe the QAC actions in the ELSA-Brasil, presenting them in the sequence in which they were developed and indicating how and when key QAC decisions were adopted in the construction of the cohort and in its follow-up.

QUALITY ASSURANCE

Quality assurance activities in the ELSA are defined by the set of actions developed before the beginning of data collection, in order to ensure the quality desired for the study's results. They are:

• development of the research protocol;

• selection of research instruments to fulfill the objectives of the study, based on literature data;

• development of the Operations Manual, composed of specific manuals for interviews, tests, and procedures, such as electrocardiogram, echocardiography, and others;

• pretesting of the instruments;

• training and certification of the data collection team;

• pilot studies;

• quality control system for data collection (for the initial stage and for the entire duration of the study);

• pretesting of the system of data entry and management.

Development of the research protocol

The basic points of the research protocol - objectives, research design, characteristics of the studied population, sample size, initial selection of the instruments to obtain information of interest, as well as logistic, operational and financial aspects - were developed by the Steering Committee and its Advisory Committees. Small adjustments to the initial protocol were performed during the operationalization of the research.²

Selection of instruments

The proposals for research instruments (for tests or interviews) were presented by the researchers in standardized forms, in which the conceptual and methodological aspects of the instrument were specified, as well as the logistics for its application, the relation to the objectives and the adequacy to the context of the ELSA (Table 1). The forms were available on the ELSA platform and presented during the discussions of the Steering Committee and its Advisory Committees, which facilitated the debate and selection of the instruments to be included in the research. For example, in 2007, it was decided to include the glycated hemoglobin test (HbA1c) in the baseline of the ELSA. The decision was based on data presented about the viability of assuring quality and standardizing the techniques involved, and also on the possibility of the HbA1c starting to be employed in the definition of cases of diabetes, a position that was effectively taken by international entities two years later. The process of selection of the questionnaires to be included in the research was similar.

Table 1
Example of a Quality Assurance and Control form

Objective/hypothesis
Justification
	Relation to study’s objectives
	Relevance, originality and innovation
Description
	Description of the test, technique, grounding, alternatives, advantages and disadvantages of more complete or more summarized procedures (for example, image tests with 1 or with 2 photos)
Validity
Precision, reliability
Logistics
	Duration
	Inconveniences to participants
	Cost
	Factors that need to be considered in standardization: fasting, rest, obesity, flu, use of medicines, and others
Bibliographic references
Appendices
	Test protocol (indicating if they are contained in Manuals from other studies)
	Instructions (indicating if they are contained in Manuals from other studies)

Pretesting of the instruments

The instruments of the study, whose validity and reliability were already known, were pretested in the research context of the ELSA-Brasil. Generally speaking, these pretests were carried out with simple protocols, aiming to meet the practical requirements that were more relevant to each situation. The Advisory Committees and/or Results Checking Centers performed, in cooperation with the Investigation Centers (ICs), as many pretests as necessary before including them in the project's Operations Manual, so as to begin the training of the research team.

In some cases, it was necessary to perform some kind of formal validation of the instrument before it was adopted by the study, like, for example, the cross-cultural adaptation of the Clinical Interview Schedule-Revised to the Portuguese language.

Development of the Operations Manual

The development of an Operations Manual, with a clear and detailed description of all the activities, was an essential task in the planning stage. The final product, totaling 19 specific manuals, constituted the study's "navigation map", a written reference to the research team and to the investigators. Produced by the Advisory Committees of the Steering Committee, the manuals, many of them revolving around specific QAC aspects, contain, besides routine procedures, mechanisms for decision-making in exceptional or unpredicted situations.

Training and Certification of the Data Collection Team

Trainings and certifications were planned before the pilot study and performed by technical nuclei with representation of the Advisory Committees and/or Reading Centers, according to the specificities of each procedure.

Training was executed in a centralized way to ensure uniformity. However, for more complex procedures or in view of a large number of people to be trained, the training team went to the other centers, and, whenever possible, teams from two or three centers were trained at the same time.

Supervisors were initially trained and certified in a central level, and then they acted as local trainers and certifiers, aiming to guarantee the uniformity of the adopted standards in all the ICs. During data collection, when it was necessary to substitute members of the research team, new trainings were performed. In some cases, centralized trainings or trainings given by local professionals were subsequently repeated to revise routines and update techniques and procedures.

Certification of the research team was carried out at the end of the training process or sometime later, in case it required more practice with the procedure. To each technique or procedure, a special type of certification was planned, such as: interviews observed by a supervisor or an expert in a specific questionnaire (cognitive function and food frequency questionnaire); review of images or signs by experts in Reading Centers; repetition of tests for reliability analyzes (anthropometry); evaluation, by experts in audio-recordings and/or printed copies, of notes made during the interviews or tests.

Pilot Studies

When the training of the team was completed, pilot studies were conducted in series and with increasing complexity. The last one included the entire research protocol - interviews, blood and urine collection, and also the processing, freezing and transport of biological samples. In each one of the pilot studies, the flowchart to execute the procedures was tested, including the operational difficulties for simultaneous execution in multiple participants, taking into account the limitations of number of rooms and equipment available at the ICs. When problems were found they were discussed and the necessary alterations were made to the protocol and manuals.

QUALITY CONTROL AT THE INITIAL STAGE OF DATA COLLECTION

Quality Control (QC) activities in the ELSA are actions performed during data collection and processing in order to monitor quality and to quantify data reliability. These tasks were carried out at the initial stage and throughout data collection.

Data collection started between August and November 2008 among the different ICs and it was concluded between March and December 2010. Factors that led to distinct periods of data collection were, among others, the number of participants and the availability of the Research Center's own physical space at each IC.

Data collection started with few studies per week and a gradual increase was planned according to the experience acquired by the research research team in relation to the protocol and to the flowchart of tests and interviews. This enabled a more intensive quality control, with the development of field diaries, which were revised at the end of the day and discussed in weekly meetings with local supervisors. Information that required personnel development, technique improvement or revision of the manual was registered and shared with the supervisors of the other centers by means of electronic mail, discussion groups in the ELSA platform, audio-conference by telephone or through the Internet, and sometimes, in face-to-face meetings. Doubts and difficulties were annotated on the manuals and communicated to the people responsible for their writing. At this stage, final amendments were made to the forms and questionnaires and their respective manuals.

As data collection was divided into two distinct phases, with research teams and physical spaces that were frequently distinct, a good interaction among the teams was essential. At this initial stage, it was possible to identify and review specific problems, like errors in the protocols followed by participants concerning 12 hours fasting and urine collection. As a result, specific reminders were generated, transmitted in the telephone calls that preceded the visit to the ELSA Research Center, called "ELSA calls".

Respect for the ethical criteria defined by the study implied the adoption of certain procedures to guarantee the confidentiality of information, like registering only the recruitment number on forms and questionnaires. As the teams (stages 1 and 2) interacted with the participants using their names, special care was taken to maintain together all the papers filled for each participant number. The list with names and recruitment numbers was under the responsibility of the supervisors, and it was consulted only in some situations, like in the delivery of test results. The research team signed a Responsibility Document about Data Confidentiality.

The local supervisors were responsible for the accomplishment of the daily tasks. The experience developed at the initial stage was shared among the ICs in face-to-face meetings, in order to conclude the necessary standardization before reaching the average number of participants/day intended at each IC. This occurred because some local aspects required flexibility of the protocol; for example, characteristics of the headquarters, size of the center, profile of the research team, the established functional hierarchy, some aspects of the participants' flow in the Research Center, recruitment strategies, delivery of test results, etc. The alterations were registered on the reports presented in meetings of the Steering Committee or in the supervisors' meetings, in visits to the centers or in other opportunities of interaction.

Organization of the data collection team and of the supervision process

Hierarchical aspects and individual and group responsibilities were defined and agreed with the research team to enable adequate supervision. Although at the beginning one or two supervisors were sufficient, the growing number of participants required the increase in the number of supervisors for specific activities. For an adequate management of the research team, and to ensure its technical excellence and the relational care with the participants, some activities were emphasized:

• meetings with the team (at least on a monthly basis and whenever necessary);

• registration of punctuality and assiduity;

• verification of the use of the uniform and overall presentation of the team;

• verification of adherence to protocols and manuals;

• control of days-off, absences, time of arrival and departure;

• register of complications on the field diary or on an adequate form;

• daily reading and verification of each team's field diaries.

Verification of the central standardization for the local definition of flows of interviews and tests

Before beginning the flow designated to each participant in the Research Center, the date and recruitment number were checked. Alterations to the flow (upon the participant's request due to some unforeseen event, delay, indisposition, loss of electricity, among others) were made with the supervisors' authorization. The limits of this flexibility complied with the norms established in the protocols of tests and measurements, and respected specifically the schedule, the fasting period, the HVAC system in the rooms, and the order of tests and questionnaires, as indicated on Table 2.

Table 2
Minimum standardization required in planning the flow of participants on the occasion of their permanence in the Investigation Center.

Tests/Standardizations	Obligatory fasting	Test must be performed 2 hours after the beginning of the ingestion of the glucose solution	Empty bladder	Test must be performed at least 30 minutes post-prandial or after the ingestion of the glucose solution	Temperature of the room: 20ºC-24ºC
Blood pressure	X		X		X
Heart rate	X
Weight and waist	X		X
Blood collection	X	X			X
Electrocardiogram				X
Heart rate variability				X
Ankle brachial index					X
Postural change					X

Quality control of equipment, rooms and materials

The temperature in the rooms for blood collection and measurement of blood pressure was monitored on a daily basis at different times (7 a.m., 9 a.m., 11 a.m., 1 p.m. and 3 p.m.), and the aim was to maintain it between 20^°C and 24°C. The registers were filed and then checked by the QAC Committee during the in loco visits.

The equipment was checked periodically at each IC, following the instructions of the specific manuals. The scales were calibrated and the measuring tapes were regularly verified. The other equipment was checked concerning its operation: good state, calibration, need of maintenance, need of substitution, battery replacement.

The organization of the rooms and the cleanness of the environment included cleaning the equipment according to the protocol; cleaning the cuffs for measuring blood pressure; and accomplishing the quantitative control of the necessary material to perform the tests and measurements (electrodes, ECG paper, gauze, alcohol 70%, conductive gel, gloves, among others).

Quality of the filing of questionnaires and forms

After being revised, the questionnaires and forms were filed with the last adopted version. The quality control registers concerning equipment, test-retest, room temperature, complications, etc. were also filed. These files were available to be accessed on the ELSA platform.

Observation of the research team's technique and attitude

Periodic observations performed by local supervisors aimed to identify which techniques needed to be improved and which points of the protocol had not been well understood. In addition, periodic observations intended to maintain the good performance of interviewers and technicians, so that the quality of their work was not reduced in the daily routine.

Generally speaking, supervisors verified at least one type of measurement of each technician per month, using specific checklists. Inadequate performances of the team triggered immediate corrective actions, and these might include re-training. Depending on the person's previous performance, on the magnitude of the error, and taking into account personal circumstances, the appropriate action could be a simple conversation, orientations to stimulate a better performance, or even the replacement of the person.

The administered questionnaires were revised in the paper format by the supervisors to identify problems and to be returned to the team of interviewers.

Quality control of the sending of tests to Reading Centers or the Laboratory/Biobank

The tests that required centralized reading assessment were sent to the respective Reading Centers. The centers that adopted the system Digital Imaging and Communications in Medicine (DICOM) sent the tests through the internet. The biological samples were stored at 70°C and transferred to the central laboratory according to the project's protocol and to the schedule established to each center. For the quality control of the supervision activities, worksheets were developed, containing information on the sending of tests, receipt of results and scheduling of pending tests.

QUALITY CONTROL DURING THE STUDY

The routines developed at the initial stage of the study, including the revision of forms and questionnaires, were maintained during the study. After the initial difficulties were overcome, studies with the maximum projected number of participants per day were encouraged.

The following quality control techniques were utilized during the study:

Team monitoring

Periodic observations by local supervisors to identify protocol deviations were made during the study and became the main periodic re-certification mechanism of the team. Specific checklists adopted at the initial stage of data collection were revised and simplified to be used during the rest of the study. The frequency of verification was stipulated in the corresponding manual. Inadequate team performances required re-training and re-certification.

For the interviews, the audio-recording mechanism was used. Periodically, all the interviews of a certain week were recorded and two were drawn to be evaluated by a supervisor. Emphasis was given to the interview's fluency, the correct completion of the answers and to adequacy in the interaction with the participant. The recordings were also evaluated by supervisors from other centers and subsequently discussed within the supervisor networks, usually by audio-conference.

When data entry occurred directly in the system, the correct marking of the interview answers (blank questions, skip errors or other inconsistencies) was made directly by the system. The other aspects were evaluated by audio-recording, which was carried out in all the interviews.

Test-retest monitoring

The technique of random repetition of measurements by the same person or by different people was used for some tests and questionnaires. In order not to tire the participant out, a maximum period that each repetition could add to the study of each participant was established. So as not to overload the team, which was already busy with flowcharts of tests and interviews for multiple participants and a limited number of available rooms, the repetitions were alternated for each procedure, demanding a maximum period of time allowed per day, which usually was not superior to 30 minutes. The repetition of interviews was performed on another day, respecting a maximum period of two weeks.

Care was taken so that test and re-test were really independent, that is, the technician/interviewer could not see/remember the previous results. In the case of waist circumference, for example, it was verified whether the ink marks made to the participant's skin had been removed. For the laboratory measurements, the samples were masked to prevent the analyst from identifying the repeated samples.

These studies allowed to estimate the proportion of total variation of a measure (interval variable) that is due to variation among individuals and, in some cases, they also enabled to quantify inter-observer and intra-observer variability.⁷ A similar technique was used to evaluate the reliability of laboratory determinations and of the readings performed in the Reading Centers (for example: ultrasonography protocol to track intrahepatic and abdominal wall fat, evaluation of the carotid intima-media thickness, echocardiography and retinography). For the analysis of the reliability of the laboratory determinations during the study, "masked" aliquots of a random sample of specimens taken to the central laboratory of the study, located in São Paulo, were utilized.

Whenever pertinent, test-retest was evaluated by the agreement verified between the two tests, using kappa coefficients.⁷

Two important benefits were achieved with the tests-retests. Firstly, the team felt stimulated to maintain its good performance and the detected deviations led to improvements in the technique. Secondly, it was possible to quantify the reliability degree of the data obtained in the study, which enabled to document the success achieved by the quality assurance and control system. Some parameters of this evaluation are illustrated on Table 3. In the future, specific papers will detail the employed methodologies.

Table 3
Estimates of the intra-class correlation coefficients and corresponding 95% confidence intervals of some tests and questionnaires re-tested during data collection. ELSA-Brasil, 2010.

Variable		ICC	95%CI
Test
	Systolic Blood Pressure	0.88	0.82;0.91
	Diastolic Blood Pressure	0.89	0.83;0.92
	Waist	0.98	0.85;1.0
	Pulse Wave Velocity	0.90	0.77;0.95
Questionnaire (Neighborhood)
	Social Cohesion	0.83	0.78;0.87
	Environment for Physical Education	0.90	0.87;0.92

New trainings, maintenance of certification and re-certification

New trainings and certifications for the expansion of the team or substitution of its members overloaded the project's local supervision, as they were very busy with the daily routine. Thus, as the study was already being conducted, the training included the observation of data collection (with the participants' consent). This allowed to simplify the initial model of centralized training. Illustrative videos of the general approach to the participant were also used.

To ensure that the interviewers/technicians maintained their skills, mechanisms were implemented to verify the certification that had been previously granted by the project. One of the criteria adopted was to guarantee the regular performance of a minimum number of interviews or tests during a certain period, for example, every two months. In case of a longer temporary license, the technique was revised with the supervisor when the professional returned. In addition, when it was observed that an interviewer/technician ignored an aspect of the technique, he/she underwent re-training/re-certification or, at the discretion of the local supervision, he/she was substituted. Protocol deviations due to routine or tiredness were immediately communicated.

Supervisors network

The construction of supervisors networks was a mechanism that was very appreciated by the research team. Two main networks, one for supervisors of interviews and the other for supervisors of measurements, were maintained during data collection. The interaction mechanisms that were used were audio-conference (usually on a monthly basis), e-mails, and some face-to-face meetings. The interaction moments enabled to share experiences and to discuss mistakes and achievements, being an opportunity for the teams to express solidarity (for example, exchange of materials and information).

Periodic site visits

Periodic site visits to each IC to check whether the project's activities were being carried out according to the protocol and to exchange experiences were performed at two moments: during data collection and after its conclusion. These visits, called cross visits, were performed according to the protocol established by the QAC committee.

During data collection, the cross visits aimed to check the standardization achieved, as well as to identify facilitating situations and problems to be considered in the planning of the second stage of the study. The visits involved the IC coordination, the local technical team and at least two representatives of the QAC Committee from other ICs. An outline containing the items to be observed was applied, concerning the performance and register of the quality control measures of interviews, measurements, laboratory tests and other aspects selected from the baseline study. Moreover, the registers of the procedures adopted to publicize the study and to recruit ELSA participants were consolidated.

After data collection, the cross visits aimed to verify the procedures adopted in the supervision of questionnaires, forms and data entry. Based on a list of participants from each center, provided by the Data Center and organized randomly, registers to be verified were identified. Approximately 1% of the files of each IC were revised according to a previously defined script. For the quality control of data entry, the data entered in the system were compared to the information contained in the handwritten data collection form.

Data monitoring and periodic reports

When collection was performed with data entry directly in the electronic system, it was possible to control some errors in the interviews and in the completion of the questionnaires, like skip errors, questions that had not been filled, outliers or discrepant values.³ Periodic reports were produced to meet the study's recruitment goals ,and reports on the data incorporated into the system were also generated, in order to evaluate the completeness of the activities performed and to request clarification of inconsistencies.

FINAL REMARKS

The study's pioneering nature in many aspects required the development of new expertise, which many times did not exist in Brazil. To build the QAC procedures, the researchers consulted manuals of projects like ARIC, MESA and HCHS/SOL. Other times, direct contacts were made with researchers who were experienced in specific topics. In this process, the ELSA-Brasil innovated and adapted procedures and manuals to the Brazilian reality and to the current moment of its conduction. Among the QAC innovations, we highlight the supervisor network, constructed to discuss errors, difficulties and achievements. The reliability evaluated by repeated measurements obtained during the ELSA data collection shows that the system adopted was successful, as it has achieved international quality standards. The acquired experience can be useful also to other Brazilian epidemiological studies.

REFERÊNCIAS

• ¹
  Almeida Filho N, Barreto ML. Epidemiologia & saúde: fundamentos, métodos e aplicações. Rio de Janeiro: Guanabara Koogan; 2011.
• ²
  Aquino EML, Vasconcellos-Silva PR, Coeli CM, Araújo MJ, Santos SM, Figueiredo RC, et al. Aspectos éticos em estudos longitudinais: o caso do ELSA-Brasil. Rev Saude Publica 2013;47(Supl 2):19-26.
• ³
  Duncan B, Vigo A, Hernandez E, Luft VC, Ahlert H, Bergmann K, Mota E. Gerência de Informação em Estudos Multicêntricos: o Estudo Longitudinal de Saúde do Adulto. Rev Saude Publica 2013;47(Supl 2):95-104.
• ⁴
  Gagnon J, Province MA, Bouchard C, Leon AS, Skinner JS, Wilmore JH, et al. The HERITAGE Family Study: quality assurance and quality control. Ann Epidemiol 1996;6(6):520-9.
• ⁵
  Hulley SB, Cummings SR, Browner WS, Grady DG, Newman TB. Delineando a pesquisa clínica: uma abordagem epidemiológica. 3.ed. Porto Alegre: Artmed; 2008.
• ⁶
  Rothman KJ, Greenland S, Lash T. Epidemiologia moderna. Porto Alegre: Artmed; 2011.
• ⁷
  Szklo M, Nieto FJ. Epidemiology: beyond the basics. 2.ed. Sudbury (MA): Jones and Bartlett Publishers; 2007.
• ⁸
  Whitney CW, Lind BK, Wahl PW. Quality assurance and quality control in longitudinal studies. Epidemiol Rev 1998;20(1):71-80.
• ⁹
  World Health Organization. Use of glycated haemoglobin (HbA1c) in the diagnosis of diabetes mellitus: abbreviated report of a WHO Consultation. Geneva; 2011. (WHO/NMH/CHP/CPM/11.1.).

• Article available from: www.scielo.br/rsp

Publication Dates

• Publication in this collection
  June 2013

History

• Received
  26 Oct 2011
• Accepted
  04 May 2012