Indexing of clinical trials in LILACS: Assessment of 721 articles published in cardiology journals

Indexação de ensaios clínicos na LILACS: avaliação de 721 artigos de periódicos de cardiologia

Abstract

Systematic reviews are considered the highest level of evidence for decision making in health care issues. One of the first steps of a SR involves identifying all relevant clinical trials on the topic of interest. However, the retrieval of clinical trials in a database partially depends on the article indexing quality. The aim of this article is to evaluate the adequacy of indexing of clinical trials as a publication type in the LILACS database in a sample of articles published in cardiology journals. This cross-sectional study analyzed the indexing quality of clinical trials published between 2008 and 2009 in cardiology journals indexed in LILACS. Two independent reviewers identified and reclassified all original studies published in these journals as being clinical trials or other types of studies. The result of their classification was compared with the indexing publication type produced by LILACS. A total of 721 articles published in 11 cardiology journals were included. The reviewers classified 63 articles as clinical trials; 44 of these were correctly indexed in LILACS, while 19 were indexed as other types of studies (false negatives). The reviewers classified 658 articles as non-clinical trials; 651 were correctly indexed and 7 were incorrectly indexed in LILACS as being clinical trials (false positives). The sensitivity, specificity and global accuracy of LILACS indexing were 69.8%, 98.9% and 96.4% (695/721), respectively. Almost one third of the clinical trials published in LILACS-indexed Cardiology journals are not adequately indexed. The indexing quality of the studies published in these journals must be improved.

Keywords:
Indexing as topic; Bibliographic databases; Clinical trials as topic; Quality control; Periodicals as topic

Resumo

As revisões sistemáticas são consideradas o mais alto nível de evidência para a tomada de decisão em questões de cuidados de saúde. Um dos primeiros passos de uma RS envolve a identificação de todos ensaios clínicos relevantes sobre o tema de interesse. Porém, a recuperação de ensaios clínicos em uma base de dados, depende em parte da qualidade da indexação. O objetivo deste artigo é avaliar a adequação da indexação dos ensaios clínicos como tipo de publicação na base de dados LILACS, em uma amostra de artigos publicados em periódicos de cardiologia. Estudo transversal de análise da indexação dos ensaios clínicos publicados entre 2008-2009 em periódicos de cardiologia na LILACS. Duas revisoras identificaram e reclassificaram, de forma independente, todos os estudos publicados nesses periódicos e compararam sua classificação tipo de publicação com a indexação LILACS. Foram incluídos 721 artigos publicados em 11 periódicos. Os revisores classificaram 63 artigos como ensaios clínicos; 44 desses haviam sido corretamente indexados na LILACS, enquanto 19 tinham sido indexados como outros tipos de estudos (falsos negativos). As revisoras classificaram 658 estudos como não ensaios clínicos; 651 haviam sido corretamente indexados como não ensaios clínicos na LILACS e 7 haviam sido indexados como ensaios clínicos na LILACS (falsos positivos). A sensibilidade, especificidade e a acurácia da indexação LILACS foram de 69,8%, 98,9% e 96,4% (695/721), respectivamente. Quase um terço dos ensaios clínicos em periódicos de cardiologia da LILACS não está corretamente indexado. É necessário melhorar a qualidade da indexação dos estudos publicados nesses periódicos.

Palavras-chave:
Indexação como assunto; Bases de dados bibliográficas; Ensaios clínicos como assunto; Controle de qualidade; Publicações periódicas como assunto

Introduction

Access to health information is essential for those involved in evidence-based research, management and decision-making (COHN et al., 2005; NEUMANN; RADA, 2014). Systematic Reviews (SR) are considered the best source of evidence for making health care decisions, especially in the area of treatment. SR identify, assess critically and synthesize the findings of several Clinical Trials (CT) (HIGGINS; GREEN, 2011). CT are primary studies that prospectively allocate individuals to different intervention groups and assess the effects of these interventions on health outcomes. Interventions may be surgical or radiological procedures, behavioral therapies or preventive care (WORLD HEALTH ORGANIZATION, 2014). The primary purpose of a CT is to clarify what is the best intervention for a specific health problem. Therefore, CT are an essential part of any SR on health care (COOK et al., 1995; DICKERSIN et al., 1994; FORREST; MILLER, 2002).

Currently, the search for CT focuses on electronic databases such as Medical Literature Analysis and Retrieval System Online (MEDLINE) and Excerpta Medica Database (EMBASE) (DICKERSIN et al., 1994; GLANVILLE et al., 2006; LEFEBVRE et al., 2008). Cochrane Central Register of Controlled Trials (CENTRAL) uses these two databases to retrieve relevant studies (GLANVILLE et al., 2006; LEFEBVRE et al., 2008). The success of any SR search depends on the search strategy created (LEFEBVRE et al., 2008), as well as on the correct indexing of studies as CT in their electronic databases. If the CT are not properly indexed as such, they cannot be retrieved and this could compromise the results of a SR (GLANVILLE et al., 2006; LEFEBVRE, 2008; TALJAARD et al., 2010).

Latin American and Caribbean Health Sciences Literature (LILACS) currently indexes 920 journals and it is an important additional database that can be used in SR (CASTRO et al., 1998; CLARK; CASTRO, 2002). The indexing system of LILACS follows the principles established by the National Library of Medicine (NLM) for indexing publications in MEDLINE (CENTRO LATINO-AMERICANO E DO CARIBE DE INFORMAÇÃO EM CIÊNCIAS DA SAÚDE, 2008). LILACS indexing is performed by the attribution of terms from DeCS terminology, a trilingual vocabulary: Descritores em Ciências da Saúde (in Portuguese), Descriptores en Ciencias de la Salud (in Spanish) and Health Sciences Descriptors in English), DeCS was developed from the Medical Subject Headings (MeSH), vocabulary of the NLM (PELLIZZON, 2004). The professionals in charge of indexing the articles in LILACS should assign a number of subject descriptors that best represent the content of each article, including the Publication Type (PT), which describes the design of the study (CENTRO LATINO-AMERICANO E DO CARIBE DE INFORMAÇÃO EM CIÊNCIAS DA SAÚDE, 2008).

Although there are several studies on the quality of indexing of other databases (DICKERSIN et al., 2002; GLANVILLE et al., 2006; LEFEBVRE et al., 2008; TALJAARD et al., 2010; WIELAND et al., 2012), there are few studies that evaluate the indexing of CT as a PT in LILACS (CASTRO et al. 1997; FREITAS et al., 2005; MORENO FERNANDEZ et al., 2013). Therefore, the objective of this study was to evaluate the adequacy of indexing of CT as a PT in LILACS in a sample of articles published in cardiology journals and to analyze the characteristics of articles indexed incorrectly.

Methodological procedures

This analytical cross-sectional study was conducted between 2011 and 2014 by researchers at the Universidade Federal de São Paulo - Escola Paulista de Medicina (UNIFESP, EPM Federal University of São Paulo - Paulista Medical School) in association with the Evidence-Based Health Graduate Program. The study was approved by the Research Ethics Committee of UNIFESP.

All articles published between 2008 and 2009 in LILACS-indexed cardiology journals (http://lilacs.bvsalud.org/) and full texts freely available online were eligible for inclusion. We only included articles written in portuguese, english and spanish, indexed with subject descriptors, check tags (called Limits in LILACS) and publication types (PT).

We excluded journals whose sites were unavailable, with restricted access or whose indexing was in process during the study period. We also excluded all articles without indexing, as well as those published in supplements or special issues. We included only primary studies, i.e., those reporting the results of an original investigation. This type of study usually includes Introduction, Methods, Results, and Discussion sections (PEREIRA, 2012).

The whole process of article screening and selection, as well as data extraction and reclassification of the studies was performed independently by two reviewers (authors of this study): M.A. Conceição and M.R.C. Silva These two reviewers are librarians, both with over 12 years of experience indexing for the MEDLINE and LILACS databases. In case of disagreement between these reviewers, another reviewer (the author G.E. Tello) was consulted. This reviewer is an information specialist from the bibliographic services division of the NLM, fluent in Portuguese and is the indexing reviewer for 30 Brazilian journals indexed for MEDLINE by Centro Latino-Americano e do Caribe de Informação em Ciências da Saúde (BIREME).

Journal and article identification and selection

Journals were identified using the ‘Advanced Search’ resource from the Portal of Journals on Health Sciences from the Virtual Health Library (VHL) (CENTRO LATINO-AMERICANO E DO CARIBE DE INFORMAÇÃO EM CIÊNCIAS DA SAÚDE, 2008). In the search area of the website, the following parameters were used: Subject: Cardiology; Indexed in: LILACS; Language: All; Full Text: Available and Free; Title status: Current; Country: All (from Latin America and the Caribbean).

Data extraction and article reclassification

(a) Identification of the article as a CT or non-CT: A specific form was created for extracting data from the articles selected for inclusion with the following fields: identification of the journal (title, year, volume, issue), record identifier number in LILACS Identification (ID), PT indexed in LILACS, reclassification according to expert review and article section of the article (Title, Objective, Methods, etc.) that contained information on the type of study.

After careful analysis of Title, Objective, Methods and Results, the two reviewers (M.A. Conceição and M.R.C. Silva) initially reclassified each article as a CT or non-clinical trials (non-CT), following the standardized process for this study. In the Title and Objective, the reviewers looked for typical words indicative of a CT, such as ‘clinical trial’, ‘randomized’, ‘random’ or variants. In the Objective, the reviewers looked for words such as ‘efficacy’, ‘effectiveness’ or ‘safety’. In the Methods section, the authors analyzed the description of the study design and the form of allocation of subjects searching for words such as ‘control’, ‘groups’, ‘randomization’ and ‘random’. When the study design was not clear in Methods, the reviewers searched for this information in the Results and Discussion sections. The flowchart in Figure 1 shows the process used for classifying the studies as CT or non-CT (CENTRO LATINO-AMERICANO E DO CARIBE DE INFORMAÇÃO EM CIÊNCIAS DA SAÚDE, 2008).

Figure 1
Steps to identify and classify the articles included in the study.

(b) Reclassification of the CT by subtype: When there was enough information, the articles categorized as CT by the reviewers were then classified with the specific CT subtype, according to the definitions used in LILACS indexing and contained in the DeCS (Chart 1), available at (http://decs.bvs.br.). The authors also used the Indexing Instructions and Rules from the Indexing Manual (CENTRO LATINO-AMERICANO E DO CARIBE DE INFORMAÇÃO EM CIÊNCIAS DA SAÚDE, 2008).

Chart 1
Definitions of clinical trials (Publication Type) in DeCS.

Clinical Trial - Work that is the report of a pre-planned clinical study of the safety, efficacy, or optimum dosage schedule of one or more diagnostic, therapeutic, or prophylactic drugs, devices, or techniques in humans selected according to predetermined criteria of eligibility and observed for predefined evidence of favorable and unfavorable effects. While most clinical trials concern humans, this publication type may be used for clinical veterinary articles meeting the requisites for humans. Specific headings for specific types and phases of clinical trials are also available. Indexing Annotation: this heading is used as a Publication Type; for original report of the conduct or results of a specific clinical trial; a different heading ‘clinical trials as topic’ is used for general design, methodology, economics, etc. of clinical trials

Controlled Clinical Trial - Work consisting of a clinical trial involving one or more test treatments, at least one control treatment, specified outcome measures for evaluating the studied intervention, and a bias-free method for assigning patients to the test treatment. The treatment may be drugs, devices, or procedures studied for diagnostic, therapeutic, or prophylactic effectiveness. Control measures include placebos, active medicine, no-treatment, dosage forms and regimens, historical comparisons, etc. When randomization using mathematical techniques, such as the use of a random numbers table, is employed to assign patients to test or a control treatment, the trial is characterized as a ‘randomized controlled trial’ - Indexing Annotation: this heading is used as a Publication Type; for original report of the conduct or results of a specific controlled clinical trial; a different heading ‘controlled clinical trials as topic’ is used for general design, methodology, economics, etc. of clinical trials.

Randomized Controlled Trial - Work consisting of a clinical trial that involves at least one test treatment and one control treatment, concurrent enrollment and follow-up of the test- and control-treated groups, and in which the treatments to be administered are selected by a random process, such as the use of a random-numbers table. Indexing Annotation: this heading is used as a Publication Type; for original report of the conduct or results of a specific randomized controlled trial; a different heading ‘randomized controlled trials as topic’ is used for general design, methodology, economics, etc. of randomized controlled trials.

Clinical Trial Phase I* - Work that is the report of a pre-planned, usually controlled, clinical study of the safety and efficacy of diagnostic, therapeutic, or prophylactic drugs, devices, or techniques based on a small number of healthy persons and conducted over the period of about a year in either the United States or a foreign country.

Clinical Trial Phase II* - Work that is a report of a pre-planned, usually controlled, clinical study of the safety and efficacy of diagnostic, therapeutic, or prophylactic drugs, devices, or techniques based on several hundred volunteers, including a limited number of patients, and conducted over a period of about two years in either the United States or a foreign country.

Clinical Trial Phase III* - Work that is a report of a pre-planned, usually controlled, clinical study of the safety and efficacy of diagnostic, therapeutic, or prophylactic drugs, devices, or techniques after phase II trials. A large enough group of patients is studied and closely monitored by physicians for adverse response to long-term exposure, over a period of about three years in either the United States or a foreign country.

Clinical Trial Phase IV* - Work that is a report of a planned post-marketing study of diagnostic, therapeutic, or prophylactic drugs, devices, or techniques that have been approved for general sale after clinical trials, phases I, II, and III. These studies, conducted in the United States or a foreign country, often garner additional data about the safety and efficacy of a product.

• Source: CENTRO LATINO-AMERICANO E DO CARIBE DE INFORMAÇÃO EM CIÊNCIAS DA SAÚDE (2016), based onWieland et al. (2012, Box 1).
• Note: *Indexing Annotation: this heading is used as a Publication Type; for original report of the conduct or results of a specific phase; different heading ‘clinical trials’, Phase I (or Phase II, or Phase III, or Phase IV) ‘as topic is’ used for general design, methodology, economics, etc.

Comparison of the two classifications

The result of the reclassification of the articles by each reviewer (M.A. Conceição and M.R.C. Silva) was compared and the articles were classified into the following groups: (a) certainly not a CT (both reviewers classified the article as a non-CT); (b) certainly a CT (both reviewers classified the article as a CT); and (c) articles with discordant classifications (articles marked as a CT by one reviewer and as a non-CT by the other). The articles with discordant classifications were sent to the reviewer (G.E. Tello) and his classification was considered final for these articles. The degree of agreement between the two reviewers was measured using the Kappa method (VIEIRA; GARRET, 2005).

Indexing adequacy of the CT in LILACS

The reviewers’ reclassification of the present study (M.A. Conceição, M.R.C. Silva and G.E. Tello) was considered the correct [true] indexing or gold standard. The accuracy of the original indexing in LILACS was analyzed by calculating sensitivity (correct LILACS indexing of the articles as a CT) and specificity (correct LILACS indexing of the articles as not being a CT). The articles indexed as another type of design in LILACS and as a CT by the reviewers of this study were classified as false negatives. The articles indexed as CT in LILACS and as another design by the reviewers in this study were classified as false positive cases. Studies categorized as CT by the reviewers and with data that allowed their classification into subtypes are presented descriptively (number, percentages).

Characteristics of studies indexed incorrectly in LILACS

We conducted a more detailed analysis of the articles reclassified by the reviewers as CT, which were incorrectly classified as another type of study in LILACS (false negatives), and the articles incorrectly indexed in LILACS as CT (false positives). We also indicate where (Title, Objective, Methods or Results) we found a clear description of the study design and the CT subtype in the article.

Results

Identification and selection of articles

The reviewers authors (reviewers) identified 16 Latin American open access Cardiology journals with articles published in 2008-2009. Five journals were excluded: one did not provide open access to the full text and four had no indexed issues (delay in indexing or indexing still in process) at the time of data collection. Eleven journals, totaling 1,396 articles, were included. We obtained the electronic version of all articles of each eligible journal and screened the titles and abstracts to exclude clearly irrelevant articles (review or update articles, biographies, editorials, letters, clinical conference reports, comments, short communications, guidelines, case reports, technical reports and points of view). At this stage, we excluded 11 articles that were not indexed and 664 that did not meet the selection criteria. The remaining 721 articles were read and reclassified by the reviewers (M.A. Conceição and M.R.C. Silva), as previously described (Figure 2).

Figure 2
Process of identification and selection of the articles included in the study.

Among the 721 articles, 670 were indexed in LILACS as other types of studies and 51 as Clinical Trials studies, divided into the following subtypes: 14 CT, 5 Controlled Clinical Trials (CCT), 31 Randomized Controlled Trials (RCT) and 1 CT phase I.

Classification of articles by the reviewers

Both reviewers (M.A. Conceição and M.R.C. Silva) agreed that 49 of the 721 articles were CT and 637 had other study designs. A total of 35 studies (4.8%) were classified differently by the two reviewers and referred to the third reviewer. The agreement in the classification of the studies between the two reviewers was 95.1% (686/721) and the Kappa coefficient was 0.74. After the third reviewer (G.E. Tello) analyzed these 35 articles with discordant classifications, 14 were reclassified as CT and 21 as other study designs. Therefore, among the 721 articles included in this study, 63 were reclassified as CT and 658 as having other types of designs. The subtypes of the 63 CT were: 45 RCT, 16 CT and 2 CCT. Most of the 63 CT (84.1%) were published in Portuguese in four Brazilian journals.

Adequacy of indexing of CT in LILACS

Among the 63 articles reclassified as being in fact a CT, 44 had been correctly indexed as such in LILACS (true positive) and 19 had been incorrectly indexed as other types of studies (false negatives). The sensitivity of the original LILACS indexing was thus 69.8% and the false negative rate was 30.2%. On the other hand, 658 of the 721 articles were reclassified as other study designs, and 651 of these (98.9%) were correctly indexed in LILACS, while 7 (1.1%) were indexed incorrectly as CT (false positive). Therefore, the specificity (correct indexing of other designs) of the original LILACS indexing was 98.9%. The overall error rate of LILACS indexing was 3.6% (26/721) and overall accuracy was 96.4% (695/721) (Table 1). The agreement coefficient (Kappa) between the original LILACS indexing and the classification by the authors of this study was 0.96.

Table 1
Analysis of Indexing in LILACS as (regarding) the type of study of 721 articles published in cardiology journals in 2008-2009.

Original indexing in LILACS	Correct indexing*		Total
	CT	Others studies
CT	44 (True positive)	7 (False positive)	51
Others studies	19 (False negative)	651 (True negative)	670
Total	63	658	721

• Source: Prepared by the authors (2016).
• Note: *Type of study according to the evaluation of the researchers of this study. Sensitivity: 69.8%; Specificity: 98.9%; Accuracy: 96.4%;
• CT: Clinical Trial.

Analysis of articles indexed incorrectly

Most of the 63 CT (84.1%, n=53) were published in Brazilian journals. Of the 19 CT indexed incorrectly as other types of studies in LILACS (false negatives), almost half (9) were published in the Revista Brasileira de Cardiologia Invasiva and the others were distributed among seven other journals (Table 2).

Table 2
Distribution of clinical trials and indexing errors by journal.

Journal	Country	Articles per journal	Number of CT*	False Negative	%	False Positive	%
Arquivos Brasileiros de Cardiologia †	Brazil	255	22	2	10,5	1	14,29
Revista Brasileira de Cirurgia Cardiovascular†	Brazil	118	18	1	5,26	2	28,57
Revista Brasileira de Cardiologia Invasiva	Brazil	96	11	9	47,37	0	0,00
Archivos de Cardiología de México†	Mexico	54	4	1	5,26	1	14,29
Revista Peruana de Cardiología	Peru	22	4	3	15,79	1	14,29
RELAMPA (Revista Latino-Americana de Marcapasso e Arritmia)	Brazil	18	2	1	5,26	2	28,57
Revista Chilena de Cardiología	Chile	40	1	1	5,26	0	0,00
Revista Colombiana Cardiología	Colombia	29	1	1	5,26	0	0,00
Jornal Vascular Brasileiro	Brazil	40	0	0	0,00	0	0,00
Revista Brasileira de Hipertensão	Brazil	25	0	0	0,00	0	0,00
Revista Uruguaya de Cardiología	Uruguay	24	0	0	0,00	0	0,00
Total		721	63	19	100,00	7	100,00

• Source: Prepared by the authors (2016).
• Note: False Negative: CT classified as other PT in LILACS; False Positive: study with other type of design that was classified as CT in LILACS; *Reviewers classification; ^† Journal also indexed in MEDLINE.
• CT: Clinical Trial.

Among the 19 CT indexed incorrectly in LILACS as other types of designs, 12 were RCT, 6 were CT and 1 was a CCT. In almost 60% of the articles (11/19) the authors did not clearly describe what was study design or this information was not highlighted and the expressions ‘clinical trial’, ‘controlled trial’, and ‘randomized controlled trials’ were not found in the text. Among the 8 articles describing the study design, in 4 the information appeared in methods, in 2 of them it was in the title, objective and methods, in 1 article the information was in the Methods and Results, and in the last article it appeared in the Title, Objective, Methods and Results.

The 7 articles indexed incorrectly in LILACS as CT (false positives), were published in five different journals, three from Brazil (Table 2). Five of these 7 articles were indexed as CT, one article as a CCT and one as a RCT. When analyzing the texts of these 7 articles, 4 did not describe the study design but they did not present the characteristics of a CT, 2 articles described the study as ‘observational’ and ‘cross-sectional’ and one study addressed CT as a subject, not as a publication type.

Discussion

Our findings indicate that there are important errors in the indexing of CT as PT in LILACS. The sensitivity of LILACS indexing in a sample of 721 articles published in cardiology journals was 69.9%. This means that over 30.0% of the CT in cardiology published in 11 Latin American journals would not be retrieved in a search that used PT. This rate is far from the recommended standards, since a sensitivity index greater than 90.0% is expected for a reasonable level of CT retrieval in databases (GLANVILLE et al., 2006). Correct indexing of a CT as a PT is essential for the development of search strategies in databases such as MEDLINE and EMBASE as the inclusion of indexing terms improves the sensitivity and specificity when retrieving these studies (DICKERSIN et al., 1994, 2002; GLANVILLE et al., 2006; LEFEBVRE et al., 2008; TALJAARD et al., 2010).

In 11 of the 19 CT indexed incorrectly in LILACS (false negatives), the authors did not clearly describe the study design in their manuscript and this may have contributed to the error of the person in charge of indexing for LILACS. Similarly, Wieland and colleagues (2012) reported that only 245 of the 572 RCT not indexed in MEDLINE as a PT described the study design. However, the study design in 40% (8/19) of the articles that were incorrectly indexed as CT in LILACS was clearly described in the title or Methods or Results sections, pointing to an evident indexing error in LILACS.

The fact that almost half (9/19) of false negative cases occurred in the Revista Brasileira de Cardiologia is intriguing because most of those articles (5/9) clearly described the study design as a CT. We would need to perform a more in depth analysis of the indexing center to know the reasons that led to these indexing errors in this specific journal. The disagreement between the native language of the indexer and the language of the text has been pointed out as a possible cause for incorrect indexing (CENTRO LATINO-AMERICANO E DO CARIBE DE INFORMAÇÃO EM CIÊNCIAS DA SAÚDE, 2008). However, this does not apply to this case because the highest rates of errors in Brazilian journals with texts in Portuguese are indexed at Brazilian centers by indexers whose native language is also Portuguese, while the journals from other Latin American countries (with articles in Spanish) are indexed by centers located in Spanish speaking countries.

The second type of indexing error (false positive) was less common but it also suggests a possible lack of training of the indexers. The analysis of the 7 articles indexed incorrectly as CT in LILACS showed that almost all (6/7) did not present any characteristics of a CT. This type of error is less serious because the authors of a SR would retrieve these articles and quickly discard them after reading them. Therefore, although this could lead to a waste of time for the SR authors, this type of indexing error would not compromise the final results of the review.

Several guidelines have been created over the last 15 years to optimize and standardize the publication of CT; this could facilitate the correct indexing of these studies in the databases (BEGG et al., 1996). One of the most frequently used guidelines is the Consolidated Standards of Reporting Trials (CONSORT) with the 1^st edition dating back to 1996 (BEGG et al., 1996). According to the Consolidated Standards of Reporting Trials recommendations, authors should describe the specific type of study in the title and abstract (which must be structured) to facilitate indexing and identification of a CT using the terms “randomised” or “randomized” to indicate that the participants were randomly allocated to the comparison group. The latest version of the CONSORT includes a checklist with 25 items that clearly specifies the rules for reporting CT (CONSOLIDATED STANDARDS OF REPORTING TRIALS, 2010). At the time of their publication, only 3 of the 11 journals included in the present study (Brazilian Archives of Cardiology, Brazilian Journal of Cardiovascular Surgery and the Latin American Journal of Pacemaker and Arrhythmia) recommended the use of the CONSORT in their ‘instructions for authors’ section.

The first study on CT identification in LILACS as a PT was conducted by Castro and colleagues in 1997 in collaboration with the Brazilian Cochrane Center. The researchers developed a search strategy based on the strategy developed by Dickersin and colleagues (1994) and the search was conducted in the January 1997 LILACS/CD-ROM, retrieving 38,261 citations that would be analyzed to identify the CT. The goal was to identify all RCT and CCT in every new edition of LILACS for the Cochrane Library (CASTRO et al., 1997). This study has led to the Centro Latino-Americano e do Caribe de Informação em Ciências da Saúde (2002) project (unpublished report) with the Brazilian Cochrane Center to identify CT in LILACS to correct the indexing of all publications indexed in the database between 1985 and 2000. Subsequently, the researchers sent a list of 1,202 reviewed articles to Centro Latino-Americano e do Caribe de Informação em Ciências da Saúde, of which 723 were CT. Of these, 324 (44.9%) were correctly indexed in LILACS as CT and 399 (55.1%) were incorrectly indexed as other types of study (false negatives). Moreover, 266/479 (55.5%) were indexed incorrectly as CT (false positives). Thus, over half of the CT would not have been retrieved from LILACS and half of the retrieved ones would have been irrelevant. One explanation for the large number of inadequately indexed CT is that, at that time, there were no specific terms for indexing them. The publication types, RCT and CCT were created by the NLM and incorporated into MeSH in 1991 and 1995 (DICKERSIN et al., 2002; WIELAND et al., 2012), and in DeCS at the same time, since these two vocabularies have the same version for each year. Our current finding of a false negative rate of 30.2% shows improvement in the indexing of CT in LILACS. This improvement over time was also observed in studies assessing the quality of indexing in MEDLINE (DICKERSIN et al., 1994, 2002; WIELAND et al., 2012).

The capacity of LILACS for retrieving RCT in Portuguese and Spanish was tested by Freitas and colleagues in 2005 using three different search strategies. The authors report that only 47 out of 5,262 articles retrieved were RCT, and sensitivity ranged from 2,0% to 53,0% (FREITAS et al., 2005). The strategy with the worst performance (2,0%) included only indexing terms. Ten years later, our study found a 68.9% sensitivity for CT indexing in LILACS, which means that about 30,0% of the CT would not have been retrieved if the strategy included only indexing terms. The main difference in our study is its wider scope because instead of using a search strategy for the selection of included articles, we downloaded all articles directly from LILACS and identified the CT by reading the entire text. Strategies to retrieve non-indexed CT by searching for words in the title and abstract run the risk of not retrieving articles that do not contain CT-suggesting words in these sections (WIELAND et al., 2012).

In order to improve the retrieval of a RCT in a database, DICKERSIN et al. (1994, 2002) recommend improving the terminology used by the authors in the description of the study design, improved indexing such as PT and improved search strategies, which should include both indexing terms and free text words. The specialized training of the indexing personnel is important to ensure the quality of indexing. The NLM offers indexing training for MEDLINE using MeSH terms. The initial module of this training is on-line (NATIONAL LIBRARY OF MEDICINE, 2015) followed by practical training at the NLM headquarters (CLEVELAND; CLEVELAND, 2013). Periodically, NLM indexing reviewers send reports to MEDLINE indexing centers (http://www.nlm.nih.gov/bsd/indexhome.html), of which Brazil is a member, pointing out their inaccuracies. LILACS, in turn, also offers courses and distance training to new indexing personnel (CENTRO LATINO-AMERICANO E DO CARIBE DE INFORMAÇÃO EM CIÊNCIAS DA SAÚDE, 2013) and the LILACS indexing system performs automatic quality control of human indexing. However, LILACS does not currently include indexing reviewers.

The present study had several strengths. It was the first study to assess the quality of CT indexing as a PT in LILACS based on the review of all the original articles. The professional experience of the team that carried out the reclassification of all articles and the use of a rigorous methodology were important elements to reduce the risk of bias and strengthen the findings of this study.

However, there were several limitations in our study. The quality of LILACS indexing was assessed in only a specific group of journals (cardiology), published some time ago [2008-2009] within a limited period of time (two years). Therefore, the results of our study cannot be generalized to other journals and may not reflect the current quality of CT indexing in LILACS. The inclusion of journals indexed in MEDLINE may have also affected the results because experienced indexers from BIREME perform the indexing for these journals and the articles are updated on LILACS. As a result, it is assumed that the quality of indexing of those journals is superior.

The articles identified in this study with inadequate indexing were sent to managers of LILACS for re-indexing of the database. Our results point to the need for developing strategies to improve the indexing of CT in LILACS, such as improvements in training and periodic evaluation of LILACS indexers. Another important strategy would be to improve the quality of articles published in these journals. The mandatory adoption of the CONSORT recommendations for the publication of CT in all LILACS-indexed journals could improve indexing and subsequent the retrieval of CT from the database.

More studies involving journals from other fields are necessary to further investigate the indexing quality of CT in LILACS. Further studies would allow the evaluation and comparison of differences in the indexing quality in different medical specialties. It would also be important to maintain and increase partnership between BIREME and researchers from the Brazilian Cochrane Center to ensure that CT not indexed as the correct PT in LILACS can be identified and corrected in that database.

Conclusion

Nearly a third of the CT published in LILACS-indexed Cardiology journals are not properly indexed as such. The findings of this study are important for the authors of SR in the area of cardiology who are searching for CT in the LILACS database, as well as for researchers interested in information to support evidence-based decisions in this area.

References

• BEGG, C. et al. Improving the quality of reporting of randomized controlled trials: The CONSORT statement. Journal of America Medical Association, v. 276, n. 8, p. 637-639, 1996.
• CASTRO, A. A.; ATALLAH, A. N.; CLARK, O. A. Locating and appraising systematic reviews. Annals of Internal Medicine, v. 128, n. 4, p. 322-323, 1998.
• CASTRO, A. A.; CLARK, O. A.; ATALLAH, A. N. Optimal search strategy for clinical trials in the Latin American and Caribbean Health Science Literature Database (LILACS). Medical Journal, v. 115, n. 3, p. 1423-1426, 1997.
• CENTRO LATINO-AMERICANO E DO CARIBE DE INFORMAÇÃO EM CIÊNCIAS DA SAÚDE. BIBLIOTECA VIRTUAL EM SAÚDE. Fontes Referenciais de Informação: relatório técnico: Projeto Cochrane. São Paulo: BIREME, 2002.
• CENTRO LATINO-AMERICANO E DO CARIBE DE INFORMAÇÃO EM CIÊNCIAS DA SAÚDE. BIBLIOTECA VIRTUAL EM SAÚDE. Manual de indexação de documentos para a base de dados LILACS. São Paulo: BIREME, 2008a. Disponível em: <http://metodologia.lilacs.bvsalud.org/download/P/LILACS-4-ManualIndexacao-pt.pdf>. Acesso em: 10 jan. 2016.
  » http://metodologia.lilacs.bvsalud.org/download/P/LILACS-4-ManualIndexacao-pt.pdf
• CENTRO LATINO-AMERICANO E DO CARIBE DE INFORMAÇÃO EM CIÊNCIAS DA SAÚDE. BIBLIOTECA VIRTUAL EM SAÚDE. Portal de revistas. São Paulo: BIREME, 2008b. Disponível em: <http://portal.revistas.bvs.br/main.php?home=true&lang=pt>. Acesso em: 10 jan. 2016.
  » http://portal.revistas.bvs.br/main.php?home=true&lang=pt
• CENTRO LATINO-AMERICANO E DO CARIBE DE INFORMAÇÃO EM CIÊNCIAS DA SAÚDE. BIBLIOTECA VIRTUAL EM SAÚDE. Cursos LILACS. São Paulo: BIREME, 2013. Disponível em: <http://cursos.lilacs.bvsalud.org/php/level.php?lang=pt&component=33&item=2>. Acesso em: 10 jan. 2016.
  » http://cursos.lilacs.bvsalud.org/php/level.php?lang=pt&component=33&item=2
• CENTRO LATINO-AMERICANO E O CARIBE DE INFORMAÇÃO EM CIÊNCIAS DA SAÚDE. BIBLIOTECA VIRTUAL EM SAÚDE. DeCS - Descritores em Ciências da Saúde. São Paulo: BIREME, 2016. Disponível em: <http://decs.bvs.br/I/homepagei.htm>. Acesso em: 10 jan. 2016.
  » http://decs.bvs.br/I/homepagei.htm
• CLARK, O. A.; CASTRO, A. A. Searching the Literatura Latino Americana e do Caribe em Ciências da Saúde (LILACS) database improves systematic reviews. International Journal of Epidemiology, v. 31, n. 1, p. 112-114, 2002. Available from: <http://ije.oxfordjournals.org/content/31/1/112.full.pdf+html>. Cited: Dec 22, 2015.
  » http://ije.oxfordjournals.org/content/31/1/112.full.pdf+html
• CLEVELAND, D. B; CLEVELAND A. D. Introduction to indexing and abstracting. Libraries Unlimited. 4^th ed. Santa Barbara: Libraries Unlimited, 2013.
• COHN, A.; WESTPHAL, M. F.; ELIAS, P. E. Informação e decisão política em saúde. Revista de Saúde Pública. v. 39, n. 1, p. 114-121, 2005. Disponível em: <http://www.scielo.br/pdf/rsp/v39n1/15.pdf>. Acesso em: 20 dez. 2015.
  » http://www.scielo.br/pdf/rsp/v39n1/15.pdf
• CONSOLIDATED Standards Of Reporting Trials. Ottawa (ON): The Consort Group; 2010. Available from: <http://www.consort-statement.org/checklists/view/32-consort/66-title>. Cited: Jan 15, 2016.
  » http://www.consort-statement.org/checklists/view/32-consort/66-title
• COOK, D. J., SACKETT, D. L.; SPITZER, W. O. Methodologic guidelines for systematic reviews of randomized control trials in health care from the Potsdam Consultation on Meta-Analysis. Journal of Clinical Epidemiology, v. 48, n. 1, p. 167-171, 1995.
• DICKERSIN, K.; SCHERER, R.; LEFEBVRE, C. Identifying relevant studies for systematic reviews. British Medical Journal, v. 309, n. 6964, p. 286-1291, 1994.
• DICKERSIN, K. et al. Development of the cochrane collaboration’s Central register of controlled clinical trials. Evaluation & the Health Professions, v. 25, n. 1, p. 38-64, 2002.
• FORREST, J.; MILLER, S.A. Enhancing your practice through evidence-based decision making: Appraising the evidence. Journal of Evidence-Based Dental Practice, v. 2, n. 2, p. 178-185, 2002.
• FREITAS, A. E. et al. Searching the LILACS database for Portuguese- and Spanish-language randomized trials in physiotherapy was difficult. Journal Clinical Epidemiology, v. 58, n. 3, p. 233-237, 2005.
• GLANVILLE, J. M. et al. How to identify randomized controlled trials in MEDLINE: Ten years on. Journal of the Medical Library Association, v. 94, n. 2, p. 130-136, 2006.
• HIGGINS, J. P.; GREEN, S (Ed.). Cochrane handbook for systematic reviews of interventions. version 5.1.0. London: The Cochrane Collaboration, 2011. Available from: <www.cochrane-handbook.org>. Cited: Jan. 15, 2016.
  » www.cochrane-handbook.org
• LEFEBVRE, C. et al. Enhancing access to reports of randomized trials published world-wide-the contribution of EMBASE records to the Cochrane Central Register of Controlled Trials (CENTRAL) in The Cochrane Library. Emerging Themes in Epidemiology, v. 5, p. 13, 2008.
• MORENO FERNÁNDEZ, L. M. et al. Consistency between indexers in the LILAC database (Latin American and Caribbean Health Science Literature). Information Research, v.18, n. 4, 2013. Available from: <http://www.informationr.net/ir/18-4/paper601.html#.VEF_RbDF9TI>. Cited: Jan. 22, 2016.
  » http://www.informationr.net/ir/18-4/paper601.html#.VEF_RbDF9TI
• NATIONAL LIBRARY OF MEDICINE. Medline Indexing Online Training Course Bethesda: National Library of Medicine, 2015. Available from: <http://www.nlm.nih.gov/bsd/indexing/training/PUB_070.html>. Cited: Jan 15, 2016.
  » http://www.nlm.nih.gov/bsd/indexing/training/PUB_070.html
• NEUMANN, I.; RADA, G. Evidence-based decision-making: When it is worthwhile. Medwave, v. 14, n. 5, p. 5966, 2014.
• PELLIZZON, R. F. Pesquisa na área da saúde: 1 - Base de dados DeCS (Descritores em Ciências da Saúde). Acta Cirúrgica Brasileira. v. 19, n. 2, p. 153-163, 2004. Disponível em: <http://www.scielo.br/scielo.php?script=sci_arttext&pid=S0102-86502004000200013&lng=en&nrm=iso>. Acesso em: 9 fev. 2016.
  » http://www.scielo.br/scielo.php?script=sci_arttext&pid=S0102-86502004000200013&lng=en&nrm=iso
• PEREIRA, M. G. Estrutura do artigo científico. Epidemiologia e Serviços de Saúde, v. 21, n. 2, p. 351-352, 2012.
• TALJAARD, M. et al. Electronic search strategies to identify reports of cluster randomized trials in MEDLINE: Low precision will improve with adherence to reporting standards. BMC Medical Research Methodology, v. 10, p. 15, 2010.
• VIEIRA, A. J.; GARRETT, J. M. Understanding interobserver agreement: The kappa statistic. Family Medicine, v. 37, n. 5, p. 360-363, 2005.
• WIELAND, L. S. et al. Understanding why evidence from randomised clinical trials may not be retrieved from Medline: Comparison of indexed and non-indexed records. British Medical Journal, v. 344, p. d7501, 2012.
• WORLD HEALTH ORGANIZATION. International Clinical Trials Registry Platform (ICTRP). Genève: WHO, 2014. Available from: <http://www.who.int/ictrp/en/>. Cited: Jan. 10, 2016.
  » http://www.who.int/ictrp/en/

Publication Dates

• Publication in this collection
  Sep-Dec 2017

History

• Received
  19 Feb 2016
• Reviewed
  12 Dec 2016
• Accepted
  20 Jan 2017