Critical analysis of the drafting of physical therapy randomized controlled trials published in Portuguese

Análise crítica da redação dos experimentos controlados e randomizados em fisioterapia publicados na Língua Portuguesa

Abstract

Introduction:

randomized controlled trials are high quality studies. Many problems related to the drafting of these studies have been identified and consequently various national and international journals, in an attempt to improve this writing, have adopted the Consolidated Standards of Reporting Trials recommendations. Regarding the studies written specifically by physical therapists, until now, the quality of the drafting in Portuguese has been unknown.

Aim:

To critically analyze the drafting of RCTs in the area of physical therapy, published in Portuguese, in relation to the CONSORT recommendations.

Materials and Methods:

On 17th Oct, 2012, 548 RCTs in Portuguese were recovered from the MEDLINE and PEDro databases, which were divided among four evaluators who, after reading the abstracts, selected those related to physical therapy. Of these studies, 78 RCTs were related to physical therapy, which were divided among the four evaluators for the analysis of the drafting according to the CONSORT recommendations. The four evaluators who participated in this study previously obtained a median kappa above 70% when their analyses were compared to the analyses of the evaluator considered the gold standard due to having greater experience.

Results:

The quantity of items of the CONSORT recommendations according to year of publication was very small, corresponding to a mean of 43% of the items in the articles analyzed.

Conclusion:

The results make very clear the need to improve the quality of the drafting of the RCTs related to physical therapy in Portuguese and to include more rigorous methodological procedures, such as sample size, randomization and blinding. The dissemination and adoption of the CONSORT recommendations by physical therapy researchers would, without doubt, be a big step towards improving this quality.

Keywords:
Evaluation; Research; Methodology; Physical Therapists; Science.

Resumo

Introdução:

os experimentos controlados e randomizados são pesquisas de alta qualidade. Entretanto, muitos problemas relacionados à redação desses estudos foram identificados e em função disso vários periódicos nacionais e internacionais, na tentativa de melhorar essa redação, estão adotando as recomendações Consolidated Standards of Reporting Trials. Em relação aos estudos redigidos especificamente por fisioterapeutas desconhece-se, até o momento, a qualidade das suas redações na língua portuguesa.

Objetivo:

Analisar criticamente a redação dos ECR, na área da fisioterapia, publicados na língua portuguesa em relação às recomendações CONSORT.

Materiais e Métodos:

Em 17 Out 2012 foram recuperados 548 ECR em língua portuguesa existentes nas bases MEDLINE e PEDro, que foram divididos entre quatro avaliadores para, após leitura do resumo, selecionarem aqueles sobre fisioterapia. Desses estudos, apenas 78 ECR tinham como escopo a fisioterapia, que foram então divididos entre os quatro avaliadores para a análise da redação conforme as recomendações CONSORT. Os quatro avaliadores que participaram deste estudo obtiveram previamente um kappa mediano acima de 70% quando comparadas as suas análises com as análises do avaliador considerado como padrão-ouro pela sua maior experiência.

Resultados:

A quantidade de itens das re­comendações CONSORT contemplados por ano de publicação foi muito pequena, correspondendo a uma média de 43% dos itens nos artigos analisados.

Conclusão:

Os resultados encontrados deixam bem evidentes a necessidade de melhorar a qualidade da redação dos ECR em fisioterapia na língua portuguesa. A divulgação e adoção das recomendações CONSORT pelos pesquisadores fisioterapeutas é, sem dúvida, um grande passo no sentido de melhorar esta qualidade.

Palavras-chave:
Avaliação; Pesquisa; Metodologia; Fisioterapeutas; Ciência.

Introduction

Throughout the last century access to scientific information was limited in Brazil, since obtaining the studies depended on inefficient procedures that made such access time-consuming and arduous. Thus, evidence-based clinical practice ⁽1^,2 was not usual in some areas, as the time to develop the knowledge base was incompatible with the needs of the health professionals. Recently, however, after the strategic massification in the use of the internet, the dissemination of scientific information has been increasingly gaining strength ⁽3. Considering the sharing of this information within the academic community, access to information has become extremely viable. This network of sharing scientific articles has made the professional actualization in short periods of time entirely possible, which is critical for the current clinical practice ⁽4. This process of accessibility to information requires heath professional to have the ability to select the best evidence ⁽5^,6, since the number of published studies is very high (ex. 21,820 randomized controlled trials (RCTs) on health, performed with humans, published in the last five years and indexed in MEDLINE) ⁽7.

In the context of physical therapy, reading is directed toward certain areas of practice ⁽5, facilitating the decision-making of the professional. However, even restricting the area of practice, the production of Brazilian researchers in physical therapy ⁽8, as well as in other areas, is increasing. Consequently, with the scientific production on a large scale, special attention should be given to the studies with the best methodological rigor - the RCTs. However, as mentioned above, the amount of RCTs published is extensive and they often do not express the performance of basic methodological procedures.

Concern about the quality of the drafting of studies is evident in the recent reviews published by Brazilian ⁽9^,10 and foreign ⁽11 researchers, as scientific findings should guide clinical practice. This concept is called evidence-based practice ⁽1^,2^,12. In this context, two types of studies are used to conduct this practice: the systematic review and randomized controlled trial ⁽13^,14.

In the systematic review a thorough overview of the state of the art on a particular topic is carried out, mainly based on RCTs, where the information of best methodological rigor is systematically included and analyzed ⁽15. Next, questions about the actual findings of these studies are made, as on some occasions the reviews do not make the important methodological requirements clear. For this reason, the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA Statement) recommendations ⁽16^,17 were created in 2009, guiding the drafting of this kind of study and improving the quality of the information provided. Perhaps due to the short time of existence of these recommendations, Padula et al. ⁽2012^{) (}9 and Costa et al. ⁽2011^{) (}10 highlighted the existence of important limitations in the drafting of the reviews.

Similar to the PRISMA Statement, with respect to the recommendations for drafting the text, there is the Consolidated Standards of Reporting Trials (CONSORT) ⁽18, established in 1999 and published in 2001, which provides basic guidelines for drafting RCTs and which increases the reliability of the information contained in these studies. Issues related to drafting have also been identified in RCTs published between 2000 and 2006, although the authors verified an improvement in this period ⁽11, and this reaffirms the need for adequation of the language used in these research reports.

Some national and international journals are adopting CONSORT as the standards for publication of RCTs. This shows the importance of the drafting of these studies ⁽19^,20. In Brazil, including the adoption of CONSORT is already a reality in some traditional journals of physical therapy ⁽21, encouraging Brazilian researchers to perfect the writing technique according to international standards. Therefore, in accordance with the latest update of CONSORT and with the requirements of the Brazilian Journal of Physical Therapy (both in the year of 2010), an investigation of the current scenario is necessary, as at the present moment it is not known whether the national production of RCTs is in fact following the international standard. The aim of this study was to critically analyze the drafting of RCTs in the area of physical therapy, published in Portuguese, in relation to the CONSORT recommendations.

Materials and methods

Search strategy

On 17^th October, 2012, all RCTs existing in the Portuguese language were recovered from the MEDLINE and PEDro bases. In the MEDLINE database the descriptors randomized controlled trial AND Portuguese were used in the publication type and language fields, respectively. In the PEDro database the descriptors clinical trial AND Portuguese were used in the method AND title only fields, respectively. A total of 455 RCTs were recovered from the MEDLINE database and 93 RCTs from the PEDro database. These databases were chosen for their importance in the scientific community and physical therapy, respectively.

Standardization of the use of the CONSORT recommendations

Despite all the evaluators already knowing or having previously used the CONSORT recommendations, three meetings were held for the standardization of their use. The evaluator with more experience in the use of the CONSORT recommendations was considered the gold standard. Immediately after the meeting, each evaluator applied the CONSORT recommendations to three RCTs in the Portuguese language, which were compared with the gold-standard evaluations for verification of the concordance index measured through kappa ⁽22. The median kappa was considered from the three kappas obtained in the three RCTs of each evaluator, with the evaluators who obtained a minimum kappa of 70% being considered concordant.

Evaluation of the RCTs

Four researchers that obtained a Kappa ≥ 7.0 participated in the evaluation of the studies. One was a physical therapist, physical education professional, doctoral candidate in exercise and sports science / UGF, with a master's degree in physical education; Two were physical therapists, with one a physical education professional, and both master's degree students in exercise and sports science / UGF; The other was a physical education professional and master's degree student in exercise and sports science / UGF.

The 455 RCTs recovered from the MEDLINE database and 93 RCTs recovered from the PEDro database were divided among the four evaluators and, after reading the abstracts of the studies, those related to physical therapy were selected. The reading of the abstracts was carried over a period of 60 days (approximately two abstracts per day per evaluator). In the MEDLINE database, 421 RCTs were not related to physical therapy resulting in 34 articles. In the PEDro database, 38 of RCTs were also found in MEDLINE or were not related to physical therapy, resulting in 55 articles to be evaluated. Of these 55 articles, 8 were excluded due to the year publication being 2001 or earlier, the year of publication of CONSORT.

The CONSORT recommendations consist of 37 items, with 25 being considered obligatory and 12 optional in a RCT. These 37 item are distributed in the following way: title and abstract ⁽2, introduction ⁽2 methods ⁽17, results ⁽10, discussion ⁽3 and other information ⁽3. Further details regarding the complete checklist can be found at http://www.consort-statement.org/.

The 81 RCTs were again divided among the four evaluators for the application of the recommendations. Each evaluator received 20 or 21 RCTs and was instructed to evaluate one or two RCTs per day to avoid exhaustion and fatigue during the evaluation procedure. Three of the 81 RCTs selected were excluded: a) "Acute effects of mechanical lumbar traction with different intensities on stature", because it was not randomized; b) "Ensaio clínico comparando três modalidades de crioterapia em mulheres não grávidas" and c) "Efeito do exercício físico no metabolismo lipídico de idosos" because they had no involvement of physical therapists in their authorship. This resulted in 78 remaining RCTs.

Results

The median Kappa for each of the three evaluators was 0.72, 0.73 and 0.78, when their results were compared to the fourth evaluator considered the gold standard.

The search process that resulted in the inclusion of the 78 RCTs to be evaluated can be seen in the flow diagram (Figure 1).

Figure 1
Flowchart of study selection

The quantity of CONSORT items covered by the 78 studies was 43 ± 12%. According to the years 2003 (n = 6 RCTs), 2004 (n = 5 RCTs), 2005 (n = 7 RCTs), 2006 (n = 10 RCTs), 2007 (n = 4 RCTs), 2008 (n = 17 RCTs), 2009 (n = 6 RCTs), 2010 (n = 9 RCTs), 2011 (n = 9 RCTs) and 2012 (n = 5 RCTs) this was 51 ± 4%, 34 ± 13%, 39 ± 6% , 39 ± 14%, 42 ± 14%, 42 ± 11%, 39 ± 6%, 50 ± 10%, 51 ± 17%, and 58 ± 12%, respectively, after excluding items 3b, 6b, 7b, 11b, 12b, 13b, 14b, 17b, 18, 19, 24 and 25, considered optional (Figure 2).

Figure 2
Percentages of the total score of the CONSORT recommendations met by year of publication (2003 - 2012). The percentage was calculated over 29 items and not the 37 of CONSORT, because they were removed 8 items that are optional. Seventy-eight studies were evaluated

The least frequent CONSORT items were 1a, 1b, 10, 14a and 23, with percentages below 10%, while the most frequent were 2a, 2b, 4a, 12a and 17a, with percentages above 80% (Figure 3).

Figure 3
Frequency of CONSORT items (n = 78). The optional utems were excluded

Discussion

The first RCT published in MEDLINE was in March 1961 in the British Journal of Ophthalmology ⁽7. In PEDro, the first RCT was published in March 1965 in the journal Klinicheskaia Meditsina ⁽23. The first physical therapy RCTs were published in Portuguese 36 and 40 years latter, respectively ⁽7. This period of approximately 40 years may be related to the creation of stricto sensu courses in Physical Therapy in Brazil: "On 20^th December, 1996, the MEC, through the Coordination for the Improvement of Higher Education Personnel (CAPES), officially authorized the operation of the first Brazilian master's degree course in Physical Therapy at the University of São Carlos (UFSCar). From this, physical therapy began to formally integrate into the Brazilian scientific community" ⁽8.

Importantly, the fact that there were no RCTs published in the Portuguese language does not imply the absence of studies published by Brazilians physical therapists in other languages or by physical therapists of countries that adopt the Portuguese language.

The number of RCTs in the Portuguese language still is very low. On December 14, 2012, there were 335,988 RCTs in MEDLINE and 18,874 RCTs in PEDro, with only 455 in MEDLINE and 93 in PEDro being in Portuguese. Of these 510 RCTs, with the removal of duplicates and the selection of RCTs related to physical therapy, this number decreased significantly to 78 (Figure 1).

This low number of RCTs in the Portuguese language may be explained by the low number of existing physical therapy PhD holders, despite Coury and Vilella (2009) finding an increase of 900% in the number of PhD holders with a degree in physical therapy between 1998 and 2008, jumping from 57 to 573 PhD holders in this period ⁽24. This large increase in the number of Physical Therapy PhD holders was not enough to improve its ranking within the health professions, with Physical Therapy occupying the penultimate position with regards to the number of PhD holders. Medicine, at the peak of the classifications, has 13 times more PhD holders than Physical Therapy ⁽8.

Comparing Physical Therapy to other professions in the health sciences area, it was observed that in 2011 there were approximately 28,111 PhD holders and 43,060 MSc holders. Of these, 1,145 were physical therapists or occupational therapists with doctorate degrees and 4,675 physical therapists or occupational therapists with master's degrees ⁽8. This represented 4% and 11%, respectively, of the stricto sensu of the overall area of Health ⁽8. In February 2013, the number of physical therapy and or occupational therapy PhD holders and MSc holders increased to 1,711 and 6,429, respectively. This represented 5% and 12%, respectively, of the stricto sensu of the overall area of Health, showing a small increase in this sector ⁽25. The dissemination and adoption by national journals of the CONSORT recommendations would certainly raise the quality of the RCTs published in Portuguese, as has occurred in other countries ⁽26. This concern with the adoption of the CONSORT recommendations and improved drafting of the RCTs has increased ⁽27.

The results show that physical therapy RCTs published in Portuguese from 2003 to 2012 only included about 43 ± 12% of the items of the CONSORT recommendations. This low rate can be partly explained by the fact that the CONSORT recommendations were recently published in 2001 and are still poorly understood by researchers. From 2004, there has been a tendency to increase the percentage rate of inclusion of items of the CONSORT recommendations, reaching close to 60% in 2012 (Figure 2), showing that over the years the CONSORT recommendations have been acknowledged and used by researchers (Figure 2).

Although all 78 studies were RCTs, only 78% of the studies described in detail the interventions in a way that would allow replication of the study (item 05) and 35% described the randomization (item 8a). In addition, only 82% established inclusion criteria (item 4a), 17% determined the sample size (item 7a), 18% had some kind of blinding (item 11a) and 91% reported the statistical methods used (item12a) (Figure 3). This shows that the designs of the physical therapy RCTs in Portuguese lack rigorous methodological procedures, particularly regarding blinding, sample size and randomization. The great difficulty of blinding of patients and of those that apply the intervention in physical therapy RCTs is recognized, however, it is quite feasible to blind the evaluation and the data analysis.

The CONSORT items with frequency below 10% were 1a, 1b, 10, 14a, and 23 (Figure 3). The low frequency of items 1a, 1b, 14a and 23 characterize the lack of knowledge of the CONSORT recommendations, as they are easy to include: to include RCT in the study title, to structure the abstract, to report the date of recruitment of the participants and to report the registration of the RCT, respectively. Item 10 also characterizes a lack of knowledge of the CONSORT recommendations, however, also the lack of use of randomization in physical therapy RCTs, as this item deals with reporting who generated the random allocation sequence, who enrolled the participants and who distributed the participants for the interventions.

Conversely, item 2a, 2b, 4a, 12a and 17a were the most frequent, with percentages above of 80% (Figure 3). The increased frequency of these items can be explained by them covering the scientific information and logical explanation for conducting the study, contained in the introduction, the aim of the study, the inclusion criteria for the participants, the statistical method used, and the results obtained, respectively.

Adherence to the CONSORT recommendations can influence the quality of the scientific production ⁽28^,29^,30 and result in a high impact factor ⁽26. Many Brazilian researchers have resorted to international journals, due to the impact factor of these tending to be higher ⁽26. It seems that the journals that have adopted the CONSORT recommendations have increased their impact and, taking into consideration the academic requirements, the higher the impact of the publication, the better ⁽26.

It is therefore critical that national periodicals adopt these recommendations in order to improve the Brazilian scientific production in physical therapy and increase the demand of national and international researchers for our journals.

Conclusion

The results make very clear the need to improve the quality of writing of the RCTs in physical therapy in the Portuguese language and to include more rigorous methodological procedures, such as sample size, randomization and blinding. The dissemination and adoption of the CONSORT recommendations by physical therapy researchers would be a big step towards improving this quality.

References

• Filippin LI, Wagner MB. Fisioterapia baseada em evidência: uma nova perspectiva. Revista Brasileira de Fisioterapia. 2008; 12(5):432-3.
• Dias RC, Dias JMD. Prática baseada em evidências: uma metodologia para a boa prática fisioterapêutica. Fisioter Mov. 2006; 19(1):11-6.
• Mueller SPM. A comunicação científica e o movimento de acesso livre ao conhe-cimento. Ciência da Informação. 2006; 35(2):27-38.
• Costa SM. Filosofia aberta, modelos de negócios e agências de fomento: elementos essenciais a uma discussão sobre o acesso aberto à informação científica. Ciência da informação. 2006; 35(2):39-50.
• Shiwa SR, Costa LOP, Moser ADDL, Aguiar IDC, Oliveira LVFD. PEDro: a base de dados de evidências em fisioterapia. Fisioter mov. 2011; 24(3):523-33.
• Savi MGM, Silva ED. O fluxo da informação na prática clínica dos médicos residentes: análise na perspectiva da medicina baseada em evidências. Ciência da Informação. 2009; 38(3):177-91.
• US National Library of MedicineNational Institutes of Health. {cited Feb 23, 2013}. Available from: Available from: http://www.ncbi.nlm.nih.gov/pubmed/
  » http://www.ncbi.nlm.nih.gov/pubmed/
• Calvalcante CCL, Rodrigues ARS, Dadalto TV, Silva EB. Evolução científica da fisioterapia em 40 anos de profissão. Fisioter Mov. 2011; 24(3):513-22.
• Padula RS, Pires RS, Alouche SR, Chiavegato LD, Lopes AD, Costa LOP. Análise da apresentação textual de revisões sistemáticas em fisioterapia publicadas em idioma português. Rev Bras Fisioter. 2012; 16(4):381-8.
• Costa LOP, Maher CG, Lopes AD, Noronha MA, Costa LCM. Como escrever de forma transparente artigos científicos relevantes para a prática da fisioterapia. Rev Bras Fisioter. 2011; 15(4):267-71.
• Hopewell S, Dutton S, Yu LM, Chan AW, Altman DG. The quality of reports of randomised trials in 2000 and 2006: comparative study of articles indexed in PubMed. BMJ. 2010;2-8.
• Dawes M, Summerskill W, Glasziou P, Cartabellotta A, Martin J, Hopayian K, Porzsolt F, Burls A Osborne J. Sicily statement on evidence-based practice. BMC Medical Education. 2005; 5(1):1-7.
• Marques AP, Peccin MS. Pesquisa em fisioterapia: a prática baseada em evidências e modelos de estudos. Fisioterapia e Pesquisa. 2005; 11(1):43-8.
• Sampaio RF, Mancini MC. Estudos de revisão sistemática: um guia para síntese criteriosa da evidência científica. Rev. Bras. Fisioter. 2007; 11(1):83-9.
• Medina EU, Pailaquilén RMB. A revisão sistemática e sua relação com a prática baseada na evidência em saúde. Rev. Latino-Am. Enfermagem. {periódico na Internet}, jul-ago 2010{cited Sep. 10, 2015}; 18(4):{08 telas}. Available from: Available from: http://www.scielo.br/pdf/rlae/v18n4/pt_23.pdf
  » http://www.scielo.br/pdf/rlae/v18n4/pt_23.pdf
• Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JPA, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Ann Intern Med. 2009; 151(4):65-94.
• Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009; 6(7): e1000100.
• Kenneth F. Schulz, Douglas G. Altman, David Moher. CONSORT 2010 Statement: Updated Guidelines for Reporting Parallel Group Randomized Trials. Ann Intern Med. 2010; 152(11):726-32.
• Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA. 1996; 276(8):637-9.
• Vaarbakken K, Ljunggren AE, Hendricks EJ. The importance of the new CONSORT statement for clinicians. Aust J Physiother. 2008; 54(3):155-6.
• Costa LO, Maher CG, Moseley AM, Sherrington C, Herbert RD, Elkins MR. Editorial: Endorsement of trial registration and the CONSORT statement by the Revista Brasileira de Fisioterapia. Rev Bras Fisioter. 2010; 14(3):183-275.
• Kottner J, Audigé L, Brorson S, Donner A, Gajewski BJ, Hróbjartsson A, Streiner DL. Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. International journal of nursing studies. 2011; 48(6):661-71.
• PEDro - Phisiotherapy evidence database. Available from : {cited Jan. 15, 2015} Available from : {cited Jan. 15, 2015} http://www.pedro.org.au
  » http://www.pedro.org.au
• Coury HJCG, Vilella I. Perfil do pesquisador fisioterapeuta brasileiro. Rev Bras Fisioter. 2009; 13(4):356-63.
• CNPq. Plataforma Lattes, 2013. {cited Feb. 4 2013}. Available from: http://lattes.cnpq.br.
• Hopewell S, Ravaud P, Baron G, Boutron I. Effect of editors' implementation of CONSORT guidelines on the reporting of abstracts in high impact medical journals: interrupted time series analysis. BMJ. 2012;1-7.
• Debono VB, Zhang S, Ye C, Paul J, Arya A, Hurlburt L, Murthy Y, Thabane L. The quality of reporting of RCTs used within a postoperative pain management meta-analysis, using the CONSORT statement. BMC Anesthesiology. 2012; 12:1-13.
• Porcino AJ. The Greater Value of the CONSORT Statement Guidelines: Guideposts for Designing and Reporting all TMB Research. Int J Ther Massage Bodywork. 2012; 5(4):1-2.
• Nojomi M, Ramezani M, Ghafari-Anvar A. Quality of reports on randomized controlled trials published in Iranian journals: application of the new version of consolidated standards of reporting trials (CONSORT). Arch Iran Med. 2013; 16(1):20-2.
• Turner L, Shamseer L, Altman DG, Schulz KF, Moher D. Does use of the CONSORT Statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane review. Syst Rev. 2012; 1(1):60 .

Data availability

Data citations

PEDro - Phisiotherapy evidence database. Available from : {cited Jan. 15, 2015} Available from : {cited Jan. 15, 2015} http://www.pedro.org.au

Publication Dates

• Publication in this collection
  Apr-Jun 2016

History

• Received
  07 Dec 2013
• Accepted
  10 Aug 2015