APPLICABILITY OF RANDOMIZED TRIALS IN HAND SURGERY: SURVEY STUDY

APLICABILIDADE DE ENSAIOS RANDOMIZADOS EM CIRURGIA DA MÃO: ESTUDO TIPO SURVEY

ABSTRACT

Objective

To assess the applicability of randomized clinical trials and whether certain factors (surgeon experience/journal impact factor) influence their applicability.

Methods

In this survey study we used the Pubmed/Medline database to select 32 consecutive randomized clinical trials published between 2013 and 2015, involving hand surgery (high/low impact). These studies were independently assessed by 20 hand surgeons (with more or less than 10 years of practice) who answered 4 questions regarding their applicability. Agreement was assessed using Cohen’s kappa and comparison of proportions via chi-square statistics. P-value <5% constituted statistical significance.

Results

A total of 640 evaluations were produced, generating 2560 responses. A weak correlation was observed between less and more experienced respondents (kappa <0.2; range 0.119–0.179). Applicability between the least and most experienced respondents was similar (p = 0.424 and p = 0.70). Stratification by journal impact factor showed no greater propensity of applicability (p = 0.29) for any of the groups.

Conclusions

Low agreement was found between the respondents for the applicability of the randomized studies. Surgeon experience and journal impact do not seem to influence this decision. Level of Evidence II, Prospective comparative study.

Clinical trial; Evidence-based medicine; Reserch

RESUMO

Objetivo

Avaliar a aplicabilidade de ensaios clínicos randomizados e se há fatores que a influenciam (experiência do cirurgião/ impacto do periódico).

Métodos

Estudo tipo survey. Selecionou-se (via Medline/Pubmed) dentre os anos de 2013 e 2015, 32 ensaios clínicos randomizados consecutivos envolvendo cirurgia da mão (estratificados como alto/ baixo impacto). Estes estudos foram avaliados de forma independente por 20 cirurgiões de mão (mais versus menos de 10 anos de prática), que responderam quatro questões dicotômicas relativas à propensão da aplicabilidade clínicados estudos. A concordância foi avaliada pelo kappa de Cohen e comparação de proporções pelo Qui-quadrado. Constituiu-se como significantes p menores que 5%.

Resultados

realizou-se 640 avaliações, envolvendo 2560 respostas. Observou-se baixa concordância entre os avaliadores (menos versus mais experientes): Kappa <0,2; alcance 0,119-0,179. A propensão para aplicabilidade foi semelhante entre os menos e mais experientes (p=0,424 e p=0,70). O mesmo ocorre quando estratificados por impacto da revista, não há maior propensão de aplicabilidade para quaisquer dos grupos (p=0,29).

Conclusões

Há baixa concordância entre os avaliadores quanto à aplicabilidade de estudos randomizados. Experiência do cirurgião e impacto do periódico parecem não influenciar nesta decisão. Nível de Evidência II; Estudo prospectivo comparativo.

Ensaio Clínico; Medicina baseada em evidências; Pesquisar

INTRODUCTION

The number of published clinical studies has been increasing dramatically, with a challenging volume of information to be evaluated and summarized.^1,2 Scientific journals are largely responsible for disseminating such information, but there is some doubt about the connection between what is published in high-impact journals and its relevance in everyday practice.^3,4

In an effort to achieve visibility for their findings, researchers suffer from the high methodological requirements necessary for publication in high-impact journals. These transform good clinical questions into studies that do not reflect real practice scenarios, sometimes controlled artificially by the particularities of controlled randomized clinical studies. This is the current criticism of the purism of evidence-based medicine.⁵

Furthermore, a confounding factor can be found in the potential conflict of interest between high-impact journals and external funding by industry or various research sources.^6,7

Even so, the consensus is that comparative clinical studies controlled by randomization are the best and most reliable for everyday application.⁸ There is great doubt, however, correlating high levels of methodological excellence and practical applicability; studies involving hand surgery and orthopedics report that approximately half of randomized clinical studies change treatment paradigms in a north American orthopedists.^9-12

This study starts from the hypothesis that applying research to daily clinical practice (external validity) is not directly related to the impact or importance of the journal in which the study was published and also not associated with experience gained over time.

The objective of this current study is to verify whether characteristics of the study/periodical (journal impact) are related to the propensity of applicability (feasibility/relevance) of these results in daily clinical practice, using the opinion of physicians (hand surgeons and their experience) as the parameter.

MATERIALS AND METHODS

This study was approved by the institutional review board (CAAE: 55969916.5.0000.5505), and participants signed an informed consent form after explanation of the study.

Strategy for identifying studies eligible for the survey

Randomized clinical trials on treatment involving the topic of hand surgery. Consecutive articles were selected by a survey strategy involving MeSH terms (hand surgery OR hand therapy OR wrist surgery) associated with the filter¹⁰ for randomized clinical trials and limited to the past 4 years (2016–2012).

• 18 studies from high-impact journals^{13-18,21,23,26,28,29,32-35,37-39} (defined as Qualis/CAPES A1, A2, B1, AND/OR impact factor above 1.5);

• 14 studies from low-impact journals^{19,20,22,24,25,27,30,31,36,40-44} (defined as Qualis/CAPES B2 and lower and/or impact factor below 1.5).

The structured abstracts of these studies were evaluated by: 20 (twenty) physicians:

• 10 hand surgeons with more than 10 years of training;

• 10 hand surgeons/orthopedists with less than 10 years of training.

Verification of applicability/relevance/feasibility

For each study, questions related to applicability/feasibility/clinical relevance were developed, and participants responded during a single session under the supervision of the researcher, who did not influence the responses:

1. Do you consider the topic (clinical research question) relevant/important?

2. Under ideal conditions, would you apply these results in your patients?

3. In everyday clinical practice, is application of these results feasible?

4. Do you consider this methodology appropriate to answer the research question?

These questions were answered categorically (Yes/No). We considered studies which earned more than 75% “yes” responses applicable, study by study.

Analysis of subgroups

The responses were evaluated and categorized according to:

1. Journal impact: low vs. high impact;

2. Surgeon’s experience: more or less than 10 years of experience.

Statistical analysis

The numerical data were presented as mean and standard deviations or percentages, and confidence intervals of 95%.

Categorical data were assessed in accordance with 2x2 contingency tables and subsequent inferential analysis by means of the chi-square test. Assessment of interobserver concordance consisted of the Kappa statistic, with the level of concordance standardized according to Cohen. P values lower than 5% were considered significant.

RESULTS

Thirty-two studies were included, and evaluated by 20 hand surgeons, totaling 640 evaluations and 2560 responses.

Applicability/Feasibility/Relevance: high versus low-impact publications

When the study-by-study responses were stratified according to “great chance of applicability” and “low chance of applicability,” and “low and high scientific impact publications,” no significant difference was seen between the propensity of applicability. (Table 1)

Table 1
Summary of studies with low and high chance of applicability, stratified by journal impact factor.

Journal impact	High chance of applicability	Low chance of applicability
High	187	173
Low	157	123

• Chi-square; 1.079; p=0.29.

Concordance: more vs. less experienced

In general, the responses regarding the studies demonstrate low reproducibility and low significance when more experienced surgeons were compared with less experienced surgeons. Of the 32 studies, only four demonstrated statistical significance. (Tables 2 and 3)

Table 2
Interobserver agreement stratified by study and potential applicability: more vs. less experienced.

Concordance	Kappa	P-value
Vanni et al13	-0.032	0.629
Van Heest et al14	0.179	0.002*
White et al.15	-0.009	0.852
Zaino et al.16	-0.021	0.757
Rubin et al.17	0.110	0.103
Dundar et al.18	-0.114	0.087*
Orlandi et al.19	0.060	0.355
Roh et al.20	0.012	0.755
Kolbenschlag et al.21	-0.027	0.677
Mickelson et al22	0.069	0.308
Ekrol et al.23	0.000	0.998
McMillan et al.24	0.174	0.010*
Costa et al.25	0.119	0.046*
Prosser et al.26	0.043	0.525
Vermeulen et al.27	0.018	0.721
Vermeulen et al.28	0.012	0.815
Walenkamp et al.29	0.059	0.376
Buijze et al.30	0.096	0.066
Paschos et al.31	0.023	0.729
Bentohami et al.32	0.036	0.593
Gradl et al.33	0.000	1.000
Rocchi et al.34	0.018	0.752
Yamazaki et al.35	0.108	0.107
Gautam et al.36	0.062	0.353
Zhang et al.37	0.110	0.092
Lindan et al.38	0.035	0.597
Østerås et al.39	-0.031	0.633
Nam et al.40	0.052	0.262
Karlsson et al.41	0.019	0.775
Drac et al.42	0.150	0.026*
Geetha et al.43	0.116	0.073
Koman et al.44	0.064	0.321

Table 3
Interobserver agreement stratified by study and potential applicability: more vs. less experienced - significant results.

Concordance	Kappa	P-value
Van Heest et al.14	0.179	0.002
McMillan et al.24	0.174	0.010
Costa et al.25	0.119	0.046
Buijze et al.30	0.150	0.026

External validity: more vs. less experienced

Among the studies, the difference between the proportions of “yes” answers (number of “yes” answers per study), stratified by experience, showed a difference for eight studies (25%). Among these studies, the proportion of “yes” responses was greater among less experienced respondents in five studies. Among the more experienced physicians, the proportion of “yes” responses was higher in three studies, with no difference between the proportions (Fisher’s exact test, p=0.70).

When “yes” answers (>75%) were categorized by study, there was no difference between the less and more experienced professionals (less experienced: 37.5% versus more experienced: 28%, chi-square = 0.637; p=0.424). (Table 4)

Table 4
Number and percentage of “yes” answers, stratified by experience.

“Yes” answer	More Experienced		Less Experienced		P-value
	N	%	N	%
Vanni et al.13	32	80.0%	34	85.0%	0.556
Van Heest et al.14	25	62.5%	31	77.5%	0.143
White et al.15	18	45.0%	32	80.0%	0.001*
Zaino et al.16	21	52.5%	18	45.0%	0.502
Rubin et al.17	28	70.0%	32	80.0%	0.302
Dundar et al.18	27	67.5%	31	77.5%	0.317
Orlandi et al.19	20	50.0%	21	52.5%	0.823
Roh et al.20	33	82.5%	39	97.5%	0.025
Kolbenschlag et al.21	17	42.5%	15	37.5%	0.648
Mickelson et al22	27	67.5%	25	62.5%	0.639
Ekrol et al.23	20	50.0%	26	65.0%	0.175
McMillan et al.24	21	52.5%	16	40.0%	0.262
Costa et al.25	34	85.0%	24	60.0%	0.012*
Prosser et al.26	17	42.5%	16	40.0%	0.820
Vermeulen et al.27	14	35.0%	26	65.0%	0.007*
Vermeulen et al.28	33	82.5%	37	92.5%	0.176
Walenkamp et al.29	35	87.5%	33	82.5%	0.531
Buijze et al.30	20	50.0%	32	80.0%	0.005*
Paschos et al.31	31	77.5%	26	65.0%	0.217
Bentohami et al.32	20	50.0%	23	57.5%	0.501
Gradl et al.33	22	55.0%	11	27.5%	0.012*
Rocchi et al.34	32	80.0%	22	55.0%	0.017*
Yamazaki et al.35	26	65.0%	21	52.5%	0.256
Gautam et al.36	23	57.5%	21	53.8%	0.744
Zhang et al.37	31	77.5%	35	87.5%	0.239
Lindan et al.38	22	59.5%	28	70.0%	0.333
Østerås et al.39	32	80.0%	28	70.0%	0.302
Nam et al.40	28	70.0%	33	91.7%	0.018*
Karlsson et al.41	11	27.5%	16	40.0%	0.237
Drac et al.42	18	45.0%	20	50.0%	0.654
Geetha et al.43	27	67.5%	31	77.5%	0.317
Koman et al.44	14	35.0%	16	40.0%	0.644

DISCUSSION

The objective of this study was to verify the factors leading to applicability of high-quality clinical research (level I, randomized clinical studies) in the spectrum of hand surgery. It is plausible that in clinical practice, surgeon experience and journal impact are relevant factors for decision-making. More experienced surgeons are expected to be more skeptical about applying new evidence compared to less experienced professionals. The same is also expected in terms of journal impact factor: higher-impact journals are expected to publish studies with greater clinical relevance and external validity. However, the results of this study refute common sense, demonstrating that these factors are not relevant.

With the recent advent of open-access journals that charge for publication, there is a tendency for some studies to migrate to these journals since they offer impartial theoretical assessment and swifter publication. This scenario permits publication of studies with high methodological quality in “open” journals with lower impact and visibility in the area. Our results demonstrate a greater tendency to applicability, when stratified by journal impact or importance.

The applicability of the evidence, as disseminated by the Canadian school, should follow the cycle of evidence, which consists of five stages: 1. formulate the clinical question; 2. look for the best evidence; 3. critical analysis, verify the effects and their applicability; 4. integrate the evidence with the experience, in the clinical setting; 5. evaluate effectiveness and efficiency. This present study systematically created a controlled environment in which we can evaluate stages 2, 3 and 4 of evidence-based medicine in an integrated manner.^3,8

The results of our study are unprecedented and there is no standard for comparison in the area literature, making our results relevant and groundbreaking. Future studies could consider larger samples and investigate other influencing factors, such as external funding, costs, and specific regional characteristics (related to assistance and economic aspects).

CONCLUSION

In summary, in this study a low concordance was observed between a group of specialized surgeons in terms of applicability of randomized clinical trials. Potentially influencing factors, such as surgeon experience and journal impact factor were not seen to be relevant for this sample.

REFERENCES

• ¹
  Moraes VY, Belloti JC, Moraes FY, Galbiatti JA, Palacio EP, Santos JBG, et al. Hierarchy of evidence relating to hand surgery in Brazilian orthopedic journals. Sao Paulo Med J. 2011;129(2):94-8.
• ²
  Moraes VY, Moreira CD, Tamaoki MJS, Faloppa F, Belloti JC. Randomized Controlled Trials in orthopedics and traumatology: systematic analysis of the national evidence. Rev Bras Ortop. 2015;45(6):601-5.
• ³
  Moraes VY, Ferrari PM, Gracitelli GC, Faloppa F, Belloti JC. Outcomes in orthopedics and traumatology: translating research into practice. Acta Ortop Bras. 2014;22(6):330-3.
• ⁴
  Paradis C. Bias in surgical research. Ann Surg. 2008;248(2):180-8.
• ⁵
  Greenhalgh T, Howick J, Maskrey N. Evidence based medicine: a movement in crisis? The BMJ. 2014;348:g3725.
• ⁶
  Bhandari M, Jonsson A, Buhren V. Conducting industry-partnered trials in orthopaedic surgery. Injury. 2006;37(4):361-6.
• ⁷
  Bhandari M, Busse JW, Jackowski D, Montori VM, Schunemann H, Sprague S, et al. Association between industry funding and statistically significant pro-industry findings in medical and surgical randomized trials. CMAJ. 2004;170(4):477-80.
• ⁸
  Bhandari M, Haynes RB. How to appraise the effectiveness of treatment. World J Surg. 2005;29(5):570-5.
• ⁹
  Bhandari M, Tornetta P, Guyatt GH. Glossary of evidence-based orthopaedic terminology. Clin Orthop Relat Res.2003;(413):158-63.
• ¹⁰
  Haynes RB, Wilczynski NL. Optimal search strategies for retrieving scientifically strong studies of diagnosis from Medline: analytical survey. BMJ. 2004;328:1040.
• ¹¹
  Matzon JL, Lutsky KF, Maloney M, Beredjiklian PK. Adherence to the AAOS upper-extremity clinical practice guidelines. Orthopedics. 2013;36(11):e1407-11.
• ¹²
  Khan H, Hussain N, Bhandari M. The influence of large clinical trials in orthopaedic trauma: do they change practice? J Orthop Trauma. 2013;27(12): e268 –74.
• ¹³
  Vanni D, Sirabella FS, Galzio R, Salini V, Magliani V. The double tunnels technique: An alternative minimally invasive approach for carpal tunnel syndrome. J Neurosurg. 2015;123(5):1230-7.
• ¹⁴
  Van Heest AE, Bagley A, Molitor F, James MA . Tendon transfer surgery in upper extremity cerebral palsy is more effective than botulinum toxin injections or regular ongoing therapy. J Bone Joint Surg Am. 2015;97(7):529-36.
• ¹⁵
  White N, Dobbs TD, Murphy GR, Khan K, Batt JP, Cogswell LK. Oxygen reduces tourniquet-associated pain: a double-blind, randomized, controlled trial for application in hand surgery. Plast Reconstr Surg. 2015;135(4):721e-30e.
• ¹⁶
  Zaino CJ, Patel MR, Arief MS, Pivec R. The effectiveness of bivalving, cast spreading, and webril cutting to reduce cast pressure in a fiberglass short arm cast. J Bone Joint Surg Am. 2015;97(5):374-80.
• ¹⁷
  Rubin G, Orbach H, Rinott M, Wolovelsky A, Rozen N. The use of prophylactic antibiotic in treatment of fingertip amputation: a randomized prospective trial. Am J Emerg Med. 2015;33(5):645-7.
• ¹⁸
  Dundar U, Turkmen U, Toktas H, Ulasli AM, Solak O. Effectiveness of high-intensity laser therapy and splinting in lateral epicondylitis; a prospective, randomized, controlled study. Lasers Med Sci. 2015;30(3):1097-107.
• ¹⁹
  Orlandi D, Corazza A, Fabbro E, Ferrero G, Sabino G, Serafini G, et al. Ultrasound-guided percutaneous injection to treat de Quervain’s disease using three different techniques: a randomized controlled trial. Eur Radiol. 2015;25(5):1512-9.
• ²⁰
  Roh YH, Lee BK, Baek JR, Noh JH, Gong HS, Baek GH. A randomized comparison of volar plate and external fixation for intra-articular distal radius fractures. J Hand Surg Am. 2015;40(1):34-41.
• ²¹
  Kolbenschlag J, Sogorski A, Harati K, Daigeler A, Wiebalck A, Lehnhardt M, et al. Upper extremity ischemia is superior to lower extremity ischemia for remote ischemic conditioning of antero–lateral thigh cutaneous blood flow. Microsurgery. 2015;35(3):211-7.
• ²²
  Mickelson DT, Noland SS, Watt AJ, Kollitz KM, Vedder NB, Huang JI. Prospective randomized controlled trial comparing 1 day versus 7 day manipulation following collagenase injection for the treatment of Dupuytren’s contracture. J Hand Surg Am. 2014;39(10):1933-41.e1.
• ²³
  Ekrol I, Duckworth AD, Ralston SH, Court-Brown CM, McQueen MM. The influence of vitamin c on the outcome of distal radial fractures a double-blind, randomized controlled trial. J Bone Joint Surg Am. 2014;96(17):1451-9.
• ²⁴
  McMillan C, Binhammer P. Steroid injection and needle aponeurotomy for dupuytren disease: long-term follow-up of a randomized controlled trial. J Hand Surg Am. 2014;39(10):1942-7.
• ²⁵
  Costa ML, Achten J, Parsons NR, Rangan A, Griffin D, Tubeuf S. Percutaneous fixation with Kirschner wires versus volar locking plate fixation in adults with dorsally displaced fracture of distal radius: randomised controlled trial. BMJ. 2014;349:g4807.
• ²⁶
  Prosser R, Hancock MJ, Nicholson L, Merry C, Thorley F, Wheen D. Rigid versus semi-rigid orthotic use following TMC arthroplasty: a randomized controlled trial. J Hand Ther. 2014;27(4):265-70.
• ²⁷
  Vermeulen GM, Spekreijse KR, Slijper H, Feitz R, Hovius SE, Selles RW. Comparison of arthroplasties with or without bone tunnel creation for thumb basal joint arthritis: a randomized controlled trial. J Hand Surg Am. 2014;39(9):1692-8.
• ²⁸
  Vermeulen GM, Brink SM, Slijper H, Feitz R, Moojen TM, Hovius SE, Selles RW . Trapeziometacarpal arthrodesis or trapeziectomy with ligament reconstruction in primary trapeziometacarpal osteoarthritis a randomized controlled trial. J Bone Joint Surg Am. 2014;96(9):726-33.
• ²⁹
  Walenkamp MM, Goslings JC, Beumer A, Haverlag R, Leenhouts PA, Verleisdonk EJ et al. Surgery versus conservative treatment in patients with type A distal radius fractures, a randomized controlled trial. BMC Musculoskelet Disorders. 2014;15:90.
• ³⁰
  Buijze GA, Goslings JC, Rhemrev SJ, Weening AA, Van Dijkman B, Doornberg JN, et al. Cast immobilization with and without immobilization of the thumb for nondisplaced scaphoid waist fractures: a multi-center randomized controlled trial. J Hand Surg Am. 2014;39(4):621-7.
• ³¹
  Paschos NK, Abuhemoud K, Gantsos A, Mitsionis GI, Georgoulis AD. Management of Proximal Interphalangeal Joint Hyperextension Injuries: A Randomized Controlled Trial. J Hand Surg Am. 2014;39(3):449-54.
• ³²
  Bentohami A, de Korte N, Sosef N, Goslings JC, Bijlsma T, Schep N. Study protocol: non-displaced distal radial fractures in adult patients: three weeks vs. five weeks of cast immobilization: a randomized trial. BMC Musculoskelet Disord. 2014;15: 24.
• ³³
  Gradl G, Mielsch N, Wendt M, Falk S, Mittlmeier T, Gierer P, et al. Intramedullary nail versus volar plate fixation of extra-articular distal radius fractures. Two year results of a prospective randomized trial. Injury. 2014;45 Suppl 1:S3-8.
• ³⁴
  Rocchi L, Merolli A, Morini A, Monteleone G, Foti C. A modified spica-splint in postoperative early-motion management of skier’s thumb lesion: a randomized clinical trial. Eur J Phys Rehabil Med. 2014;50(1):49-57.
• ³⁵
  Yamazaki H, Uchiyama S, Komatsu M, Hashimoto S, Kobayashi Y, Sakurai T, et al. Arthroscopic assistance does not improve the functional or radiographic outcome of unstable intra-articular distal radial fractures treated with a volar locking plate: a randomised controlled trial. Bone Joint J. 2015;97-B(7):957-62.
• ³⁶
  Gautam VK, Verma S, Batra S, Bhatnagar N, Arora S. Platelet-rich plasma versus corticosteroid injection for recalcitrant lateral epicondylitis: clinical and ultrasonographic evaluation. J Orthop Surg (Hong Kong). 2015;23(1):1-5.
• ³⁷
  Zhang X, Li Y, Wen S, Zhu H, Shao X, Yu Y. Carpal tunnel release with subneural reconstruction of the transverse carpal ligament compared with isolated open and endoscopic release. Bone Joint J. 2015;97-B(2):221-8.
• ³⁸
  Lidman G, Nachemson A, Peny-Dahlstrand M, Himmelmann K. Botulinum toxin A injections and occupational therapy in children with unilateral spastic cerebral palsy: A randomized controlled trial. Dev Med Child Neurol. 2015; 57(8):754-61.
• ³⁹
  Østerås N, Hagen KB, Grotle M, Sand-Svartrud AL, Mowinckel P, Kjeken I. Limited effects of exercises in people with hand osteoarthritis: Results from a randomized controlled trial. Osteoarthritis Cartilage. 2014;22(9):1224-33.
• ⁴⁰
  Nam SH, Kim J, Lee JH, Ahn J, Kim YJ, Park Y. Palpation versus ultrasound-guided corticosteroid injections and short-term effect in the distal radioulnar joint disorder: a randomized, prospective single-blinded study. Clin Rheumatol. 2014;33(12):1807-14
• ⁴¹
  Karlsson M, Lindgren M, Jarnhed-Andersson I, Tarpila E. Dressing the split-thickness skin graft donor site. Adv Skin Wound Care. 2014;27(1):20-5.
• ⁴²
  Drac P, Cizmar I, Manak P, Hrbek J, Reska M, Filkuka P, et al. Comparison of the results and complications of palmar and dorsal miniinvasive approaches in the surgery of scaphoid fractures. A prospective randomized study. Biomed Pap Med Fac Univ Palacky Olomouc Czech Repub. 2014;158(2):277-81.
• ⁴³
  Geetha K, Hariharan NC, Mohan J. Early ultrasound therapy for rehabilitation after zone II flexor tendon repair. Indian J Plast Surg. 2014; 47(1):85-91.
• ⁴⁴
  Koman LA, Smith BP, Williams R, Richardson R, Naughton M, Griffin L, et al. Upper extremity spasticity in children with cerebral palsy: a randomized, double-blind, placebo-controlled study of the short-term outcomes of treatment with botulinum A toxin. J Hand Surg Am. 2013;38(3):435-46.e1.

Publication Dates

• Publication in this collection
  May-Jun 2018

History

• Received
  03 Oct 2016
• Accepted
  17 Jan 2018