Assessing the Level of Evidence of Presented Studies at the Brazilian Congress of Coloproctology

Abstract

Introduction

Scientific studies in Brazil grew around 10.7% compared to previous years. However, the level of quality of evidence has been decreasing. The aim in our study is to examine the meeting abstracts of the Brazilian congress of coloproctology and analyze the level of evidence in trends and variables.

Methods

A descriptive bibliometric study, working with secondary data to review scientific abstracts in the annals of the coloproctology congress from 2015 to 2019.

Results

A total of 1756 abstracts of the Brazilian Congress of Coloproctology were analyzed for 5 years (2015-2019). There was a higher trend of abstracts presented with lower levels of evidence (level of evidence 5: 52.3% and 3: 30%), being the majority composed of case reports (49.4%) and retrospective studies (30.4%). The last two years analyzed (2018: 55.2% and 2019: 59.3%) had a predominance above average of case reports. From 2017 to 2019 there was a significant decrease in the number of level 2 evidence studies (18.10%,11.80% and 5.50%), while the number of studies with level 5 evidence showed an increase (45.60%, 56.60% and 61.40%). Statistical analysis occurred in only 17%, with an important decrease for the last two years (2018: 13.6%; 2019: 12.1%).

Conclusions

Although the data of this study is from the Brazilian coloproctology point of view, they are important for the global scientific community, as they allow a quantitative evaluation of the relative contribution from the level of evidence of Brazilian coloproctology researchers to the scientific scenario.

Keywords
level of evidence; colorectal surgery; bibliometrics; congresses

Introduction

Employing methodological principles of evidence-based medicine from increasingly controlled studies has the potential to improve the quality and safety of medical practice in the community.¹ However, few studies address the level of evidence in congresses and scientific meetings around the world. Especially in Brazil, scientific publications have grown about 10.7% compared to previous years.² Nonetheless, the percentage of studies that have a high level of evidence is much lower when compared to the total number of published studies.

The Brazilian Society of Coloproctology today occupies second place worldwide in number of associates and public participation in the annual congress; and, to date, its main scientific event (Brazilian Congress of Coloproctology) has not been evaluated for the level of evidence of the presented studies, even so the associated factors and publications derived from them.

Methods

This is a descriptive study, working with secondary data from the scientific abstract's reviews presented in the annals of the national congresses of coloproctology from 2015 to 2019 and the evaluation of their level of evidence. Through a bibliometric study, two different examiners used a standard form for data collection, which were tabulated in the Microsoft Excel 2019 program. Papers that were published in peer-reviewed journals were identified through a standardized search, according to previous publications,³-⁵ in the MEDLINE (PubMed), SciELO and Google Scholar databases.

Only oral presentations and posters were included, excluding presentations in video format. As an exclusion factor, abstracts that were not complete, titles which did not match the content of the text and absence of authors, were not included in the sample.

To the detriment of the study design classification and the level of evidence, according to the already available medical literature,⁶-⁸ the abstracts were separated according to the Center of Evidence-Based Medicine classification from the University of Oxford (►Table 1), with the level of evidence ranging from 1 to 5 (1 being higher level of evidence and 5 lower level of evidence).⁹ Still, for statistical categorization purposes, levels 1 to 2 were grouped into “High” and levels 3 to 5 were grouped into “Low”.

Table 1
Level of evidence

Study design	Level of Evidence	Categorization
Meta-analysis of a randomized clinical trial (RCT) or high quality RCT	Level 1	High
Low Quality RCT or Prospective Studies	Level 2
Control case or retrospective studies	Level 3	Low
Series of cases without comparison or control group	Level 4
Case reports, experimental studies or personal experience	Level 5

Source:⁹

Due to the collection of secondary data and the public nature of the annals of the Brazilian Congress of Coloproctology and published articles, there was no need for evaluation and approval from the ethics committee.

Statistical Analysis

For descriptive analysis, means and standard deviations were used for the metric variables and percentages for categorical variables. The year-on-year variability of the level of evidence was analyzed. Analysis of variance, equality of two proportions, paired Student's t-test, chi-square, and confidence interval for the mean tests were used for statistical comparisons. Bivariate and multivariate logistic regression analyses were performed to determine which independent abstract variables (year, period, number of authors, number of sample individuals, coloproctology areas, presence of statistical analysis and study designs) were significant predictors for higher level of evidence (dependent variable). The Software Program Statistical Package for Social Science (SPSS version 20.0 for Windows, SPSS, Chicago, IL) was adopted in all analyses. The values were considered significant with a 95% confidence interval (p < 0.05).

Results

A total of 1,756 abstracts presented at the Brazilian Congress of Coloproctology were analyzed over 5 years, from 2015 to 2019. Being 1169 (66.6%) in the poster category and, others, 33.4% in the oral category, considering that the abstracts presented in the video category were excluded from the sample. There was a heterogeneity over the years in the number of abstracts presented, including the presentation format. We highlight the largest number of abstracts presented in 2019 (N = 420) and the highest percentage of the poster category, while the lowest number was in 2015 (N = 307) and the predominance of presentation in the oral category was in 2016.

There was a higher trend of abstracts presented with lower levels of evidence (level of evidence 5: 52.3% and 3: 30%); among these, the most common types of study designs were case reports (49.4%) and retrospective studies (30.4%). The last two years analyzed (2018: 55.2% and 2019: 59.3%) had a predominance above average of case reports, but the percentage of retrospective studies remained around 30% for each year. Of this total of abstracts, it was observed that 59.7% belonged to a university center and only 8.4% to some multicenter study. The presence of statistical analysis occurred in only 33.6%, observing a growth from 2015 to 2017 (33.1%; 34.1%; 39.1%), but with a decrease for the last two years (2018: 30.3% and 2019: 29.8%) with the lowest rates found.

From 2015 to 2017, award-winning abstracts were appointed, which only represented 2.3% of the sample per year. It was not possible to obtain these data from the years 2018 and 2019 because it did not contain nomination of the papers in the researched source, congress website or website of the Brazilian society of coloproctology.

As observed in ►Table 2, from 2017 to 2019 there was a significant decrease in the number of studies with level 2 evidence (18.1%, 11.8% and 5.5%), while the number of studies with level 5 evidence increased (45.60%, 56.6% and 61.4%).

Table 2
Distribution of level of evidence among the editions of the congress

Level Evidence	2015		2016		2017		2018		2019		TOTAL
	N	%	N	%	N	%	N	%	N	%	N	%
Level 1	0	0,0%	6	1,8%	3	0,8%	1	0,3%	2	0,5%	12	0,7%
Level 2	42	13,7%	61	18,5%	64	18,1%	41	11,8%	23	5,5%	231	13,2%
Level 3	92	30,0%	96	29,1%	110	31,2%	101	29,2%	127	30,2%	526	30,0%
Level 4	16	5,2%	10	3,0%	11	3,1%	6	1,7%	2	0,5%	45	2,6%
Level 5	149	48,5%	154	46,7%	161	45,6%	196	56,6%	258	61,4%	918	52,3%
Not rated	8	2,6%	3	0,9%	4	1,1%	1	0,3%	8	1,9%	24	1,4%

Analysis made with Chi-Square test.

Moreover, in ►Table 3, the analysis of bivariate relationship of the level of evidence in congress with other qualitative factors, using Chi-Square test, demonstrated that it has a statistically significant relationship with some factors such as the category, where we have that the index of presentations in the oral format was 27.7% among the studies of low level of evidence against 68.7% among the studies with high level of evidence. The poster index was 72.3% down and 31.3% for high levels of evidence (p-value <0.001). Another example that we mention is the association of the level of evidence with publication, where we found that the positive index (answer “Yes”) was 5.4% in Low and 16.5% in High (p-value <0.001).

Table 3
Association of Level of Evidence with Qualitative Factors

Variable		Low Evidence Level		High Level of Evidence		Total		P-value
		N	%	N	%	N	%
Category	Oral	412	27,7%	167	68,7%	579	33,4%	<0,001
	Poster	1.077	72,3%	76	31,3%	1.153	66,6%
University Center	No	605	40,6%	100	41,2%	705	40,7%	0,878
	Yes	884	59,4%	143	58,8%	1.027	59,3%
Published	No	1.409	94,6%	203	83,5%	1.612	93,1%	<0,001
	Yes	80	5,4%	40	16,5%	120	6,9%
Authors' Number Classes	1-3	72	4,8%	22	9,1%	94	5,4%	0,017
	4-5	173	11,6%	22	9,1%	195	11,3%
	≥6	1.244	83,5%	199	81,9%	1.443	83,3%
Local	Midwest	101	7,0%	8	3,4%	109	6,5%	0,003
	International	23	1,6%	8	3,4%	31	1,8%
	Northeast	215	14,8%	49	20,9%	264	15,6%
	North	8	0,6%	3	1,3%	11	0,7%
	Southeast	936	64,4%	149	63,7%	1.085	64,3%
	South	170	11,7%	17	7,3%	187	11,1%
Area	Colonoscopy	83	5,6%	17	7,0%	100	5,8%	<0,001
	Benign Anoretal Diseases	164	11,0%	33	13,6%	197	11,4%
	Pelvic floor diseases	91	6,1%	48	19,8%	139	8,0%
	Inflammatory Bowel Diseases	229	15,4%	38	15,6%	267	15,4%
	Malignant and premalignant diseases of the colon	548	36,8%	50	20,6%	598	34,5%
	Sexually Transmitted Diseases	31	2,1%	3	1,2%	34	2,0%
	Experimental Studies in Coloproctology	31	2,1%	3	1,2%	34	2,0%
	Miscellany	312	21,0%	51	21,0%	363	21,0%

Analysis made using the Chi-Square test.

As it is a cross-sectional observational study, it is possible to calculate the prevalence ratio of publication in dichotomous factors, observed in ►Table 4. A variable with prevalence ratio (PR) of statistical significance, occurred with “Awarded”, where the PR was 4.35 (2.11 - 8.96), that is, awarded works have a higher chance of being published 4.35 times than the unawarded ones. Moreover, when the variable “level of evidence” was evaluated as a favorable factor for publication, we found that studies with a higher level of evidence are 3.06 (2.16 - 4.34) times more likely to be published than studies with lower level of evidence.

Table 4
Comparison between the years of qualitative factors related to the abstracts presented

		Published	Unpublished	Prevalence	P-value	Prevalence Ratio
Awarded	Yes	6	10	37,50	<0,001	4,35 (2,11 - 8,96)
	No	84	890	8,62
Level Evidence	High	40	203	16,46	<0,001	3,06 (2,16 - 4,34
	Low	80	1.409	5,37

When performing the bivariate analysis of the complete period of 2015 to 2019, using the Chi-Square test, it was verified that publication has a statistically significant relationship with several factors (►Table 5). For example, an interesting relationship is studies of high level of evidence (analysis of the level of evidence grouped) that are present in 33.3% of those published, in contrast, 12.6% of the unpublished.

Table 5
Publication Associated with Qualitative Factors

		Unpublished		Published		Total		P-value
		N	%	N	%	N	%
Level Evidence (Grouped)	High	203	12,6%	40	33,3%	243	14,0%	<0,001
	Low	1.409	87,4%	80	66,7%	1.489	86,0%
Level Evidence	Level 1	11	0,7%	1	0,8%	12	0,7%	<0,001
	Level 2	192	11,9%	39	32,5%	231	13,3%
	Level 3	482	29,9%	44	36,7%	526	30,4%
	Level 4	39	2,4%	6	5,0%	45	2,6%
	Level 5	888	55,1%	30	25,0%	918	53,0%

Analysis made with Chi-Square test.

When we see specific levels of evidence, we observe that evidence level 2 is present in 32.5% of those published compared to 11.9% of unpublished ones. On the other hand, the relationship is inverse when analyzing the level of evidence 5, in which the proportion of studies with this level is higher among the unpublished (55.1%) versus the published (25%).

Discussion

Congresses are rich environments of discussion and presentation of new themes, reassessment of concepts already pre-established and composition of protocols that are best appropriate to the present. In fact, the congresses have an important impact on the continuity of medical education,⁶ mainly tied to the hierarchical gradation of levels of evidence that allow an overview about quality of the theme addressed, facilitating choice of work that should be integrated into practice and those without significant value. Today, although still less frequent, bibliometric study about congresses and papers presented around the world has become more common,¹,⁶-⁸,¹⁰-¹⁴ especially in plastic surgery and orthopedics.

We evaluated a specific congress, as it is the largest scientific colorectal surgery event in Brazil and the 2nd largest coloproctology congress in the world. Although our data are mainly from the point of view of national coloproctology and in an event in which the main language is Portuguese, this measurement is still important for global scientific community, as it allows a quantitative bibliometric evaluation of the relative contribution of the level of evidence of Brazilian researchers in the area to the scientific scenario. This bibliometric report also allows us to initiate a critical reflection on the real level of evidence with potential planning modifications by the specialized society and the academic community.

The levels of evidence qualify scientific studies, facilitating the evaluation of results that should be incorporated into practice and those without significant value. This study showed a considerable portion of abstracts presentations with low levels of evidence (level 5: 52.3% and 3: 30%), with many case reports (49.4%) and retrospective studies (30.4%). In addition, it was observed that from 2017 to 2019 there was a significant decrease in the number of studies with level 2 evidence, while the number of studies with level 5 evidence increased. This data was in contradiction with the literature about other congresses, especially in nephrology and orthopedics,¹ where an improvement in the quality of the level of evidence has been observed over the last few years. According to Zamir et al, the quality of evidence presented at the annual congress of the Canadian Society of Nephrology has increased in the last 5 years, and these results are comparable to those of similar studies in other disciplines; for example, the quality of evidence presented at the annual orthopedic surgery congresses has also increased over time.⁷

The proportion of studies with level 1 evidence was only 0.7%. Other studies have shown similar proportions, such as 0.9%¹⁵ and 4%.¹¹ This indicates that the lack of studies with the highest quality in scientific evidence is not exclusive to Brazilian congresses.

The scientific production of papers with a lower level of evidence (level 3-5) is more attractive from the curricular point of view, since the revision of retrospective medical records, such as case report, has a shorter execution time,⁶ requiring less effort and resource detachment - besides being easily delegable to less graduated professionals and medical students. Often the medical congress is a way to only boost individual curriculum quality, physicians end up opting for simpler forms of academic production, and this preference is a growing aspect in the Brazilian coloproctology congresses, as observed in this study. Nevertheless, this data should not be considered positive, since the production of work without relevance or low level of evidence does not add changes or scientific growth in the area, preventing improvements and technical innovations.

The proportion of studies with high levels of evidence was the same inside and outside university centers, showing that it is possible to develop science outside the academic environment, given its importance, since most medical practice occurs outside universities. However, although this analysis is scarce in literature, European studies have shown a greater relationship of publication with university centers. In addition, such studies describe that the quality and impact of research in different European countries vary according to the characteristics of each institution (its size, reputation, time of existence, etc.),¹⁶-¹⁸ variables that would be subjective in nature and difficult to measure or control.

As for the category of works (oral or poster), in general, an evaluating panel, from the congress itself, is the one who distinguishes each work to its belonging category; and, in the 5 years analyzed in this study, it is possible to observe an increasing in poster presentations and a decreasing in “oral” presentations. Studies with a higher level of evidence tend to need a larger and more elaborate presentation, being preferred for oral presentation. On the other hand, studies with lower level of evidence, being the main representative of this group the “case report”, would not require oral presentation, because the poster gathers all the necessary information and is faster to be presented, occupying less time and physical space, and requiring a smaller number of evaluators.

Our statistical results of univariate and bivariate analyses showed that a study with a high level of evidence impacts a higher chance of publication (PR = 3.06 [2.16 - 4.34]), according to the literature about the same theme.⁴,¹⁹ On the other hand, the variables “presence of statistical analysis”, “number of authors” and “multicenter” proved to be important during our literature review - being trends that would boost the level of evidence of presentations in international congresses. However, when analyzing the Brazilian congress of coloproctology, we did not obtain statistical support for these variables related to a higher level of evidence.

Measurements of greater rigor in the approval of these papers presented or even propose the submission of complete papers and not only of their abstracts could filter those with lower level of evidence. Until recently, there were awards at the congress for case reports and even awards aimed at undergraduate students; the award in this analysis proved to be determinant for publication and greater gradation of evidence, another possibility to stimulate a higher level of evidence of the event would be to include more awards and possibly direct them only to works of certain types of study or level of evidence (level 1, 2 and 3).

The massive amount of data processed for this study may have been subject to measurement bias, despite all care in the training of examiners and having followed criteria already well established in the literature. Still, considering the evaluation of only one congress and in a specific time interval, we cannot transpose the reality of the data found for all medical meetings of coloproctology or other Brazilian medical areas.

Moreover, it should be in mind that the gradation of the oxford level of evidence⁹ is a subjective classification, being susceptible to discrete changes depending on variables (such as diagnostic screening, prognosis, treatment and etc.), but used by much of the literature in studies of similar theme.⁶,⁷,¹¹

Conclusion

Data on the quality and possible determinants of the level of evidence of papers presented at an important scientific meeting were provided, revealing, in general, a decrease in studies with a high level of evidence and an increase in studies with a low level of evidence. Resolutory measures may be promoted by the scientific board of the coloproctology society or organizing committee of its events. In addition, new studies on the same topic should be encouraged and conducted, alerting the scientific community about the potential risks of reducing the number of studies with a high level of evidence in the long term, mainly due to the potential harm to society and its development.

Acknowledgments

We are grateful for the support of the Pontifícia Universidade Católica de Campinas/PUC-Campinas for reating a prosperous environment for research and development in the academic environment by its professors and students.

References

• ¹
  Zamir N, Gholami A, Jajarmi Y, Jackson Chornenki NL, Patel A, Dore KL. Assessing the Quality of Evidence Presented at Annual General Meetings: A 5-Year Retrospective Study. J Contin Educ Health Prof 2019;39(02):152–157
• ²
  de Almeida ECE, Guimarães JA. Brazil's growing production of scientific articles—how are we doing with review articles and other qualitative indicators? Scientometrics 2013;97(02):287–315
• ³
  Denadai R, Pinho AS, Samartine H, Denadai R, Raposo-Amaral CE. Conversion of Plastic Surgery meeting abstract presentations to full manuscripts: a brazilian perspective. Rev Col Bras Cir 2017;44 (01):17–26
• ⁴
  Scherer RW, Langenberg P, von Elm E, Schmucker C, Schwarzer G, von Elm E. Full publication of results initially presented in abstracts. Cochrane Database Syst Rev 2018;(11):
• ⁵
  Nascimento S, Rahal RMS, Soares LR, de Souza Pimentel HJ, Kamimura TO, Freitas-Junior R. Publication rate of scientific papers presented at the largest event on breast cancer research in Latin America. Ecancermedicalscience 2021;15:1259
• ⁶
  Denadai R, Pinho AS, Junior HS, Denadai R, Raposo-Amaral CE. Level of Evidence of Abstract Presentations at Brazilian Plastic Surgery Annual Meetings. J Craniofac Surg 2016;27(05):1239–1243
• ⁷
  Shoker M, Hahn L, Patel A, Zamir N. Assessing the Quality of Evidence Presented at the Annual Conferences of Diabetes Canada. Can J Diabetes 2021;45(04):369–374
• ⁸
  Gazendam AM, Nucci NW, Ekhtiari S, Lanting BA, MacDonald SJ, Wood TJ. Quantifying the Level of Evidence of Podium Presentations at the American Association of Hip and Knee Surgeons From 2015 to 2019. J Arthroplasty 2021;36(06):2219–2222
• ⁹
  OCEBM LoEWG. -. “The Oxford Levels of Evidence 2”: Oxford Centre for Evidence-Based Medicine. [Available from: https://www.cebm.ox.ac.uk/resources/levels-of-evidence/ocebm-levelsof-evidence
  » https://www.cebm.ox.ac.uk/resources/levels-of-evidence/ocebm-levels-of-evidence
• ¹⁰
  Daniel SJ, Bouchard M-J, Tremblay M. Rethinking Our Annual Congress-Meeting the Needs of Specialist Physicians by Partnering With Provincial Simulation Centers. J Contin Educ Health Prof 2022;42(01):e83–e87
• ¹¹
  Lerman DM, Cable MG, Thornley P, et al. Has the Level of Evidence of Podium Presentations at the Musculoskeletal Tumor Society Annual Meeting Changed Over Time? Clin Orthop Relat Res 2017; 475(03):853–860
• ¹²
  Rifkin WJ, Yang JH, DeMitchell-Rodriguez E, Kantar RS, Diaz-Siso JR, Rodriguez ED. Levels of Evidence in Plastic Surgery Research: A 10-Year Bibliometric Analysis of 18,889 Publications From 4 Major Journals. Aesthet Surg J 2020;40(02):220–227
• ¹³
  Gallo L, Wakeham S, Dunn E, Avram R, Thoma A, Voineskos S. The Reporting Quality of Randomized Controlled Trial Abstracts in Plastic Surgery. Aesthet Surg J 2020;40(03):335–341
• ¹⁴
  Parikh RP, Sharma K, Qureshi AA, Franco MJ, Myckatyn TM. Quality of Surgical Outcomes Reporting in Plastic Surgery: A 15-Year Analysis of Complication Data. Plast Reconstr Surg 2018;141(06):1332–1340
• ¹⁵
  Chattopadhyay A, Wu R, Wan D, Momeni A. AAPS Podium Presentations-Has the Level of Evidence Changed over the Past Decade? Plast Reconstr Surg Glob Open 2021;9(05):e3588
• ¹⁶
  Bonaccorsi A, Secondi L. The determinants of research performance in European universities: a large scale multilevel analysis. Scientometrics 2017;112(03):1147–1178
• ¹⁷
  Bruni R, Catalano G, Daraio C, Gregori M, Moed H. Studying the heterogeneity of European higher education institutions. Scientometrics 2020;125:1117–1144
• ¹⁸
  Lepori B, Seeber M, Bonaccorsi A. Competition for talent. Country and organizational-level effects in the internationalization of European higher education institutions. Res Policy 2014;•••:44
• ¹⁹
  Saygili ES, Yildiz BO. Publication outcome of research presented at the European Congress of Endocrinology: a web scraping-based analysis and critical appraisal. Endocrine 2021;72(02):385–391

Publication Dates

• Publication in this collection
  22 Dec 2023
• Date of issue
  Dec 2023

History

• Received
  16 Jan 2023
• Accepted
  21 June 2023