Undergraduate Medical Student’s Perception about the Progress Testing

Abstract:

Introduction:

The Progress Test was created to address the necessity of measuring the level of knowledge consolidation along the years of Medical school. The test is administered periodically to all students in a curriculum, assessing the student’s cognitive growth throughout their journey at undergraduate level. In addition to assessing the student individually, the test evaluates the institution, showing in which areas its curriculum base should be improved. The aim is to assess the Universidade do Extremo Sul Catarinense student’s perception of the Progress Test.

Methods:

A cross-sectional study was performed. Data was collected through questionnaires created by the researchers and applied to medical students - the ones who took the Progress Test at least once - from October 15^th to November 30^th, 2018. The statistical analysis was performed with a 95% confidence interval.

Results:

A response rate of 70.41% was obtained, with a total of 424 questionnaires being included in the research. Demographic data showed a predominance of female gender (60,4%) and white ethnicity (96,2%) in the population and a mean age of 23 years. In all semesters (early, intermediate and final ones) the participants knew the goal of the progress test, and most students consider it important. It was also observed that the majority of the students considered clinical surgery and collective health as their worst performance in the test. In clinical medicine, pediatrics, and gynecology-obstetrics, the students of the intermediate and final semesters were satisfied with their level of knowledge. “To evaluate the student’s progress/performance” was highlighted as the most positive point. Among the negative ones “decrease the number of questions so the test is not as extensive” was emphasized.

Conclusion:

The students of the sample consider the Progress Test important and know about its purpose. The final third of the Medical School is the one who feels most prepared to face the test. The main fields to which the students attributed their worst performance were clinical surgery and collective health. Regarding clinical medicine, pediatrics, gynecology, and obstetrics the students were satisfied with their knowledge.

Keywords:
Progress Test; Medical Education; Knowledge; Perception; Students

Resumo:

Introdução:

O Teste de Progresso serve para medir a consolidação do conhecimento ao longo da formação acadêmica médica. O teste é administrado periodicamente a todos os alunos de um currículo, com o propósito de avaliar o crescimento cognitivo deles na graduação. Além de avaliar o estudante de forma individual, o teste avalia a instituição, de modo a mostrar em quais áreas a sua base curricular deve ser melhorada. O objetivo deste estudo foi avaliar a percepção de acadêmicos de Medicina da Universidade do Extremo Sul Catarinense em relação ao Teste de Progresso.

Método:

Realizou-se um estudo transversal, em que se coletaram dados por meio de questionários elaborados pelos pesquisadores, aplicados aos alunos do curso de Medicina - que fizeram o Teste de Progresso em algum momento do curso - no período de 15 de outubro a 30 de novembro de 2018. A análise estatística foi realizada com um nível de confiança de 95%.

Resultados:

Obteve-se uma taxa de resposta de 70,41%, com um número de 424 questionários incluídos na pesquisa. Os dados demográficos revelam uma população de predominância feminina (60,4%), branca (96,2%), com uma média de 23 anos de idade. Nas fases iniciais, intermediárias e finais da pesquisa, os participantes sabiam o objetivo do Teste de Progresso, bem como a maioria dos estudantes, em todas as fases, considerou-o importante. Observou-se também que a maioria dos estudantes considera clínica cirúrgica e saúde coletiva como seu pior desempenho no Teste de Progresso. Já nas áreas de clínica médica, pediatria e ginecologia obstetrícia, os estudantes das fases intermediárias e finais se consideram satisfeitos com seu nível de conhecimento. “Avaliar a evolução/o desempenho do acadêmico” foi o ponto positivo mais citado. Já entre os negativos, destacou-se “diminuir o número de questões para que a prova não fique tão extensa”.

Conclus

ões: Os estudantes da amostra consideram o Teste de Progresso importante e conhecem o objetivo dele. Reconheceram que se sentiram mais preparados nas fases finais. As principais áreas a que os estudantes atribuíram seu pior desempenho foram: clínica cirúrgica e saúde coletiva. Já nas áreas de clínica médica, pediatria e ginecologia obstetrícia, eles se mostraram satisfeitos com o próprio nível de conhecimento.

Palavras-chave:
Teste de Progresso; Educação Médica; Conhecimento; Percepção; Alunos

INTRODUCTION

With the advancement of medical education towards competence-based education, the assessment aims to monitor the student’s learning process in a continuous and systematic way. The progress test (PT) was introduced in the late 1970s¹. It has been used to evaluate the student’s cognitive ability in a longitudinal way through multiple-choice items periodically applied to all students², being an important type of assessment, especially in curricular matrices based on active student-centered methodologies³.

The PT aims to assess not only the performance of students during the growth of longitudinal knowledge, but also the course itself, allowing managers to perform a broad analysis, guiding possible changes that would benefit the undergraduate program as a whole⁴. When several schools participate in PT, comparative information can be obtained, which is even more useful for curriculum evaluation⁵. In the Netherlands, six of the eight medical schools have combined forces and collaboratively develop and manage progress tests^6),(7.

The UNESC Medical course, which is part of the Interinstitutional South Pedagogical Support Nucleus II (NAPISUL II, Núcleo de Apoio Pedagógico Interinstitucional Sul II), which was created in 2011, participated in all editions of the Progress Test⁸. There are currently 12 PT centers in operation in Brazil. The PT has been applied in the UNESC Medical course to evaluate the institution and the individual evolution of the students, who individually receive their analysis compared to their phase and the mean of the respective phase of all schools that constitute the center. The student’s perception of the test importance is crucial for adherence, as well as the commitment to answer the items.

The aim of the present study was to understand the perception of UNESC medical students about the importance of the PT.

MATERIALS AND METHODS

A cross-sectional observational study was carried out, with collection of primary data and quantitative approach. The project was approved by the Research Ethics Committee of Universidade do Extremo Sul Catarinense - UNESC under Opinion N. 2,744,900 (CAAE: 91936318.7.0000.0119) in 2018, and N. 3,286,259 (CAAE: 91936318.7.0000.0119) of 2019, and all participants signed the free and informed consent form.

The sample was calculated using the formula proposed by Callegari-Jacques (2004)⁹, where P refers to the proportion that maximizes the minimum sample size, 0.5, z is the standard normalized statistic linked to the significance level α = 0.05, which is 1.96, ɛ is the maximum tolerable sampling error, 0.05 and, n₀, the minimum sample size, which resulted in 385 questionnaires. Considering the target population of this research, consisting of 622 students, the value previously found was adjusted using the formula proposed by Barbetta (2007)¹⁰, resulting in a minimum sample size of 238 participants, which were distributed in all phases of the course. The formulas used are shown below:

$n_0=\frac{P.\left(1-P\right).{z_{\frac{\alpha }{2}}}^2}{{\epsilon }^2}$

$n=\frac{N.n_0}{N+n_0}$

The inclusion criterion was that the students must have taken the Progress Test at least once, at any time during the course.

The PT applied by the NAPISUL II Center, which consists of 12 medical schools, is in its eighth edition and contains 120 questions, with different levels of difficulty divided into six main areas: clinical medicine, clinical surgery, pediatrics, gynecology and obstetrics, collective health and basic sciences, and must be answered within a four-hour period. The questions have a reference matrix, and all are supported by the literature that becomes available to students after the test is applied ^8),(11. The test is performed at annual intervals throughout the course, and what is expected is an upward curve of the results at the end of the training period². After performing the test, the participants receive a score of their performance allowing them to compare it with the score of their school year and with the mean of the same school year in the Center, thus allowing a self-assessment¹².

Data collection was carried out through questionnaires applied to students at the 12 phases of undergraduate medical school at UNESC, which is located in the city of Criciúma, in the state of Santa Catarina, from October 15 to November 30, 2018, after providing to the students the answers to the PT applied in 2018. The questionnaire, prepared by the authors, was applied to the students, between classes, by two undergraduate medical students. The questionnaire contained closed and open questions related to the students’ profile (age, gender, year, ethnicity), their perceptions regarding the Progress Test (objective, importance, influence on professional career, student training, reading of questions, access to feedback and dedication), level of student satisfaction in the areas assessed in the test and test strengths and weaknesses (^{Annex 1}).

In the statistical analysis, the open responses were distributed into categories according to the topics that the students referred to. The obtained data were transcribed and analyzed using the IBM Statistical Package for the Social Sciences (SPSS) software, version 21.0. Age (years) was expressed as means and standard deviation. Qualitative variables were expressed as frequency and percentage. Inferential analyses were performed with a significance level of α = 0.05 and, therefore, 95% of confidence. Age distribution (years) was assessed for normality using the Kolmogorov-Smirnov test.

The association between qualitative variables was assessed using Pearson’s Chi-square and Likelihood Ratio, followed by residual analysis when statistical significance was observed.

RESULTS

The questionnaires were answered by 438 medical students at UNESC, which is equivalent to a response rate of 70.41%, with this n being higher than the estimated sample calculation. Of these, 14 were excluded because they did not meet the inclusion criteria for the test at some point during the course; thus, 424 questionnaires were included in the study.

The demographic data show a predominantly female (60.4%), white (96.2%) population, with a mean age of 23 years. Of these, 33.25% (n = 141) were from the initial phases (1^st to 4^th), 33.25% (n = 141) from the intermediate phases (5^th to 8^th) and 33.49% (n = 142) from the final phases (9^th to 12^th). The adherence in the assessed sample to the last test performed at the university was 83.3% (Table 1).

Table 1
Demographic characteristics

	Mean ± Standard Deviation, n (%)
	n = 424
Age (years)	23.03 ± 3.49
Gender
Female	256 (60.4)
Male	168 (39.6)
Ethnicity
White	408 (96.2)
Brown / Mulatto	13 (3.1)
Yellow / Eastern origin	2 (0.5)
Black	1 (0.2)
Semester
1st Phase	42 (9.9)
2nd Phase	31 (7.3)
3rd Phase	26 (6.1)
4th Phase	42 (9.9)
5th Phase	36 (8.5)
6th Phase	31 (7.3)
7th Phase	47 (11.1)
8th Phase	27 (6.4)
9th Phase	33 (7.8)
10th Phase	34 (8.0)
11th Phase	33 (7.8)
12th Phase	42 (9.9)
Did you take the test on 10/3/18?
Yes	353 (83.3)
No	71 (16.7)

• Source: Research data, 2018.

Table 2 represents the students’ perceptions about the Progress Test related to the phase they are in. In the initial, as well as in the intermediate and final phases, the participants knew the objective of the Progress Test (p = 0.570). Of the seven evaluated questions, statistical significance was obtained in five of them, namely: Q2, Q3, Q4, Q5 and Q6 (p <0.05). The students who consider the Progress Test to be more important (Q2) are those who belong to the initial phases (p = 0.031). Again, students in the early phases are the ones who most often believe that the Progress Test can positively contribute to their professional career (Q3), while students in the final phases are among those who do not believe it (p <0.001). On the other hand, students in the final phases are the ones who feel the most prepared to take the test (Q4), whereas the opposite is observed in the initial and intermediate phases, where the majority of students do not feel prepared to do it (p <0.001). The students who most often read all the test questions to answer it (Q5) are students in the final years, while among those in the initial phases do not usually read all the test questions (p = 0.014). Regarding the access to the correct answers after taking the test (Q6), undergraduate students in the final phases are the ones that most often access it, while those in the initial phases most often do not access it (p = 0.009). In all phases of the sample, most students do not usually increase their dedication in areas where their performance was worse on the test (Q7) (p = 0.107).

Table 2
Students’ perception of the Progress Test related to the phases

		Phases, n (%)		p-value†
	Initial *	Intermediate**	Final***
Q1 (n=423)
Yes	129 (91.5)	126 (89.4)	131 (92.9)	0.570
No	12 (8.5)	15 (10.6)	10 (7.1)
Q2 (n=421)
Yes	131 (92.9)b	119 (85.0)	116 (82.9)	0.031
No	10 (7.1)	21 (15.0)	24 (17.1)
Q3 (n=422)
Yes	92 (65.7)b	53 (37.6)	55 (39.0)	<0.001
No	15 (10.7)	38 (26.9)	44 (31.2)b
Perhaps	33 (23.6)	50 (35.5)	42 (29.8)
Q4 (n=421)
Yes	18 (12.9)	49 (34.8)	115 (82.1)b	<0.001
No	122 (87.1)b	92 (65.2)b	25 (17.9)
Q5 (n=421)
Yes	90 (64.3)	101 (71.6)	112 (80.0)b	0.014
No	50 (35.7)b	40 (28.4)	28 (20.0)
Q6 (n=420)
Yes	60 (43.2)	73 (51.8)	86 (61.4)b	0.009
No	79 (56.8)b	68 (48.2)	54 (38.6)
Q7 (n=420)
Yes	56 (40.3)	44 (31.2)	60 (42.9)	0.107
No	83 (59.7)	97 (68.8)	80 (57.1)

• ^† Values obtained through Pearson’s Chi-square test. ^b Statistically significant value found after residual analysis. Q1: Do you know what the objective of the Progress Test is? Q2: Do you consider the Progress Test important? Q3: Do you believe that the Progress Test can somehow positively contribute to your professional career? Q4: Do you feel prepared to take the test? Q5: Do you usually read all the questions in the Progress Test to solve it? Q6: After taking the test, do you usually access the correct answers? Q7: Do you try to dedicate yourself more to areas where your performance was worse on the test? *Initial phases: 1^st to 4^th. **Intermediate phases: 5^th to 8^th. ***Final phases: 9^th to 12^th. Source: Research data, 2018.

Data were collected on the students’ satisfaction with their level of knowledge in each main area of medicine. All participants answered these questions (n = 424). Most students said they were satisfied with their knowledge in the areas of clinical medicine (50.6%), pediatrics (40.3%), gynecology and obstetrics (44.0%), collective health (36.2%) and basic sciences (46.7%). However, in clinical surgery, most participants (40.9%) mentioned being dissatisfied with their level of knowledge.

Another addressed question was which area the participants considered to be the one with their worst performance, and the total “n” for this question was 405, as some participants did not answer it. The most often mentioned area was clinical surgery (42.2%), followed by collective health (25.7%), gynecology and obstetrics (12.8%), pediatrics (9.1%), basic sciences (6.4%) and clinic medicine (3.7%).

Table 3 associates the phases with students’ satisfaction with their level of knowledge in each major area assessed in the Progress Test. In clinical medicine, it was observed that the students who most often considered themselves to be very dissatisfied, dissatisfied or indifferent with their level of knowledge were from the initial phases, while the students from the intermediate and final phases were the ones who declared they were satisfied with their level of learning in clinical medicine ( p <0.001). In clinical surgery, the students of the initial phases were the ones who most often answered they were very dissatisfied or indifferent with their performance, whereas the students of the intermediate phases were the ones most often dissatisfied with their level of knowledge; students of the final phases were among those who most often declared to be satisfied; however, even the majority of students of the final phases (40.7%) declared themselves to be dissatisfied (p <0.001). In the large area of pediatrics, the phases which the students are most often satisfied with their knowledge are the intermediate and final ones, while the students of the initial phases are the ones that most often declared to be very dissatisfied or indifferent about the subject (p <0.001). The same is true for gynecology and obstetrics, where the students of the initial phases were those that most often considered themselves to be very dissatisfied, dissatisfied and indifferent. While the intermediate and final phases were the ones that most often believed that their knowledge in the area were satisfactory, with a tendency to be very satisfied in the final phases (p <0.001). In relation to collective health, the most indifferent were the students in the initial phases, whereas those belonging to the intermediate phases were the most dissatisfied and the students in the final phases were most often those who claimed a level of great dissatisfaction, (p <0.001). In the initial, as well as in the intermediate and final phases, the participants were satisfied with their knowledge of basic sciences, with no statistically significant difference (p = 0.345).

Table 3
Satisfaction with the level of knowledge related to the phases

	Phases, n (%)			p-value
	Initial*	Intermediate**	Final***
Clinical Medicine (n = 421)
Very dissatisfied	12 (8.6)b	1 (0.7)	1 (0.7)	<0.001†
Dissatisfied	37 (26.4)b	31 (22.0)	17 (12.1)
Indifferent	59 (42.1)b	18 (12.8)	16 (11.4)
Satisfied	29 (20.7)	86 (61.0)b	98 (70.0)b
Very satisfied	3 (2.1)	5 (3.5)	8 (5.7)
Clinical Surgery (n=421)
Very dissatisfied	22 (15.7)b	10 (7.1)	9 (6.4)	<0.001†
Dissatisfied	43 (30.7)	72 (51.1)b	57 (40.7)
Indifferent	57 (40.7)b	23 (16.3)	21 (15.0)
Satisfied	17 (12.1)	31 (22.0)	47 (33.6)b
Very satisfied	1 (0.7)	5 (3.5)	6 (4.3)
Pediatrics (n=419)
Very dissatisfied	35 (25.2)b	2 (1.4)	0 (0.0)	<0.001†
Dissatisfied	26 (18.7)	31 (22.0)	17 (12.2)
Indifferent	69 (49.6)b	33 (23.4)	24 (17.3)
Satisfied	9 (6.5)	69 (48.9)b	91 (65.5)b
Very satisfied	0 (0.0)	6 (4.3)	7 (5.0)
Gynecology and Obstetrics (n=420)
Very dissatisfied	36 (25.9)b	1 (0.7)	0 (0.0)	<0.001†
Dissatisfied	27 (19.4)b	22 (15.6)	5 (3.6)
Indifferent	70 (50.4)b	21 (14.9)	14 (10.0)
Satisfied	5 (3.6)	83 (58.9)b	97 (69.3)b
Very satisfied	1 (0.7)	14 (9.9)	24 (17.1)b
Collective Health (n=420)
Very dissatisfied	7 (5.0)	25 (17.7)	27 (19.3)b	<0.001†
Dissatisfied	18 (12.9)	43 (30.5)b	35 (25.0)
Indifferent	48 (34.5)b	27 (19.1)	23 (16.4)
Satisfied	58 (41.7)	42 (29.8)	52 (37.1)
Very satisfied	8 (5.8)	4 (2.8)	3 (2.1)
Basic Sciences (n=420)
Very dissatisfied	6 (4.3)	8 (5.7)	7 (5.0)	0.345‡
Dissatisfied	16 (11.5)	28 (19.9)	29 (20.7)
Indifferent	45 (32.4)	30 (21.3)	31 (22.1)
Satisfied	63 (45.3)	67 (47.5)	66 (47.1)
Very satisfied	9 (6.5)	8 (5.7)	7 (5.0)

• ^†Values obtained by applying the Likelihood Ratio test. ^‡Values obtained by applying Pearson’s chi-square test. ^bStatistically significant value found after residual analysis. *Initial phases: 1^st to 4^th. **Intermediate phases: 5^th to 8^th. ***Final phases: 9^th to 12^th. Source: Research data, 2018.

When asked about which area they considered to be their worst performance (Table 4), students in the early phases predominantly believed it was clinical surgery and were also the main ones who considered the same about pediatrics and gynecology and obstetrics. Among the students of intermediate phases, the most often mentioned area was also clinical surgery. Regarding those who mentioned collective health as their worst performance, students of the intermediate and final phases were the main ones. The area most often chosen by students in the final phases was collective health. Those in the final phases were the ones that most often cited the basic sciences as their worst performance (p <0.001).

Table 4
Area of worst performance related to the phases

	Phases, n=405, n (%)			p-value
	Initial*	Intermediate**	Final***
Clinical Medicine	5 (3.9)	7 (5.0)	3 (2.2)	<0.001†
Clinical Surgery	54 (42.2)	66 (46.8)	51 (37.5)
Pediatrics	18 (14.1)b	11 (7.8)	8 (5.9)
Gynecology and Obstetrics	43 (33.6)b	6 (4.3)	3 (2.2)
Collective Health	7 (5.5)	45 (31.9)b	52 (38.2)b
Basic Sciences	1 (0.8)	6 (4.3)	19 (14.0)b

• ^†Value obtained by applying the Likelihood Ratio test. ^bStatistically significant value found after residual analysis. *Initial phases: 1^st to 4^th. **Intermediate phases: 5^th to 8^th. ***Final phases: 9^th to 12^th. Source: Research data, 2018.

There were two open questions in which each participant was asked to name three positive points of the test and three points in which the test needs improvement. However, not all respondents mentioned three items in each question, and thus 968 responses were obtained for positive points and 703 for points that need improvement. The frequencies with which the answers were mentioned are related to the total “n” of the sample in Table 5. When asked to highlight three positive points about the Progress Test, most participants provided answers that mentioned “evaluating the student’s evolution/performance”, mentioned 286 times, that is, 67.5% of the participants cited this item. The second most frequent positive point was “evaluating the institution’s teaching performance aiming at improvements”, mentioned by 178 (42.0%) of those assessed. A total of 153 (36.0%) students mentioned that the Progress Test “helps in the search for knowledge by identifying personal deficits”.

Table 5
Positive points and suggestions for improvements to the Progress Test

	n (%)*
Highlight three positive points of the Progress Test:	n = 424
Assess the student’s evolution/performance	286 (67.5)
Evaluate the institution’s teaching performance, pointing out improvements	178 (42.0)
Helps in the search for knowledge by identifying personal deficits	153 (36.0)
Train for tests in the future, such as for the residency tests	112 (26.4)
Self-evaluation	78 (18.4)
Compare results with other schools	74 (17.5)
Well-formulated questions	42 (9.9)
Evaluate the phases with each other	42 (9.9)
Others	3 (0.7)
Highlight three points that the Progress Test needs to improve:	n = 424
Decrease the number of questions so that the test is not so long	179 (42.2)
Improve the formulation of questions and / or topics	89 (21.0)
Specific tests according to the phases	69 (16.3)
Asking more objective questions	65 (15.3)
Encourage students by clarifying the purpose of the test	47 (11.0)
Have questions corrected in class with teachers	38 (9.0)
Questions more similar to those of exams / residency tests	36 (8.5)
Be effective for course changes	31 (7.3)
Generate some benefit for students or earn points in phases or residency	30 (7.1)
It should improve adherence to the test, or the test should be mandatory	28 (6.6)
Start at more advanced phases	28 (6.6)
The test should not take place during regular exams’ weeks	23 (5.4)
Commented feedback should be available faster and answers should be clearer	22 (5.2)
Allow the students to take the resolved questions with them	9 (2.1)
Others	9 (2.1)

• Source: Research data, 2018. *Percentage calculated in relation to the total “n” of the sample (n = 424).

When asked about possible improvements in the Progress Test, the main answers were “decrease the number of questions so that the test is not so long”, “improve the formulation of questions and / or subjects” and “specific tests according to the phases”, with such statements being mentioned by 179 (42.2%), 89 (21.0%) and 69 (16.3%) students, respectively.

DISCUSSION

Our study showed that UNESC medical students’ mean age is 23 years, with a predominance of females and Caucasians, which meets the demographic profile shown in other studies carried out with medical students in Santa Catarina⁸, and also in other states such as Goiás¹³, Paraíba¹⁴ and Rio Grande do Norte¹⁵.

As with all cross-sectional studies, ours also has some limitations. The fact that there is no validated questionnaire for the investigation that was performed, implying the need to use one’s own instrument and the non-inclusion of all students, can be considered important limitations, since the study included 438 (70.41%) of those enrolled in the second semester of 2018, and of these, 424 were included, totaling 68.16% of those enrolled. The application of questionnaires opens the possibility for a series of missing data, which happened in parts of our study.

Regarding the knowledge related to the PT objective, most students answered affirmatively, which can be attributed to a wide dissemination and information about the test objectives by the course. Similarly, the students consider the PT an important type of assessment. Similar data were found in the study by Ali et al. (2018), performed with undergraduate students in medicine, dentistry and dental therapy, in which they invited participants to anonymously answer an online questionnaire to share their perceptions and experiences about progress tests, and among the students who answered, it was observed they also considered the PT as a useful form of assessment to improve their knowledge¹⁶.

This study also identified that, in the final phases of the course, students tend to feel more confident when taking the test, and they are also more interested in searching for the correct answers after the end of it. Similar data were found in the study by Nascimento et al. (2018). In this study they included 152 students from the 2^nd to the 12^th period of a medical course at a philanthropic university in the city of Goiânia (GO). The 3^rd and 5^th-year students, in comparison with the 2^nd-year students, were the ones that most often considered the PT an important way to guarantee knowledge through memorization, and consequently appropriation of knowledge¹³. There was also a demand by these students that there should be discussions with the teachers about the test and feedback, after they had taken it. This information was anticipated, since during the course it is expected that new knowledge will be acquired and therefore students will feel more prepared. Likewise, their interest in questions and feedback also increases, since they should have already mastered this knowledge. In the early phases, students are often less interested, because they have not yet had contact with the required content, even though they are the ones who most often consider the PT to be important and most of them declare they know its goals.

These data show that the difficulty in adequately answering the PT is not related to the lack of knowledge about the objectives and importance of the test, but it may be related to the lack of an objective contribution to the curriculum, either through the extra grades in the semiannual curriculum evaluations or scores in medical residency tests and examinations. In the final phases, students usually read the test and seek the correct answers, although they do not consider that the test can contribute to their professional career so much, due to greater concern and proximity to the medical residency tests and the fact they will applying to positions at the end of graduation; thus, the PT acts as a type of preparation for such assessments.

As in the initial phases, it was observed that the big problem is the lack of interest in taking the test, and some measures could be taken, such as starting taking the test at more advanced phases, when the student already has a greater mastery of the knowledge to analyze the questions, and validate the test as a score for the medical residency entrance examinations, thus increasing the importance that the test has in the student’s curriculum.

In the initial, as well as in the intermediate and final phases, a degree of dissatisfaction was found regarding the level of knowledge in the area of clinical surgery among the students in the sample. Similar information was found in a cross-sectional study, carried out by Pinheiro et al. (2015), which analyzed the performance of medical students in PT, with one test being applied in 2008 and another, unchanged, in 2011, disclosing that little or no progression of knowledge was observed in the area of ​clinical surgery¹⁰. In the areas of clinical medicine, pediatrics and obstetrics and gynecology, students in the intermediate and final phases considered themselves satisfied with their level of knowledge, since according to the curricular matrix n.05¹⁷ of the assessed university, pediatrics and obstetrics start being studied in the fifth phase, gynecology in the sixth phase and clinical medicine is mainly addressed during the intermediate phases. The areas of clinical surgery and public health were the most often mentioned as the areas where the students had their worst performance in the test, whereas clinical surgery was the main one in the initial and intermediate phases, and the collective health area in the final phases. These data are consistent with data found in the study by Rosa et al. (2017)⁸, also carried out at UNESC, in which the Progress Test applied at the university in 2011, showed that the overall mean score of the School of Medicine of Universidade do Extremo Sul Catarinense was lower than that expected in most phases of the course, when compared to the overall mean score of the schools that belonged to the Núcleo de Apoio Pedagógico Interinstitucional Sul II in the area of collective health, which was previously classified as public health institution. Even though students were included in the basic health network since the first phase, this showed to be insufficient. Therefore, new activities such as lectures, seminars and lectures were implemented at the university⁸.

The research observed that the positive points most often mentioned by the students were the possibility of evaluating the evolution/performance of students throughout the course, as well as evaluating the institution’s teaching performance, which can help with improvements. Such statements are consistent with the objectives of the Progress Test found in the studies by Van Der Vleuten et al. (1996) and Ito and Janowski (2003), who mention among the test’s advantages the ability to provide individual performance information to the students, and also evaluate the university’s curriculum and compare it with other schools. The third most frequently mentioned positive point was the help in the search for knowledge that the test provides, since it stimulates long-term knowledge retention^3),(18).

When asked about aspects in which the Progress Test needs improvement, the most frequently mentioned point was the test length, which is in line with what was stated in a study, which declared a minimum value of three minutes per question, so that all students are able to read and try to solve the questions in a timely manner¹⁹. A longer interval is required per question, as the test is applied to the entire student body, which has different levels of knowledge about the topics and, therefore, some groups need more time to understand and solve the test. Students also frequently mentioned that it is necessary to improve the formulation of questions, and that they should be more objective, which is in line with Teixeira’s recommendations²⁰ for the creation of multiple-choice questions, which says that the questions should be as short as possible, with a statement containing all pertinent information and with alternatives that are easy to understand and with one only correct option.

The third point that was most frequently mentioned by the students was that the tests should be specific according to the phases, which goes against the main objective of the test of being a way to assess the students’ evolution throughout the course, by making it just one more test included in the curriculum. Moreover, it would miss the possibility of comparison with other medical schools that have different curricula^{21), (22}. However, for students in the early stages, who have a lower level of knowledge, the tests become difficult, which causes them to lose reliability. With this in mind, an alternative that is being researched for medical areas is the use of Computerized Adaptive Tests (CAT), which create personalized and adaptive tests using algorithms, based on the answers given by students, thus adapting it to the different levels of knowledge²³. This type of evaluation has shown more homogeneous reliability estimates during the different phases. In many cases it can reduce the number of questions, reducing the duration of the test, and this is particularly advantageous for first-year students, a period known to be less reliable in traditional on-paper tests. CATs would optimize evaluations according to the student’s level of knowledge, thus assisting in the main complaint of students, which is the extended time of traditional Progress Tests^{23), (24}.

Our research is one of the few on this subject and needs to be refined over time to allow a more qualified understanding of the students’ perception of the Progress Test; to do so, new studies are required in other medical schools to compare and solidify the obtained results, and also to establish new forms of assessment and compare them to the current ones.

REFERENCES

• ¹
  Bierer SB, Dannefer EF, Taylor C, Hall P, Hull AL. Methods to assess students’ acquisition, application and integration of basic science knowledge in an innovative competency-based curriculum. Med. teach. 2008;30(7):e171-7.
• ²
  Blake JM, Norman GR, Keane DR, Mueller CB, Cunnington J, Didyk N. Introducing progress testing in McMaster University’s problem-based medical curriculum: psychometric properties and effect on learning. Acad. med. 1996;71(9):1002-7.
• ³
  Van Der Vleuten CPM, Verwijnen GM, Wijnen WHFW. Fifteen years of experience with progress testing in a problem-based learning curriculum. Med. teac. 1996;18(2):103-9.
• ⁴
  Pugh D, Regehr G. Taking the sting out of assessment: is there a role for progress testing? Med. educ. 2016;50:721-9.
• ⁵
  Muijtjens AM, Timmermans I, Donkers J, Peperkamp R, Medema H, Cohen-Schotanus J, Van der Vleuten CP. Flexible electronic feedback using the virtues of progress testing. Med. teach . 2010;32:491-5.
• ⁶
  Van der Vleuten CP, Schuwirth LW, Muijtjens AM, Thoben AJ, Cohen-Schotanus J, van Boven CP. Cross institutional collaboration in assessment: a case on progress testing. Med. teach . 2004;26:719-25.
• ⁷
  Van der Vleuten CP, Heeneman S. On the issue of costs in programmatic assessment. Perspect Med Educ 2016;5:303-7.
• ⁸
  Rosa MI da, Isoppo CC, Cattaneo HD, Madeira K, Adami F, Ferreira FOF. O Teste de Progresso como indicador para melhorias em curso de graduação em Medicina. Rev. bras. educ. med. 2017;41(1):58-68.
• ⁹
  Callegari-Jacques S. Bioestatística: princípios e aplicações. Porto Alegre: Artmed; 2004.
• ¹⁰
  Barbetta PA. Estatística aplicada às ciências sociais. 7a ed. Florianópolis: UFSC; 2007.
• ¹¹
  Sakai MH, Filho OFF, Almeida MJ, Mashima DA, Marchese MC. Teste de Progresso e avaliação do curso: dez anos de experiência da medicina da Universidade Estadual de Londrina. Rev. bras. educ. med . 2008;32(2):254-63.
• ¹²
  Pinheiro OL, Spadella MA, Moreira HM, Ribeiro ZMT, Guimarães APC, Almeida Filho OM, et al. Progress Test: an evaluative tool for academic management. Rev. bras. educ. med . 2015;39(1):68-78.
• ¹³
  Nascimento HCF, Ferreira Júnior WA, Silva AMTC, Carvalho IGM, Bastos GCFC, Almeida RJ. Análise dos níveis de empatia de estudantes de Medicina. Rev. bras. educ. med . 2018; 42(1):150-8.
• ¹⁴
  Costa GPO, Herculano TB, Gama ALH, Cabral RP, Campos DB, Oliveira DVS. Enfrentamentos do estudante na iniciação da Semiologia Médica. Rev. bras. educ. med . 2018; 42(2):79-88.
• ¹⁵
  Cardoso Filho FAB, Magalhães JF, Silva KML, Pereira ISSD. Perfil do estudante de Medicina da Universidade do Estado do Rio Grande do Norte (UERN). Rev. bras. educ. med . 2015;39(1):32-40.
• ¹⁶
  Ali K, Cockerill J, Zahra D, Tredwin C, Ferguson C. Impact of Progress Testing on the learning experiences of students in medicine, dentistry and dental therapy. BMC med. educ. 2018;18(1):218-53.
• ¹⁷
  Colegiado Unasau. Resolução nº 01/2015/Unasau. Criciúma: Unidade Acadêmica de Ciências da Saúde-UNESC; 2015 [acesso 20 maio 2019]. Disponível em: Disponível em: http://www.unesc.net/portal/capa/index/20/8842
  » http://www.unesc.net/portal/capa/index/20/8842
• ¹⁸
  Ito K, Janowski E. Resultados dos Testes de Progresso da medicina realizados no período de 1998 a 2002. Londrina: Colegiado de Medicina da Universidade Estadual de Londrina; 2003.
• ¹⁹
  Morgado F, Barbosa NM, Mota EFRO. Criando, validando e aplicando testes de progresso nos cursos de engenharia do Unifeso. Anais do 40° Congresso Brasileiro de Educação em Engenharia; 2012 set 3-6; Belém, Brasil; 2012.
• ²⁰
  Teixeira G. Avaliação da aprendizagem. Practical Suggestions for Writing Multiple-Choice Questions. 2012 [acesso 20 maio 2019]. Disponível em: Disponível em: https://www.academia.edu/5716219/WRITING_MULTIPLE
  » https://www.academia.edu/5716219/WRITING_MULTIPLE
• ²¹
  Morgan HL, Petry AF, Licks PAK, Ballester AO, Teixeira KN, Dumith SC. Consumo de estimulantes cerebrais por estudantes de Medicina de uma universidade do Extremo Sul do Brasil: prevalência, motivação e efeitos percebidos. Rev. bras. educ. med . 2017;41(1):102-9.
• ²²
  Matsuyama Y, Muijtjens AMM, Kikukawa M, Stalmeijer R, Murakami R, Ishikawa S, et al. A first report of East Asian students’ perception of progress testing: a focus group study. BMC med. educ . 2016;16(1):216-45.
• ²³
  Collares CF, Fernandes DC. When I say ... computerised adaptive testing. Med. educ. 2019;53(2):115-6.
• ²⁴
  Vleuten C, Freeman A, Collares CF. Progress test utopia. Perspect Med Educ 2018; 7(2):136-8.

ATTACHMENT 1

This is a questionnaire for academic purposes, which will be used as a research instrument for the Medical Course / UNESC course Term Paper by medical students Laís Büttner Sartor and Luana Lanzarini da Rosa, supervised by Prof. Dr. Kristian Madeira. All data collected here will be used confidentially and your sincere responses are very important for the quality of our work.

PROFILE

A1. How old are you? _______ years

A2. What is your gender?

( )Male ( )Female

A3. How do you self-declare your ethnicity?

( )White

( )Black

( )Brown/Mulatto

( )Yellow/Eastern origin

( )Native/Indigenous origin

( )Other:_________________________________________________

A4. What phase are you currently studying? ___ phase

A5. Have you ever taken a Progress Test?

( )Yes ( )No

A6. Did you take the Progress Test that too place on 10/03/2018?

( )Yes ( )No

If not, why? ________________________________________________________.

PERCEPTIONS

B1. Do you know the purpose of the Progress Test?

( )Yes ( )No

If yes, what is it?________________________ ______________________________________.

B2. Do you consider the Progress Test important?

( )Yes ( )No

Why? ________________________________________________________________

B3. Do you believe that the Progress Test can somehow positively contribute to your professional career?

( )Yes ( )No. ( )Perhaps

Why?__ _____________________________________________________________________

B4. Do you feel prepared to take the test?

( )Yes ( )No

Why? _________________________________________________________________________

B5. Do you usually read all the questions in the Progress Test to solve it?

( )Yes ( )No

Why? __________________________________________________________________

B6. After taking the test, do you usually access the correct answers?

( )Yes ( )No.

Why? ________________________________________________________________________

B7. Do you intend to dedicate yourself more to the areas in the test where your performance was worse?

( )Yes ( )No

Why?_____________________________________________________________________

SATISFACTION EVALUATION

In the next 6 questions, mark the alternative that represents your satisfaction with the level of knowledge you have in each of the following areas:

C1. Clinical Medicine:

( )Very Dissatisfied ( )Dissatisfied ( )Indifferent ( )Satisfied ( )Very satisfied

C2. Clinical Surgery:

( )Very Dissatisfied ( )Dissatisfied ( )Indifferent ( )Satisfied ( )Very satisfied

C3. Pediatrics:

( )Very Dissatisfied ( )Dissatisfied ( )Indifferent ( )Satisfied ( )Very satisfied

C4. Gynecology and Obstetrics:

( )Very Dissatisfied ( )Dissatisfied ( )Indifferent ( )Satisfied ( )Very satisfied

C5. Collective Health:

( )Very Dissatisfied ( )Dissatisfied ( )Indifferent ( )Satisfied ( )Very satisfied

C6. Basic Sciences:

( )Very Dissatisfied ( )Dissatisfied ( )Indifferent ( )Satisfied ( )Very satisfied

C7. Which area do you think was that of your worst performance?

( ) Clinical Medicine ( ) Clinical Surgery ( )Pediatrics ( )Gynecology and Obstetrics ( )Collective Health ( )Basic Sciences

C8. Highlight three positive points of the Progress Test:

1 _____________________________________________________________________________

2 ____________________________________________________________________________

3 __________________________________________________________________________

C9. Highlight three points that the Progress Test needs improvement:

1 ___________________________________________________________________________

2 ___________________________________________________________________________

3 ____________________________________________________________________________

Thank you for your participation.

Publication Dates

• Publication in this collection
  08 May 2020
• Date of issue
  2020

History

• Received
  20 Jan 2020
• Accepted
  26 Feb 2020