Factors associated with student performance on the medical residency test

SUMMARY OBJECTIVE: To determine whether the scores of the Progress test, the Skills and Attitude test, and the medical internship are correlated with the medical residency exam performance of students who started medical school at the Federal University of São Paulo in 2009 METHODS: The scores of 684 Progress tests from years 1-6 of medical school, 111 Skills and Attitude exams (5 th year), 228 performance coefficients for the 5 th and 6 th years of internship, and 211 scores on the medical residency exam were analyzed longitudinally. Correlations between scores were assessed by Pearson’s correlation. Factors associated with medical residency scores were analyzed by linear regression. RESULTS: Scores of Progress tests from years 1-6 and the Skills and Attitude test showed at least one moderate and significant correlation with each other. The theoretical exam and final exam scores in the medical residency had a moderate correlation with performance in the internship. The score of the theoretical medical residency exam was associated with performance in internship year 6 ( β =0.833; p<0.001), and the final medical residency exam score was associated with the Skills and Attitude score ( β =0.587; p<0.001), 5 th -year internship score, ( β =0.060; p=0.025), and 6 th -year Progress test score ( β =0.038; p=0.061). CONCLUSIONS: The scores of these tests showed significant correlations. The medical residency exam scores were positively associated with the student’s performance in the internship and on the Skills test, with a tendency for the final medical residency exam score to be associated with the 6 th -year Progress test. tests


INTRODUCTION
The form and content of medical student evaluations of should cover the acquired knowledge as well as specific skills and elements of their affective nature, such as attitudes towards professional practice. 1 The "Miller pyramid" is a model to define the learning outcomes in terms of the skills students should acquire, and it provides the cognitive bases ("knows" and "knows how") of professional practice ("does") and the need to evaluate practical skills and competencies ("shows how"). 2 "Knows" refers to methods that evaluate knowledge, which can be recovered from memory, and supports the construction of more complex capabilities. Both "knows" and "knows how" belong to the cognitive domain and therefore should be evaluated by appropriate methods for measuring knowledge Avaliação em Educação Médica (Interinstitutional Center for Studies and Practices of Medical Training Evaluation).
The voluntary SA consisted of 10 stations with different clinical tasks, such as obtaining a clinical history, performing the clinical examination, evaluating a radiograph or electrocardiographic tracing, and giving the diagnosis and/or instructing the patient. This test was designed and administered similarly to the Objective Structured Clinical Examination (OSCE).
The PC is an index that measures academic performance at the end of each academic period. It is calculated based on the final grade and workload in each curricular unit.
The MR test was performed in two phases. The first phase had a theoretical test with 100 assertive questions with short answers and a computerized test with 50 questions with images. The second phase was in the form of four practical stations (12 boxes). The final result of the MR test was the sum of the theoretical test score (weight 5), the practical test score (weight 4), and the interview score (weight 1).
The scores of students who did not complete the PT in all years of the medical course were excluded.
Reliability analysis of the collected data was performed, the correlations between the scores of all tests were calculated, and the factors associated with the scores of the theoretical test and the final result of the MR test were identified.

Statistical analysis
The numerical variables are expressed as the mean and standard deviation and were compared by the paired t-test. Data reliability was assessed by Cronbach's α coefficient, 3 which was considered adequate when > 0.7. 4 For the correlation analysis, Pearson's correlation coefficient was calculated, where 0.1 to 0.3 was considered a weak correlation, 0.3 to 0.6 was considered a moderate correlation, and 0.7 to 1.0 was considered a strong correlation. 5 Two linear regression models were constructed. The dependent variable of one of the models was the theoretical test score, while the dependent variable of the other model was the final result of the MR test. In the univariate linear regressions, the scores of the various tests were regressed on, and those with p < 0.2 were included in the multiple linear regression model.
Statistical analyses were performed in SPSS ® software (IBM SPSS Statistics, Somers, NY, USA), with a significance level of p < 0.05. acquisition. However, they differ in the nature of the knowledge they refer to: "Knows" is more related to the theoretical domain, whereas "knows how" is an applied type of knowledge. Thus, the tests proposed for this stratum should target the use of knowledge for decision-making and problem-solving within a clinical context. The "shows how" refers to the evaluation of clinical skills and competencies that are performed in the context of training. This skill set is usually evaluated through practical exams involving clinical tasks. The evaluation of "does" corresponds to the practice in the work environment. It is tested in the student at the end of the course, in the professional training stages, where training targets the practice of "does", with the student exercising the clinical practice under supervision.
Among the various forms of applied evaluations, the Paulista School of Medicine of the Federal University of São Paulo (EPM-UNIFESP) has evaluated the performance of students using the Progress test (PT), the Skills and Attitude test (SA), and the performance coefficient (PC) during the medical internship.
In this context, the objective of this study was to determine whether the PT scores from the first to the sixth year of medical school, the SA score, and the PC during the medical internship were correlated with the medical residency (MR) scores of the students who started medical school at the Paulista School of Medicine -Federal University of São Paulo (EPM-UNIFESP) in 2009.

METHODS
This was a retrospective study that employed a longitudinal analysis of the PT scores, SA score, and PC during the 5 th and 6 th medical school years (PC 5 th and PC 6 th ) and the exam of the MR. This study was approved by the Research Ethics Committee of UNI-FESP. The data included the scores on the PT from the 1 st to the 6 th medical school years, the score on the SA performed in the 5 th year, the PC in the 5 th and 6 th years, and the score on the theoretical test and the final test result of the 2015 MR among the students who in 2009 were enrolled in the 1 st year of medical school at EPM-UNIFESP.
The PT, which was voluntarily taken by the students, consisted of 120 multiple-choice questions about medicine and was written and administered together with the institutions that are part of the Núcleo Interinstitucional de Estudos e Práticas de

RESULTS
In 2009, 123 students enrolled in the 1 st year of the medical school at EPM-UNIFESP. Of these, 114 (92.7%) performed all PT tests from 2009 to 2014, and 111 (90.2%) performed the SA in the 5 th year. Of the 114 students included in the study, 106 (93.0%) completed the theoretical MR test, and 105 (92.1%) completed all stages of the MR tests at EPM-UNIFESP ( Figure 1).
The data reliability analysis showed internal consistency between the scores of all tests analyzed (0.730, 95% CI: 0.646 to 0.802 (p <0.001)). The data reliability had an α of 0.749 (95% CI: 0.669 to 0.815 (p < 0.001)) if the scores of the 1 st -year PT or the SA score were deleted. This difference was small, so the scores of all tests were included in the analysis.
Some students received a score of 0 on the PT, to wit, four students in the 1 st school year, one student in the 2 nd year, one student in the 3 rd year, two students in the 4 th year, six students in the 5 th year, and five students in the 6 th year. The mean ± SD of PT scores (n = 114 for each year) were as follows: 1 st year (2.67 ± 0.97), 2 nd year (3.01 ± 0.70), 3 rd year (4.19 ± 1.13), 4 th year (4.01 ± 1.05), 5 th year (5.19 ± 1.56), and 6 th year (6.38 ± 1.80) ( Figure 2). There was a significant increase (p <0.05) between the PT scores of consecutive years, except from the 3 rd to the 4 th school year. Between the 1 st and 6 th school years, there was a considerable increase in the PT score ( Figure 3). The mean of the 111 SA scores was 7.53 ± 0.56. The scores of PC 5 th (n = 114) were 8.32 ± 0.29 and PC 6 th (n = 114) were 8.26 ± 0.28 (p <0.001) ( Figure 2).
Of the 114 students included in the study, 106     (Figure 2).
The correlations between the scores obtained are shown in Table 1. The factors associated with the MR score were tested through two linear regression models. The first model considered the score of the theoretical MR test as the dependent variable, while the other model considered the final result of the MR test as the dependent variable (Tables 2 and 3). The final model of multiple linear regression, which was included the PT scores from the 4 th to the 6 th school year, the SA score, and the performance in the medical internship because they met the cutoff in the univariate linear regression model ( Table 2), showed that each additional point in PC 6 th increased the theoretical test score by 0.833 points (Table 3). In the second multivariate regression model, which included the same variables mentioned above (Table 2), each additional point in PC 5 th increased the final MR score by 0.587, and each additional point on the SA increased the final MR score by 0.06. The PT of the 6 th school year showed an association trend without statistical significance (Table 3).

DISCUSSION
In the present study, the PT scores showed a progressive increase over time, except between the 3 rd and 4 th years. Between the 1 st and the 6 th years, there was a substantial increase of 3.71 points on the PT, which was an increase of 139%. This highlights the degree of knowledge acquisition in the medical school period. Similarly, a study conducted in the EPM from 1996 to 2001 showed a similar increase of 2.79 points to 3.90 points in the PT score from the 1 st to the 6 th school year. 6 At the University of Londrina, there was an average increase of 2.89 points in the PT from the 1 st to the 6 th school year from 1998 to 2006. 7 In general, the PT scores obtained in the present study were similar to those of other institutions. The national 2015 PT conducted in 23,065 students at 57 medical schools and showed pass rates of 32.38%, 35.23%, 39.71%, 44.85%, 51.98%, and 61.28% from the 1 st to the 6 th school year, respectively, which were higher than those at UNIFESP in the first two school years and lower than that at UNIFESP in the 6 th school year. 8 At the State University of Campinas (UNICAMP), the PT grades from 2011 to 2014 among the 6 th -year students had a mean of 6.7 ± 0.7, 6.1 ± 1.0, 6.6 ± 0.7, and 7.1 ± 0.7, respectively, which were slightly lower than the means of the present study. 9 A study at the University of Missouri, USA, reported lower percentages of correct answers on the PT than those of the present study for the 1 st to the 6 th school years, with percentages of 6.1%, 16.1%, 30.7%, 41.6%, 50.9%, and 56.0%, respectively. 10 Blake et al 11 reported the PT scores of three classes of students at McMaster University, which were 10-20% at the beginning of the course but increased almost linearly until reaching 50% on the 5 th exam 20 months later. The scores of the theoretical MR test in this study were similar to those observed at UNICAMP, which were 6.8 ± 0.8, 7.6 ± 0.9, 7.3 ± 0.8, and 7.5 ± 0.9 from 2011 to 2014, respectively. 9 In this study, there was a weak correlation between the PT scores of the 1 st and 2 nd school years. From the 2 nd school year, the most relevant correlations occurred between the closest years. There was a strong correlation between the 3 rd and 4 th school years and between the 5 th and 6 th school years, which suggests greater consistency in the knowledge acquired in the clinical area. However, the strength of correlations decreased for the MR tests, which might show the insufficient preparedness for that test at that time.
Ferreira 9 compared the PT score during the 6 th school year with the score on the MR theoretical examination of UNICAMP students and observed moderate correlations of 0.588 in 2011, 0.610 in 2012, 0.671 in 2013, and 0.476 in 2014, in contrast to the present study, which showed a significant but weak correlation between the 6 th school year PT score and the final MR score.
The analysis of internal consistency showed that the scores of PT, SA, PC 5 th , PC 6 th , and MR together showed an α coefficient greater than 0.7. Another study showed that PT scores in medical school had a predictive validity of 0.6 for success on the medical licensing exam. 11 In the present study, the removal of the 1 st -year PT score or the SA score from the analysis increased the internal consistency of the study. This result suggests that the PT has a large number of questions considered "difficult" for the 1 st -year students and therefore has less discriminatory power for student performance. Regarding the SA test, a possible explanation would be the diverse nature of this test in comparison to PT test, in addition to the small number of questions, since studies have highlighted the importance of the number of questions on this test. 12 Other studies have found stronger predictive relationships between the clinical performance of the student and OSCE when the test had 18 5-minute stations 13 or 35 2-minute stations, 14 which suggests that longer OSCE assessments may be better predictors of performance. A systematic review reinforced this hypothesis, finding that the best reliability was associated with a greater number of stations. 15 In another systematic review, 12 of the 15 reviewed articles that reported the best relationships between OSCE scores and clinical performance were precisely those with the most questions. 16 In the literature, the importance of OSCE in health education programs is well established. It is an evaluation mode specifically designed to provide a valid and reliable measure of the clinical competence of students in a simulated environment. 17 Reinforcing this idea, the final linear regression model obtained in the present study showed the importance of the SA score to the final result of the MR test. Each additional point on the SA increased the final MR test score by 0.060 points. However, the variable most strongly associated with performance on the final exam of the MR was PC 5 th , each point of which increased the score on the final MR exam by 0.58 points. Likewise, PC 6 th was significantly associated with the score of the theoretical MR exam, each point of which increased the theoretical MR exam score by 0.83 points. A study at São Paulo State University from 2009 to 2011 showed a moderate correlation between PT 6 th and the score on the first phase of the multiple-choice MR test, but this correlation did not persist in the second phase of the test, which consisted of the OSCE and an interview. 18 Another study 19 showed an association between good performance on the medical internship exam and student performance in clinical practice (0.8 points), in the theoretical course of clinical medicine (0.5 points), and in basic science (0.4 points). Similarly, other authors found that performance on previous tests influenced future results. 20. 21 The results of the present study suggest that students should be encouraged to take the PT and SA seriously to improve their performance year by year. In addition, greater participation in practical activities, especially in medical internships, can improve the final result on the MR test.
The strength of this study was the longitudinal analysis of the scores of the tests taken during the medical course and the statistical treatment applied, which showed results that would be expected in practice but were reinforced by the relevant statistical analysis. The limitations of the study were the inclusion of student grades in only a certain period, and in the linear regression, external factors were not considered, such as having taken preparatory courses, which could interfere in the performance on the MR test.

CONCLUSION
The data reliability obtained in this study was adequate and showed a significant correlation between the scores of the tests analyzed. The performance on the SA and in the internship was positively associated with the performance on the MR tests.
Author's Contribution Maria Cristina de Andrade: Study design, data collection and analysis, manuscript writing, critical review, and final approval of the manuscript. Maria Wany Louzada Strufaldi; Rimarcs Gomes Ferreira; Gilmar Fernandes do Prado; Rosana Fiorini Puccini: Data collection and analysis, critical review, and final approval of the manuscript.
Amélia Miyashiro Nunes dos Santos: Study design, statistical analysis of data, writing of the manuscript, critical review, and final approval of the manuscript.