The effect of formative assessment in Brazilian university physics courses

Efeito da avaliação formativa em cursos de física em universidades brasileiras

DESENVOLVIMENTO EM ENSINO DE FÍSICA

The effect of formative assessment in Brazilian university physics courses

Efeito da avaliação formativa em cursos de física em universidades brasileiras

Émerson Cruz^I; Hélio Dias^II; Gerd KortemeyerIII, ¹

^IDepartment of Physics and Astronomy, Michigan State University, East Lansing, MI, USA

^IIInstituto de Física, Universidade de São Paulo, São Paulo, SP, Brasil

^IIILyman Briggs College and Department of Physics and Astronomy, Michigan State University, East Lansing, MI, USA

ABSTRACT

Most postsecondary physics courses in Brazil offer no meaningful formative assessment opportunities. We implemented online homework with immediate feedback in two courses, one with traditional learners at a public university, and one with non-traditional learners at a private university. In addition, at the public university, clickers were used in lecture. While surveys showed broad acceptance of these techniques by the students and the belief that they helped in learning, grades did not significantly improve — instead, we observed a narrowing of the grade distribution toward mid-range grades at the public university, and no difference at the private university. Our study also identifies a number of logistical and organizational hurdles that need to be overcome before a hopefully more successful implementation of these techniques should be attempted.

Keywords: online assessment, formative assessment.

RESUMO

A maioria dos cursos de física de nivel superior no Brasil não oferecem significativas oportunidades de avaliação formativa. Neste trabalho, implementamos atividades de tarefas online com feedback imediato em dois cursos: um com alunos tradicionais em uma universidade publica, e outra com alunos não-tradicionais em uma universidade privada. Implementamos, também, na universidade publica, atividades com os denominados "clickers" em sala de aula. Posterior pesquisa de opinião com os alunos mostrou ampla aceitação destas técnicas e avaliação positiva de que tais técnicas ajudaram na aprendizagem do conteúdo. Todavia as notas não apresentaram melhora significativa. Na universidade publica observou-se um estreitamento na distribuição de notas em torno dos valores medianos, enquanto na universidade privada não foi observada diferença significativa na distribuição de notas. Nosso estudo também identificou uma série de dificuldades, tanto logísticas como organizacionais, que precisam ser superadas antes de uma implementação mais eficaz das técnicas exploradas nesse trabalho.

Palavras-chave: avaliação online, avaliação formativa.

1. Introduction

Formative assessment is an essential part of how people learn [1], as it enables learners to assess their own progress through the material and practice relevant techniques. The most efficient methods provide immediate feedback and allow learners to interact with the materials, peers, and instructors before possibly reattempting a question or problem [1]. Formative assessment not only informs the learner, but also the instructor [2-4].

To provide meaningful formative assessment in spite of frequently large enrollment and limited personnel, many of the techniques rely on technology. Examples of formative assessment within the classroom include peer-instruction [5,6] and Classroom Formative Assessment [7]. while outside the classroom, computer-based homework has become increasingly popular, due to the net positive effect on student learning that was confirmed by a number of studies [8-10]. The range of the effect size, though, is considerable [11], and compared to written homework, it can be detrimental to certain aspects of desirable problem solving strategies [12].

While these techniques and technologies are well-researched within the USA, little research exists on their applicability and effect in other countries. In this study, we are focusing on Brazil. Brazil's university system is closer to the European tradition, which at the postsecondary level favors specialization rather than the American ideal of a "well-rounded" education. Much of the responsibility for learning the materials is put on the learners, and little guidance is given by the instructors. Traditionally, courses do not follow a particular textbook, and most of the grade depends on summative evaluations, such as written or oral exams. Efforts in the exploration of technology-mediated formative assessment have been mostly exploratory (e.g., Otsuka and Vieira da Rocha [13]).

In this study, we are attempting to use online homework and clickers in Brazilian physics courses for engineers in order to study which if any results from research on technology-mediated formative assessment are transferable. Our study took place in two courses, one in a public university with traditional learners, and one in a private university with non-traditional learners. Results from the public university course were compared to results from similar courses at Michigan State University.

2. Courses

2.1. University of São Paulo

We introduced technology-mediated formative assessment in an introductory calculus-based physics course for engineers at the University of São Paulo (USP) in Fall 2009. USP is one of the largest public universities in Brazil and has one of the leading physics departments in the country. This large enrollment course (total enrollment about 1000 students) is taught in 12 sections of about 80 students each by a number of instructors. Traditionally, no graded homework is offered, and the course is taught in pure lecture "transmission format" with no quizzes or other feedback to the learners, as there is insufficient grading personnel to offer formative assessment. The curriculum for the course is predetermined, and the instructors are bound to the same timing and grading criteria across all sections.

The course grade is determined by a series of three midterm exams and a possible remedial exam. The two-hour exams are free-form with numerical and symbolic-manipulation problems (usually "broken-intoparts" and "complex and/or multi-step" [14]), calculators are not allowed. There are usually no qualitative or "real world" problems on the exams. The exams are hand-graded on a scale from 0 (worst) to 10 (best), and students fail the class with an averaged grade lower than 5.

We deployed both online homework and clickers in two of the 12 sections of this course. These two sections had 157 students. Personalized data was collected based on usage of the online homework system (LON-CAPA [15]).

The clickers (iClicker^TM) were on loan from the company, and there were only enough clickers for one section at a time. Thus, clickers were distributed to the students at the beginning of lecture sessions and collected again at the end, but not registered to a particular student. As a result, we have no personalized data on clicker usage. After some negotiations and with assistance from a senior faculty member, we were able

to secure a room with the required infrastructure of video projectors and screens. As there are no required textbooks for Brazilian physics courses, there is also no mechanism for requiring students to buy clickers, and no distribution infrastructure similar to college textbook stores in the USA. Clickers would need to be purchased by the university, which for class sizes like ours presents a considerable expense.

The original plan of carefully introducing both technologies with both authors in the classroom failed, as the start of the Fall 2009 semester at USP was "indefinitely postponed" due to swine-flu concerns. The semester eventually was off with a few weeks delay, and some of the examinations rolled over into the Spring semester.

We also administered a post-survey asking students about their usage of LON-CAPA and iClicker^TM, but the delayed end of the semester did not allow for us to administer this survey in class. Thus, we used an online survey, which ended up having only 86 responses (a little more than half of the student population in the two sections).

Exam data was available from sections taught traditionally by the same instructor (one of the authors, E.C.) in previous years, and from other sections of the same course taught traditionally in the same year. Exams were designed by a committee without input from the instructor teaching the two experimental sections, and the same exams were given across all sections in a given year.

Due to the rigid grading structure of the course (taught by several instructors with the need for uniformity among the sections), students could not earn points for doing online homework or using clickers.

2.2. Unianhanguera

We also used online homework in Fall 2009 in a two-month course on transport phenomena (fluid dynamics and thermodynamics) at Unianhanguera. The private for-profit Unianhanguera mostly serves non-traditional students who take evening courses while holding daytime jobs, and is one of the largest such institutions in the country with over 50 campuses. The grade in the course is determined by an exam similar in nature to the exams at USP, however, students can earn up to two bonus points by doing additional projects, visiting seminars, writing essays, etc., to make up for points lost on the exam (due to this policy, it is rare that students completely fail the course).

Online homework problems, but no clickers, were also offered in one of the two sections (57 and 55 students, respectively) of this course, both taught by the same instructor (E.C.). The exam was written by the instructor and the same for both sections. The rather open bonus-point option in this course offered an opportunity to reward students for doing online homework. 28 out of 55 students in the study group filled out the survey.

3. Usage of formative assessment opportunities

3.1. University of São Paulo 3.1.1. Online homework

We compared the use of online homework in the large enrollment calculus-based introductory physics course at USP to similar calculus-based courses at Michigan State University. While similar in many respects, an important difference is that at Michigan State University, completion of the online homework contributed to the course grade (5% and 10%, respectively, in the two courses under consideration).

It became apparent that internet access was a hurdle for the USP students: only 54 out of the 157 user accounts were actually used, and many of those only sporadically. The majority of our students did not have internet access at home, much less laptops, and the number of public workstations at the university is limited. We thus do not have meaningful data from the online system and need to rely on self-reported usage information.

In apparent contradiction to the fact that only 54 accounts were used, 76 of the 86 participants in the online survey stated that they took advantage of the online homework at least at some point during the semester. While it comes at no surprise that users of the online homework system would be over-represented in this online survey, the numbers are only explained by another result from the online survey: 13 of the 76 online homework users stated that they worked in groups, with an average of three students in front of one workstation.

Lack of internet access may also explain some of the differences between usage of LON-CAPA in the USA and in Brazil. One of the most welcome effects of randomizing online homework is an increase in student discussions and peer-teaching [16], which, for a sizable portion of students, is the first thing they do when encountering a new online problem [17]. As Fig. 1 shows, for both populations, about one sixth of the students immediately seeks interaction with others, however, in Brazil, only a tiny amount of this interaction takes place online. Also, Brazilian students state that they do not interact with course personnel; most interaction takes place in person with peers. On the other hand, there are remarkable similarities in gender differences of the first approach to new problems. While, due to the small sample size of the online survey, Brazilian results are somewhat less reliable than the results obtained in the USA [18], it is striking how the percentages agree between the genders in the two introductory courses.

Maybe as a result of increased group work, better understanding of instructor expectations, online homework not being mandatory, or the perception that online work should be quicker, a sizable portion of Brazilian students who used the online homework stated that due to it, they spent less time on this course than they spent on comparable courses. Fig. 2 shows self-reported additional time-on-task versus perceived helpfulness of online homework. The majority of both populations found online homework to be helpful or very helpful in learning, however, students in the USA spent significantly more time with it (and hardly any students in the USA reported that they spent less time). [19]

Given the hurdles and overall sporadic use of the system, one might expect that students would recommend its discontinuation. However, 17 out of 86 students strongly recommended using the system again in a coming semester, 50 recommended it, 18 were indifferent, and only one student would like to see it discontinued.

3.1.2. Clickers

Clicker were frequently used during lectures; to our knowledge, this may well have been the only physics class in Brazil using them. However, participation did not count toward the course grade, and unfortunately, due to lack of clicker registration, not much data was collected.

While the lack of grade relevance and anonymity may lead to not taking the clicker questions seriously, 68 out of 81 survey respondents stated that they always answered the questions to the best of their knowledge, and over 80 percent of the students stated that they took the clicker questions very seriously. When asked if clickers helped them in their learning, 58 out of 81 respondents agreed or strongly agreed, 17 respondents were indifferent, and six respondents stated that they did not or not at all help them learn. When asked if they recommend continued use of clickers, 60 out of 81 respondents recommended or strongly recommended continued use, 18 respondents were indifferent, and three respondents recommended discontinuation. While the recommendations regarding online homework need to be taken with a grain of salt, given that ungraded online homework does not infringe on the students, the responses regarding clickers are significant:

clickers took away lecture time, and, as the vast majority of students took them seriously, meant extra work for them. Still, over 70 percent of the survey respondents considered them helpful in learning and would like to see them used again.

The instructor observed that classroom dynamics changed considerably toward a much higher level of student engagement. Traditionally, there is very little interaction between instructors and students: the instructor lectures, the student copy from the blackboard into their notebooks. Clickers changed the dynamics of the lecture venue (Fig. 3).

3.2. Unianhanguera

Unianhanguera offered a vastly different picture: online homework was rewarded by bonus points, and most students had internet access and laptop computers connected with their daytime jobs. All 55 students completed almost all homework problems offered over the two months. In spite of having their own workstations, 18 out of 28 survey participants reported that they worked in groups of 2 to 5 students. The vast majority of students in this course was male, so gender differences in online homework could not meaningfully be assessed. As Fig. 1 shows, the non-traditional students in the Unianhanguera course react to homework very differently from the traditional students in the introductory courses at the public universities in both countries. It is striking that "reading up on the topic" does not appear as a first action when encountering a new online problem. It seems that students in this course mostly interact with each other to solve problems.

Also the online discussion boards in this course showed a lot more activity than at USP, and a few students stated that this is where they first turned for help. Interestingly, many of the postings on the discussion boards started with "Professor, how do I ... ?" — it is usually not part of the Brazilian university culture that students would directly ask a professor for help (reflected in Fig. 1), yet, in this public online medium, no such hesitation seemed to exist. Several of the questions addressed to the professor were eventually answered by students.

Once again, surprisingly, as Fig. 2 shows, a sizable percentage of students stated that they spent less time on this course than on comparable courses due to online homework. All students participating in the survey stated that online homework was helpful or very helpful, and all survey participants recommended or strongly recommended using online homework again in other semesters.

4. Influence on grades

4.1. University of São Paulo

The grade distribution in our reformed sections was compared to the distribution of grades of a traditionally taught section with the same instructor in the past, and with the grade distribution of the traditional sections in the same year (see middle panel of Fig. 4). It turns out that particularly students who would fail the course profited from the formative assessment, as the percentage of students with scores ranging from 0 to 4 went down. However, also the percentage of students with grades from 7 to 10 went down; overall, the distribution more strongly peaks around the mid-range grades. This effect is different from what was observed in the USA when online homework was introduced in an otherwise similar large enrollment physics course for scientists and engineers (left panel of Fig. 4): while also here, students that were about to fail the class with a grade lower than 2 profited the most, the overall distribution was skewed toward higher grades [15,20] (the same effect was seen in a non-calculus based courses for non-majors and at other universities in the USA [15, 21]).

In previous studies, it was also found that in the USA, the benefit from online homework is gender-dependent, with females benefitting more strongly from the formative assessment provided [15, 18, 21]. The same gender-differential benefit was seen due to interactive engagement classroom strategies including peer-instruction with clickers [22]. Unfortunately, also this effect could not be verified in the Brazilian classroom, see Fig. 5. If anything, the grade distribution of female students in the reformed section is even more strongly peaked around mid-range grades.

Regarding the online homework, it is tempting to simply compare the final grades of the 54 students who used their online accounts with the final grades of the remainder of the students. As it turns out, the average grade and standard deviation of grades of students using online homework is 5.4±1.0, while the average grade of students not using online homework is 4.8±1.4. Thus, while students using the online homework do better, the difference is not significant.

Associating online account usage with individual users does not take into account the fact that students worked in groups. In an attempt to capture also these students, the online survey participants instead of the whole class were taken as the sample, and we looked at the self-reported usage of online homework. The average grade of student stating that they regularly worked on online homework was compared to the average grade of students who stated that they rarely or never did online homework; the results were 5.4±1.3 and 5.4±1.5, i.e., no difference was found within the sample of online survey participants.

Regarding clickers, taking the survey respondents as sample, students who stated that they always answered the clicker questions to the best of their knowledge had an average grade of 5.5±1.2, while students admitting to not taking the clickers seriously had an average grade of 4.8±1.7 — students taking advantage of the formative assessment do better, but not significantly. Interestingly, on the other hand, students who stated that they only sometimes replied to questions (i.e., students who admitted to frequently not bothering to answer) did significantly worse (4.0±1.3 versus 5.5±1.3), which is the only significant grade difference that could be extracted from the survey. Students who frequently did not participate in the clicker activities were significantly more likely going to fail the course, or vice versa.

4.2. Unianhanguera

At Unianhanguera, the online homework, even though perceived as helpful by the students, made no difference in the grades. As remedial bonus points were given for online homework, in the right panel of Fig. 4, we only considered the final exam grades. The distributions are essentially identical, and so is the average grade of 5.1 for both sections. As almost all students in the course were male, no meaningful gender-specific data could be gained.

5. Discussion and conclusion

Student acceptance of technology-mediated formative assessment in the Brazilian courses was high. Students overwhelmingly reported that the technology was helpful in learning the materials (e.g., Fig. 2), yet, in contrast, there was little or no effect on grades in the courses (Fig. 4). How can we begin to explain the discrepancy between perceived and measured effectiveness in terms of exam grades?

A major difference between data from Brazil and the USA is that Brazilian students reported time-savings as a result of online homework. In the USA, online homework invariably results in more reported time-on-task, which may be the biggest contributing factor to its effectiveness. Brazilian students may have re-allocated time to online homework which they otherwise may have used for studying from books or lecture notes. Also, clicker questions focussed on basic physics concepts, while the exams required elaborate problem solving strategies, calculations, and analytical skills. Some students can "solve" these problems without any understanding of the underlying concepts [23, 24].

Evaluations of the system may also have been unrealistically positive, since the students had "nothing to lose" by the introduction of this non-mandatory learning tool and did not invest much time into it. The survey results might suffer from a systematic sampling error due to lacking internet access: students who had access to the online survey likely also had access to the online homework; students who were unable to access the online homework likely were not able to access the survey, either.

The results we got from this study were overshadowed by the unexpected hurdles we encountered in carrying it out. Innovations in teaching and learning are rarely "plug-and-play," instead, they require a systemic change in many course components, as well as instructor and learner expectations, to be successful [25-27]. The appropriation, deployment, and research design encountered a series of setbacks: lack of internet access kept students from doing online homework, shortage of clickers prevented personalized data collection, schedules did not accommodate survey administration in class, grading policies did not allow for the incorporation of formative assessment scores, etc. In retrospect, it was naïve to assume that techniques could simply be transplanted between educational cultures, even when course topics and populations are comparable. The value system of a course is communicated to the learners by what is graded and put on exams [14,23].

It is our hope that work can go forward; as Fig. 4 shows, a sizable portion of the student population is failing the course at USP, and would fail the course at Unianhanguera if it wasn't for the bonus points — we believe that formative assessment could make a significant difference. In future deployments, it will be important to secure full support from senior faculty. Grading structures would need to be changed to offer (even minimal) credit for doing online homework and participating in clicker questions and peer teaching, means for selling or loaning clickers to individual students would need to be found, and departments would need to offer accessible help rooms with internet access for students.

Acknowledgments

We would like to thank the iClicker^TM company for loaning us a classroom set of clickers for the duration of this study.

References

[1] J.R. Bransford, A.L. Brown and R.R. Cocking, eds., How People Learn (National Academy Press, Washington, 2000).

[2] G.M. Novak, E.T. Patterson, A.D. Gavrin and W. Christian, Just-in-Time Teaching: Blending Active Learning with Web Technology (Prentice Hall, Upper Saddle River, 1999).

[3] J.W. Pellegrino, N. Chudowsky and R. Glaser, eds., Knowing What Students Know (National Academy Press, Washington, 2001).

[4] G. Kortemeyer, M. Hall, Joyce Parker, B. Minaei-Bidgoli, G. Albertelli II, W. Bauer and E. Kashy, Journal of Asynchronous Learning Networks 9 (2005).

[5] E. Mazur, Peer Instruction (Prentice Hall, Upper Saddle River, 1997) .

[6] C.H. Crouch and E. Mazur, Am. J. Phys. 69, 970 (2001).

[7] R.J. Dufresne and W.J. Gerace, Phys. Teach. 42, 428 (2004).

[8] R. Azevedo and R.M. Bernard, J. Educ. Comp. Res. 13, 111 (1995).

[9] R.J. Dufresne, D. Hart, J.P. Mestre and K. Rath, Journal of Computers in Mathematics and Science Teaching 21, 229 (2002).

[10] K.K. Cheng, B.A. Thacker, R.L. Cardenas and Catherine Crouch, Am. J. Physics 72, 1447 (2004).

[11] S.W. Bonham, R.J. Beichner and D.L. Deardorff, Phys. Teach. 39, 293 (2001).

[12] A. Pascarella, Proceedings of the NARST 2004 Annual Meeting, Vancouver, BC (2004).

[13] J.L. Otsuka and H.V. da Rocha, Proceedings of the IADIS International Conference WWW/Internet 2007, 35 (2007).

[14] E. Yerushalmi, E. Cohen, K. Heller, P. Heller and C. Henderson, Phys. Rev. ST Phys. Educ. Res. 6, 020108 (2010).

[15] G. Kortemeyer, E. Kashy, W. Benenson and W. Bauer, Am. J. Phys 76, 438 (2008). [16] G. Kortemeyer, Am. J. Phys 74, 526 (2006).

[17] G. Kortemeyer and P. Riegler, Zeitschrift fuer E-Learning 5, 8 (2010).

[18] G. Kortemeyer, Phys. Rev. ST-PER 5, 010107 (2009).

[19] E. Kashy, M. Thoennessen, Y. Tsai, N.E. Davis and S.L. Wolfe, J. Eng. Educ. 87, 385 (1998).

[20] D.A. Kashy, G. Albertelli, E. Kashy and M. Thoennessen, J. Eng. Educ. 89, 499 (2001).

[21] P. Kotas, Homework Behavior in an Introductory Physics Course, Master's thesis, Central Michigan University (2000).

[22] M. Lorenzo, C. H. Crouch and E. Mazur, Am. J. Phys. 74, 118 (2006).

[23] H. Lin, Phys. Teach. 20, 151 (1982).

[24] A. van Heuvelen, Am. J. Physics 59, 891 (1991).

[25] A. De Ambrosis and O. Levrini, Phys. Rev. ST Phys. Educ. Res. 6, 020107 (2010).

[26] R.M. Goertzen, R.E. Scherr, and A. Elby, Phys. Rev. ST Phys. Educ. Res. 5, 020109 (2009).

[27] C. Turpen and N.D. Finkelstein, Phys. Rev. ST Phys. Educ. Res. 5, 020101 (2009).

Recebido em 1/6/2011

Aceito em 12/10/2011

Publicado em 29/11/2011

Publication Dates

• Publication in this collection
  02 Mar 2012
• Date of issue
  Dec 2011

History

• Received
  01 June 2011
• Accepted
  12 Oct 2011