ABSTRACT
This paper describes the steps of transforming a paper-and-pencil English proficiency test into an computerized adaptive test (TAI-PI) based on an Item Response Theory (IRT) model. The exam is composed of multiple choice items administered according to the Admissible Probability Measurement Procedure, adopted by the graduate program at the Institute of Mathematics and Computer Sciences at the University of São Paulo (ICMC-USP). Despite the fact that the program accepts various internationally recognized tests that attest non-native speakers English proficiency, such as the Test of English as a Foreign Language (TOEFL), the International English Language Testing System (IELTS) and the Cambridge English: Proficiency (CPE), for instance, its requirement is incompatible with the way the Brazilian public university operates due to the cost, which ranges from US$ 200.00 to US$ 300.00 per exam. The TAI-PI software (Computerized Adaptive Test for English Proficiency), which was developed in Java language and SQLite, started to be used to assess the English proficiency of students on the program from October, 2013. The statistical methodology used was defined considering the history and aims of the test and adopted Samejima’s Graded Response Model, the Kullback-Leibler information criterion for item selection, the a posteriori estimation method for latent trait and the Shadow Test approach to impose restrictions (content and test length) on the test composition of each individual. A description of the test design, the statistical methods used, and the results of a real application of TAI-PI for graduate students are presented in this paper, as well as the validation studies of the new methodology for pass or fail classification, showing the good quality of the new evaluation system and examination of improvement using the IRT and CAT methods.
Keywords:
computerized adaptive testing; item response theory; shadow test
