'Your Model Is Predictive-- but Is It Useful?' Theoretical and Empirical Considerations of a New Paradigm for Adaptive Tutoring Evaluation

International Educational Data Mining Society. e-mail: admin@educationaldatamining.org; Web site: http://www.educationaldatamining.org

N

8

Intelligent Tutoring Systems; Evaluation Methods; Program Evaluation; Student Behavior; Mathematics; Student Participation; Responses; Drills (Practice); Skill Development; Statistical Analysis; Middle School Students; Mathematics Instruction; Mathematics Skills; Prediction; Models; Classification; Simulation

Classification evaluation metrics are often used to evaluate adaptive tutoring systems-- programs that teach and adapt to humans. Unfortunately, it is not clear how intuitive these metrics are for practitioners with little machine learning background. Moreover, our experiments suggest that existing convention for evaluating tutoring systems may lead to suboptimal decisions. We propose the Learner Effort-Outcomes Paradigm (Leopard), a new framework to evaluate adaptive tutoring. We introduce Teal and White, novel automatic metrics that apply Leopard and quantify the amount of effort required to achieve a learning outcome. Our experiments suggest that our metrics are a better alternative for evaluating adaptive tutoring. [For complete proceedings, see ED560503.]

As Provided

22

2015

ED560590