Computer-Assisted Assessment in Open-Ended Activities through the Analysis of Traces: A Proof of Concept in Statistics with R Commander
More details
Hide details
Section of Statistics, Faculty of Biology, Universitat de Barcelona, SPAIN
Department of Applied Physics, Universitat de Barcelona, SPAIN
Quantitative Methods Department, IQS Universitat Ramon Llull, Barcelona, SPAIN
Online publish date: 2019-04-12
Publish date: 2019-04-12
EURASIA J. Math., Sci Tech. Ed 2019;15(9):em1743
Open-ended tasks are common in Science, Technology, Engineering and Mathematics (STEM) education. However, as far as we know, no tools have been developed to assist in the assessment of the solution process of open-ended questions. In this paper, we propose the use of analysis of traces as a tool to address this need. To illustrate this approach, we developed a modified version of R Commander that collects traces of students’ actions and described a way to analyze them by using regular expressions. We used this tool in an undergraduate introductory statistics course. The traces were analyzed by comparing them to predefined problem-solving steps, arranged by the instructor. The analyses provide information about the time students spent on the activity, their work intensity and the choices they made when solving open-ended questions. This automated assessment tool provides grades highly correlated to those obtained by a traditional test and traditional grading scheme.
Anderson-Cook, C., & Dorai-Raj, S. (2003). Making the concepts of power and sample size relevant and accessible to students in introductory statistics courses using applets. Journal of Statistics Education, 10.
Bahar, A., & Maker, C. (2015). Cognitive Backgrounds of Problem Solving: A Comparison of Open-ended vs. Closed Mathematics Problems. Eurasia Journal of Mathematics, Science & Technology Education, 11(6).
Baker, R., Corbett, A., & Koedinger, K. (2004). Detecting student misuse of intelligent tutoring systems. In: Intelligent tutoring systems, Heidelberg: Springer, pp. 54–76.
Basturk, R. (2005). The effectiveness of computer-assisted instruction in teaching introductory statistics. Educational Technology & Society, 8, 170–178.
Cai, J., & Lester, F. (2010). Why is teaching with problem solving important to student learning. Problem Solving: Research Brief.
Carnegie Mellon Open Learning Initiative (2017). Statistics course. Retrieved on 24 January 2018 from
Chang, W., Cheng, J., Allaire, JJ., Xie, Y. & McPherson, J. (2018). shiny: Web Application Framework for R. R package version 1.2.0. Retrieved from
Cocea, M., & Weibelzahl, S. (2009). Log file analysis for disengagement detection in e-learning environments. User Modeling and User-Adapted Interaction, 19, 341–385.
Conijn, R., Snijders, C., Kleingeld, A., & Matzat, U. (2017). Predicting student performance from LMS data: A comparison of 17 blended courses using Moodle LMS. IEEE Transactions on Learning Technologies, 10(1), 17-29.
De Marsico, M., Sciarrone, F., Sterbini, A., & Temperini, M. (2017). Supporting Mediated Peer-Evaluation to Grade Answers to Open-Ended Questions. Eurasia Journal of Mathematics, Science and Technology Education, 13(4), 1085-1106.
Dinov, I., Sanchez, J., & Christou, N. (2008). Pedagogical utilization and assessment of the statistic online computational resource in introductory probability and statistics courses. Computers & Education, 50, 284–300.
English, L., & Sriraman, B. (2010). Problem solving for the 21st century. In Theories of mathematics education (pp. 263-290). Springer, Berlin, Heidelberg.
Fox, J. (2005). The R Commander: a basic statistics graphical user interface to R. Journal of Statistical Software, 14(9).
Gonzalez, J., & Muñoz, P. (2006). e-status: An automatic web-based problem generator-applications to statistics. Computer Applications in Engineering Education, 14, 151–159.
González, J., Jover, L., Cobo, E., & Muñoz, P. (2010). A web-based learning tool improves student performance in statistics: A randomized masked trial. Computers & Education, 55, 704–713.
Grün, B., & Zeileis, A. (2009). Automatic generation of exams in R. Journal of Statistical Software, 29(10), 1-14.
Hardin, L. E. (2003). Problem-solving concepts and theories. Journal of veterinary medical education, 30(3), 226-229.
Kinnebrew, J. S., Segedy, J. R., & Biswas, G. (2017). Integrating model-driven and data-driven techniques for analyzing learning behaviors in open-ended learning environments. IEEE Transactions on Learning Technologies, 10(2), 140-153.
Lane, D., & Scott, D. (2000). Simulations, case studies, and an online text: a web-based resource for teaching statistics. Metrika, 51, 67–90.
Larreamendy-Joerns, J., Leinhardt, G., & Corredor, J. (2005). Six online statistics courses: Examination and review. The American Statistician, 59, 240–251.
MERLOT (2018). Multimedia Educational Resource for Learning and Online Teaching. Retrieved on 24 January 2018 from
Mostow, J., & Beck, J. (2009). What, how, and why should tutors log? In: Proceedings of Educational Data Mining 2009, 269–278.
Muenchen, R. A. (2017, June 19). The Popularity of Data Science Software [Web log message]. Retrieved from
Piatetsky, G. (2017, May 22). New Leader, Trends, and Surprises in Analytics, Data Science, Machine Learning Software Poll [Web log message]. Retrieved from
PSLC (2013), Guide to the tutor message format. a standard XML vocabulary for logging student and tutor actions. Retrieved on 24 January 2018 from https://pslcdatashop.web.cmu.e....
R Core Team (2017). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Retrieved from
Ritter, S., & Blessing, S. (1998). Authoring tools for component-based learning environments. Journal of the Learning Sciences, 7, 107–132.
Ritter, S., & Koedinger, K. (1996). An architecture for plug-in tutor agents. Journal of Artificial Intelligence in Education, 7, 315-347.
Romero, C., & Ventura, S. (2010). Educational data mining: a review of the state of the art. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 40, 601–618.
Romero, C., Ventura, S., & Garcia, E. (2008). Data mining in course management systems: Moodle case study and tutorial. Computers & Education, 51, 368–384.
Rosales, F., Garcia, A., Rodriguez, S., Pedraza, J., Méndez, R., & Nieto, M. (2008). Detection of plagiarism in programming assignments. IEEE Transactions on Education, 51, 174–183.
Song, C., & Ma, K. (2008). Applications of data mining in the education resource based on XML. In: 2008 International Conference on Advanced Computer Theory and Engineering, IEEE, 943–946.
Stevens, R., Beal, C., & Sprang, M. (2009). Tracking the development of problem solving skills with learning trajectories. In: Proceedings of the 17th International Conference on Computers in Education, 99-106.
Suanpang, P., Petocz, P., & Kalceff, W. (2004). Student attitudes to learning business statistics: Comparison of online and traditional methods. Educational Technology & Society, 7, 9–20.
Symanzik, J., & Vukasinovic, N. (2003). Teaching experiences with a course on web-based statistics. The American Statistician, 57, 46–50.
VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R., Weinstein, A., & Billings, B. (2009). What’s in a step? toward general, abstract representations of tutoring system log data. User Modeling 2007, 455–459.
West, R., & Ogden, R. (1998). Interactive demonstrations for statistics education on the world wide web. Journal of Statistics Education, 6(3).
Zeileis, A., Umlauf, N., & Leisch, F. (2014). Flexible Generation of E-Learning Exams in R: Moodle Quizzes, OLAT Assessments, and Beyond. Journal of Statistical Software, 58(1), 1-36.